Revision history [back]

click to hide/show revision 1
initial version

Unable to start nova-conductor

Hi, I'm running into an issue starting the nova-conductor after install and configuration of the controller node. Because of nova-conductor being down, this affects openstack-nova-compute.service not able to start as well. I'm seeing in the nova-conductor.log the error messages, "ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile". Has anyone seen this before and how do I fix this? Thanks in advance!

Output from nova-conductor.log: 2017-01-18 15:18:06.413 25139 ERROR oslo_service.service 2017-01-18 15:18:07.147 11081 INFO oslo_service.service [req-b39952e9-10b5-4ec5-add3-aa71111de907 - - - - -] Child 25135 exited with status 1 2017-01-18 15:18:07.149 11081 INFO oslo_service.service [req-b39952e9-10b5-4ec5-add3-aa71111de907 - - - - -] Forking too fast, sleeping 2017-01-18 15:18:07.152 25141 INFO nova.service [-] Starting conductor node (version 14.0.2-1.el7) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service [req-e0dd9fcb-d06d-4862-a91f-1a1e3ffb5a5c - - - - -] Error starting thread. 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service Traceback (most recent call last): 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in run_service 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service service.start() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 153, in start 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.rpcserver.start() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 268, in wrapper 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service log_after, timeout_timer) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 188, in run_once 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service post_fn = fn() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 267, in <lambda> 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service states[state].run_once(lambda: fn(self, args, kwargs), 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 420, in start 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.listener = self._create_listener() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 121, in _create_listener 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return self.transport._listen(self._target, 1, None) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 112, in _listen 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service batch_timeout) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 471, in listen 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service conn = self._get_connection(rpc_common.PURPOSE_LISTEN) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 371, in _get_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service purpose=purpose) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py", line 394, in __init__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.connection = connection_pool.create(purpose) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py", line 144, in create 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return self.connection_cls(self.conf, self.url, purpose) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 582, in __init__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.ensure_connection() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 681, in ensure_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.ensure(method=lambda: self.connection.connection) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 791, in ensure 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service ret, channel = autoretry_method() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return fun(args, **kwargs) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 507, in __call__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.revive(create_channel()) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 242, in channel 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service chan = self.transport.create_channel(self.connection) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self._connection = self._establish_connection() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service conn = self.transport.establish_connection() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service conn = self.Connection(opts) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 180, in __init__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service (10, 30), # tune 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.channel_id, allowed_methods) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 273, in _wait_method 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.wait() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return self.dispatch_method(method_sig, args, content) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return amqp_method(self, args) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service (class_id, method_id), ConnectionError) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile. 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service

Here's the output showing nova-conductor being down. [root@controller nova]# nova service-list +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+ | 1 | nova-consoleauth | controller | internal | enabled | up | 2017-01-18T20:10:41.000000 | - | | 2 | nova-conductor | controller | internal | enabled | down | - | - | | 5 | nova-scheduler | controller | internal | enabled | up | 2017-01-18T20:10:34.000000 | - | +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+

Unable to start nova-conductor

Hi, Hi,

I'm running into an issue starting the nova-conductor after install and configuration of the controller node. Because of nova-conductor being down, this affects openstack-nova-compute.service not able to start as well. I'm seeing in the nova-conductor.log the error messages, "ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile". Has anyone seen this before and how do I fix this? Thanks in advance!

Output from nova-conductor.log: nova-conductor.log:

2017-01-18 15:18:06.413 25139 ERROR oslo_service.service 2017-01-18 15:18:07.147 11081 INFO oslo_service.service [req-b39952e9-10b5-4ec5-add3-aa71111de907 - - - - -] Child 25135 exited with status 1 2017-01-18 15:18:07.149 11081 INFO oslo_service.service [req-b39952e9-10b5-4ec5-add3-aa71111de907 - - - - -] Forking too fast, sleeping 2017-01-18 15:18:07.152 25141 INFO nova.service [-] Starting conductor node (version 14.0.2-1.el7) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service [req-e0dd9fcb-d06d-4862-a91f-1a1e3ffb5a5c - - - - -] Error starting thread. 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service Traceback (most recent call last): 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in run_service 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service service.start() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 153, in start 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.rpcserver.start() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 268, in wrapper 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service log_after, timeout_timer) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 188, in run_once 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service post_fn = fn() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 267, in <lambda> 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service states[state].run_once(lambda: fn(self, args, kwargs), 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 420, in start 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.listener = self._create_listener() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 121, in _create_listener 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return self.transport._listen(self._target, 1, None) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 112, in _listen 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service batch_timeout) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 471, in listen 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service conn = self._get_connection(rpc_common.PURPOSE_LISTEN) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 371, in _get_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service purpose=purpose) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py", line 394, in __init__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.connection = connection_pool.create(purpose) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py", line 144, in create 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return self.connection_cls(self.conf, self.url, purpose) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 582, in __init__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.ensure_connection() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 681, in ensure_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.ensure(method=lambda: self.connection.connection) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 791, in ensure 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service ret, channel = autoretry_method() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return fun(args, **kwargs) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 507, in __call__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.revive(create_channel()) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 242, in channel 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service chan = self.transport.create_channel(self.connection) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self._connection = self._establish_connection() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service conn = self.transport.establish_connection() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service conn = self.Connection(opts) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 180, in __init__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service (10, 30), # tune 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.channel_id, allowed_methods) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 273, in _wait_method 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.wait() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return self.dispatch_method(method_sig, args, content) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return amqp_method(self, args) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service (class_id, method_id), ConnectionError) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile. 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service

Here's the output showing nova-conductor being down. [root@controller nova]# nova service-list +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+ | 1 | nova-consoleauth | controller | internal | enabled | up | 2017-01-18T20:10:41.000000 | - | | 2 | nova-conductor | controller | internal | enabled | down | - | - | | 5 | nova-scheduler | controller | internal | enabled | up | 2017-01-18T20:10:34.000000 | - | +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+

Unable to start nova-conductor

Hi,

I'm running into an issue starting the nova-conductor after install and configuration of the controller node. Because of nova-conductor being down, this affects openstack-nova-compute.service not able to start as well. I'm seeing in the nova-conductor.log the error messages, "ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile". Has anyone seen this before and how do I fix this? Thanks in advance!

Output from nova-conductor.log:

2017-01-18 15:18:06.413 25139 ERROR oslo_service.service 2017-01-18 15:18:07.147 11081 INFO oslo_service.service [req-b39952e9-10b5-4ec5-add3-aa71111de907 - - - - -] Child 25135 exited with status 1 2017-01-18 15:18:07.149 11081 INFO oslo_service.service [req-b39952e9-10b5-4ec5-add3-aa71111de907 - - - - -] Forking too fast, sleeping 2017-01-18 15:18:07.152 25141 INFO nova.service [-] Starting conductor node (version 14.0.2-1.el7) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service [req-e0dd9fcb-d06d-4862-a91f-1a1e3ffb5a5c - - - - -] Error starting thread. 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service Traceback (most recent call last): 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in run_service 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service service.start() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/nova/service.py", line 153, in start 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.rpcserver.start() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 268, in wrapper 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service log_after, timeout_timer) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 188, in run_once 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service post_fn = fn() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 267, in <lambda> 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service states[state].run_once(lambda: fn(self, args, kwargs), 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 420, in start 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.listener = self._create_listener() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 121, in _create_listener 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return self.transport._listen(self._target, 1, None) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 112, in _listen 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service batch_timeout) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 471, in listen 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service conn = self._get_connection(rpc_common.PURPOSE_LISTEN) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 371, in _get_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service purpose=purpose) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py", line 394, in __init__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.connection = connection_pool.create(purpose) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py", line 144, in create 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return self.connection_cls(self.conf, self.url, purpose) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 582, in __init__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.ensure_connection() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 681, in ensure_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.ensure(method=lambda: self.connection.connection) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 791, in ensure 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service ret, channel = autoretry_method() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return fun(args, **kwargs) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 507, in __call__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.revive(create_channel()) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 242, in channel 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service chan = self.transport.create_channel(self.connection) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self._connection = self._establish_connection() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service conn = self.transport.establish_connection() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service conn = self.Connection(opts) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 180, in __init__ 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service (10, 30), # tune 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.channel_id, allowed_methods) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 273, in _wait_method 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service self.wait() 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return self.dispatch_method(method_sig, args, content) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service return amqp_method(self, args) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service (class_id, method_id), ConnectionError) 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile. 2017-01-18 15:18:07.420 25141 ERROR oslo_service.service

Here's the output showing nova-conductor being down. [root@controller nova]# nova service-list +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason | +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+ | 1 | nova-consoleauth | controller | internal | enabled | up | 2017-01-18T20:10:41.000000 | - | | 2 | nova-conductor | controller | internal | enabled | down | - | - | | 5 | nova-scheduler | controller | internal | enabled | up | 2017-01-18T20:10:34.000000 | - | +----+------------------+------------+----------+---------+-------+----------------------------+-----------------+

| 1 | nova-consoleauth | controller | internal | enabled | up | 2017-01-18T20:10:41.000000 |

| 2 | nova-conductor | controller | internal | enabled | down | - | - |

| 5 | nova-scheduler | controller | internal | enabled | up | 2017-01-18T20:10:34.000000 | -

+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+

Unable to start nova-conductor

Hi,

I'm running into an issue starting the nova-conductor after install and configuration of the controller node. Because of nova-conductor being down, this affects openstack-nova-compute.service not able to start as well. I'm seeing in the nova-conductor.log the error messages, "ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile". Has anyone seen this before and how do I fix this? Thanks in advance!

Output from nova-conductor.log:

2017-01-18 15:18:06.413 25139 ERROR oslo_service.service
2017-01-18 15:18:07.147 11081 INFO oslo_service.service [req-b39952e9-10b5-4ec5-add3-aa71111de907 - - - - -] Child 25135 exited with status 1
2017-01-18 15:18:07.149 11081 INFO oslo_service.service [req-b39952e9-10b5-4ec5-add3-aa71111de907 - - - - -] Forking too fast, sleeping
2017-01-18 15:18:07.152 25141 INFO nova.service [-] Starting conductor node (version 14.0.2-1.el7)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service [req-e0dd9fcb-d06d-4862-a91f-1a1e3ffb5a5c - - - - -] Error starting thread.
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service Traceback (most recent call last):
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 708, in run_service
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     service.start()
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py", line 153, in start
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     self.rpcserver.start()
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 268, in wrapper
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     log_after, timeout_timer)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 188, in run_once
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     post_fn = fn()
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 267, in <lambda>
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     states[state].run_once(lambda: fn(self, args, kwargs),
*args, **kwargs),
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/server.py", line 420, in start
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     self.listener = self._create_listener()
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 121, in _create_listener
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     return self.transport._listen(self._target, 1, None)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 112, in _listen
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     batch_timeout)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 471, in listen
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     conn = self._get_connection(rpc_common.PURPOSE_LISTEN)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 371, in _get_connection
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     purpose=purpose)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/common.py", line 394, in __init__
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     self.connection = connection_pool.create(purpose)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/pool.py", line 144, in create
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     return self.connection_cls(self.conf, self.url, purpose)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 582, in __init__
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     self.ensure_connection()
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 681, in ensure_connection
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     self.ensure(method=lambda: self.connection.connection)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/impl_rabbit.py", line 791, in ensure
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     ret, channel = autoretry_method()
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 436, in _ensured
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     return fun(args, fun(*args, **kwargs)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 507, in __call__
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     self.revive(create_channel())
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 242, in channel
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     chan = self.transport.create_channel(self.connection)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 741, in connection
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     self._connection = self._establish_connection()
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/connection.py", line 696, in _establish_connection
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     conn = self.transport.establish_connection()
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 116, in establish_connection
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     conn = self.Connection(opts)
self.Connection(**opts)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 180, in __init__
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     (10, 30),  # tune
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 67, in wait
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     self.channel_id, allowed_methods)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 273, in _wait_method
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     self.wait()
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     return self.dispatch_method(method_sig, args, content)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     return amqp_method(self, args)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service     (class_id, method_id), ConnectionError)
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2017-01-18 15:18:07.420 25141 ERROR oslo_service.service

oslo_service.service

Here's the output showing nova-conductor being down. down.

[root@controller nova]# nova service-list
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host       | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+

+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+ | 1 | nova-consoleauth | controller | internal | enabled | up | 2017-01-18T20:10:41.000000 |

| | 2 | nova-conductor | controller | internal | enabled | down | - | - |

| | 5 | nova-scheduler | controller | internal | enabled | up | 2017-01-18T20:10:34.000000 | -

+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+

+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+