Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Changed host name on controller; all compute services are now "down"

Hi,

I'm starting to install OpenStack on a controller and compute node (both running CentOS 7) with the official installation guide. I've just changed the hostname of my controller server from "localhost.localdomain" to "controller"; I had forgotten to do so earlier. Everything has just broken. When I run the command openstack compute service list, all the services are "down."

+----+------------------+------------+----------+---------+-------+----------------------------+
| ID | Binary           | Host       | Zone     | Status  | State | Updated At                 |
+----+------------------+------------+----------+---------+-------+----------------------------+
| 11 | nova-compute     | compute1   | nova     | enabled | down  | 2017-07-06T15:52:01.000000 |
| 12 | nova-consoleauth | controller | internal | enabled | down  | None                       |
| 13 | nova-conductor   | controller | internal | enabled | down  | None                       |
| 14 | nova-scheduler   | controller | internal | enabled | down  | None                       |
+----+------------------+------------+----------+---------+-------+----------------------------+

The error messages on the controller are all very similar.

tail -n 50 /var/log/nova/nova-scheduler.log

…

2017-07-06 10:29:14.306 28323 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 273, in _wait_method
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service     self.wait()
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 69, in wait
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service     return self.dispatch_method(method_sig, args, content)
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/abstract_channel.py", line 87, in dispatch_method
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service     return amqp_method(self, args)
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/amqp/connection.py", line 529, in _close
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service     (class_id, method_id), ConnectionError)
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service AccessRefused: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN. For details see the broker logfile.
2017-07-06 10:29:14.306 28323 ERROR oslo_service.service

The compute node cannot start the service openstack-nova-compute.service. This was working properly before the controller hostname was changed and was "up" on openstack compute service list.

systemctl start openstack-nova-compute.service
Job for openstack-nova-compute.service failed because the control process exited with error code. See "systemctl status openstack-nova-compute.service" and "journalctl -xe" for details.

The commands it recommends are not too helpful.

journalctl -xe
-- Unit openstack-nova-compute.service has begun starting up.
Jul 06 10:35:38 compute1 systemd[1]: openstack-nova-compute.service: main proces
Jul 06 10:35:38 compute1 systemd[1]: Failed to start OpenStack Nova Compute Serv
-- Subject: Unit openstack-nova-compute.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit openstack-nova-compute.service has failed.
-- 
-- The result is failed.

The error messages are generic.

systemctl status openstack-nova-compute.service
● openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled; vendor preset: disabled)
   Active: activating (start) since Thu 2017-07-06 10:39:25 MDT; 767ms ago
 Main PID: 8461 (nova-compute)
   CGroup: /system.slice/openstack-nova-compute.service
           └─8461 /usr/bin/python2 /usr/bin/nova-compute

Jul 06 10:39:25 compute1 systemd[1]: Starting OpenStack Nova Compute Server...

The services stay in the "starting" state, which leads me to believe they are not able to connect to some network resource and are trying repeatedly.

systemctl | grep openstack
openstack-glance-api.service                                                                                                                loaded active     running         OpenStack Image Service (code-named Glance) API server
openstack-glance-registry.service                                                                                                           loaded active     running         OpenStack Image Service (code-named Glance) Registry server
openstack-nova-api.service                                                                                                                  loaded active     running         OpenStack Nova API Server
openstack-nova-conductor.service                                                                                                            loaded active     running         OpenStack Nova Conductor Server
openstack-nova-consoleauth.service                                                                                                          loaded activating start     start OpenStack Nova VNC console auth Server
openstack-nova-novncproxy.service                                                                                                           loaded active     running         OpenStack Nova NoVNC Proxy Server
openstack-nova-scheduler.service                                                                                                            loaded activating start     start OpenStack Nova Scheduler Server

I have both hosts defined in /etc/hosts and I have verified that the IP addresses are correct. I can ping both servers from each other and both have Internet access. My best guess is that some service has been set up with the hostname hard-coded, but I am not sure what could possibly cause this, as nothing I have configured has required I enter the hostname and everything works off the definitions in /etc/hosts (this worked before the hostname was changed). Everything seems like it should be working—and it was, prior to the change—so I'm at a loss for what could have happened.

I'd be happy to run other commands if you need more information. Right now, I'm thinking the only option is to re-install everything, as the error messages aren't very helpful. I'm dreading this, however, and I'm hopeful there is a better way that won't take a day's work.

Thanks in advance!