Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Compute services not starting

I am running an openstack newton setup with a single controller and three computes.

Now i have noticed that the openstack-nova-compute services are not starting on the compute nodes.

FYI,

[root@controller mongodb]# nova hypervisor-list
+----+--------------------------------+-------+---------+
| ID | Hypervisor hostname            | State | Status  |
+----+--------------------------------+-------+---------+
| 1  | compute-253.os.mbs.comviva.com | up    | enabled |
| 4  | compute-31.os.nfv.comviva.com  | down  | enabled |
| 5  | compute-252.os.nfv.comviva.com | down  | enabled |
+----+--------------------------------+-------+---------+

Also, I have matched the nova.conf of all the three compute nodes and they are exactly the same.

When i debugged a little further, i have noticed that the libvirtd service on both of the down compute nodes are throwing some errors.

FYI,

[root@compute-252 tmp]# systemctl status libvirtd -l
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2017-03-29 11:41:17 IST; 2h 57min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 1337 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           ├─1337 /usr/sbin/libvirtd
           ├─2662 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           └─2664 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

Mar 29 12:05:20 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:24 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:28 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:32 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:35 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:39 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:43 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error

Also,

[root@compute-31 ~]# grep -i error /var/log/nova/nova-compute.log
[root@compute-31 ~]#
[root@compute-31 ~]#

Now as i understand the libvirtd is throwing some I/O error and that is possibly not letting the compute services start up.

=====

I did further research and then restarted the dbus.service on my compute nodes and then restarted the libvirtd and nova-compute on the affected nodes.

What I have noticed that libvirtd I/O error vanished on one of the nodes(still coming on the second node), but the nova-compute service is still not starting on either of them.

When i tried to manually start the compute service "/usr/bin/nova-compute", I am getting the below error.

[root@compute-31 ~]# /usr/bin/nova-compute
Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2017-03-29 15:54:28.343 19403 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-03-29 15:54:28.397 19403 ERROR nova.virt.driver [req-7f509aa4-85f9-4374-b856-c91c877307f5 - - - - -] Compute driver option required, but not specified

I researched further and added the following lines in the nova.conf file on the compute and the controller node.

vif_plugging_is_fatal: false
vif_plugging_timeout: 0
compute_driver=libvirt.LibvirtDriver

Even these changes are not helping. Now when i try to manually start the service, this is what i get:

[root@compute-252 tmp]# /usr/bin/nova-compute
Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2017-03-29 15:56:56.258 94021 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 will no longer be registered in a future release, so please use SIGUSR2 to generate reports.
2017-03-29 15:56:56.306 94021 INFO nova.virt.driver [req-590cdcbd-56fc-4f27-830b-94df57610124 - - - - -] Loading compute driver 'libvirt.LibvirtDriver'
2017-03-29 15:56:56.595 94021 INFO os_vif [req-590cdcbd-56fc-4f27-830b-94df57610124 - - - - -] Loaded VIF plugin class '<class 'vif_plug_ovs.ovs.OvsPlugin'>' with name 'ovs'
2017-03-29 15:56:56.596 94021 INFO os_vif [req-590cdcbd-56fc-4f27-830b-94df57610124 - - - - -] Loaded VIF plugin class '<class 'vif_plug_linux_bridge.linux_bridge.LinuxBridgePlugin'>' with name 'linux_bridge'
2017-03-29 15:56:56.615 94021 INFO os_brick.initiator.connectors.disco [req-590cdcbd-56fc-4f27-830b-94df57610124 - - - - -] Init DISCO connector
2017-03-29 15:56:56.616 94021 WARNING os_brick.initiator.connectors.remotefs [req-590cdcbd-56fc-4f27-830b-94df57610124 - - - - -] Connection details not present. RemoteFsClient may not initialize properly.
2017-03-29 15:56:56.618 94021 WARNING oslo_config.cfg [req-590cdcbd-56fc-4f27-830b-94df57610124 - - - - -] Option "rabbit_host" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2017-03-29 15:56:56.618 94021 WARNING oslo_config.cfg [req-590cdcbd-56fc-4f27-830b-94df57610124 - - - - -] Option "rabbit_userid" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2017-03-29 15:56:56.619 94021 WARNING oslo_config.cfg [req-590cdcbd-56fc-4f27-830b-94df57610124 - - - - -] Option "rabbit_password" from group "oslo_messaging_rabbit" is deprecated for removal.  Its value may be silently ignored in the future.
2017-03-29 15:56:56.937 94021 INFO nova.service [-] Starting compute node (version 14.0.3-1.el7)
2017-03-29 15:56:56.952 94021 INFO nova.virt.libvirt.driver [-] Connection event '1' reason 'None'
2017-03-29 15:56:57.115 94021 INFO oslo.privsep.daemon [req-06ad52b3-8de7-4d45-974c-1cc2d66638b3 - - - - -] Running privsep helper: ['sudo', 'nova-rootwrap', '/etc/nova/rootwrap.conf', 'privsep-helper', '--config-file', '/etc/nova/nova.conf', '--config-dir', [], '--privsep_context', 'vif_plug_linux_bridge.privsep.vif_plug', '--privsep_sock_path', '/tmp/tmpkZmIPu/privsep.sock']
2017-03-29 15:56:57.133 94021 ERROR os_vif [req-06ad52b3-8de7-4d45-974c-1cc2d66638b3 - - - - -] Failed to plug vif VIFBridge(active=True,address=fa:16:3e:46:be:a0,bridge_name='brq7b8f536f-c3',has_traffic_filtering=True,id=56ade7f3-b34c-4c0e-ac63-f2d97cabbdfa,network=Network(7b8f536f-c364-4f92-8bf0-2db602184fd0),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap56ade7f3-b3')
2017-03-29 15:56:57.133 94021 ERROR os_vif Traceback (most recent call last):
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib/python2.7/site-packages/os_vif/__init__.py", line 77, in plug
2017-03-29 15:56:57.133 94021 ERROR os_vif     plugin.plug(vif, instance_info)
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_linux_bridge/linux_bridge.py", line 105, in plug
2017-03-29 15:56:57.133 94021 ERROR os_vif     linux_net.ensure_bridge(bridge_name, iface)
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
2017-03-29 15:56:57.133 94021 ERROR os_vif     return f(*args, **kwargs)
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib/python2.7/site-packages/vif_plug_linux_bridge/linux_net.py", line 101, in ensure_bridge
2017-03-29 15:56:57.133 94021 ERROR os_vif     _ensure_bridge_privileged(bridge, interface, net_attrs, gateway, filtering)
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 203, in _wrap
2017-03-29 15:56:57.133 94021 ERROR os_vif     self.start()
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib/python2.7/site-packages/oslo_privsep/priv_context.py", line 214, in start
2017-03-29 15:56:57.133 94021 ERROR os_vif     channel = daemon.RootwrapClientChannel(context=self)
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib/python2.7/site-packages/oslo_privsep/daemon.py", line 304, in __init__
2017-03-29 15:56:57.133 94021 ERROR os_vif     proc = subprocess.Popen(cmd, shell=False, stderr=_fd_logger())
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib/python2.7/site-packages/eventlet/green/subprocess.py", line 55, in __init__
2017-03-29 15:56:57.133 94021 ERROR os_vif     subprocess_orig.Popen.__init__(self, args, 0, *argss, **kwds)
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
2017-03-29 15:56:57.133 94021 ERROR os_vif     errread, errwrite)
2017-03-29 15:56:57.133 94021 ERROR os_vif   File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
2017-03-29 15:56:57.133 94021 ERROR os_vif     raise child_exception
2017-03-29 15:56:57.133 94021 ERROR os_vif TypeError: execv() arg 2 must contain only strings
2017-03-29 15:56:57.133 94021 ERROR os_vif
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service [req-06ad52b3-8de7-4d45-974c-1cc2d66638b3 - - - - -] Error starting thread.
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service Traceback (most recent call last):
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/oslo_service/service.py", line 722, in run_service
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service     service.start()
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/service.py", line 117, in start
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service     self.manager.init_host()
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1150, in init_host
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service     self._init_instance(context, instance)
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 953, in _init_instance
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service     self.driver.plug_vifs(instance, net_info)
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 702, in plug_vifs
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service     self.vif_driver.plug(instance, vif)
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 817, in plug
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service     self._plug_os_vif(instance, vif_obj, vif)
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/vif.py", line 783, in _plug_os_vif
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service     raise exception.NovaException(msg)
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service NovaException: Failure running os_vif plugin plug method: Failed to plug VIF VIFBridge(active=True,address=fa:16:3e:46:be:a0,bridge_name='brq7b8f536f-c3',has_traffic_filtering=True,id=56ade7f3-b34c-4c0e-ac63-f2d97cabbdfa,network=Network(7b8f536f-c364-4f92-8bf0-2db602184fd0),plugin='linux_bridge',port_profile=<?>,preserve_on_delete=False,vif_name='tap56ade7f3-b3'). Got error: execv() arg 2 must contain only strings
2017-03-29 15:56:57.155 94021 ERROR oslo_service.service

Can someone suggest how to troubleshoot this.

Also note that even without adding the compute driver details to my nova.conf file, i was actually able to spawn instances on the compute node earlier. This makes me believe that this is not the problem in the first case.