Compute services not starting
I am running an openstack newton setup with a single controller and three computes.
Now i have noticed that the openstack-nova-compute services are not starting on the compute nodes.
FYI,
[root@controller mongodb]# nova hypervisor-list
+----+--------------------------------+-------+---------+
| ID | Hypervisor hostname | State | Status |
+----+--------------------------------+-------+---------+
| 1 | compute-253.os.mbs.comviva.com | up | enabled |
| 4 | compute-31.os.nfv.comviva.com | down | enabled |
| 5 | compute-252.os.nfv.comviva.com | down | enabled |
+----+--------------------------------+-------+---------+
Also, I have matched the nova.conf of all the three compute nodes and they are exactly the same.
When i debugged a little further, i have noticed that the libvirtd service on both of the down compute nodes are throwing some errors.
FYI,
[root@compute-252 tmp]# systemctl status libvirtd -l
● libvirtd.service - Virtualization daemon
Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2017-03-29 11:41:17 IST; 2h 57min ago
Docs: man:libvirtd(8)
http://libvirt.org
Main PID: 1337 (libvirtd)
CGroup: /system.slice/libvirtd.service
├─1337 /usr/sbin/libvirtd
├─2662 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
└─2664 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
Mar 29 12:05:20 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:24 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:28 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:32 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:35 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:39 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:43 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Also,
[root@compute-31 ~]# grep -i error /var/log/nova/nova-compute.log
[root@compute-31 ~]#
[root@compute-31 ~]#
Now as i understand the libvirtd is throwing some I/O error and that is possibly not letting the compute services start up.
=====
I did further research and then restarted the dbus.service on my compute nodes and then restarted the libvirtd and nova-compute on the affected nodes.
What I have noticed that libvirtd I/O error vanished on one of the nodes(still coming on the second node), but the nova-compute service is still not starting on either of them.
When i tried to manually start the compute service "/usr/bin/nova-compute", I am getting the below error.
[root@compute-31 ~]# /usr/bin/nova-compute
Option "rpc_backend" from group "DEFAULT" is deprecated for removal. Its value may be silently ignored in the future.
2017-03-29 15:54:28.343 19403 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 ...