I am running an openstack newton setup with a single controller and three computes.

Now i have noticed that the openstack-nova-compute services are not starting on the compute nodes.

FYI,

[root@controller mongodb]# nova hypervisor-list
+----+--------------------------------+-------+---------+
| ID | Hypervisor hostname            | State | Status  |
+----+--------------------------------+-------+---------+
| 1  | compute-253.os.mbs.comviva.com | up    | enabled |
| 4  | compute-31.os.nfv.comviva.com  | down  | enabled |
| 5  | compute-252.os.nfv.comviva.com | down  | enabled |
+----+--------------------------------+-------+---------+

Also, I have matched the nova.conf of all the three compute nodes and they are exactly the same.

When i debugged a little further, i have noticed that the libvirtd service on both of the down compute nodes are throwing some errors.

FYI,

[root@compute-252 tmp]# systemctl status libvirtd -l
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2017-03-29 11:41:17 IST; 2h 57min ago
     Docs: man:libvirtd(8)
           http://libvirt.org
 Main PID: 1337 (libvirtd)
   CGroup: /system.slice/libvirtd.service
           ├─1337 /usr/sbin/libvirtd
           ├─2662 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper
           └─2664 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf --leasefile-ro --dhcp-script=/usr/libexec/libvirt_leaseshelper

Mar 29 12:05:20 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:24 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:28 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:32 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:35 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:39 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error
Mar 29 12:05:43 compute-252.os.nfv.comviva.com libvirtd[1337]: End of file while reading data: Input/output error

Also,

[root@compute-31 ~]# grep -i error /var/log/nova/nova-compute.log
[root@compute-31 ~]#
[root@compute-31 ~]#

Now as i understand the libvirtd is throwing some I/O error and that is possibly not letting the compute services start up.

=====

I did further research and then restarted the dbus.service on my compute nodes and then restarted the libvirtd and nova-compute on the affected nodes.

What I have noticed that libvirtd I/O error vanished on one of the nodes(still coming on the second node), but the nova-compute service is still not starting on either of them.

When i tried to manually start the compute service "/usr/bin/nova-compute", I am getting the below error.

[root@compute-31 ~]# /usr/bin/nova-compute
Option "rpc_backend" from group "DEFAULT" is deprecated for removal.  Its value may be silently ignored in the future.
2017-03-29 15:54:28.343 19403 WARNING oslo_reports.guru_meditation_report [-] Guru meditation now registers SIGUSR1 and SIGUSR2 by default for backward compatibility. SIGUSR1 ...