Ask Your Question
0

nova-compute State down

asked 2015-01-09 06:22:27 -0500

IvanAK gravatar image

Hi, I have installed Openstack and working perfect. But after New Year some problem came out. The first think i notes was one of my VM's (instance) when i gave a reboot action, it went to reboot state / Task and never came back. I gave here an Shutdown and still nothing. Then i list the nova service-list and i see that 3 compute nodes are down. I can't get it back. also i note the following in the libvirtd.log.

2014-12-17 17:31:21.684+0000: 21073: info : libvirt version: 1.2.2
2014-12-17 17:31:21.684+0000: 21073: warning : virSecurityManagerNew:179 : Configured security driver "none" disables default policy to create confined guests
2014-12-18 03:08:17.264+0000: 21064: warning : qemuDomainObjTaint:1628 : Domain id=2 name='instance-00000003' uuid=f6a0c2c2-771f-4890-807d-f235ec9858de is tainted: high-privileges
2015-01-08 13:29:03.762+0000: 21062: error : virNetSocketReadWire:1454 : End of file while reading data: Input/output error
2015-01-08 13:34:08.630+0000: 21062: error : virNetSocketReadWire:1454 : End of file while reading data: Input/output error

I restart the services, I restart the compute node and still nothing. Any solution or suggestions about this scenario ?

Here is some other output, and i note the time is not right. how can change ....

Binary           Host                                 Zone             Status     State Updated_At
nova-consoleauth node-32                              internal         enabled    :-)   2015-01-09 12:17:06
nova-scheduler   node-32                              internal         enabled    :-)   2015-01-09 12:17:09
nova-conductor   node-32                              internal         enabled    :-)   2015-01-09 12:17:11
nova-cert        node-32                              internal         enabled    :-)   2015-01-09 12:17:09
nova-consoleauth node-33                              internal         enabled    :-)   2015-01-09 12:17:09
nova-scheduler   node-33                              internal         enabled    :-)   2015-01-09 12:17:09
nova-conductor   node-33                              internal         enabled    :-)   2015-01-09 12:17:09
nova-consoleauth node-36                              internal         enabled    :-)   2015-01-09 12:17:09
nova-scheduler   node-36                              internal         enabled    :-)   2015-01-09 12:17:09
nova-conductor   node-36                              internal         enabled    :-)   2015-01-09 12:17:12
nova-cert        node-33                              internal         enabled    :-)   2015-01-09 12:17:09
nova-cert        node-36                              internal         enabled    :-)   2015-01-09 12:17:09
nova-compute     node-24                              nova             enabled    XXX   2015-01-05 17:16:42
nova-compute     node-37                              nova             enabled    :-)   2015-01-09 12:17:12
nova-compute     node-34                              nova             enabled    XXX   2015-01-05 17:28:53
nova-compute     node-35                              nova             enabled    XXX   2015-01-05 17:28:53
edit retag flag offensive close merge delete

Comments

Can you check if any selinux/apparmor/iptables rules are preventing libvirtd? Take a look at syslog for a clue

Poonam Agarwal gravatar imagePoonam Agarwal ( 2015-01-09 11:24:11 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2015-01-09 16:02:21 -0500

IvanAK gravatar image

We have found the problem. First of all i have forgot to mention that all Openstack are on Ubuntu. Second we lost the communication with the services rabbitmq, and then we find out that Ceph not working on those nodes. So we fix it and reboot the nodes and all things come back again. Thanks for the reply ! I hope someone will get benefit of this question ;).

edit flag offensive delete link more

Comments

Hi Ivan, can you provide more information about the failed communication with RabbitMQ service and why Ceph stopped working on the compute node? Also, is nova-conductor running on the same host as the compute node?

Tabi T gravatar imageTabi T ( 2015-07-10 18:55:37 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

3 followers

Stats

Asked: 2015-01-09 06:15:31 -0500

Seen: 4,729 times

Last updated: Jan 09 '15