We have run openstackļ¼icehouse on ubt 12.04/14.04ļ¼ for a year. We setup RabbitMQ in master/slave mode . Openstack component connect to MQ via its vip. Some days ago ,the MQ master node went down , so the vip floated to slave node, after master node rebooted , vip floated back to master node.
And we have a problem now: nova service-list shows that all compute node are "down".
But in fact , there exists connection between compute node and MQ
root@computenode:~# sudo netstat -antp |grep 5672
tcp 0 0 10.0.0.10:28846 10.0.2.7:5672 ESTABLISHED 53206/python
tcp 0 0 10.0.0.10:42368 10.0.2.7:5672 ESTABLISHED 2519/python
tcp 0 0 10.0.0.10:28847 10.0.2.7:5672 ESTABLISHED 53206/python
tcp 0 0 10.0.0.10:42367 10.0.2.7:5672 ESTABLISHED 2519/python
tcp 0 0 10.0.0.10:28845 10.0.2.7:5672 ESTABLISHED 53206/python
pid 2519 is neutron-openvswitch-agent pid 53206 is nova-compute
I know that if I restart nova-compute on the compute node, it will fix the problem , but it will cause a short network outage(some seconds) for the VM on that compute node .
My question is that : Is there any way to fix this problem without causing a short network outage for the VM ? Thank you .