Compute nodes always showing MessagingTimeOut errors

asked 2017-11-14 04:22:59 -0600

vathanlal gravatar image

updated 2017-11-14 04:24:17 -0600

Hello,

I have an OpenStack test cluster with 10 compute nodes and 1 controller node. Everything is fine I can launch instance, migrate, attach volumes, snapshot instances all is fine. But in all my compute nodes I can see MessagingTimeout errors regularly. It clearly reflect in my rabitmq logs also. I can also see DB exceeded retry limit error in nova-conductor.log. But after couple of seconds everything will be back and fine. And all the services are seems to be running fine I can launch instances. Iam really curious to know why this happens. Any help is really appreciated.

/var/log/nova/nova-compute.log

 2017-11-14 08:53:09.812 42708 WARNING nova.scheduler.client.report [req-724b0dad-4823-421a-a158-721375f881ea - - - - -] No authentication information found for placement API. Placement is optional in Newton, but required in Ocata. Please enable the placement service before upgrading.
2017-11-14 08:53:09.813 42708 WARNING nova.scheduler.client.report [req-724b0dad-4823-421a-a158-721375f881ea - - - - -] Unable to refresh my resource provider record
2017-11-14 08:53:09.814 42708 INFO nova.compute.resource_tracker [req-724b0dad-4823-421a-a158-721375f881ea - - - - -] Compute_service record updated for server-1:server-1
2017-11-14 08:53:18.786 42708 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 35.71 sec
2017-11-14 08:53:27.172 42708 INFO nova.compute.resource_tracker [req-724b0dad-4823-421a-a158-721375f881ea - - - - -] Auditing locally available compute resources for node server-1
2017-11-14 08:54:18.798 42708 WARNING nova.servicegroup.drivers.db [-] Lost connection to nova-conductor for reporting service status.
2017-11-14 08:54:18.800 42708 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 50.01 sec
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager [req-724b0dad-4823-421a-a158-721375f881ea - - - - -] Error updating resources for node server-1.
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager Traceback (most recent call last):
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 6442, in update_available_resource_for_node
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager     rt.update_available_resource(context)
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 511, in update_available_resource
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager     resources = self.driver.get_available_resource(self.nodename)
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 5385, in get_available_resource
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager     disk_over_committed = self._get_disk_over_committed_size_total()
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 6955, in _get_disk_over_committed_size_total
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager     ctx, filters, use_slave=True)
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 177, in wrapper
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager     args, kwargs)
2017-11-14 08:54:35.190 42708 ERROR nova.compute.manager   File "/usr/lib/python2.7/dist-packages/nova ...
(more)
edit retag flag offensive close merge delete