Timeout between Nova-Conductor and Nova-Compute on ARM64 (RaspberryPi, RockPi)
Hi there,
currently we are trying to run Openstack Stein on multiple RaspberryPis and RockPis. The goal is to make some performance tests on Load Balancing, High Availability and Real-Time Capabilities.
Current Set-Up:
1 Rock Pi (Ubuntu Server 18.04): Keystone, Horizon, Glance, Nova(-Server)
3 Raspberry Pi Model 3 (Ubuntu Server 18.04 and Raspbian): Nova-Compute
The implementation of the RockPi as Controller Node was successfull. We were able to connect it to a Nova-Compute Node running for test purpose in a VirtualBox and start a VM via the Dashboard. The Management network is realized with VLAN. The public interface is the normal eth0 interface without VLAN Tag.
Now we wanted to implement the Nova-Compute Node on a RaspberryPi, but got stuck there: The installation of nova-compute was successfull. nova-compute and libvirt services are running. The problem is the initial connection of controller node with the compute node. As far as logs showed us, the compute node is telling the controller node its HW capabilites, but after that a TIMEOUT occures on the compute node (see logfile as picture). We think that there is a service/module/appliction not running correctly where the compute node does not answer correctly on a message of the controller. We analyzed some IP Traffic between a) controller - compute (on Virtualbox) which was successfull and b) controller - compute (RaspberryPi) which wasn't successfull.
The Log file:
2019-08-18 15:13:25.986 3076 WARNING nova.virt.libvirt.driver [-] The libvirt driver is not tested on qemu/aarch64 by the OpenStack project and thus its quality can not be ensured. For more information, see: https://docs.openstack.org/nova/latest/user/support-matrix.html (https://docs.openstack.org/nova/lates...)
2019-08-18 15:14:25.993 3076 WARNING nova.virt.libvirt.driver [req-d67c6d73-cbc4-450e-8d94-546612678d33 - - - - -] Cannot update service status on host "compute04" due to an unexpected exception.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID e9939a447dd94c05b0874fb03f05e533
2019-08-18 15:14:25.993 3076 ERROR nova.virt.libvirt.driver Traceback (most recent call last):
2019-08-18 15:14:25.993 3076 ERROR nova.virt.libvirt.driver File "/usr/lib/python3/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 393, in get
2019-08-18 15:14:25.993 3076 ERROR nova.virt.libvirt.driver return self._queues[msg_id].get(block=True, timeout=timeout)
2019-08-18 15:14:25.993 3076 ERROR nova.virt.libvirt.driver File "/usr/lib/python3/dist-packages/eventlet/queue.py", line 322, in get
2019-08-18 15:14:25.993 3076 ERROR nova.virt.libvirt.driver return waiter.wait()
2019-08-18 15:14:25.993 3076 ERROR nova.virt.libvirt.driver File "/usr/lib/python3/dist-packages/eventlet/queue.py", line 141, in wait
2019-08-18 15:14:25.993 3076 ERROR nova.virt.libvirt.driver return get_hub().switch()
2019-08-18 15:14:25.993 3076 ERROR nova.virt.libvirt.driver File "/usr/lib/python3/dist-packages/eventlet/hubs/hub.py", line 298, in switch
2019-08-18 15:14:25.993 3076 ERROR nova.virt.libvirt.driver return self.greenlet.switch()
2019-08-18 15:14:25.993 3076 ERROR ...