Cannot launch instances anymore due to RabbitMQ errors
I used to be able to launch instances from this setup (Packstack single node install, Queens) with no problem. I haven't made changes to the network or anything within OpenStack in months. Now, for whatever reason, I can no longer launch instances - they get stuck in Scheduling. Looking in the nova-scheduler log, I can see rabbitmq errors, and the rabbitmq log is full of:
=ERROR REPORT==== 27-May-2020::15:58:26 ===
closing AMQP connection <0.17487.0> (<node's ip:47928="" -=""> <node's ip="">:5672):
{handshake_timeout,frame_header}
I've tried restarting the service and even full on rebooting, I've reset the guest password to guest, and blown away mnesia - none of these made any difference at all.
Looking back at earlier logs, I can see these errors occasionally were in there as far back as I have logs (January), but they used to still work sometimes. But now it's just giving an error every time.
rabbitmqctl list_connections shows a number of running connections, but the last one in there shows in the logs as initiating 4 days ago. I've rebooted and restarted numerous times since then, so that seems interesting.
What about the status of the cluster : rabbitmqctl cluster_status ? And queue list rabbitmqctl list_queue ?
list_queues gives a very long list of queues, which are all at 0. Not sure if there's a specific one I should be looking at/for? scheduler.<my node="" name=""> is there. And cluster_status shows the node as running.