Ask Your Question
1

AMQP server on controller1:5672 is unreachable: timed out.

asked 2016-10-27 16:14:23 -0600

kaleem gravatar image

updated 2016-10-27 17:02:55 -0600

running openstack mitaka on ubuntu 14.04 with 3 nodes, controller1, compute1, block1.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The thing this error is affecting is causing my nova-compute instance to be in state down and all of my neutron services on both controller and compute node are down. I've tried restarting the services, rabbitmq, and no luck.

1 thing actually did work for me temporarily was adding this entry: "ram_allocation_ratio=3" in the default section of /etc/nova/nova.conf. I got a "Reconnected to AMQP server on controller1:5672 via [amqp] client" message in the nova log and I was able to launch an instance but after 10 minutes the errors below re-emerged.

The error I get in dashboard when trying to launch an instance is "not enough hosts available" and I know it's connected to rabbitmq's port 5672 on my controller blocking off messages from my compute node. I have tried looking at iptables and I have ran this command to try and enable this rabbit port: "iptables -A INPUT -p tcp --dport 5672 -j ACCEPT," and that still didn't work for me.

I have tried adding cpu_allocation_ratio=16 under the default section of nova.conf to see if it was an issue if nova couldn't see my storage, but it did nothing.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

tail -f /var/log/nova/nova.conf on compute1 node gives me this output:

2016-10-27 13:56:11.252 8670 ERROR oslo.messaging._drivers.impl_rabbit [req-bb092395-1187-4ec2-b5f2-10530d0e7905 - - - - -] The broker has blocked the connection: low on disk 2016-10-27 13:56:21.253 8670 ERROR oslo.messaging._drivers.impl_rabbit [req-bb092395-1187-4ec2-b5f2-10530d0e7905 - - - - -] AMQP server on controller1:5672 is unreachable: timed out. Trying again in 1 seconds.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

nova-consoleauth | controller1 | internal | enabled | up

nova-scheduler | controller1 | internal | enabled | up

nova-conductor | controller1 | internal | enabled | up

nova-compute | compute1 | nova | enabled | down

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

DHCP agent | controller1 | nova | xxx | True

Metadata agent | controller1 | | xxx | True

Linux bridge agent | compute1 | | xxx | True

Linux bridge agent | controller1 | | xxx | True

L3 agent | controller1 | nova | xxx | True

edit retag flag offensive close merge delete

2 answers

Sort by ยป oldest newest most voted
0

answered 2016-11-01 04:06:59 -0600

yprokule gravatar image

@kaleem - some other things to check.

On controller - rabbitmqctl status to ensure rmq is up-n-running

On compute - can controller1 be resolved - ping controller1 Access port - nc controller1 5672

Check rabbit credentials in nova.conf both on controller and compute nodes.

Also, enable debug and restart services this might give additional info.

edit flag offensive delete link more

Comments

Also, are either compute or controller low on disk space like the error suggests?

rabbitmqctl cluster_status will show if any rabbit partitions are present. May need to restart rabbit or rejoin the rabbit cluster.

The allocation_ratios in nova.conf are more for scheduling instances.

aludwar gravatar imagealudwar ( 2017-01-10 16:06:33 -0600 )edit
0

answered 2016-10-29 03:15:55 -0600

mesutaygun gravatar image

try ufw disable

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2016-10-27 16:14:23 -0600

Seen: 3,602 times

Last updated: Nov 01 '16