Ask Your Question
1

AMQP server on 10.0.0.11:5672 is unreachable

asked 2014-11-27 01:24:11 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

Hi All,

Could someone please help me to resolve the issue which i am facing. I done everything on controller what we needed on it. Now i am working on compute node when i installed packages and changes done in /etc/nova/nova.conf finally when i start the service getting below error.

"[root@compute nova]# systemctl start openstack-nova-compute.service
Job for openstack-nova-compute.service failed. See 'systemctl status openstack-nova-compute.service' and 'journalctl -xn' for details."

and found some below logs in /var/log/nova/nova-compute.log.

"[root@compute nova]# tailf nova-compute.log
2014-11-26 14:56:30.963 28832 INFO oslo.messaging._drivers.impl_rabbit [req-f4de0a1c-52d5-47bc-aa80-c69ece3480c4 ] Delaying reconnect for 1.0 seconds...
2014-11-26 14:56:31.966 28832 INFO oslo.messaging._drivers.impl_rabbit [req-f4de0a1c-52d5-47bc-aa80-c69ece3480c4 ] Connecting to AMQP server on 10.0.0.11:5672
2014-11-26 14:56:31.994 28832 ERROR oslo.messaging._drivers.impl_rabbit [req-f4de0a1c-52d5-47bc-aa80-c69ece3480c4 ] AMQP server on 10.0.0.11:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 7 seconds."

BR, Kishore

edit retag flag offensive close merge delete

7 answers

Sort by » oldest newest most voted
4

answered 2014-11-27 03:48:34 -0500

Ram.Meena gravatar image

First I would suggest you to verify the 'RabbitMQ status' and make sure that all OpenStack nodes are able to ping/connect with RabbitMQ server.

To verify if RabbitMQ service is running on your rabbit server, run below command: -

#systemctl status rabbitmq-server.service -l

To verify the RabbitMQ server status run the below command: -

#rabbitmqctl status

This command should print a detailed output of various information related to AMQP server. If the status is good then you need to make sure that rabbitmq port is allowed to make connection between your openstack nodes through firewall.

On RabbitMQ server you may run the below command to check if your compute node is able to establish connection with rabbit server:-

   # lsof -i :5672|grep 'compute_node_ip'

Make sure that there is no time lag between your openstack node, all the nodes should be synced with a central server time using NTP.

edit flag offensive delete link more

Comments

Do you mean raun these all three commands on compute-node ?

Kishore Devkate gravatar imageKishore Devkate ( 2014-11-27 04:50:38 -0500 )edit

If your RabbitMQ server is running on controller node then you need to run these commands on controller node not on compute node. These commands are for RabbitMQ service diagnosis.

Ram.Meena gravatar imageRam.Meena ( 2014-11-27 05:52:34 -0500 )edit

@Ram.Meena, I ran all the command in the last post.. on controller node which rabbitmq installed on controller only.

Kishore Devkate gravatar imageKishore Devkate ( 2014-11-27 06:19:19 -0500 )edit

Did you check if firewall is blocking the connection on '5672' port between controller and compute node?. You should check by disabling the firewall and disabling selinux for diagnosis purposes.

Ram.Meena gravatar imageRam.Meena ( 2014-11-27 08:36:51 -0500 )edit

@Ram.Meena,

yes it was giving trouble so I disabled it. now working fine. Thanks Ram Meena for the help.

BR, Kishore

Kishore Devkate gravatar imageKishore Devkate ( 2014-11-27 09:26:03 -0500 )edit
3

answered 2015-08-19 17:59:51 -0500

Rajan433 gravatar image

I had the same problem. Added firewall rule to allow port 5672 and issue got resolved.

edit flag offensive delete link more

Comments

1

this helped me on controller node (where rabbitmq is running)

sudo iptables -A IN_public_allow -p tcp -m tcp --dport 5672 -m conntrack --ctstate NEW -j ACCEPT
vovailchenko gravatar imagevovailchenko ( 2015-12-29 11:01:37 -0500 )edit

Hi when I tried this command it is not accepting --dport

ramraj.s@ktr.srmuniv.ac.in gravatar imageramraj.s@ktr.srmuniv.ac.in ( 2016-08-31 07:38:28 -0500 )edit

this helped me in controller node (Ubuntu 16.04 server):

 # sudo iptables -I INPUT -p tcp --dport 5672 -j ACCEPT
 # sudo netfilter-persistent save
 # sudo netfilter-persistent reload
michelgokan3 gravatar imagemichelgokan3 ( 2017-10-17 04:47:44 -0500 )edit
0

answered 2015-06-16 09:22:33 -0500

Karan Chhabra gravatar image

updated 2015-06-16 09:23:13 -0500

@ RamMeena. Its not firewall! I disabled and did check but still was not able to find nova-compute through my controller node. I am running nova service-list and in result i see nova-cert, nova-scheduler, nova-conductor and nova-consoleauth up and running but can't see nova compute!

Please help me out! (RDO Kilo)

edit flag offensive delete link more
0

answered 2014-11-27 05:30:07 -0500

this post is marked as community wiki

This post is a wiki. Anyone with karma >75 is welcome to improve it.

@Ram.Meena,

Below is the status which you requested to check.

"[root@controller ~]# systemctl status rabbitmq-server.service -l

rabbitmq-server.service - RabbitMQ broker
   Loaded: loaded (/usr/lib/systemd/system/rabbitmq-server.service; enabled)
   Active: active (running) since Wed 2014-11-26 19:15:17 IST; 27min ago
  Process: 47237 ExecStopPost=/usr/bin/rm /var/run/rabbitmq/pid (code=exited, status=0/SUCCESS)
  Process: 47210 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
  Process: 47268 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmqctl wait /var/run/rabbitmq/pid (code=exited, status=0/SUCCESS)
  Process: 47239 ExecStartPre=/bin/sh -c /usr/lib/rabbitmq/bin/rabbitmqctl status > /dev/null 2>&1 (code=exited, status=2)
 Main PID: 47267 (beam)
   CGroup: /system.slice/rabbitmq-server.service
           ââ47267 /usr/lib64/erlang/erts-5.10.4/bin/beam -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../ebin -noshell -noinput -s rabbit boot -sname rabbit@controller -boot start_sasl -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/rabbit@controller.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/rabbit@controller-sasl.log"} -rabbit enabled_plugins_file "/etc/rabbitmq/enabled_plugins" -rabbit plugins_dir "/usr/lib/rabbitmq/lib/rabbitmq_server-3.1.5/sbin/../plugins" -rabbit plugins_expand_dir "/var/lib/rabbitmq/mnesia/rabbit@controller-plugins-expand" -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/var/lib/rabbitmq/mnesia/rabbit@controller"
           ââ47368 inet_gethost 4
           ââ47369 inet_gethost 4
Nov 26 19:15:15 controller rabbitmqctl[47268]: pid is 47267 ...
Nov 26 19:15:15 controller rabbitmq-server[47267]: RabbitMQ 3.1.5. Copyright (C) 2007-2013 GoPivotal, Inc.
Nov 26 19:15:15 controller rabbitmq-server[47267]: ##  ##      Licensed under the MPL.  See http://www.rabbitmq.com/
Nov 26 19:15:15 controller rabbitmq-server[47267]: ##  ##
Nov 26 19:15:15 controller rabbitmq-server[47267]: ##########  Logs: /var/log/rabbitmq/rabbit@controller.log
Nov 26 19:15:15 controller rabbitmq-server[47267]: ######  ##        /var/log/rabbitmq/rabbit@controller-sasl.log
Nov 26 19:15:15 controller rabbitmq-server[47267]: ##########
Nov 26 19:15:16 controller rabbitmq-server[47267]: Starting broker... completed with 0 plugins.
Nov 26 19:15:17 controller rabbitmqctl[47268]: ...done.
Nov 26 19:15:17 controller systemd[1]: Started RabbitMQ broker."


"[root@controller ~]# rabbitmqctl status

Status of node rabbit@controller ...
[{pid,47267},
 {running_applications,[{rabbit,"RabbitMQ","3.1.5"},
                        {os_mon,"CPO  CXC 138 46","2.2.14"},
                        {xmerl,"XML parser","1.3.6"},
                        {mnesia,"MNESIA  CXC 138 12","4.11"},
                        {sasl,"SASL  CXC 138 11","2.3.4"},
                        {stdlib,"ERTS  CXC 138 10","1.19.4"},
                        {kernel,"ERTS  CXC 138 10","2.16.4"}]},
 {os,{unix,linux}},
 {erlang_version,"Erlang R16B03-1 (erts-5.10.4) [source] [64-bit] [async-threads:30] [hipe] [kernel-poll:true]\n"},
 {memory,[{total,35439832},
          {connection_procs,36904},
          {queue_procs,5440},
          {plugins,0},
          {other_proc,13412016},
          {mnesia,57232},
          {mgmt_db,0},
          {msg_index,33936},
          {other_ets,727128},
          {binary,670208},
          {code,16441148},
          {atom,594537},
          {other_system,3461283}]},
 {vm_memory_high_watermark,0.4},
 {vm_memory_limit,414043340},
 {disk_free_limit,1000000000},
 {disk_free,7424532480},
 {file_descriptors,[{total_limit,924},
                    {total_used,4},
                    {sockets_limit,829},
                    {sockets_used,2}]},
 {processes,[{limit,1048576},{used,127}]},
 {run_queue,0},
 {uptime,1657}]
...done."

"[root@controller ...
(more)
edit flag offensive delete link more

Comments

RabbitMQ server seems to be running fine. However, there is no connection established with compute node so it seems that firewall is blocking the access. You should disable the firewall for diagnosis and check it. Also make sure that there is no lag between controller and compute nodes.

Ram.Meena gravatar imageRam.Meena ( 2014-11-27 08:33:54 -0500 )edit
0

answered 2014-11-27 10:59:02 -0500

amitabh sinha gravatar image

I have faced this issue before...you first change your rabbitmq password rabbitmqctl change-password guest guest. then after that reinstall ovs. It seems very wired but trust me it worked out for me.!

edit flag offensive delete link more

Comments

Changing the password helped for me.

Tuxfux gravatar imageTuxfux ( 2015-09-21 14:47:40 -0500 )edit
0

answered 2014-11-27 03:19:48 -0500

dbaxps gravatar image
edit flag offensive delete link more

Comments

@dbaxps,

Yet i did not installed neutron service. I stucked at compute configuration on compute node.

BR, Kishore

Kishore Devkate gravatar imageKishore Devkate ( 2014-11-27 07:00:03 -0500 )edit

It doesn't matter the link is about rabbitmq-sever ( AMQP Broker ) access . Same actions on Controller, no matter which service is suffering due to rabbitmq access.

dbaxps gravatar imagedbaxps ( 2014-11-27 07:11:44 -0500 )edit
0

answered 2015-11-25 16:32:41 -0500

rushi gravatar image

most of the errors from this type is usually the rabbitmq user password. At any point, after you change password you need to execute OVS setup. OVS Setup is the fix for any issues related to neutron.

so follow the 3 steps and this issue will never show up.

First install OVS ( i use SUSE 11 SP3 so the tool is "zypper") you can use the tools part of LINUX distros with)

zypper -n in openvswitch openvswitch-switch python-openvswitch

Start the service

systemctl start openvswitch-switch.service

Add the interface or bridge (in my case it was br-ex) ovs-vsctl --may-exist add-br br-ex

enable the service to start from boot.

systemctl enable openvswitch-switch.service

Tha't it.

BR, Rushi

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2014-11-27 01:24:11 -0500

Seen: 25,555 times

Last updated: Nov 25 '15