Stuck while creating a volume in Cinder, AMQP seems broken
Hello everyone, I'm following a course in EDX for getting an introduction to Openstack and I'm stuck for creating a volume in Cinder.
I have installated Devstack Ocata on Ubuntu 16.04 LTS in a VM with the Compute, Storage and Network nodes. The installation works fine, I can have an access to Openstack by the Dashboard (Horizon) and also by the CLI.
I can make instances, networks, images but something is wrong about creating volume. When I try to create one volume, it gets stuck at "Creating". So I decided to use the CLI for making a "cinder force-delete" command and retrying to recreate a volume, same trouble...
Into the c-api.logs
(using pipe grep ERROR
), I got this :
2017-04-24 09:49:10.522 ERROR oslo.messaging._drivers.impl_rabbit [req-0ed54049-e1a8-45f4-9cb9-9f75dbacf191 myproject myuser] [062a0bfb-b45f-44dd-9462-94366f70123b] AMQP server on 192.168.30.155:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds. Client port: 49878
It seems that Cinder can't communicate with RabbitMQ (the AMQP service). So I've checked if RabbitMQ-Server works on my server :
service rabbitmq-server status
rabbitmq-server.service - RabbitMQ Messaging Server
Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
Active: active (running) since lun. 2017-04-24 11:12:17 CEST; 36min ago
Process: 656 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
Process: 776 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
Main PID: 775 (rabbitmq-server) Tasks: 70
Memory: 569.0M
CPU: 11.648s
CGroup: /system.slice/rabbitmq-server.service
├─ 775 /bin/sh /usr/sbin/rabbitmq-server
├─ 780 /bin/sh -e /usr/lib/rabbitmq/bin/rabbitmq-server
├─ 869 /usr/lib/erlang/erts-7.3/bin/epmd -daemon
├─ 916 /usr/lib/erlang/erts-7.3/bin/beam -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s
├─1027 inet_gethost 4
└─1028 inet_gethost 4 avril 24 11:12:11 openstack systemd[1]: Starting RabbitMQ Messaging Server...
avril 24 11:12:13 openstack rabbitmq[776]: Waiting for rabbit@openstack ...
avril 24 11:12:13 openstack rabbitmq[776]: pid is 780 ...
avril 24 11:12:17 openstack systemd[1]: Started RabbitMQ Messaging Server.
Apparently, no trouble, so after that, I've checked the state of network ports and here's the result :
netstat -an | grep 5672
tcp 0 0 0.0.0.0:25672 0.0.0.0:* LISTEN
tcp 0 0 192.168.30.155:33740 192.168.30.155:5672 ESTABLISHED
tcp 0 0 192.168.30.155:57618 192.168.30.155:5672 ESTABLISHED
tcp 0 0 192.168.30.155:57948 192.168.30.155:5672 ESTABLISHED
tcp 0 0 192.168.30.155:57668 192.168.30.155:5672 ESTABLISHED
tcp 0 0 192.168.30.155:57960 192.168.30.155:5672 ESTABLISHED
tcp 0 0 192.168.30.155:57664 192.168.30.155:5672 ESTABLISHED
tcp 0 0 192.168.30.155:57774 192 ...
49878 is the client port. No wonder nothing is listening on that port. Your problem is somewhere else.
Broken Pipe means that the amqp server closed the connection. I don't know where the amqp logs are on devstack, but I would guess some more information could be found there. Wrong amqp credentials perhaps?