Stuck while creating a volume in Cinder, AMQP seems broken

asked 2017-04-24 07:39:01 -0500

1 ●1 ●1 ●1

Hello everyone, I'm following a course in EDX for getting an introduction to Openstack and I'm stuck for creating a volume in Cinder.

I have installated Devstack Ocata on Ubuntu 16.04 LTS in a VM with the Compute, Storage and Network nodes. The installation works fine, I can have an access to Openstack by the Dashboard (Horizon) and also by the CLI.

I can make instances, networks, images but something is wrong about creating volume. When I try to create one volume, it gets stuck at "Creating". So I decided to use the CLI for making a "cinder force-delete" command and retrying to recreate a volume, same trouble...

Into the c-api.logs (using pipe grep ERROR), I got this :

2017-04-24 09:49:10.522 ERROR oslo.messaging._drivers.impl_rabbit [req-0ed54049-e1a8-45f4-9cb9-9f75dbacf191 myproject myuser] [062a0bfb-b45f-44dd-9462-94366f70123b] AMQP server on 192.168.30.155:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds. Client port: 49878

It seems that Cinder can't communicate with RabbitMQ (the AMQP service). So I've checked if RabbitMQ-Server works on my server :

 service rabbitmq-server status 
rabbitmq-server.service - RabbitMQ Messaging Server
       Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
       Active: active (running) since lun. 2017-04-24 11:12:17 CEST; 36min ago
      Process: 656 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
      Process: 776 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
     Main PID: 775 (rabbitmq-server) Tasks: 70
       Memory: 569.0M
          CPU: 11.648s
       CGroup: /system.slice/rabbitmq-server.service
               ├─ 775 /bin/sh /usr/sbin/rabbitmq-server
               ├─ 780 /bin/sh -e /usr/lib/rabbitmq/bin/rabbitmq-server
               ├─ 869 /usr/lib/erlang/erts-7.3/bin/epmd -daemon
               ├─ 916 /usr/lib/erlang/erts-7.3/bin/beam -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s
               ├─1027 inet_gethost 4
               └─1028 inet_gethost 4 avril 24 11:12:11 openstack systemd[1]: Starting RabbitMQ Messaging Server...
    avril 24 11:12:13 openstack rabbitmq[776]: Waiting for rabbit@openstack ...
    avril 24 11:12:13 openstack rabbitmq[776]: pid is 780 ...
    avril 24 11:12:17 openstack systemd[1]: Started RabbitMQ Messaging Server.

Apparently, no trouble, so after that, I've checked the state of network ports and here's the result :

netstat -an | grep 5672
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN
tcp        0      0 192.168.30.155:33740    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57618    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57948    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57668    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57960    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57664    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57774    192 ...

(more)

Comments

49878 is the client port. No wonder nothing is listening on that port. Your problem is somewhere else.

Bernd Bausch ( 2017-04-24 17:19:35 -0500 )edit

Broken Pipe means that the amqp server closed the connection. I don't know where the amqp logs are on devstack, but I would guess some more information could be found there. Wrong amqp credentials perhaps?

Bernd Bausch ( 2017-04-24 23:42:00 -0500 )edit

add a comment

answered 2017-04-25 03:55:23 -0500

Loïck75

1 ●1 ●1 ●1

updated 2017-04-25 07:40:35 -0500

PROBLEM RESOLVED.

The fact is that's not about AMQP problems, but about cinder-volume & cinder-scheduler that are not working, as I said before.

To see in real time what's going on during an instance creation on RabbitMQ, I've seen in real time logs by using the command tail -f /var/log/rabbitmq/rabbit@openstack.log (logs for RMQ are located here). And there's no "ERROR" log during the execution of instance creation. Here's what I got :

=INFO REPORT==== 25-Apr-2017::09:48:52 ===
accepting AMQP connection <0.13986.0> (192.168.30.155:53962 -> 192.168.30.155:5672)

As Bernd said before, the problem is somewhere else. So I decided to see if there's services in execution about Cinder by executing the command ps -an | grep cinder and I saw that there's no cinder-scheduler & cinder-volume service in running state.

I've tried to find if there's a path to execute these services in order to revive correctly Cinder. By using the command head -n 10 for c-sch.log and c-vol.log, I've found the anwser to my question.

/usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf

/usr/local/bin/cinder-scheduler --config-file /etc/cinder/cinder.conf

By using the screen command, I've created two screens by naming them c-sch and c-vol.

1) screen -S c-sch

2) Then execute the services related to as shown below and let the magic operate

3) Ctrl-A + D for being detached from the screen

4) Do the same thing for c-vol

According to their names, I've executed the services related to. Detaching from the screens and executing the command openstack volume service list and I can see that there are now working !

+------------------+-----------------------+------+---------+-------+----------------------------+
| Binary           | Host                  | Zone | Status  | State | Updated At                 |
+------------------+-----------------------+------+---------+-------+----------------------------+
| cinder-scheduler | openstack             | nova | enabled | up    | 2017-04-25T08:49:01.000000 |
| cinder-volume    | openstack@lvmdriver-1 | nova | enabled | up    | 2017-04-25T08:49:01.000000 |
+------------------+-----------------------+------+---------+-------+----------------------------+

I don't know if it's the good way to resolve this issue but now, I can create volumes without doubt and attach them to my instances.

edit flag offensive delete link

Comments

You could have figured out the command by going into the screens of c-vol and c-sch, then going back one step in the shell command history. Normally the command should be shown then. But what you did looks correct as well.

Bernd Bausch ( 2017-04-25 04:25:08 -0500 )edit

So why didn't they start in the first place? Perhaps a race condition or other timing-related glitch. As we say here in Japan: Omedeto (félicitations).

Bernd Bausch ( 2017-04-25 04:25:48 -0500 )edit

I've edited my last post about how I figured out by using screen to launch these services, it's better now for those who's having the same trouble.

I remembered that I've done a mistake at the beginning by killing two screens where (maybe) these services runs. Don't ask me why, maybe curiosity!

Loïck75 ( 2017-04-25 07:45:03 -0500 )edit

And Arigato !

Loïck75 ( 2017-04-25 07:45:29 -0500 )edit

@Loick75 , I still didn't get it , how is attach and dettach the screen reoslved the issue of scheduler and volume , in my case controller runs scheduler service and cinder-volume service is run on compute nodes , how did you actually solve this problem ? The Attach and Dettach seems magical ?

vgdub ( 2018-11-15 19:15:49 -0500 )edit

add a comment

Stuck while creating a volume in Cinder, AMQP seems broken

Comments

1 answer

Comments

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Stuck while creating a volume in Cinder, AMQP seems broken edit

Comments

1 answer

Comments

Get to know Ask OpenStack

Question Tools

Stats

Related questions

Feedback About This Page

OpenStack

Community

Documentation

Branding & Legal

Stuck while creating a volume in Cinder, AMQP seems broken