Ask Your Question
0

Stuck while creating a volume in Cinder, AMQP seems broken

asked 2017-04-24 07:39:01 -0600

Loïck75 gravatar image

Hello everyone, I'm following a course in EDX for getting an introduction to Openstack and I'm stuck for creating a volume in Cinder.

I have installated Devstack Ocata on Ubuntu 16.04 LTS in a VM with the Compute, Storage and Network nodes. The installation works fine, I can have an access to Openstack by the Dashboard (Horizon) and also by the CLI.

I can make instances, networks, images but something is wrong about creating volume. When I try to create one volume, it gets stuck at "Creating". So I decided to use the CLI for making a "cinder force-delete" command and retrying to recreate a volume, same trouble...

Into the c-api.logs (using pipe grep ERROR), I got this :

2017-04-24 09:49:10.522 ERROR oslo.messaging._drivers.impl_rabbit [req-0ed54049-e1a8-45f4-9cb9-9f75dbacf191 myproject myuser] [062a0bfb-b45f-44dd-9462-94366f70123b] AMQP server on 192.168.30.155:5672 is unreachable: [Errno 32] Broken pipe. Trying again in 1 seconds. Client port: 49878

It seems that Cinder can't communicate with RabbitMQ (the AMQP service). So I've checked if RabbitMQ-Server works on my server :

 service rabbitmq-server status 
rabbitmq-server.service - RabbitMQ Messaging Server
       Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
       Active: active (running) since lun. 2017-04-24 11:12:17 CEST; 36min ago
      Process: 656 ExecStop=/usr/sbin/rabbitmqctl stop (code=exited, status=0/SUCCESS)
      Process: 776 ExecStartPost=/usr/lib/rabbitmq/bin/rabbitmq-server-wait (code=exited, status=0/SUCCESS)
     Main PID: 775 (rabbitmq-server) Tasks: 70
       Memory: 569.0M
          CPU: 11.648s
       CGroup: /system.slice/rabbitmq-server.service
               ├─ 775 /bin/sh /usr/sbin/rabbitmq-server
               ├─ 780 /bin/sh -e /usr/lib/rabbitmq/bin/rabbitmq-server
               ├─ 869 /usr/lib/erlang/erts-7.3/bin/epmd -daemon
               ├─ 916 /usr/lib/erlang/erts-7.3/bin/beam -W w -A 64 -P 1048576 -K true -B i -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.5.7/sbin/../ebin -noshell -noinput -s
               ├─1027 inet_gethost 4
               └─1028 inet_gethost 4 avril 24 11:12:11 openstack systemd[1]: Starting RabbitMQ Messaging Server...
    avril 24 11:12:13 openstack rabbitmq[776]: Waiting for rabbit@openstack ...
    avril 24 11:12:13 openstack rabbitmq[776]: pid is 780 ...
    avril 24 11:12:17 openstack systemd[1]: Started RabbitMQ Messaging Server.

Apparently, no trouble, so after that, I've checked the state of network ports and here's the result :

netstat -an | grep 5672
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN
tcp        0      0 192.168.30.155:33740    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57618    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57948    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57668    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57960    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57664    192.168.30.155:5672     ESTABLISHED
tcp        0      0 192.168.30.155:57774    192 ...
(more)
edit retag flag offensive close merge delete

Comments

49878 is the client port. No wonder nothing is listening on that port. Your problem is somewhere else.

Bernd Bausch gravatar imageBernd Bausch ( 2017-04-24 17:19:35 -0600 )edit

Broken Pipe means that the amqp server closed the connection. I don't know where the amqp logs are on devstack, but I would guess some more information could be found there. Wrong amqp credentials perhaps?

Bernd Bausch gravatar imageBernd Bausch ( 2017-04-24 23:42:00 -0600 )edit
add a comment

1 answer

Sort by » oldest newest most voted
0

answered 2017-04-25 03:55:23 -0600

Loïck75 gravatar image

updated 2017-04-25 07:40:35 -0600

PROBLEM RESOLVED.

The fact is that's not about AMQP problems, but about cinder-volume & cinder-scheduler that are not working, as I said before.

To see in real time what's going on during an instance creation on RabbitMQ, I've seen in real time logs by using the command tail -f /var/log/rabbitmq/rabbit@openstack.log (logs for RMQ are located here). And there's no "ERROR" log during the execution of instance creation. Here's what I got :

=INFO REPORT==== 25-Apr-2017::09:48:52 ===
accepting AMQP connection <0.13986.0> (192.168.30.155:53962 -> 192.168.30.155:5672)

As Bernd said before, the problem is somewhere else. So I decided to see if there's services in execution about Cinder by executing the command ps -an | grep cinder and I saw that there's no cinder-scheduler & cinder-volume service in running state.

I've tried to find if there's a path to execute these services in order to revive correctly Cinder. By using the command head -n 10 for c-sch.log and c-vol.log, I've found the anwser to my question.

/usr/local/bin/cinder-volume --config-file /etc/cinder/cinder.conf

&

/usr/local/bin/cinder-scheduler --config-file /etc/cinder/cinder.conf

By using the screen command, I've created two screens by naming them c-sch and c-vol.

1) screen -S c-sch

2) Then execute the services related to as shown below and let the magic operate

3) Ctrl-A + D for being detached from the screen

4) Do the same thing for c-vol

According to their names, I've executed the services related to. Detaching from the screens and executing the command openstack volume service list and I can see that there are now working ! 

+------------------+-----------------------+------+---------+-------+----------------------------+
| Binary           | Host                  | Zone | Status  | State | Updated At                 |
+------------------+-----------------------+------+---------+-------+----------------------------+
| cinder-scheduler | openstack             | nova | enabled | up    | 2017-04-25T08:49:01.000000 |
| cinder-volume    | openstack@lvmdriver-1 | nova | enabled | up    | 2017-04-25T08:49:01.000000 |
+------------------+-----------------------+------+---------+-------+----------------------------+

I don't know if it's the good way to resolve this issue but now, I can create volumes without doubt and attach them to my instances.

edit flag offensive delete link more

Comments

You could have figured out the command by going into the screens of c-vol and c-sch, then going back one step in the shell command history. Normally the command should be shown then. But what you did looks correct as well.

Bernd Bausch gravatar imageBernd Bausch ( 2017-04-25 04:25:08 -0600 )edit

So why didn't they start in the first place? Perhaps a race condition or other timing-related glitch. As we say here in Japan: Omedeto (félicitations).

Bernd Bausch gravatar imageBernd Bausch ( 2017-04-25 04:25:48 -0600 )edit

I've edited my last post about how I figured out by using screen to launch these services, it's better now for those who's having the same trouble.

I remembered that I've done a mistake at the beginning by killing two screens where (maybe) these services runs. Don't ask me why, maybe curiosity!

Loïck75 gravatar imageLoïck75 ( 2017-04-25 07:45:03 -0600 )edit

And Arigato !

Loïck75 gravatar imageLoïck75 ( 2017-04-25 07:45:29 -0600 )edit

@Loick75 , I still didn't get it , how is attach and dettach the screen reoslved the issue of scheduler and volume , in my case controller runs scheduler service and cinder-volume service is run on compute nodes , how did you actually solve this problem ? The Attach and Dettach seems magical ?

vgdub gravatar imagevgdub ( 2018-11-15 19:15:49 -0600 )edit
add a comment

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2017-04-24 05:10:11 -0600

Seen: 551 times

Last updated: Apr 25 '17

Related questions

Cinder: find who created some volume

AMQP server on localhost:5672 is unreachable: socket closed [closed]

Cannot start instance after reboot

Cinder Volume Cache

error in cinder-volume

Cinder Bad Request (HTTP 400)

Setup on compute node with no cinder storage

Can I create a snapshot from a server (vm)?

Intermittent issues with Cinder block storage attaching.

cinder type-create fails