Cinder volume creation status creating error

asked 2017-07-18 15:33:44 -0600

daniel hernandez gravatar image

updated 2017-07-18 18:48:47 -0600

Ocata installation in Ubuntu 16.04. after followed official installation for Block Storage service in controller and storage nodes and verify installation, everything seems OK.

openstack volume service list
+------------------+------------+------+---------+-------+----------------------------+
| Binary           | Host       | Zone | Status  | State | Updated At                 |
+------------------+------------+------+---------+-------+----------------------------+
| cinder-scheduler | controller | nova | enabled | up    | 2017-07-18T20:14:04.000000 |
| cinder-volume    | cinder@lvm | nova | enabled | up    | 2017-07-18T20:14:10.000000 |
+------------------+------------+------+---------+-------+----------------------------+

I create a volume of 1G however after more than one hour it stuck as creating

openstack volume list
+--------------------------------------+--------------+----------+------+-------------+
| ID                                   | Display Name | Status   | Size | Attached to |
+--------------------------------------+--------------+----------+------+-------------+
| 1030e06f-d722-4bb6-94cd-ee3d4c58dd8d | volume1      | creating |    1 |             |
+--------------------------------------+--------------+----------+------+-------------+

I checked in cinder-scheduler.log at controller node and it does not display any error message. also I cheched in 2 error messages are displayed

ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on controller:5672 is unreachable: timed out. Trying again in 8 seconds.
ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on controller:5672 is unreachable: [Errno 111] ECONNREFUSED. Trying again in 6 seconds.

back to controller node I see that port is listening

netstat -an | grep 5672
tcp        0      0 0.0.0.0:25672           0.0.0.0:*               LISTEN

then I checked in rabbit@controller.log which seems that communication is not successfully.

=INFO REPORT==== 18-Jul-2017::14:03:39 ===
accepting AMQP connection <0.1386.0> (10.1.1.103:44624 -> 10.1.1.101:5672)

=ERROR REPORT==== 18-Jul-2017::14:03:39 ===
closing AMQP connection <0.1386.0> (10.1.1.103:44624 -> 10.1.1.101:5672):
{handshake_timeout,handshake}

I dont find cases similar to it for a workaround, please help me with this.

my cinder.conf in controller is:

[DEFAULT]
transport_url = rabbit://openstack:root@controller
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
my_ip = 10.1.1.101

[database]
connection = mysql+pymysql://cinder:cinder@controller/cinder

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = cinder

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp

my cinder.conf in node is:

[DEFAULT]
rootwrap_config = /etc/cinder/rootwrap.conf
api_paste_confg = /etc/cinder/api-paste.ini
iscsi_helper = tgtadm
volume_name_template = volume-%s
volume_group = cinder-volumes
verbose = True
auth_strategy = keystone
state_path = /var/lib/cinder
lock_path = /var/lock/cinder
volumes_dir = /var/lib/cinder/volumes
transport_url = rabbit://openstack:root@controller
my_ip = 10.1.1.103
glance_api_servers = http://controller:9292
enabled_backends = lvm

[database]
connection = mysql+pymysql://cinder:cinder@controller/cinder

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_name = default
user_domain_name = default
project_name = service
username = cinder
password = cinder

[lvm]
# ...
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes
iscsi_protocol = iscsi
iscsi_helper = tgtadm

[oslo_concurrency]
lock_path = /var/lib/cinder/tmp
edit retag flag offensive close merge delete

Comments

You say "also I cheched in 2 error messages are displayed".

Where did you find the AMQP error messages, and is their time stamp related to the volume creation time?

The messages can be a red herring. They might be issued at startup time, when the message queue server is not ready yet.

Bernd Bausch gravatar imageBernd Bausch ( 2017-07-18 18:53:00 -0600 )edit

Also check the LVM config on the storage node. Is the volume group in place, can you create logical volumes etc.

Bernd Bausch gravatar imageBernd Bausch ( 2017-07-18 18:53:40 -0600 )edit

did you find a solution for this. Having the exact same error - but with cinder-backup. happening when i am creating to many backups at once.

hojgaard gravatar imagehojgaard ( 2018-03-22 09:19:34 -0600 )edit