Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Cinder SQL & Volume Creation Failure

I have a multinode IceHouse OpenStack setup right now. I am having trouble getting cinder to work. As far as I can tell this may be a sql connection issue though I'm not too sure what's causing it...

Here is some information which I have gathered:

Computer-01 Node (Volume host)

The volume group information of the "cinder-volumes" vg found on the compute-01 node:

[root@compute-01 ~]# pvdisplay 
  --- Physical volume ---
  PV Name               /dev/sda3
  VG Name               cinder-volumes
  PV Size               100.00 GiB / not usable 4.00 MiB
  Allocatable           yes 
  PE Size               4.00 MiB
  Total PE              25599
  Free PE               25599
  Allocated PE          0
  PV UUID               8VhheA-tqOW-4k7L-ceMI-Dz2e-nTSW-GXyji8

  ...

[root@compute-01 ~]# vgdisplay 
  --- Volume group ---
  VG Name               cinder-volumes
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                0
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               100.00 GiB
  PE Size               4.00 MiB
  Total PE              25599
  Alloc PE / Size       0 / 0   
  Free  PE / Size       25599 / 100.00 GiB
  VG UUID               f5H7Dh-QYkb-7NY6-VMzY-0WLy-eIAW-rHwYEA

  ...

Relavant /etc/cinder/cinder.conf information of compute-01 node:

[DEFAULT]
auth_strategy = keystone
rpc_backend = cinder.openstack.common.rpc.impl_qpid
qpid_hostname = controller
glance_host = controller

[keystone_authtoken]
auth_uri = http://controller:5000
auth_host = controller
auth_protocol = http
auth_port = 35357
admin_user = cinder
admin_tenant_name = service
admin_password = K_Pass

[database]
connection = mysql://cinder:DB_Pass@controller/cinder

Relavant /etc/tgt/targets.conf information:

include /etc/cinder/volumes/*

/etc/lvm/lvm.conf:

devices {
...
filter = ["a/.*/"]
...
}

/var/log/cinder/volume.log for the compute-01 node on which the "cinder-volumes" volume group is found:

2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup Traceback (most recent call last):
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line 125, in wait
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     x.wait()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/threadgroup.py", line 47, in wait
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return self.thread.wait()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return self._exit_event.wait()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return hubs.get_hub().switch()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return self.greenlet.switch()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/service.py", line 486, in run_service
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     service.start()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/service.py", line 103, in start
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     self.manager.init_host()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/volume/manager.py", line 253, in init_host
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     volumes = self.db.volume_get_all_by_host(ctxt, self.host)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/db/api.py", line 217, in volume_get_all_by_host
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return IMPL.volume_get_all_by_host(context, host)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.py", line 119, in wrapper
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return f(*args, **kwargs)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.py", line 1187, in volume_get_all_by_host
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return _volume_get_query(context).filter_by(host=host).all()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.py", line 137, in wrapper
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return f(*args, **kwargs)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.py", line 1129, in _volume_get_query
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     project_only=project_only).\
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/db/sqlalchemy/api.py", line 178, in model_query
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     session = kwargs.get('session') or get_session()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/db/sqlalchemy/session.py", line 376, in get_session
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     engine = get_engine(sqlite_fk=sqlite_fk)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/db/sqlalchemy/session.py", line 498, in get_engine
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     sqlite_fk=sqlite_fk)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/cinder/openstack/common/db/sqlalchemy/session.py", line 616, in create_engine
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     engine.connect()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1641, in connect
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return self._connection_cls(self, **kwargs)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 59, in __init__
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     self.__connection = connection or engine.raw_connection()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1699, in raw_connection
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return self.pool.unique_connection()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 220, in unique_connection
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return _ConnectionFairy(self).checkout()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 431, in __init__
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     rec = self._connection_record = pool._do_get()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 772, in _do_get
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return self._create_connection()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 225, in _create_connection
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return _ConnectionRecord(self)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 318, in __init__
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     self.connection = self.__connect()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     connection = self.__pool._creator()
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return dialect.connect(*cargs, **cparams)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return self.dbapi.connect(*cargs, **cparams)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     return Connection(*args, **kwargs)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup     super(Connection, self).__init__(*args, **kwargs2)
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on 'controller' (113)") None None
2014-06-24 09:29:46.219 4323 TRACE cinder.openstack.common.threadgroup 
2014-06-24 09:29:46.938 4508 WARNING cinder.openstack.common.db.sqlalchemy.session [req-ac0b1f16-4177-4d37-a245-f7bf82b7740c - - - - -] SQL connection failed. 10 attempts left.
2014-06-24 09:29:56.949 4508 WARNING cinder.openstack.common.db.sqlalchemy.session [req-ac0b1f16-4177-4d37-a245-f7bf82b7740c - - - - -] SQL connection failed. 9 attempts left.
2014-06-24 09:30:06.959 4508 WARNING cinder.openstack.common.db.sqlalchemy.session [req-ac0b1f16-4177-4d37-a245-f7bf82b7740c - - - - -] SQL connection failed. 8 attempts left.
2014-06-24 09:30:16.970 4508 WARNING cinder.openstack.common.db.sqlalchemy.session [req-ac0b1f16-4177-4d37-a245-f7bf82b7740c - - - - -] SQL connection failed. 7 attempts left.
2014-06-24 09:30:26.981 4508 WARNING cinder.openstack.common.db.sqlalchemy.session [req-ac0b1f16-4177-4d37-a245-f7bf82b7740c - - - - -] SQL connection failed. 6 attempts left.

Controller Node

Information when attempting to create a volume:

[root@controller ~]# cinder create --display-name testvolume 1
+---------------------+--------------------------------------+
|       Property      |                Value                 |
+---------------------+--------------------------------------+
|     attachments     |                  []                  |
|  availability_zone  |                 nova                 |
|       bootable      |                false                 |
|      created_at     |      2014-06-24T14:04:01.396540      |
| display_description |                 None                 |
|     display_name    |              testvolume              |
|      encrypted      |                False                 |
|          id         | a59eac93-e0bc-4679-876b-92c4d417e5e4 |
|       metadata      |                  {}                  |
|         size        |                  1                   |
|     snapshot_id     |                 None                 |
|     source_volid    |                 None                 |
|        status       |               creating               |
|     volume_type     |                 None                 |
+---------------------+--------------------------------------+
[root@controller ~]# cinder list
+--------------------------------------+--------+--------------+------+-------------+----------+-------------+
|                  ID                  | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+--------------------------------------+--------+--------------+------+-------------+----------+-------------+
| a59eac93-e0bc-4679-876b-92c4d417e5e4 | error  |  testvolume  |  1   |     None    |  false   |             |
+--------------------------------------+--------+--------------+------+-------------+----------+-------------+
[root@controller ~]# tail /var/log/cinder/scheduler.log 
...
2014-06-24 10:04:01.487 1564 WARNING cinder.context [-] Arguments dropped when creating context: {'user': u'1e84baaa848a4c7983fce0b2497fd7d3', 'tenant': u'42b6d8d6e47346f0ba1f3b28a12ad1b5', 'user_identity': u'1e84baaa848a4c7983fce0b2497fd7d3 42b6d8d6e47346f0ba1f3b28a12ad1b5 - - -'}
2014-06-24 10:04:01.494 1564 ERROR cinder.scheduler.flows.create_volume [req-af9c7ff7-b13c-4307-886b-8ad264e8ee68 1e84baaa848a4c7983fce0b2497fd7d3 42b6d8d6e47346f0ba1f3b28a12ad1b5 - - -] Failed to schedule_create_volume: No valid host was found.

Connection with MySQL using cinder credentials using DP_Pass:

[root@controller ~]# mysql -u cinder -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 319
Server version: 5.1.73 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use cinder;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql>

Iptables relavant information:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9292 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8774 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 5000 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9696 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 35357 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 5672 -j ACCEPT
-A INPUT -p tcp -m multiport --dports 6080 -j ACCEPT

Every other service is working at this point without this issue. I don't know why cinder is the only one having the problem. Does anyone have any ideas? Please let me know of anything else that I could post that would help diagnose the problem.

Thanks.