Revision history [back]

click to hide/show revision 1
initial version

cinder mysql connection failed errno 113

 Hello, I have a problem with cinder, command tail -f /var/log/cinder/volume.log shows :


2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume Traceback (most recent call last):
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/cmd/volume.py", line 100, in _launch_service
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     cluster=cluster)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 403, in create
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     cluster=cluster)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/service.py", line 156, in __init__
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     *args, **kwargs)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 218, in __init__
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     *args, **kwargs)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/manager.py", line 183, in __init__
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     self.scheduler_rpcapi = scheduler_rpcapi.SchedulerAPI()
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/rpc.py", line 207, in __init__
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     obj_version_cap = self.determine_obj_version_cap()
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/rpc.py", line 260, in determine_obj_version_cap
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     cinder.context.get_admin_context(), cls.BINARY)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/objects/service.py", line 222, in get_minimum_obj_version
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     binary)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/objects/service.py", line 192, in _get_minimum_version
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     services = ServiceList.get_all_by_binary(context, binary)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/objects/service.py", line 256, in get_all_by_binary
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     disabled=disabled)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/api.py", line 141, in service_get_all
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     return IMPL.service_get_all(context, backend_match_level, **filters)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 164, in wrapper
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     return f(*args, **kwargs)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 524, in service_get_all
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     **filters)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 444, in _service_query
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     read_deleted=read_deleted)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 253, in model_query
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     session = kwargs.get('session') or get_session()
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 92, in get_session
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     return main_context_manager._factory.get_legacy_facade().get_session(
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 347, in get_legacy_facade
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     self._start()
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 491, in _start
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     engine_args, maker_args)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line 515, in _setup_for_connection
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     sql_connection=sql_connection, **engine_kwargs)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/debtcollector/renames.py", line 43, in decorator
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     return wrapped(*args, **kwargs)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 184, in create_engine
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     test_conn = _test_connection(engine, max_retries, retry_interval)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 370, in _test_connection
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume     six.reraise(type(de_ref), de_ref)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume   File "<string>", line 2, in reraise
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'controller' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8)
2019-04-03 10:39:51.000 11911 ERROR cinder.cmd.volume
2019-04-03 10:39:51.008 11911 ERROR cinder.cmd.volume [req-7d0ef732-c954-4a9b-a071-7b3b6201d122 - - - - -] No volume service(s) started successfully, terminating.
2019-04-03 10:39:53.702 11929 WARNING oslo_db.sqlalchemy.engines [req-b04eefc6-70cb-4306-bc3b-00878717391e - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on 'controller' ([Errno 113] EHOSTUNREACH)") (Background on this error at: http://sqlalche.me/e/e3q8)

my cinder.conf on storage node contains  that data : 
ofc xxxx is my private password

cinder.conf on storage node

[DEFAULT]
transport_url = rabbit://openstack:xxxx@controller
auth_strategy = keystone
my_ip = 192.168.100.6
enabled_backends = lvm
glance_api_servers = http://controller:9292

[lvm]
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
#volume_driver = cinder.volume.drivers.nexenta.ns5.iscsi.NexentaISCSIDriver
#volume_driver = cinder.volume.drivers.nexenta.ns5.nfs.NexentaNfsDriver
volume_group = cinder-volumes
iscsi_helper = lioadm
iscsi_protocol = iscsi

[database]
connection = mysql+pymysql://cinder:xxxx@controller/cinder

[keystone_authtoken]
auth_uri = http://controller:5000
auth_url = http://controller:35357
memcached_servers = controller:11211
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = xxxx



cinder.conf on controller node : 


[DEFAULT]
transport_url = rabbit://openstack:xxxx@controller
auth_strategy = keystone
my_ip = controller
[database]
connection = mysql+pymysql://cinder:xxxx@controller/cinder
iscsi_protocol = iscsi
iscsi_helper = tgtadm

[keystone_authtoken]
auth_uri = http://controller:5000
#auth_url = http://controller:35357
auth_url = http://controller:5000
memcached_servers = controller:11211
auth_type = password
project_domain_id = default
user_domain_id = default
project_name = service
username = cinder
password = xxxx



other information :

openstack volume service list (controller node)

+------------------+-------------+------+---------+-------+----------------------------+
| Binary           | Host        | Zone | Status  | State | Updated At                 |
+------------------+-------------+------+---------+-------+----------------------------+
| cinder-volume    | storage@lvm | nova | enabled | down  | 2018-10-31T14:24:52.000000 |
| cinder-scheduler | controller  | nova | enabled | up    | 2019-04-03T12:47:42.000000 |
| cinder-backup    | controller  | nova | enabled | down  | 2019-04-03T11:48:22.000000 |
+------------------+-------------+------+---------+-------+----------------------------+


mysql -u root -p  (contoler node)
Enter password:
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 1105
Server version: 10.1.20-MariaDB MariaDB Server

Copyright (c) 2000, 2016, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> SHOW DATABASES;

+--------------------+
| Database           |
+--------------------+
| cinder             |



Can someone help me or guide how to progress on that issue, please ??
I do not know how to change cinder-volume to UP status also.