nova-api mysql too many connections
I'm running into an issue that is quite annoying and I can't seem to get it fixed. Every so often as I use my cluster I will start getting errors in the nova-api log that say the following:
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions [req-4f5b00b6-ae2c-4997-85b3-ce3879a8d20d de69db66f61846968edb6eae85f19030 f663d581469341b49372921a5b682e49 - default default] Unexpected exception in API method
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions return f(*args, **kwargs)
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions return func(*args, **kwargs)
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 214, in detail
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions servers = self._get_servers(req, is_detail=True)
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 357, in _get_servers
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions sort_keys=sort_keys, sort_dirs=sort_dirs)
...
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions err.raise_mysql_exception(self._data)
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions raise errorclass(errno, errval)
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions
Now I have increased the max connections multiple times in multiple places:
/etc/my.cnf
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]
#
# This group is read by the server
#
[mysqld]
max_connections = 4096
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
#
# include all files from the config directory
#
!includedir /etc/my.cnf.d
and
/etc/my.cnf.d/openstack.cnf
[mysqld]
bind-address = 192.168.60.9
default-storage-engine = innodb
innodb_file_per_table = on
#max_connection = 4096
max_connection = 51200
collation-server = utf8_general_ci
character-set-server = utf8
I am at a lost and need some help. I'm probably ...
I ask just because you didn't explicitly say you did it already, have you restarted mysql and nova services?
is there a load sharing proxy like haproxy in between?
Were you able to fix this? I have the same problem.