Ask Your Question
0

nova-api mysql too many connections

asked 2017-06-27 14:36:55 -0500

danigrith gravatar image

updated 2017-06-27 14:37:13 -0500

I'm running into an issue that is quite annoying and I can't seem to get it fixed. Every so often as I use my cluster I will start getting errors in the nova-api log that say the following:

2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions [req-4f5b00b6-ae2c-4997-85b3-ce3879a8d20d de69db66f61846968edb6eae85f19030 f663d581469341b49372921a5b682e49 - default default] Unexpected exception in API method
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions Traceback (most recent call last):
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/extensions.py", line 338, in wrapped
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions     return f(*args, **kwargs)
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/validation/__init__.py", line 181, in wrapper
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions     return func(*args, **kwargs)
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 214, in detail
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions     servers = self._get_servers(req, is_detail=True)
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/servers.py", line 357, in _get_servers
2017-06-27 13:29:22.210 1796 ERROR nova.api.openstack.extensions     sort_keys=sort_keys, sort_dirs=sort_dirs)
...
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions     err.raise_mysql_exception(self._data)
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions     raise errorclass(errno, errval)
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions OperationalError: (pymysql.err.OperationalError) (1040, u'Too many connections')
2017-06-27 13:29:25.852 1796 ERROR nova.api.openstack.extensions

Now I have increased the max connections multiple times in multiple places:

/etc/my.cnf
#                                                 
# This group is read both both by the client and the server                                          
# use it for options that affect everything       
#                                                 
[client-server]                                   

#                                                 
# This group is read by the server                
#                                                 
[mysqld]                                          
max_connections = 4096                            
# Disabling symbolic-links is recommended to prevent assorted security risks                         
symbolic-links=0                                  

#                                                 
# include all files from the config directory     
#                                                 
!includedir /etc/my.cnf.d

and

/etc/my.cnf.d/openstack.cnf
[mysqld]                                          
bind-address = 192.168.60.9                       

default-storage-engine = innodb                   
innodb_file_per_table = on                        
#max_connection = 4096                            
max_connection = 51200                            
collation-server = utf8_general_ci                
character-set-server = utf8

I am at a lost and need some help. I'm probably ... (more)

edit retag flag offensive close merge delete

Comments

I ask just because you didn't explicitly say you did it already, have you restarted mysql and nova services?

eblock gravatar imageeblock ( 2017-06-29 02:57:11 -0500 )edit

is there a load sharing proxy like haproxy in between?

Andreas Merk gravatar imageAndreas Merk ( 2017-06-30 03:34:22 -0500 )edit

Were you able to fix this? I have the same problem.

Campos gravatar imageCampos ( 2018-02-23 07:37:28 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2018-06-20 01:55:55 -0500

Christian Zunker gravatar image

We had the same problem and solved it by reducing the connection pool size of our nova services:

max_pool_size

We reduced this setting in nova.conf and restartet the different services. Nova was using up most of our MySQL DB connections.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2017-06-27 14:36:55 -0500

Seen: 2,176 times

Last updated: Jun 20 '18