Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

asked 2014-06-26 14:30:25 -0600

JulesFakhoury gravatar image

Help creating instance in Openstack Dashboard

I have Open-Stack with Red Hat Linux. In the Dashboard, when I create an instance from an image or a volume, I get an error: task status not updating and stating at "scheduling", or "local host not found". Can you help resolve the problem and provide the detailed procedure to create an instance? Thanks!

click to hide/show revision 2
No.2 Revision

updated 2014-06-30 15:54:23 -0600

smaffulli gravatar image

Help creating instance in Openstack Dashboard

I have Open-Stack with Red Hat Linux. In the Dashboard, when I create an instance from an image or a volume, I get an error: task status not updating and stating at "scheduling", or "local host not found". Can you help resolve the problem and provide the detailed procedure to create an instance? Thanks!instance?

2014-06-26 13:43:43.790 15386 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2014-06-26 13:43:43.852 15386 AUDIT nova.service [-] Starting scheduler node (version 2014.1.1-1.el6)
2014-06-26 13:43:45.090 15386 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 192.34.76.101:5672
2014-06-26 13:53:40.821 15386 WARNING nova.openstack.common.db.sqlalchemy.session [-] Database server has gone away: (2006, 'MySQL server has gone away')
2014-06-26 13:53:40.824 15386 ERROR nova.servicegroup.drivers.db [-] model server went away
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py", line 95, in _report_state
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     service.service_ref, state_catalog)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/conductor/api.py", line 218, in service_update
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return self._manager.service_update(context, service, values)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 966, in wrapper
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return func(*args, **kwargs)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return func(*args, **kwargs)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 491, in service_update
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     svc = self.db.service_update(context, service['id'], values)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 148, in service_update
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return IMPL.service_update(context, service_id, values)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return f(*args, **kwargs)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 532, in service_update
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     with_compute_node=False, session=session)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 437, in _service_get
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     result = query.first()
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2282, in first
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     ret = list(self[0:1])
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2149, in __getitem__
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return list(res)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return self._execute_and_instances(context)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     close_with_result=True)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     **kw)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     close_with_result=close_with_result)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return self.transaction._connection_for_bind(engine)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     conn = bind.contextual_connect()
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     self.pool.connect(),
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return _ConnectionFairy(self).checkout()
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 516, in checkout
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     self.connection = self._connection_record.get_connection()
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     self.connection = self.__connect()
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     connection = self.__pool._creator()
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:53:40.824 15386 TRACE nova.servicegroup.drivers.db
2014-06-26 13:54:28.312 15386 ERROR nova.openstack.common.periodic_task [-] Error during SchedulerManager._expire_reservations: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     task(self, context)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 266, in _expire_reservations
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     QUOTAS.expire(context)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1370, in expire
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     self._driver.expire(context)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 620, in expire
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     db.reservation_expire(context)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1135, in reservation_expire
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     return IMPL.reservation_expire(context)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     return f(*args, **kwargs)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3360, in reservation_expire
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     for reservation in reservation_query.join(models.QuotaUsage).all():
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     return list(self)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     return self._execute_and_instances(context)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     close_with_result=True)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     **kw)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     close_with_result=close_with_result)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     return self.transaction._connection_for_bind(engine)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     conn = bind.contextual_connect()
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     self.pool.connect(),
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     return _ConnectionFairy(self).checkout()
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 433, in __init__
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     conn = self.connection = self._connection_record.get_connection()
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     self.connection = self.__connect()
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     connection = self.__pool._creator()
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     return Connection(*args, **kwargs)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:54:28.312 15386 TRACE nova.openstack.common.periodic_task
2014-06-26 13:55:00.967 15386 ERROR nova.servicegroup.drivers.db [-] Recovered model server connection!
2014-06-26 13:55:50.968 15386 WARNING nova.openstack.common.db.sqlalchemy.session [-] Database server has gone away: (2006, 'MySQL server has gone away')
2014-06-26 13:55:50.971 15386 ERROR nova.servicegroup.drivers.db [-] model server went away
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py", line 95, in _report_state
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     service.service_ref, state_catalog)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/conductor/api.py", line 218, in service_update
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return self._manager.service_update(context, service, values)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 966, in wrapper
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return func(*args, **kwargs)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return func(*args, **kwargs)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 491, in service_update
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     svc = self.db.service_update(context, service['id'], values)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 148, in service_update
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return IMPL.service_update(context, service_id, values)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return f(*args, **kwargs)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 532, in service_update
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     with_compute_node=False, session=session)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 437, in _service_get
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     result = query.first()
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2282, in first
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     ret = list(self[0:1])
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2149, in __getitem__
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return list(res)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return self._execute_and_instances(context)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     close_with_result=True)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     **kw)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     close_with_result=close_with_result)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return self.transaction._connection_for_bind(engine)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     conn = bind.contextual_connect()
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     self.pool.connect(),
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return _ConnectionFairy(self).checkout()
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 516, in checkout
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     self.connection = self._connection_record.get_connection()
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     self.connection = self.__connect()
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     connection = self.__pool._creator()
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:55:50.971 15386 TRACE nova.servicegroup.drivers.db
2014-06-26 13:56:28.348 15386 ERROR nova.openstack.common.periodic_task [-] Error during SchedulerManager._expire_reservations: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     task(self, context)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 266, in _expire_reservations
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     QUOTAS.expire(context)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1370, in expire
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     self._driver.expire(context)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 620, in expire
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     db.reservation_expire(context)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1135, in reservation_expire
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     return IMPL.reservation_expire(context)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     return f(*args, **kwargs)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3360, in reservation_expire
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     for reservation in reservation_query.join(models.QuotaUsage).all():
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     return list(self)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     return self._execute_and_instances(context)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     close_with_result=True)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     **kw)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     close_with_result=close_with_result)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     return self.transaction._connection_for_bind(engine)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     conn = bind.contextual_connect()
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     self.pool.connect(),
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     return _ConnectionFairy(self).checkout()
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 433, in __init__
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     conn = self.connection = self._connection_record.get_connection()
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     self.connection = self.__connect()
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     connection = self.__pool._creator()
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     return Connection(*args, **kwargs)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:56:28.348 15386 TRACE nova.openstack.common.periodic_task
2014-06-26 13:57:28.358 15386 ERROR nova.openstack.common.periodic_task [-] Error during SchedulerManager._expire_reservations: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     task(self, context)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 266, in _expire_reservations
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     QUOTAS.expire(context)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1370, in expire
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     self._driver.expire(context)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 620, in expire
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     db.reservation_expire(context)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1135, in reservation_expire
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     return IMPL.reservation_expire(context)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     return f(*args, **kwargs)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3360, in reservation_expire
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     for reservation in reservation_query.join(models.QuotaUsage).all():
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     return list(self)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     return self._execute_and_instances(context)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     close_with_result=True)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     **kw)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     close_with_result=close_with_result)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     return self.transaction._connection_for_bind(engine)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     conn = bind.contextual_connect()
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     self.pool.connect(),
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     return _ConnectionFairy(self).checkout()
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 433, in __init__
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     conn = self.connection = self._connection_record.get_connection()
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     self.connection = self.__connect()
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     connection = self.__pool._creator()
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     return Connection(*args, **kwargs)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:57:28.358 15386 TRACE nova.openstack.common.periodic_task
2014-06-26 13:58:28.378 15386 ERROR nova.openstack.common.periodic_task [-] Error during SchedulerManager._expire_reservations: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     task(self, context)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 266, in _expire_reservations
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     QUOTAS.expire(context)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1370, in expire
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     self._driver.expire(context)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 620, in expire
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     db.reservation_expire(context)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1135, in reservation_expire
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     return IMPL.reservation_expire(context)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     return f(*args, **kwargs)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3360, in reservation_expire
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     for reservation in reservation_query.join(models.QuotaUsage).all():
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     return list(self)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     return self._execute_and_instances(context)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     close_with_result=True)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     **kw)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     close_with_result=close_with_result)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     return self.transaction._connection_for_bind(engine)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     conn = bind.contextual_connect()
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     self.pool.connect(),
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     return _ConnectionFairy(self).checkout()
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 433, in __init__
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     conn = self.connection = self._connection_record.get_connection()
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     self.connection = self.__connect()
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     connection = self.__pool._creator()
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     return Connection(*args, **kwargs)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:58:28.378 15386 TRACE nova.openstack.common.periodic_task
2014-06-26 13:59:28.391 15386 ERROR nova.openstack.common.periodic_task [-] Error during SchedulerManager._expire_reservations: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     task(self, context)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 266, in _expire_reservations
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     QUOTAS.expire(context)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1370, in expire
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     self._driver.expire(context)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 620, in expire
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     db.reservation_expire(context)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1135, in reservation_expire
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     return IMPL.reservation_expire(context)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     return f(*args, **kwargs)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3360, in reservation_expire
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     for reservation in reservation_query.join(models.QuotaUsage).all():
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     return list(self)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     return self._execute_and_instances(context)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     close_with_result=True)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     **kw)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     close_with_result=close_with_result)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     return self.transaction._connection_for_bind(engine)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     conn = bind.contextual_connect()
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     self.pool.connect(),
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     return _ConnectionFairy(self).checkout()
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 433, in __init__
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     conn = self.connection = self._connection_record.get_connection()
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     self.connection = self.__connect()
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     connection = self.__pool._creator()
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     return Connection(*args, **kwargs)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 13:59:28.391 15386 TRACE nova.openstack.common.periodic_task
2014-06-26 13:59:51.205 15386 ERROR nova.servicegroup.drivers.db [-] Recovered model server connection!
2014-06-26 14:05:28.473 15386 WARNING nova.openstack.common.db.sqlalchemy.session [-] Database server has gone away: (2006, 'MySQL server has gone away')
2014-06-26 14:05:28.475 15386 ERROR nova.openstack.common.periodic_task [-] Error during SchedulerManager._expire_reservations: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     task(self, context)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 266, in _expire_reservations
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     QUOTAS.expire(context)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1370, in expire
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     self._driver.expire(context)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 620, in expire
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     db.reservation_expire(context)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1135, in reservation_expire
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     return IMPL.reservation_expire(context)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     return f(*args, **kwargs)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3360, in reservation_expire
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     for reservation in reservation_query.join(models.QuotaUsage).all():
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     return list(self)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     return self._execute_and_instances(context)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     close_with_result=True)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     **kw)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     close_with_result=close_with_result)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     return self.transaction._connection_for_bind(engine)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     conn = bind.contextual_connect()
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     self.pool.connect(),
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     return _ConnectionFairy(self).checkout()
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 516, in checkout
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     self.connection = self._connection_record.get_connection()
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     self.connection = self.__connect()
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     connection = self.__pool._creator()
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     return Connection(*args, **kwargs)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 14:05:28.475 15386 TRACE nova.openstack.common.periodic_task
2014-06-26 14:05:31.494 15386 ERROR nova.servicegroup.drivers.db [-] model server went away
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py", line 95, in _report_state
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     service.service_ref, state_catalog)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/conductor/api.py", line 218, in service_update
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return self._manager.service_update(context, service, values)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 966, in wrapper
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return func(*args, **kwargs)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return func(*args, **kwargs)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 491, in service_update
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     svc = self.db.service_update(context, service['id'], values)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 148, in service_update
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return IMPL.service_update(context, service_id, values)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return f(*args, **kwargs)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 532, in service_update
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     with_compute_node=False, session=session)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 437, in _service_get
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     result = query.first()
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2282, in first
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     ret = list(self[0:1])
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2149, in __getitem__
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return list(res)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return self._execute_and_instances(context)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     close_with_result=True)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     **kw)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     close_with_result=close_with_result)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return self.transaction._connection_for_bind(engine)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     conn = bind.contextual_connect()
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     self.pool.connect(),
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return _ConnectionFairy(self).checkout()
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 433, in __init__
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     conn = self.connection = self._connection_record.get_connection()
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     self.connection = self.__connect()
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     connection = self.__pool._creator()
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 14:05:31.494 15386 TRACE nova.servicegroup.drivers.db
2014-06-26 14:06:28.488 15386 ERROR nova.openstack.common.periodic_task [-] Error during SchedulerManager._expire_reservations: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     task(self, context)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 266, in _expire_reservations
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     QUOTAS.expire(context)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1370, in expire
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     self._driver.expire(context)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 620, in expire
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     db.reservation_expire(context)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1135, in reservation_expire
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     return IMPL.reservation_expire(context)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     return f(*args, **kwargs)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3360, in reservation_expire
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     for reservation in reservation_query.join(models.QuotaUsage).all():
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     return list(self)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     return self._execute_and_instances(context)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     close_with_result=True)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     **kw)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     close_with_result=close_with_result)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     return self.transaction._connection_for_bind(engine)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     conn = bind.contextual_connect()
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     self.pool.connect(),
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     return _ConnectionFairy(self).checkout()
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 433, in __init__
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     conn = self.connection = self._connection_record.get_connection()
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     self.connection = self.__connect()
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     connection = self.__pool._creator()
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     return Connection(*args, **kwargs)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task     super(Connection, self).__init__(*args, **kwargs2)
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-26 14:06:28.488 15386 TRACE nova.openstack.common.periodic_task
2014-06-26 14:07:01.642 15386 ERROR nova.servicegroup.drivers.db [-] Recovered model server connection!
2014-06-27 05:24:41.167 3140 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2014-06-27 05:24:41.675 3140 AUDIT nova.service [-] Starting scheduler node (version 2014.1.1-1.el6)
2014-06-27 05:24:43.958 3140 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 192.34.76.101:5672
2014-06-27 05:34:29.416 3140 WARNING nova.openstack.common.db.sqlalchemy.session [-] Database server has gone away: (2006, 'MySQL server has gone away')
2014-06-27 05:34:29.418 3140 ERROR nova.servicegroup.drivers.db [-] model server went away
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/servicegroup/drivers/db.py", line 95, in _report_state
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     service.service_ref, state_catalog)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/conductor/api.py", line 218, in service_update
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return self._manager.service_update(context, service, values)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 966, in wrapper
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return func(*args, **kwargs)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return func(*args, **kwargs)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 491, in service_update
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     svc = self.db.service_update(context, service['id'], values)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 148, in service_update
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return IMPL.service_update(context, service_id, values)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return f(*args, **kwargs)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 532, in service_update
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     with_compute_node=False, session=session)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 437, in _service_get
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     result = query.first()
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2282, in first
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     ret = list(self[0:1])
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2149, in __getitem__
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return list(res)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return self._execute_and_instances(context)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     close_with_result=True)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     **kw)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     close_with_result=close_with_result)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return self.transaction._connection_for_bind(engine)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     conn = bind.contextual_connect()
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     self.pool.connect(),
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return _ConnectionFairy(self).checkout()
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 516, in checkout
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     self.connection = self._connection_record.get_connection()
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     self.connection = self.__connect()
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     connection = self.__pool._creator()
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return dialect.connect(*cargs, **cparams)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return self.dbapi.connect(*cargs, **cparams)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     return Connection(*args, **kwargs)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db     super(Connection, self).__init__(*args, **kwargs2)
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-27 05:34:29.418 3140 TRACE nova.servicegroup.drivers.db
2014-06-27 05:35:08.142 3140 ERROR nova.openstack.common.periodic_task [-] Error during SchedulerManager._expire_reservations: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     task(self, context)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 266, in _expire_reservations
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     QUOTAS.expire(context)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1370, in expire
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     self._driver.expire(context)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 620, in expire
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     db.reservation_expire(context)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1135, in reservation_expire
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     return IMPL.reservation_expire(context)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     return f(*args, **kwargs)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3360, in reservation_expire
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     for reservation in reservation_query.join(models.QuotaUsage).all():
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     return list(self)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     return self._execute_and_instances(context)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     close_with_result=True)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     **kw)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     close_with_result=close_with_result)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     return self.transaction._connection_for_bind(engine)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     conn = bind.contextual_connect()
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     self.pool.connect(),
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     return _ConnectionFairy(self).checkout()
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 433, in __init__
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     conn = self.connection = self._connection_record.get_connection()
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     self.connection = self.__connect()
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     connection = self.__pool._creator()
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     return Connection(*args, **kwargs)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task     super(Connection, self).__init__(*args, **kwargs2)
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-27 05:35:08.142 3140 TRACE nova.openstack.common.periodic_task
2014-06-27 05:36:08.179 3140 ERROR nova.openstack.common.periodic_task [-] Error during SchedulerManager._expire_reservations: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 182, in run_periodic_tasks
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     task(self, context)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 266, in _expire_reservations
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     QUOTAS.expire(context)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1370, in expire
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     self._driver.expire(context)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/quota.py", line 620, in expire
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     db.reservation_expire(context)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1135, in reservation_expire
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     return IMPL.reservation_expire(context)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     return f(*args, **kwargs)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3360, in reservation_expire
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     for reservation in reservation_query.join(models.QuotaUsage).all():
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2241, in all
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     return list(self)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2353, in __iter__
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     return self._execute_and_instances(context)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2366, in _execute_and_instances
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     close_with_result=True)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/query.py", line 2357, in _connection_from_session
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     **kw)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 799, in connection
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     close_with_result=close_with_result)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 803, in _connection_for_bind
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     return self.transaction._connection_for_bind(engine)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/orm/session.py", line 299, in _connection_for_bind
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     conn = bind.contextual_connect()
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1661, in contextual_connect
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     self.pool.connect(),
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 272, in connect
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     return _ConnectionFairy(self).checkout()
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 433, in __init__
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     conn = self.connection = self._connection_record.get_connection()
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 346, in get_connection
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     self.connection = self.__connect()
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     connection = self.__pool._creator()
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     return dialect.connect(*cargs, **cparams)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     return self.dbapi.connect(*cargs, **cparams)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     return Connection(*args, **kwargs)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task     super(Connection, self).__init__(*args, **kwargs2)
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (2003, "Can't connect to MySQL server on '192.34.76.101' (111)") None None
2014-06-27 05:36:08.179 3140 TRACE nova.openstack.common.periodic_task
2014-06-27 05:36:19.561 3140 ERROR nova.servicegroup.drivers.db [-] Recovered model server connection!
2014-06-27 06:44:18.300 3241 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2014-06-27 06:44:18.732 3241 AUDIT nova.service [-] Starting scheduler node (version 2014.1.1-1.el6)
2014-06-27 06:44:20.865 3241 ERROR nova.openstack.common.threadgroup [-] (OperationalError) (1045, "Access denied for user 'nova'@'vm1' (using password: YES)") None None
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     x.wait()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 168, in wait
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/event.py", line 116, in wait
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/eventlet/greenthread.py", line 194, in main
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/service.py", line 486, in run_service
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     service.start()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/service.py", line 168, in start
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     self.host, self.binary)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/conductor/api.py", line 193, in service_get_by_args
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     binary=binary)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/utils.py", line 966, in wrapper
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return func(*args, **kwargs)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return func(*args, **kwargs)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/conductor/manager.py", line 441, in service_get_all_by
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     result = self.db.service_get_by_args(context, host, binary)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 134, in service_get_by_args
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return IMPL.service_get_by_args(context, host, binary)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 146, in wrapper
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return f(*args, **kwargs)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 499, in service_get_by_args
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     result = model_query(context, models.Service).\
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 234, in model_query
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     session = kwargs.get('session') or get_session(use_slave=use_slave)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 122, in get_session
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     facade = _create_facade_lazily(use_slave)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 104, in _create_facade_lazily
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     **dict(CONF.database.iteritems())
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 831, in __init__
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     retry_interval=kwargs.get('retry_interval', 10))
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.6/site-packages/nova/openstack/common/db/sqlalchemy/session.py", line 648, in create_engine
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     engine.connect()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1641, in connect
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return self._connection_cls(self, **kwargs)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 59, in __init__
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     self.__connection = connection or engine.raw_connection()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/base.py", line 1699, in raw_connection
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return self.pool.unique_connection()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 220, in unique_connection
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return _ConnectionFairy(self).checkout()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 431, in __init__
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     rec = self._connection_record = pool._do_get()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 772, in _do_get
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return self._create_connection()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 225, in _create_connection
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return _ConnectionRecord(self)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 318, in __init__
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     self.connection = self.__connect()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/pool.py", line 379, in __connect
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     connection = self.__pool._creator()
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/strategies.py", line 80, in connect
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return dialect.connect(*cargs, **cparams)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/sqlalchemy/engine/default.py", line 283, in connect
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return self.dbapi.connect(*cargs, **cparams)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/MySQLdb/__init__.py", line 81, in Connect
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     return Connection(*args, **kwargs)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 187, in __init__
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup     super(Connection, self).__init__(*args, **kwargs2)
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup OperationalError: (OperationalError) (1045, "Access denied for user 'nova'@'vm1' (using password: YES)") None None
2014-06-27 06:44:20.865 3241 TRACE nova.openstack.common.threadgroup
2014-06-27 06:52:02.743 14517 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2014-06-27 06:52:02.975 14517 AUDIT nova.service [-] Starting scheduler node (version 2014.1.1-1.el6)
2014-06-27 06:52:05.398 14517 INFO oslo.messaging._drivers.impl_rabbit [-] Connected to AMQP server on 192.34.76.101:5672