Hey Guys,
we have issues with the Nova-Compute Nodes in Havana.
The Services is correct installed. with Nova service-list we get :
+------------------+------------------+----------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+------------------+----------+---------+-------+----------------------------+-----------------+
| nova-cert | openstackc-core1 | internal | enabled | down | 2013-12-12T15:55:49.000000 | None |
| nova-conductor | openstackc-core1 | internal | enabled | up | 2013-12-13T08:37:51.000000 | None |
| nova-consoleauth | openstackc-core1 | internal | enabled | down | 2013-12-12T15:55:50.000000 | None |
| nova-scheduler | openstackc-core1 | internal | enabled | up | 2013-12-13T08:37:51.000000 | None |
+------------------+------------------+----------+---------+-------+----------------------------+--------
there is no nova-compute service....
The Log of Nova-Compute:
2013-12-13 09:29:39.109 29437 ERROR nova.servicegroup.drivers.db [-] Recovered model server connection!
2013-12-13 09:29:45.069 29437 ERROR nova.openstack.common.periodic_task [-] Error during ComputeManager.update_available_resource: Compute host 2 could not be found.
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
**args)
File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
result = getattr(proxyobj, method)(ctxt, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 450, in compute_node_update
prune_stats)
File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 216, in compute_node_update
return IMPL.compute_node_update(context, compute_id, values, prune_stats)
File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 108, in wrapper
return f(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 668, in compute_node_update
compute_ref = _compute_node_get(context, compute_id, session=session)
File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 502, in _compute_node_get
raise exception.ComputeHostNotFound(host=compute_id)
ComputeHostNotFound: Compute host 2 could not be found.
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/periodic_task.py", line 180, in run_periodic_tasks
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task task(self, context)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4863, in update_available_resource
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task rt.update_available_resource(context)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/lockutils.py", line 246, in inner
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task return f(*args, **kwargs)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 325, in update_available_resource
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task self._sync_compute_node(context, resources)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 356, in _sync_compute_node
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task self._update(context, resources, prune_stats=True)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/compute/resource_tracker.py", line 428, in _update
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task context, self.compute_node, values, prune_stats)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 263, in compute_node_update
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task prune_stats)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 386, in compute_node_update
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task prune_stats=prune_stats)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in call
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task return self._invoke(self.proxy.call, ctxt, method, **kwargs)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in _invoke
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task return cast_or_call(ctxt, msg, **self.kwargs)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 126, in call
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task result = rpc.call(context, real_topic, msg, timeout)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 139, in call
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task return _get_impl().call(CONF, context, topic, msg, timeout)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 816, in call
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task rpc_amqp.get_connection_pool(conf, Connection))
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 574, in call
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task rv = list(rv)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 539, in __iter__
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task raise result
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task ComputeHostNotFound_Remote: Compute host 2 could not be found.
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 461, in _process_data
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task **args)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/dispatcher.py", line 172, in dispatch
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task result = getattr(proxyobj, method)(ctxt, **kwargs)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 450, in compute_node_update
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task prune_stats)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 216, in compute_node_update
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task return IMPL.compute_node_update(context, compute_id, values, prune_stats)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 108, in wrapper
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task return f(*args, **kwargs)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 668, in compute_node_update
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task compute_ref = _compute_node_get(context, compute_id, session=session)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 502, in _compute_node_get
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task raise exception.ComputeHostNotFound(host=compute_id)
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task ComputeHostNotFound: Compute host 2 could not be found.
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:45.069 29437 TRACE nova.openstack.common.periodic_task
2013-12-13 09:29:48.837 29437 ERROR nova.servicegroup.drivers.db [-] model server went away
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/db.py", line 96, in _report_state
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db service.service_ref, state_catalog)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 269, in service_update
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db return self._manager.service_update(context, service, values)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 397, in service_update
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db service=service_p, values=values)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in call
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db return self._invoke(self.proxy.call, ctxt, method, **kwargs)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in _invoke
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db return cast_or_call(ctxt, msg, **self.kwargs)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 126, in call
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db result = rpc.call(context, real_topic, msg, timeout)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 139, in call
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db return _get_impl().call(CONF, context, topic, msg, timeout)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 816, in call
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db rpc_amqp.get_connection_pool(conf, Connection))
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 574, in call
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db rv = list(rv)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 539, in __iter__
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db raise result
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db ServiceNotFound_Remote: Service 7 could not be found.
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py", line 420, in catch_client_exception
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db return func(*args, **kwargs)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 459, in service_update
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db svc = self.db.service_update(context, service['id'], values)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 170, in service_update
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 108, in wrapper
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db return f(*args, **kwargs)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 482, in service_update
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db with_compute_node=False, session=session)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 389, in _service_get
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db raise exception.ServiceNotFound(service_id=service_id)
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db ServiceNotFound: Service 7 could not be found.
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:29:48.837 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:29:59.118 29437 ERROR nova.servicegroup.drivers.db [-] Recovered model server connection!
2013-12-13 09:30:08.847 29437 ERROR nova.servicegroup.drivers.db [-] model server went away
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/servicegroup/drivers/db.py", line 96, in _report_state
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db service.service_ref, state_catalog)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/conductor/api.py", line 269, in service_update
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db return self._manager.service_update(context, service, values)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 397, in service_update
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db service=service_p, values=values)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in call
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db return self._invoke(self.proxy.call, ctxt, method, **kwargs)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in _invoke
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db return cast_or_call(ctxt, msg, **self.kwargs)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py", line 126, in call
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db result = rpc.call(context, real_topic, msg, timeout)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/__init__.py", line 139, in call
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db return _get_impl().call(CONF, context, topic, msg, timeout)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_kombu.py", line 816, in call
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db rpc_amqp.get_connection_pool(conf, Connection))
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 574, in call
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db rv = list(rv)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/amqp.py", line 539, in __iter__
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db raise result
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db ServiceNotFound_Remote: Service 7 could not be found.
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db Traceback (most recent call last):
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/common.py", line 420, in catch_client_exception
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db return func(*args, **kwargs)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 459, in service_update
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db svc = self.db.service_update(context, service['id'], values)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 170, in service_update
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db return IMPL.service_update(context, service_id, values)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 108, in wrapper
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db return f(*args, **kwargs)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 482, in service_update
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db with_compute_node=False, session=session)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 389, in _service_get
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db raise exception.ServiceNotFound(service_id=service_id)
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db ServiceNotFound: Service 7 could not be found.
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:30:08.847 29437 TRACE nova.servicegroup.drivers.db
2013-12-13 09:30:19.087 29437 ERROR nova.servicegroup.drivers.db [-] Recovered model server connection!
e here
have anyone any idea, please ?