Revision history [back]

Can't launch instance in secondary compute node

Hi all,

I had 1 Controller node and 1 Compute node with Juno and recently i add other Compute node, both use nova-network, all nodes use ubuntu 14.04.1, the problem is when i try launch one instance in new server, i have this error in nova-compute log:

2014-12-21 12:21:21.882 15317 AUDIT nova.compute.manager [req-0482429d-52f4-4100-9eed-76a9864b9d62 None] [instance: a8000f54-3b54-4edf-8104-eda25b266a91] Starting instance...
2014-12-21 12:21:21.974 15317 AUDIT nova.compute.claims [-] [instance: a8000f54-3b54-4edf-8104-eda25b266a91] Attempting claim: memory 512 MB, disk 8 GB
2014-12-21 12:21:21.974 15317 AUDIT nova.compute.claims [-] [instance: a8000f54-3b54-4edf-8104-eda25b266a91] Total memory: 7969 MB, used: 512.00 MB
2014-12-21 12:21:21.975 15317 AUDIT nova.compute.claims [-] [instance: a8000f54-3b54-4edf-8104-eda25b266a91] memory limit: 11953.50 MB, free: 11441.50 MB
2014-12-21 12:21:21.975 15317 AUDIT nova.compute.claims [-] [instance: a8000f54-3b54-4edf-8104-eda25b266a91] Total disk: 9 GB, used: 0.00 GB
2014-12-21 12:21:21.975 15317 AUDIT nova.compute.claims [-] [instance: a8000f54-3b54-4edf-8104-eda25b266a91] disk limit not specified, defaulting to unlimited
2014-12-21 12:21:21.987 15317 AUDIT nova.compute.claims [-] [instance: a8000f54-3b54-4edf-8104-eda25b266a91] Claim successful
2014-12-21 12:21:22.137 15317 INFO nova.scheduler.client.report [-] Compute_service record updated for ('ftbhost3.ftb.local', 'ftbhost3.ftb.local')
2014-12-21 12:21:22.257 15317 ERROR nova.compute.manager [-] [instance: a8000f54-3b54-4edf-8104-eda25b266a91] Failed to allocate network(s)
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91] Traceback (most recent call last):
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2196, in _build_resources
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     requested_networks, security_groups)
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1689, in _build_networks_for_instance
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     requested_networks, macs, security_groups, dhcp_options)
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 1723, in _allocate_network
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     instance.save(expected_task_state=[None])
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 189, in wrapper
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     ctxt, self, fn.__name__, args, kwargs)
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/conductor/rpcapi.py", line 351, in object_action
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     objmethod=objmethod, args=args, kwargs=kwargs)
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 152, in call
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     retry=self.retry)
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 90, in _send
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     timeout=timeout, retry=retry)
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 408, in send
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     retry=retry)
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py", line 399, in _send
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     raise result
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91] RemoteError: Remote error: OperationalError (OperationalError) (1048, "Column 'instance_uuid' cannot be null") 'UPDATE instance_extra SET updated_at=%s, instance_uuid=%s WHERE instance_extra.id = %s' (datetime.datetime(2014, 12, 21, 11, 21, 22, 249208), None, 68L)
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91] [u'Traceback (most recent call last):\n', u'  File "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 400, in _object_dispatch\n    return getattr(target, method)(context, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/objects/base.py", line 204, in wrapper\n    return fn(self, ctxt, *args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/objects/instance.py", line 500, in save\n    columns_to_join=_expected_cols(expected_attrs))\n', u'  File "/usr/lib/python2.7/dist-packages/nova/db/api.py", line 746, in instance_update_and_get_original\n    columns_to_join=columns_to_join)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 143, in wrapper\n    return f(*args, **kwargs)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2289, in instance_update_and_get_original\n    columns_to_join=columns_to_join)\n', u'  File "/usr/lib/python2.7/dist-packages/nova/db/sqlalchemy/api.py", line 2380, in _instance_update\n    session.add(instance_ref)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 470, in __exit__\n    self.rollback()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 467, in __exit__\n    self.commit()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 377, in commit\n    self._prepare_impl()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 357, in _prepare_impl\n    self.session.flush()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 1919, in flush\n    self._flush(objects)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2037, in _flush\n    transaction.rollback(_capture_exception=True)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 60, in __exit__\n    compat.reraise(exc_type, exc_value, exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2001, in _flush\n    flush_context.execute()\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 372, in execute\n    rec.execute(self)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 526, in execute\n    uow\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 60, in save_obj\n    mapper, table, update)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 518, in _emit_update_statements\n    execute(statement, params)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute\n    return meth(self, multiparams, params)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection\n    return connection._execute_clauseelement(self, multiparams, params)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement\n    compiled_sql, distilled_params\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context\n    context)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1156, in _handle_dbapi_exception\n    util.raise_from_cause(newraise, exc_info)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause\n    reraise(type(exception), exception, tb=exc_tb)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context\n    context)\n', u'  File "/usr/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute\n    cursor.execute(statement, parameters)\n', u'  File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 174, in execute\n    self.errorhandler(self, exc, value)\n', u'  File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler\n    raise errorclass, errorvalue\n', u'OperationalError: (OperationalError) (1048, "Column \'instance_uuid\' cannot be null") \'UPDATE instance_extra SET updated_at=%s, instance_uuid=%s WHERE instance_extra.id = %s\' (datetime.datetime(2014, 12, 21, 11, 21, 22, 249208), None, 68L)\n'].
2014-12-21 12:21:22.257 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91] 
2014-12-21 12:21:22.283 15317 INFO nova.scheduler.client.report [-] Compute_service record updated for ('ftbhost3.ftb.local', 'ftbhost3.ftb.local')
2014-12-21 12:21:22.284 15317 ERROR nova.compute.manager [-] [instance: a8000f54-3b54-4edf-8104-eda25b266a91] Build of instance a8000f54-3b54-4edf-8104-eda25b266a91 aborted: Failed to allocate the network(s), not rescheduling.
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91] Traceback (most recent call last):
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2036, in _do_build_and_run_instance
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     filter_properties)
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2135, in _build_and_run_instance
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     'create.error', fault=e)
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/openstack/common/excutils.py", line 82, in __exit__
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     six.reraise(self.type_, self.value, self.tb)
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2108, in _build_and_run_instance
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     block_device_mapping) as resources:
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     return self.gen.next()
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]   File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2211, in _build_resources
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]     reason=msg)
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91] BuildAbortException: Build of instance a8000f54-3b54-4edf-8104-eda25b266a91 aborted: Failed to allocate the network(s), not rescheduling.
2014-12-21 12:21:22.284 15317 TRACE nova.compute.manager [instance: a8000f54-3b54-4edf-8104-eda25b266a91]

I can't find where is the problem. Thanks