Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

nova_compute.log - Compute Node - Message Timeout when creating networking for new instance

I'm trying to get KILO working and have followed the instructions at http://docs.openstack.org/kilo/install-guide/install/yum/content/ch_overview.html

I've a three node setup and have been able to verify the configuration according to the install instructions and have attempted to setup neutron networking rather than nova networking.

When it comes to launching an instance with the following command

nova boot --flavor m1.tiny --image cirros-0.3.4-x86_64 --nic net-id=3b017a7e-0e90-4b09-952d-ba6d8b164f2a --security-group default --key-name demo-key demo-instance1

It fails and remains in the following state.

+--------------------------------------+----------------+--------+------------+-------------+----------+
| ID                                   | Name           | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+--------+------------+-------------+----------+
| 225609ff-c600-42ea-9cfe-fd1d41713e72 | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 2cd55483-3951-4bea-8c07-506dfefb3fee | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 2ea3bf33-32ed-44fb-a162-213e2b3f1602 | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 63f9c7db-b85d-4399-b8ce-52fdce06080a | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 64a30f6a-d5db-490d-ae90-7e81d1d6b6bf | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 7bb9d34c-66f2-443c-a5da-986ea7e89fbd | demo-instance1 | ERROR  | -          | NOSTATE     |          |
| 80ec5331-fc60-46f4-8f97-08c06142dff8 | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| a46e771f-422b-4c40-8a14-5bd45e4b0da6 | demo-instance1 | ERROR  | -          | NOSTATE     |          |
| cbca6c9d-f9d1-4ed7-a14d-9baa68d9eb7c | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
+--------------------------------------+----------------+--------+------------+-------------+----------+

I've turned on lots of debug and see the following on the compute node indicating some timeout related to the network setup.

2015-08-10 09:04:51.658 1302 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager Traceback (most recent call last):
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1770, in _allocate_network_async
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     dhcp_options=dhcp_options)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapped
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     return func(self, context, *args, **kwargs)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 64, in wrapper
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     res = f(self, context, *args, **kwargs)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 281, in allocate_for_instance
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 152, in allocate_for_instance
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     macs=jsonutils.to_primitive(macs))
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     retry=self.retry)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     timeout=timeout, retry=retry)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     retry=retry)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     result = self._waiter.wait(msg_id, timeout)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     message = self.waiters.get(msg_id, timeout=timeout)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     'to message ID %s' % msg_id)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager MessagingTimeout: Timed out waiting for a reply to message ID aa74ba23208449b792ab67daba5c4887
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager

I am unable to delete the instances either by resetting their state to ACTIVE etc. the Delete fails with a similar MessageTimeout error on the compute node.

I can't see any obvious errors in the debug etc. on either the controller, or network nodes I've setup.

Any ideas where to look - why this message timeout could occur?

Thanks Richard

nova_compute.log - Compute Node - Message Timeout when creating networking for new instance

I'm trying to get KILO working and have followed the instructions at http://docs.openstack.org/kilo/install-guide/install/yum/content/ch_overview.html

I've a three node setup and have been able to verify the configuration according to the install instructions and have attempted to setup neutron networking rather than nova networking.

When it comes to launching an instance with the following command

nova boot --flavor m1.tiny --image cirros-0.3.4-x86_64 --nic net-id=3b017a7e-0e90-4b09-952d-ba6d8b164f2a --security-group default --key-name demo-key demo-instance1

It fails and remains in the following state.

+--------------------------------------+----------------+--------+------------+-------------+----------+
| ID                                   | Name           | Status | Task State | Power State | Networks |
+--------------------------------------+----------------+--------+------------+-------------+----------+
| 225609ff-c600-42ea-9cfe-fd1d41713e72 | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 2cd55483-3951-4bea-8c07-506dfefb3fee | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 2ea3bf33-32ed-44fb-a162-213e2b3f1602 | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 63f9c7db-b85d-4399-b8ce-52fdce06080a | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 64a30f6a-d5db-490d-ae90-7e81d1d6b6bf | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| 7bb9d34c-66f2-443c-a5da-986ea7e89fbd | demo-instance1 | ERROR  | -          | NOSTATE     |          |
| 80ec5331-fc60-46f4-8f97-08c06142dff8 | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
| a46e771f-422b-4c40-8a14-5bd45e4b0da6 | demo-instance1 | ERROR  | -          | NOSTATE     |          |
| cbca6c9d-f9d1-4ed7-a14d-9baa68d9eb7c | demo-instance1 | ERROR  | spawning   | NOSTATE     |          |
+--------------------------------------+----------------+--------+------------+-------------+----------+

I've turned on lots of debug and see the following on the compute node indicating some timeout related to the network setup.

2015-08-10 09:04:51.658 1302 ERROR nova.compute.manager [-] Instance failed network setup after 1 attempt(s)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager Traceback (most recent call last):
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1770, in _allocate_network_async
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     dhcp_options=dhcp_options)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 49, in wrapped
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     return func(self, context, *args, **kwargs)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/base_api.py", line 64, in wrapper
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     res = f(self, context, *args, **kwargs)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 281, in allocate_for_instance
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     nw_info = self.network_rpcapi.allocate_for_instance(context, **args)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 152, in allocate_for_instance
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     macs=jsonutils.to_primitive(macs))
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 156, in call
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     retry=self.retry)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/transport.py", line 90, in _send
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     timeout=timeout, retry=retry)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 350, in send
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     retry=retry)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 339, in _send
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     result = self._waiter.wait(msg_id, timeout)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 243, in wait
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     message = self.waiters.get(msg_id, timeout=timeout)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager   File "/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 149, in get
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager     'to message ID %s' % msg_id)
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager MessagingTimeout: Timed out waiting for a reply to message ID aa74ba23208449b792ab67daba5c4887
2015-08-10 09:04:51.658 1302 TRACE nova.compute.manager

I am unable to delete the instances either by resetting their state to ACTIVE etc. the Delete fails with a similar MessageTimeout error on the compute node.

I can't see any obvious errors in the debug etc. on either the controller, or network nodes I've setup.

The agents and services are all up and appear to be working ok !

[root@controller ~]# nova service-list
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary           | Host       | Zone     | Status  | State | Updated_at                 | Disabled Reason |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
| 1  | nova-consoleauth | controller | internal | enabled | up    | 2015-08-11T08:19:37.000000 | -               |
| 2  | nova-scheduler   | controller | internal | enabled | up    | 2015-08-11T08:19:37.000000 | -               |
| 3  | nova-cert        | controller | internal | enabled | up    | 2015-08-11T08:19:37.000000 | -               |
| 4  | nova-conductor   | controller | internal | enabled | up    | 2015-08-11T08:19:37.000000 | -               |
| 5  | nova-compute     | compute1   | nova     | enabled | up    | 2015-08-11T08:19:40.000000 | -               |
+----+------------------+------------+----------+---------+-------+----------------------------+-----------------+
[root@controller ~]# neutron agent-list
+--------------------------------------+--------------------+----------+-------+----------------+---------------------------+
| id                                   | agent_type         | host     | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+----------+-------+----------------+---------------------------+
| 331d5364-cdcc-4f99-91b3-f148014fdab0 | Open vSwitch agent | compute1 | :-)   | True           | neutron-openvswitch-agent |
| 3a82c02b-10cb-4af9-a3d9-3b0ac4c450f8 | Metadata agent     | network  | :-)   | True           | neutron-metadata-agent    |
| 62c4b328-c11d-419c-a162-8934b2c0a1b7 | Open vSwitch agent | network  | :-)   | True           | neutron-openvswitch-agent |
| f7af9ea5-d29e-41e6-96d3-da52b64fbe27 | DHCP agent         | network  | :-)   | True           | neutron-dhcp-agent        |
| fe5e30a8-4ae2-4e7e-bddd-600379bee0cc | L3 agent           | network  | :-)   | True           | neutron-l3-agent          |
+--------------------------------------+--------------------+----------+-------+----------------+---------------------------+

Any ideas where to look - why this message timeout could occur?

Thanks Richard