Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Neutron metadata agent "unexpected error"

I am still struggling with the not-so-co-operative neutron. This is now a two-node test installation using virtualbox instances, Ubuntu 14.04, Icehouse. Control node (.101) runs everything except computing, which is done by compute (.103).
After lengthy trial and error session I now manage to launch instances but they are unreachable as the tenant network does not appear to get set up properly. The only real error in logs is in neutron metadata log:

2014-08-06 14:53:01.576 2048 ERROR neutron.agent.metadata.agent [req-ca7f07b7-671f-4ee6-99c0-56ae53db9ef0 None] Unexpected error.
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 110, in __call__
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent     instance_id, tenant_id = self._get_instance_and_tenant_id(req)
....
....
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent     endpoint_type=self.endpoint_type)
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 81, in url_for
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent     raise exceptions.EndpointNotFound()
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent EndpointNotFound: Could not find Service or Region in Service Catalog.

This appears to be a simple keystone database problem but I fail to see where it is. Neutron is configured to use the service tenant.

visti@control2:~$ keystone service-list
+----------------------------------+----------+----------+------------------------------+
|                id                |   name   |   type   |         description          |
+----------------------------------+----------+----------+------------------------------+
| 6c25e11229194961aac8f9b8e0a78746 |  cinder  |  volume  |   OpenStack Volume Service   |
| 9c325003ddb74c6c8720ff62092a9709 |  glance  |  image   |   OpenStack Image Service    |
| 69f3f9a792bc4d4abffdecb8d27d8175 | keystone | identity |      OpenStack Identity      |
| e8d4d4f3482949f7bf7245fb555136f3 | neutron  | network  | OpenStack Networking service |
| 79b28afb127041099dcb36017bbf84cb |   nova   | compute  |  OpenStack Compute Service   |
+----------------------------------+----------+----------+------------------------------+
visti@control2:~$ keystone endpoint-list
+----------------------------------+-----------+-------------------------------------------+-------------------------------------------+-------------------------------------------+----------------------------------+
|                id                |   region  |                 publicurl                 |                internalurl                |                  adminurl                 |            service_id            |
+----------------------------------+-----------+-------------------------------------------+-------------------------------------------+-------------------------------------------+----------------------------------+
| 77357fbc77154140b19898179cc30dc9 | RegionOne |        http://172.16.1.101:9292/v2        |        http://10.10.10.101:9292/v2        |        http://10.10.10.101:9292/v2        | 9c325003ddb74c6c8720ff62092a9709 |
| 8d7f8cb0eee143dda00eea78dce7179e | RegionOne |         http://172.16.1.101:9696/         |         http://10.10.10.101:9696/         |         http://10.10.10.101:9696/         | e8d4d4f3482949f7bf7245fb555136f3 |
| 9b99110cf7d4457b8dda19bbfd00f326 | RegionOne | http://172.16.1.101:8776/v1/$(tenant_id)s | http://10.10.10.101:8776/v1/$(tenant_id)s | http://10.10.10.101:8776/v1/$(tenant_id)s | 6c25e11229194961aac8f9b8e0a78746 |
| a1a1655f16ad4c90a1f6bd318c1e0f82 | RegionOne |       http://172.16.1.101:5000/v2.0       |       http://10.10.10.101:5000/v2.0       |       http://10.10.10.101:35357/v2.0      | 69f3f9a792bc4d4abffdecb8d27d8175 |
| a307e4fa6f5e4ff386ad0d2664127f2f | RegionOne | http://172.16.1.101:8774/v2/$(tenant_id)s | http://10.10.10.101:8774/v2/$(tenant_id)s | http://10.10.10.101:8774/v2/$(tenant_id)s | 79b28afb127041099dcb36017bbf84cb |
+----------------------------------+-----------+-------------------------------------------+-------------------------------------------+-------------------------------------------+----------------------------------+
visti@control2:~$ keystone tenant-list
+----------------------------------+---------+---------+
|                id                |   name  | enabled |
+----------------------------------+---------+---------+
| 6faa4fd099c345b79dc0496b5a29119e |  admin  |   True  |
| 566c2813ebe64b0093e6bb7bd24d95e6 |   demo  |   True  |
| e75489fa55fd4c85a3c7da7fc4ac0716 | service |   True  |
+----------------------------------+---------+---------+

Everything else works. Neutron commands to create ext-net, demo-net and router between these work absolutely fine. It is just when launching an instance, these errors appear. The instance launches and it gets a private ip address. From the instance console the router is even reachable, but there is no connectivity to the router from the compute host or any other host for that matter.

Any ideas?

Neutron metadata agent "unexpected error"

I am still struggling with the not-so-co-operative neutron. This is now a two-node test installation using virtualbox instances, Ubuntu 14.04, Icehouse. Control node (.101) runs everything except computing, which is done by compute (.103).
After lengthy trial and error session I now manage to launch instances but they are unreachable as the tenant network does not appear to get set up properly. The only real error in logs is in neutron metadata log:

2014-08-06 14:53:01.576 2048 ERROR neutron.agent.metadata.agent [req-ca7f07b7-671f-4ee6-99c0-56ae53db9ef0 None] Unexpected error.
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent Traceback (most recent call last):
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 110, in __call__
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent     instance_id, tenant_id = self._get_instance_and_tenant_id(req)
....
....
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent     endpoint_type=self.endpoint_type)
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent   File "/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 81, in url_for
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent     raise exceptions.EndpointNotFound()
2014-08-06 14:53:01.576 2048 TRACE neutron.agent.metadata.agent EndpointNotFound: Could not find Service or Region in Service Catalog.

This appears to be a simple keystone database problem but I fail to see where it is. Neutron is configured to use the service tenant.

visti@control2:~$ keystone service-list
+----------------------------------+----------+----------+------------------------------+
|                id                |   name   |   type   |         description          |
+----------------------------------+----------+----------+------------------------------+
| 6c25e11229194961aac8f9b8e0a78746 |  cinder  |  volume  |   OpenStack Volume Service   |
| 9c325003ddb74c6c8720ff62092a9709 |  glance  |  image   |   OpenStack Image Service    |
| 69f3f9a792bc4d4abffdecb8d27d8175 | keystone | identity |      OpenStack Identity      |
| e8d4d4f3482949f7bf7245fb555136f3 | neutron  | network  | OpenStack Networking service |
| 79b28afb127041099dcb36017bbf84cb |   nova   | compute  |  OpenStack Compute Service   |
+----------------------------------+----------+----------+------------------------------+
visti@control2:~$ keystone endpoint-list
+----------------------------------+-----------+-------------------------------------------+-------------------------------------------+-------------------------------------------+----------------------------------+
|                id                |   region  |                 publicurl                 |                internalurl                |                  adminurl                 |            service_id            |
+----------------------------------+-----------+-------------------------------------------+-------------------------------------------+-------------------------------------------+----------------------------------+
| 77357fbc77154140b19898179cc30dc9 | RegionOne |        http://172.16.1.101:9292/v2        |        http://10.10.10.101:9292/v2        |        http://10.10.10.101:9292/v2        | 9c325003ddb74c6c8720ff62092a9709 |
| 8d7f8cb0eee143dda00eea78dce7179e | RegionOne |         http://172.16.1.101:9696/         |         http://10.10.10.101:9696/         |         http://10.10.10.101:9696/         | e8d4d4f3482949f7bf7245fb555136f3 |
| 9b99110cf7d4457b8dda19bbfd00f326 | RegionOne | http://172.16.1.101:8776/v1/$(tenant_id)s | http://10.10.10.101:8776/v1/$(tenant_id)s | http://10.10.10.101:8776/v1/$(tenant_id)s | 6c25e11229194961aac8f9b8e0a78746 |
| a1a1655f16ad4c90a1f6bd318c1e0f82 | RegionOne |       http://172.16.1.101:5000/v2.0       |       http://10.10.10.101:5000/v2.0       |       http://10.10.10.101:35357/v2.0      | 69f3f9a792bc4d4abffdecb8d27d8175 |
| a307e4fa6f5e4ff386ad0d2664127f2f | RegionOne | http://172.16.1.101:8774/v2/$(tenant_id)s | http://10.10.10.101:8774/v2/$(tenant_id)s | http://10.10.10.101:8774/v2/$(tenant_id)s | 79b28afb127041099dcb36017bbf84cb |
+----------------------------------+-----------+-------------------------------------------+-------------------------------------------+-------------------------------------------+----------------------------------+
visti@control2:~$ keystone tenant-list
+----------------------------------+---------+---------+
|                id                |   name  | enabled |
+----------------------------------+---------+---------+
| 6faa4fd099c345b79dc0496b5a29119e |  admin  |   True  |
| 566c2813ebe64b0093e6bb7bd24d95e6 |   demo  |   True  |
| e75489fa55fd4c85a3c7da7fc4ac0716 | service |   True  |
+----------------------------------+---------+---------+

Everything else works. Neutron commands to create ext-net, demo-net and router between these work absolutely fine. It is just when launching an instance, these errors appear. The instance launches and it gets a private ip address. From the instance console the router is even reachable, but there is no connectivity to the router from the compute host or any other host for that matter.

Any ideas?

Added neutron agent-list:

visti@control2:~$ neutron agent-list
+--------------------------------------+--------------------+----------+-------+----------------+
| id                                   | agent_type         | host     | alive | admin_state_up |
+--------------------------------------+--------------------+----------+-------+----------------+
| 054d0b46-be1f-4f5a-ac0f-680bd000281f | Metadata agent     | control2 | :-)   | True           |
| 14977d4c-ab0d-4978-a357-735ed965194b | Open vSwitch agent | compute2 | :-)   | True           |
| 42a6dad4-10fd-466a-a34f-e2943ac01ef2 | DHCP agent         | control2 | :-)   | True           |
| 9bf73a76-78de-48c5-b5d0-39b83521619d | Open vSwitch agent | control2 | :-)   | True           |
| dc869f4f-f93d-4e29-bc48-87f993c7696f | L3 agent           | control2 | :-)   | True           |
+--------------------------------------+--------------------+----------+-------+----------------+