Metadata via the Quantum router : Can't get metadata 403 Forbiden

asked 2013-12-05 11:40:45 -0500

sdo gravatar image

updated 2013-12-05 13:25:57 -0500

darragh-oreilly gravatar image

Hi, I have an openstack grizzly 2013 platform on CentOS 6.4 with : multi vlan with overlapping_ips and namespace support. 1 controller node (running : nova-api, nova-cert, nova-console, nova-conductor, nova-consoleauth, nova-novncproxy, nova-scheduler) 1 network node (running : quantum-linuxbridge, quantum-dhcp-agent, quantum-l3_agent, quantum-metadata-agent) 2 compute nodes (running : libvirtd, nova-compute, nova-novncproxy, quantum-metadata-agent, quantum-linuxbridge)

Everything works well. But my instances can't get metadata

From my instance:

# curl -I http://169.254.169.254/2009-04-04/meta-data/instance-id
HTTP/1.1 500 Internal Server Error
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Date: Thu, 05 Dec 2013 17:22:50 GMT

From network node :

tail -f /var/log/quantum/quantum-ns-metadata-proxyaf476d84-2f59-45e4-baee-5526616eb639.log <==
2013-12-05 18:22:50    DEBUG [quantum.agent.metadata.namespace_proxy] Request: HEAD /2009-04-04/meta-data/instance-id HTTP/1.0
Accept: */*
Content-Type: text/plain
Host: 169.254.169.254
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.13.6.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
2013-12-05 18:22:50    ERROR [quantum.agent.metadata.namespace_proxy] Unexpected error.
Traceback (most recent call last):
  File "/usr/lib/python2.6/site-packages/quantum/agent/metadata/namespace_proxy.py", line 81, in __call__
    req.body)
  File "/usr/lib/python2.6/site-packages/quantum/agent/metadata/namespace_proxy.py", line 129, in _proxy_request
    raise Exception(_('Unexpected response code: %s') % resp.status)
Exception: Unexpected response code: 403

From controller node :

# tail -f /var/log/nova/api.log 
2013-12-05 18:22:50.224 24640 WARNING nova.api.metadata.handler [-] X-Instance-ID-Signature: 12abe6db365db3bdbf0f0865407d2822cba9fca3ed8d23f3050d732d60594735 does not match the expected value: e3ab6fd79552113a89d180935109b38b6c744aec400cb9d13cf51cafa360766
2013-12-05 18:22:50.224 24640 INFO nova.api.ec2 [-] 0.647s 10.225.0.23 HEAD /2009-04-04/meta-data/instance-id None:None 403 [Python-httplib2/0.7.7 (gzip)] text/plain text/html
2013-12-05 18:22:50.225 24640 INFO nova.metadata.wsgi.server [-] 10.225.33.66,10.225.0.23 "HEAD /2009-04-04/meta-data/instance-id HTTP/1.1" status: 403 len: 122 time: 0.0012341

Looks like metadata receives the request but cannot authenticate because of the X-Instance-ID-Signature mismatch.

Don't know where that X-Instance-ID comes from and how to solve that issue ? Any help would be appreciated.


SDO

edit retag flag offensive close merge delete

Comments

Does metadata_proxy_shared_secret in metadata_agent.ini match quantum_metadata_proxy_shared_secret in nova.conf?

darragh-oreilly gravatar imagedarragh-oreilly ( 2013-12-05 13:29:22 -0500 )edit

2 answers

Sort by ยป oldest newest most voted
0

answered 2013-12-06 04:27:12 -0500

sdo gravatar image

Hi,

Yes already checked that quantum_metadata_proxy_shared_secret is the same on nova.conf and metadata_agent.ini...

But I have some more news... From the following error I tried to reproduce the X-Instance-ID-Signature

/var/log/nova/api.log (on controller node) 2013-12-04 16:44:49.110 25890 WARNING nova.api.metadata.handler [-] X-Instance-ID-Signature: c5d87439bb2c0cf2d6ccf910b2b5355d2798681e7de3d4c90e201ed3e2094244 does not match the expected value: c1e1eb6703a58b5c6e7a6beb7d7da313be57b8296d140e8c7c3339a57886662 2 for id: a4d6af1a-a316-487b-a28d-e5a9b10d7c76. Request From: 10.225.33.98

Lets check into the nova DB what is this instance ID:

mysql> select uuid, host, deleted, deleted_at from instances where uuid='a4d6af1a-a316-487b-a28d-e5a9b10d7c76';              
+--------------------------------------+----------------------+---------+---------------------+
| uuid                                 | host                 | deleted | deleted_at          |
+--------------------------------------+----------------------+---------+---------------------+
| a4d6af1a-a316-487b-a28d-e5a9b10d7c76 | comp-02 |     130 | 2013-12-05 09:30:01 |
+--------------------------------------+----------------------+---------+---------------------+

Looks like that instance was deleted. Here are running instance in my platform :

mysql> select uuid, host, deleted, deleted_at from instances where deleted = 0 ; 
+--------------------------------------+----------------------+---------+------------+
| uuid                                 | host                 | deleted | deleted_at |
+--------------------------------------+----------------------+---------+------------+
| b04eb38f-4131-4798-a01e-3184f4f37332 | comp-01 |       0 | NULL       |
| a56eae7e-6fe7-4e1f-a8b5-72a77937dd04 | comp-02 |       0 | NULL       |
+--------------------------------------+----------------------+---------+------------+

Looks like that when I'm making a metadata request from one of the instances above the ID sent is not the right one.

How can it be ? Any idea.

Next step: removing all my instances. cleaning nova, quantum and cinder Databases and restart the process again. And let's see if that will solve the issue. ....

edit flag offensive delete link more

Comments

It looks like the metadata-agent looked up the wrong instance ID in Quantum. Is there anything in the metadata-agent.log? Maybe Quantum has some stale ports from old deleted instances. But new ones should not be created with a used IP. Run quantum port-list for that network.

darragh-oreilly gravatar imagedarragh-oreilly ( 2013-12-06 06:00:53 -0500 )edit

I found that while my VM tries to access metadata, the calculated X-Instance-ID-signature inside the request (quantum/agent/metadata/agent.py) from my network node is different with that one calculated by the controller node (nova/api/metadata/handler.py) even if the quantum_metadata_proxy_shared_secret and the instance-id is the same on both side. Looks like the following piece of code inside nova/api/metadata/handler.py and nova/api/metadata/handler.py does not give the same result. hmac.new(self.conf.metadata_proxy_shared_secret, instance_id, hashlib.sha256).hexdigest() But when a make a file with the code above and run it on controller and on network node I have the same result. Looks like that when it's called by quantum-ns-metadata-proxy it does not give the same result. To figure it out I've commented following the check of X-Instance-ID-siganture inside nova/api/metadata/handler.py and magically my VM can ...(more)

sdo gravatar imagesdo ( 2013-12-10 14:03:47 -0500 )edit
0

answered 2014-03-08 05:35:40 -0500

fbwfbi gravatar image

Hi, I met the same issue with your in the OpenStack Havana. However , I nailed it finally ......But , I want to confirm that ..........Do you really set that metadata_proxy_shared_secret in metadata_agent.ini match neutron_metadata_proxy_shared_secret(havana) in nova.conf ?

NOTE: Nova uses a different key: "neutron_metadata_proxy_shared_secret"(havana), instead of the key: "metadata_proxy_shared_secret" by the metadata_agent.ini in Neutron !

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2013-12-05 11:40:45 -0500

Seen: 1,452 times

Last updated: Mar 08 '14