Ask Your Question
1

Metadata server cannot be reached: which port is used by quantum service? [closed]

asked 2013-05-22 18:52:10 -0500

spcla1 gravatar image

updated 2013-06-11 10:25:36 -0500

fifieldt gravatar image

I have setup Openstack Grizzly on multiple machines, 1 controller node, 1 network node (running dhcp L3 agent and Openvswitch agent) and multiple compute nodes. Everything seems to be working ok. I can start a VM and can assign a floating ip. I can ping the VM using both the private and floating ip but can't ssh due to the ssh key not being loaded (metadata server can't be reached). Got the following error from the console:

2013-05-22 23:04:01,665 - util.py[WARNING]: 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [9/120s]: http error [500]

I found this link https://bugs.launchpad.net/quantum/+bug/1160955 and I thought this was related to my problem but even after applying the patch, I was still having the same problem. One thing I noticed was I have 2 quantum-ns-metadata-proxy running and they are using 2 different metadata ports, here's the output of "ps -ef " grep quantum-ns" (I removed some info to make the output shorter). One of them is using port 80 and one is using 9697.

python /usr/bin/quantum-ns-metadata-proxy --pid_file=/var/lib/quantum/external/pids/3bb5723a-94d4-41d7-8daf-2139320c5ee9.pid --network_id=3bb5723a-94d4-41d7-8daf-2139320c5ee9 --state_path=/var/lib/quantum --metadata_port=80 

python /usr/bin/quantum-ns-metadata-proxy --pid_file=/var/lib/quantum/external/pids/5e6a567d-c9ae-432c-89ab-bd67eab5692f.pid --router_id=5e6a567d-c9ae-432c-89ab-bd67eab5692f --state_path=/var/lib/quantum --metadata_port=9697

The iptables command:

ip netns exec qrouter-5e6a567d-c9ae-432c-89ab-bd67eab5692f iptables -L -t nat

shows the following which is port 9697:

REDIRECT   tcp  --  anywhere             169.254.169.254      tcp dpt:http redir ports 9697

I have port 9697 in my l3_agent.ini which I believe is the default port.

Here's what I have on the metadata-agent.log and file /var/log/quantum/quantum-ns-metadata-proxy{{ router-ns-uuid }}.log is empty.

ERROR [quantum.agent.metadata.agent] Unexpected error. Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/quantum/agent/metadata/agent.py", line 88, in callreturn self.proxyrequest(instance_id, req)
File "/usr/lib/python2.7/dist-packages/quantum/agent/metadata/agent.py", line 137, in proxyrequest resp, content = h.request(url, headers=headers)
File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1444, in request (response, content) = self.request(conn, authority, uri, requesturi, method, body, headers, redirections, cachekey)   
File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1196, inrequest (response, content) = self.connrequest(conn, requesturi, method, body, headers)    
File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 1132, in connrequest conn.connect()    
File "/usr/lib/python2.7/dist-packages/httplib2/init.py", line 798, in connect raise socket.error, msg error: [Errno 111] ECONNREFUSED

ip netns exec {{ dhcp-ns-uuid}} netstat -an shows the following:

Destination Gateway Genmask Flags MSS Window irtt Iface    
169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ns-468d9253-00    
172.20.1.0 0.0.0.0 255.255.255.0 U 0 0 0 ns-468d9253-00

I have the metadata-agent.ini configured with all the information you mentioned above and also have this entry on the nova.conf file:

enabledapis=ec2,osapicompute ...
(more)
edit retag flag offensive reopen merge delete

Closed for the following reason the question is answered, right answer was accepted by spcla1
close date 2014-03-06 19:19:12.392837

2 answers

Sort by ยป oldest newest most voted
2

answered 2013-05-23 11:17:15 -0500

spcla1 gravatar image

updated 2013-05-24 13:28:57 -0500

I got it working now. I don't really know what resolved the problem but the only thing I changed was the /etc/sysctl.conf on all the compute nodes, set the net.ipv4.ip_forward to 1, restarted all the quantum and nova services on both the compute nodes, network nodes and controller.Works now! Thanks!

edit flag offensive delete link more
0

answered 2013-05-23 02:41:34 -0500

darragh-oreilly gravatar image

updated 2013-05-23 17:04:33 -0500

I'd check the logs /var/log/quantum/quantum-ns-metadata-proxy{{ router-ns-uuid }}.log and metadata-agent.log first.

I reckon the quantum-ns-metadata-proxy running on port 80 was started by the dhcp agent because of https://bugs.launchpad.net/quantum/+bug/1159015 - is it seen in 'ip netns exec {{ dhcp-ns-uuid}} netstat -an'?

metadata_agent.ini needs to have nova_metadata_ip and nova_metadata_port pointing to the Nova metadata server, and the credentials to access the main Quantum service. The machine running the metadata needs to have the word metadata as part of the enabled_apis list in nova.conf.

The BP spec https://docs.google.com/document/d/1wixS-CrHe37Fv4my9MxUVeQKDb3mUJJCwPnireQ1gn8/edit helped me to get it working.

Update: Hi, I haven't tried the DHCP way yet and I don't know how it works. But the stacktrace indicates a low level socket connection problem. Check the nova metadata service on the controller node is ok: 'grep metadata /var/log/nova/nova-api.log' and 'netstat -an | grep 8775'. Do you have nova_metadata_port=8775 in metadata_agent.ini ?

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2013-05-22 18:52:10 -0500

Seen: 7,408 times

Last updated: May 23 '13