Neutron metadata agent is not working - 500 Internal Server Error

asked 2018-12-25 05:28:26 -0500

Am Ab gravatar image

updated 2018-12-25 06:26:28 -0500

Hello,

I've a packstack installation with 3 nodes (1 control + 1 compute + 1network ), but metadata proxy is not working. here is the test results:

$ curl -v http://169.254.169.254

GET / HTTP/1.1 User-Agent: curl/7.24.0 (x86_64-pc-linux-gnu) libcurl/7.24.0 OpenSSL/1.0.0j zlib/1.2.6 Host: 169.254.169.254 Accept: /

< HTTP/1.1 500 Internal Server Error < Content-Length: 207 < Content-Type: text/html; charset=UTF-8 < Date: Tue, 25 Dec 2018 10:13:17 GMT < <html> <head> <title>500 Internal Server Error</title> </head> <body>

  • Also tcpdump result on network node:

13:40:15.006881 IP 169.254.169.254.http > 10.20.30.8.47759: Flags [P.], seq 1:344, ack 143, win 227, options [nop,nop,TS val 1290763920 ecr 2367402], length 343: HTTP: HTTP/1.1 500 Internal Server Error

  • The nova_metadata_ip=<control-node-ip> and metadata_proxy_shared_secret seems to be OK.

  • Here is the metadata-agent.log:

2018-12-25 10:57:28.640 2121 INFO eventlet.wsgi.server [-] 10.20.30.8,<local> "GET /2009-04-04/meta-data/instance-id HTTP/1.1" status: 500 len: 362 time: 0.2060108 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent [-] Unexpected error.: error: [Errno 111] ECONNREFUSED 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent Traceback (most recent call last): 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 91, in __call__ 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent return self._proxy_request(instance_id, tenant_id, req) 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/neutron/agent/metadata/agent.py", line 201, in _proxy_request 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent body=req.body) 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1621, in request 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey) 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1363, in _request 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent (response, content) = self._conn_request(conn, request_uri, method, body, headers) 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 1284, in _conn_request 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent conn.connect() 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent File "/usr/lib/python2.7/site-packages/httplib2/__init__.py", line 934, in connect 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent raise socket.error, msg 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent error: [Errno 111] ECONNREFUSED 2018-12-25 10:57:30.876 2121 ERROR neutron.agent.metadata.agent

edit retag flag offensive close merge delete

Comments

Anything in the metadata server's log file?

Bernd Bausch gravatar imageBernd Bausch ( 2018-12-25 05:34:41 -0500 )edit

I've edited original question and attached the log

Am Ab gravatar imageAm Ab ( 2018-12-25 06:47:50 -0500 )edit

This log entry is not related to the failed access to 169.254.169.254. See the timestamp. I guess, however, that there are other, similar errors in the log.

Bernd Bausch gravatar imageBernd Bausch ( 2018-12-25 16:30:01 -0500 )edit

I would think that metadata agent parameters nova_metadata_host and nova_metadata_port are at fault. nova_metadata_port is normally 8775. I would double-check these config values.

Then: Is anything listening at port 8775? Can you make a manual http GET request to http://controller:8775?

Bernd Bausch gravatar imageBernd Bausch ( 2018-12-25 16:35:08 -0500 )edit

I have just tried to do this at home. This is what it should look like:

$ curl http://192.168.1.200:8775
1.0
2007-01-19
...etc...

This is a Rocky Devstack. The Metadata API is implemented by processes named nova-api-metauWSGI.

Bernd Bausch gravatar imageBernd Bausch ( 2018-12-25 20:55:30 -0500 )edit