Revision history [back]

click to hide/show revision 1
initial version

[Juno] Neutron-metadata-agent can't start

Hi there,

I've been trying to deploy OpenStack Juno on a 4-node (controller-network-computerx2 cluster of real server and am facing now the issue of metadata related services being unable to start.

here is the output of /var/log/upstart/neutron-metadata-agent.log in Network node

Traceback (most recent call last):
  File "/usr/bin/neutron-metadata-agent", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 378, in main
    config.init(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/neutron/common/config.py", line 150, in init
    n_rpc.init(cfg.CONF)
  File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 57, in init
    aliases=TRANSPORT_ALIASES)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 185, in get_transport
    invoke_kwds=kwargs)
  File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 45, in __init__
    verify_requirements=verify_requirements,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 55, in __init__
    verify_requirements)
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 170, in _load_plugins
    self._on_load_failure_callback(self, ep, err)
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 162, in _load_plugins
    verify_requirements,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 123, in _load_one_plugin
    verify_requirements,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 187, in _load_one_plugin
    obj = plugin(*invoke_args, **invoke_kwds)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1131, in __init__
    conf, conf.oslo_messaging_rabbit.rpc_conn_pool_size,
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2309, in __getattr__
    return self._conf._get(name, self._group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2043, in _get
    value = self._do_get(name, group, namespace)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2061, in _do_get
    info = self._get_opt_info(name, group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2185, in _get_opt_info
    group = self._get_group(group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2169, in _get_group
    raise NoSuchGroupError(group_name)
oslo.config.cfg.NoSuchGroupError: no such group: <oslo_config.cfg.OptGroup object at 0x7feb0689ba50>

And output of /var/log/nova/nova-api-metadata.log in Compute node

2015-06-20 10:03:14.163 62820 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2015-06-20 10:03:14.347 62820 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-06-20 10:03:14.350 62820 ERROR nova.wsgi [-] Could not bind to 130.104.230.109:8775
2015-06-20 10:03:14.351 62820 CRITICAL nova [-] error: [Errno 99] Cannot assign requested address
2015-06-20 10:03:14.351 62820 TRACE nova Traceback (most recent call last):
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/bin/nova-api-metadata", line 10, in <module>
2015-06-20 10:03:14.351 62820 TRACE nova     sys.exit(main())
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/api_metadata.py", line 52, in main
2015-06-20 10:03:14.351 62820 TRACE nova     server = service.WSGIService('metadata', use_ssl=should_use_ssl)
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 351, in __init__
2015-06-20 10:03:14.351 62820 TRACE nova     max_url_len=max_url_len)
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/wsgi.py", line 141, in __init__
2015-06-20 10:03:14.351 62820 TRACE nova     self._socket = eventlet.listen(bind_addr, family, backlog=backlog)
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 38, in listen
2015-06-20 10:03:14.351 62820 TRACE nova     sock.bind(addr)
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/socket.py", line 224, in meth
2015-06-20 10:03:14.351 62820 TRACE nova     return getattr(self._sock,name)(*args)
2015-06-20 10:03:14.351 62820 TRACE nova error: [Errno 99] Cannot assign requested address

As a consequence, I suppose this missing of metadata services result in problem of retrieving metadata info for VM instance: (/var/log/neutron/neutron-ns-metadata-proxy-27fc9d3c-fa3b-4637-a486-dbcf0cfab454.log on Network node)

2015-06-20 01:45:36.135 3275 INFO neutron.wsgi [-] (3275) accepted ('11.0.0.6', 49583)
2015-06-20 01:45:36.136 3275 ERROR neutron.agent.metadata.namespace_proxy [-] Unexpected error.
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy Traceback (most recent call last):
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", line 72, in __call__
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     req.body)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", line 103, in _proxy_request
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     connection_type=UnixDomainHTTPConnection)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1569, in request
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1316, in _request
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     (response, content) = self._conn_request(conn, request_uri, method, body, headers)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1251, in _conn_request
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     conn.connect()
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", line 46, in connect
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     self.sock.connect(cfg.CONF.metadata_proxy_socket)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 235, in connect
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     while not socket_connect(fd, address):
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 37, in socket_connect
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     raise socket.error(err, errno.errorcode[err])
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy error: [Errno 111] ECONNREFUSED
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy
2015-06-20 01:45:36.138 3275 INFO neutron.wsgi [-] 11.0.0.6 - - [20/Jun/2015 01:45:36] "GET /2009-04-04/meta-data/instance-id HTTP/1.1" 500 343 0.002100

Besides, neutron-server doesn't seem working properly either (as in /var/log/upstart/neutron-server.log on Controller)

2015-06-20 16:00:51.826 3375 DEBUG neutron.context [req-04653d26-5bd5-4a22-ba44-204de746e671 None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} __init__ /usr/lib/python2.7/dist-packages/neutron/context.py:83

As i've been trying to deploy these, all network resources (network, subnet, router, etc.) can still be allocated and VM instance can be initialised and running. But i believe the missing of metadata makes it impossible to ssh to VM (i've tried several cloud images including CirrOS, Ubuntu, Fedora yet none worked). Another thing that might be worth noting is that lots of packages require removal and reinstallation, especially those relating to python on the Network node. This didn't occur when i tested on Virtualbox clusters or Amazon AWS.

I appreciate your help!

cheers,

Tuan Anh

[Juno] Neutron-metadata-agent can't start

Hi there,

I've been trying to deploy OpenStack Juno on a 4-node (controller-network-computerx2 cluster of real server and am facing now the issue of metadata related services being unable to start.

here is the output of /var/log/upstart/neutron-metadata-agent.log in Network node

Traceback (most recent call last):
  File "/usr/bin/neutron-metadata-agent", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 378, in main
    config.init(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/neutron/common/config.py", line 150, in init
    n_rpc.init(cfg.CONF)
  File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 57, in init
    aliases=TRANSPORT_ALIASES)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 185, in get_transport
    invoke_kwds=kwargs)
  File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 45, in __init__
    verify_requirements=verify_requirements,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 55, in __init__
    verify_requirements)
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 170, in _load_plugins
    self._on_load_failure_callback(self, ep, err)
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 162, in _load_plugins
    verify_requirements,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 123, in _load_one_plugin
    verify_requirements,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 187, in _load_one_plugin
    obj = plugin(*invoke_args, **invoke_kwds)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1131, in __init__
    conf, conf.oslo_messaging_rabbit.rpc_conn_pool_size,
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2309, in __getattr__
    return self._conf._get(name, self._group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2043, in _get
    value = self._do_get(name, group, namespace)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2061, in _do_get
    info = self._get_opt_info(name, group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2185, in _get_opt_info
    group = self._get_group(group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2169, in _get_group
    raise NoSuchGroupError(group_name)
oslo.config.cfg.NoSuchGroupError: no such group: <oslo_config.cfg.OptGroup object at 0x7feb0689ba50>

And output of /var/log/nova/nova-api-metadata.log in Compute nodenode (full log: http://pastebin.com/ZJtJX8zP)

2015-06-20 10:03:14.163 62820 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2015-06-20 10:03:14.347 62820 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-06-20 10:03:14.350 62820 ERROR nova.wsgi [-] Could not bind to 130.104.230.109:8775
2015-06-20 10:03:14.351 62820 CRITICAL nova [-] error: [Errno 99] Cannot assign requested address
2015-06-20 10:03:14.351 62820 TRACE nova Traceback (most recent call last):
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/bin/nova-api-metadata", line 10, in <module>
2015-06-20 10:03:14.351 62820 TRACE nova     sys.exit(main())
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/api_metadata.py", line 52, in main
2015-06-20 10:03:14.351 62820 TRACE nova     server = service.WSGIService('metadata', use_ssl=should_use_ssl)
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/service.py", line 351, in __init__
2015-06-20 10:03:14.351 62820 TRACE nova     max_url_len=max_url_len)
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/wsgi.py", line 141, in __init__
2015-06-20 10:03:14.351 62820 TRACE nova     self._socket = eventlet.listen(bind_addr, family, backlog=backlog)
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/dist-packages/eventlet/convenience.py", line 38, in listen
2015-06-20 10:03:14.351 62820 TRACE nova     sock.bind(addr)

(...)

2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/socket.py", line 224, in meth
2015-06-20 10:03:14.351 62820 TRACE nova     return getattr(self._sock,name)(*args)
2015-06-20 10:03:14.351 62820 TRACE nova error: [Errno 99] Cannot assign requested address

As a consequence, I suppose this missing of metadata services result in problem of retrieving metadata info for VM instance: (/var/log/neutron/neutron-ns-metadata-proxy-27fc9d3c-fa3b-4637-a486-dbcf0cfab454.log on Network node)node. Full log: http://pastebin.com/jKmfTCCM)

2015-06-20 01:45:36.135 3275 INFO neutron.wsgi [-] (3275) accepted ('11.0.0.6', 49583)
2015-06-20 01:45:36.136 3275 ERROR neutron.agent.metadata.namespace_proxy [-] Unexpected error.
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy Traceback (most recent call last):
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", line 72, in __call__
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     req.body)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", line 103, in _proxy_request
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     connection_type=UnixDomainHTTPConnection)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1569, in request
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     (response, content) = self._request(conn, authority, uri, request_uri, method, body, headers, redirections, cachekey)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1316, in _request
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     (response, content) = self._conn_request(conn, request_uri, method, body, headers)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1251, in _conn_request
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     conn.connect()
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", line 46, in connect
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     self.sock.connect(cfg.CONF.metadata_proxy_socket)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 235, in connect
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     while not socket_connect(fd, address):
(...)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 37, in socket_connect
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     raise socket.error(err, errno.errorcode[err])
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy error: [Errno 111] ECONNREFUSED
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy
2015-06-20 01:45:36.138 3275 INFO neutron.wsgi [-] 11.0.0.6 - - [20/Jun/2015 01:45:36] "GET /2009-04-04/meta-data/instance-id HTTP/1.1" 500 343 0.002100

Besides, neutron-server doesn't seem working properly either (as in /var/log/upstart/neutron-server.log on Controller)

2015-06-20 16:00:51.826 3375 DEBUG neutron.context [req-04653d26-5bd5-4a22-ba44-204de746e671 None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} __init__ /usr/lib/python2.7/dist-packages/neutron/context.py:83

As i've been trying to deploy these, all network resources (network, subnet, router, etc.) can still be allocated and VM instance can be initialised and running. But i believe the missing of metadata makes it impossible to ssh to VM (i've tried several cloud images including CirrOS, Ubuntu, Fedora yet none worked). Another thing that might be worth noting is that lots of packages require removal and reinstallation, especially those relating to python on the Network node. This didn't occur when i tested on Virtualbox clusters or Amazon AWS.

I appreciate your help!

cheers,

Tuan Anh

[Juno] Neutron-metadata-agent can't start

Hi there,

I've been trying to deploy OpenStack Juno on a 4-node (controller-network-computerx2 cluster of real server and am facing now the issue of metadata related services being unable to start.

here is the output of /var/log/upstart/neutron-metadata-agent.log in Network nodenode (full log: http://pastebin.com/QG2edmkt)

Traceback (most recent call last):
  File "/usr/bin/neutron-metadata-agent", line 10, in <module>
    sys.exit(main())
  File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line 378, in main
    config.init(sys.argv[1:])
  File "/usr/lib/python2.7/dist-packages/neutron/common/config.py", line 150, in init
    n_rpc.init(cfg.CONF)
  File "/usr/lib/python2.7/dist-packages/neutron/common/rpc.py", line 57, in init
    aliases=TRANSPORT_ALIASES)
  File "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line 185, in get_transport
    invoke_kwds=kwargs)
  File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 45, in __init__
    verify_requirements=verify_requirements,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 55, in __init__
    verify_requirements)
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 170, in _load_plugins
    self._on_load_failure_callback(self, ep, err)
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 162, in _load_plugins
    verify_requirements,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 123, in _load_one_plugin
    verify_requirements,
  File "/usr/local/lib/python2.7/dist-packages/stevedore/extension.py", line 187, in _load_one_plugin
    obj = plugin(*invoke_args, **invoke_kwds)
  File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1131, in __init__
    conf, conf.oslo_messaging_rabbit.rpc_conn_pool_size,
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2309, in __getattr__
    return self._conf._get(name, self._group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2043, in _get
    value = self._do_get(name, group, namespace)

(...)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2061, in _do_get
    info = self._get_opt_info(name, group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2185, in _get_opt_info
    group = self._get_group(group)
  File "/usr/lib/python2.7/dist-packages/oslo/config/cfg.py", line 2169, in _get_group
    raise NoSuchGroupError(group_name)
oslo.config.cfg.NoSuchGroupError: no such group: <oslo_config.cfg.OptGroup object at 0x7feb0689ba50>

And output of /var/log/nova/nova-api-metadata.log in Compute node (full log: http://pastebin.com/ZJtJX8zP)http://pastebin.com/ZJtJX8zP)

2015-06-20 10:03:14.163 62820 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
2015-06-20 10:03:14.347 62820 INFO nova.openstack.common.periodic_task [-] Skipping periodic task _periodic_update_dns because its interval is negative
2015-06-20 10:03:14.350 62820 ERROR nova.wsgi [-] Could not bind to 130.104.230.109:8775
2015-06-20 10:03:14.351 62820 CRITICAL nova [-] error: [Errno 99] Cannot assign requested address
2015-06-20 10:03:14.351 62820 TRACE nova Traceback (most recent call last):
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/bin/nova-api-metadata", line 10, in <module>
2015-06-20 10:03:14.351 62820 TRACE nova     sys.exit(main())
2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/dist-packages/nova/cmd/api_metadata.py", line 52, in main
2015-06-20 10:03:14.351 62820 TRACE nova     server = service.WSGIService('metadata', use_ssl=should_use_ssl)

(...)

2015-06-20 10:03:14.351 62820 TRACE nova   File "/usr/lib/python2.7/socket.py", line 224, in meth
2015-06-20 10:03:14.351 62820 TRACE nova     return getattr(self._sock,name)(*args)
2015-06-20 10:03:14.351 62820 TRACE nova error: [Errno 99] Cannot assign requested address

As a consequence, I suppose this missing of metadata services result in problem of retrieving metadata info for VM instance: (/var/log/neutron/neutron-ns-metadata-proxy-27fc9d3c-fa3b-4637-a486-dbcf0cfab454.log on Network node. Full log: http://pastebin.com/jKmfTCCM)http://pastebin.com/jKmfTCCM)

2015-06-20 01:45:36.135 3275 INFO neutron.wsgi [-] (3275) accepted ('11.0.0.6', 49583)
2015-06-20 01:45:36.136 3275 ERROR neutron.agent.metadata.namespace_proxy [-] Unexpected error.
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy Traceback (most recent call last):
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py", line 72, in __call__
(...)
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy   File "/usr/local/lib/python2.7/dist-packages/eventlet/greenio/base.py", line 37, in socket_connect
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy     raise socket.error(err, errno.errorcode[err])
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy error: [Errno 111] ECONNREFUSED
2015-06-20 01:45:36.136 3275 TRACE neutron.agent.metadata.namespace_proxy
2015-06-20 01:45:36.138 3275 INFO neutron.wsgi [-] 11.0.0.6 - - [20/Jun/2015 01:45:36] "GET /2009-04-04/meta-data/instance-id HTTP/1.1" 500 343 0.002100

Besides, neutron-server doesn't seem working properly either (as in /var/log/upstart/neutron-server.log on Controller)

2015-06-20 16:00:51.826 3375 DEBUG neutron.context [req-04653d26-5bd5-4a22-ba44-204de746e671 None] Arguments dropped when creating context: {u'project_name': None, u'tenant': None} __init__ /usr/lib/python2.7/dist-packages/neutron/context.py:83

As i've been trying to deploy these, all network resources (network, subnet, router, etc.) can still be allocated and VM instance can be initialised and running. But i believe the missing of metadata makes it impossible to ssh to VM (i've tried several cloud images including CirrOS, Ubuntu, Fedora yet none worked). Another thing that might be worth noting is that lots of packages require removal and reinstallation, especially those relating to python on the Network node. This didn't occur when i tested on Virtualbox clusters or Amazon AWS.

I appreciate your help!

cheers,

Tuan Anh