Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

Connection to Neutron failed: HTTPConnectionPool

I have installed Openstack on a 3 node setup(Controller & Neutron on VMs and Compute is a bare-metal). I was able to create instances. But then i tried a "systemctl restart nova-compute" on my Compute node. That would never bring up my nova-compute service, even though the libvirtd was running properly. Hence, i went ahead and started the nova-compute process manually(without any conf params. I am not aware if any of it is required). Now i am unable to launch instances. Now i see a connection to neutron failed message in the /var/log/nova/nova-compute.log.

> 2015-10-01 10:32:28.550 26821 ERROR nova.compute.manager [-] [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4] An error occurred while refreshing the network cache.
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4] Traceback (most recent call last):
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 5348, in _heal_instance_info_cache
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     self._get_instance_nw_info(context, instance, use_slave=True)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1234, in _get_instance_nw_info
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     instance)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 598, in get_instance_nw_info
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     port_ids)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 612, in _get_instance_nw_info
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     port_ids)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/api.py", line 1356, in _build_network_info_model
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     data = client.list_ports(**search_opts)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     ret = obj(*args, **kwargs)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 98, in with_params
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     ret = self.function(instance, *args, **kwargs)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 312, in list_ports
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     **_params)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     ret = obj(*args, **kwargs)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1334, in list
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     for r in self._pagination(collection, path, **params):
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1347, in _pagination
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     res = self.get(path, params=params)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     ret = obj(*args, **kwargs)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1320, in get
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     headers=headers, params=params)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     ret = obj(*args, **kwargs)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1297, in retry_request
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     headers=headers, params=params)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/nova/network/neutronv2/__init__.py", line 84, in wrapper
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     ret = obj(*args, **kwargs)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/v2_0/client.py", line 1232, in do_request
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     self.httpclient.authenticate_and_fetch_endpoint_url()
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 150, in authenticate_and_fetch_endpoint_url
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     self.authenticate()
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 247, in authenticate
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     self._authenticate_keystone()
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 226, in _authenticate_keystone
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     allow_redirects=True)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]   File "/usr/lib/python2.7/site-packages/neutronclient/client.py", line 133, in _cs_request
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4]     raise exceptions.ConnectionFailed(reason=e)
2015-10-01 10:32:28.550 26821 TRACE nova.compute.manager [instance: f19610ae-ea77-45ea-aac2-faebeaf380d4] ConnectionFailed: Connection to neutron failed: HTTPConnectionPool(host='172.19.1.118', port=8080): Read timed out. (read timeout=30)

I have checked a lot of similar posts and they suggest to restart the neutron-server service on the controller node. I have even done that and there is also no conncetivity problem in reaching the Rabbit-mq service. Also the process is listening on port 9696. FYI,

/var/log/neutron/server.log

>2015-10-01 10:30:42.576 19764 INFO keystonemiddleware.auth_token [-] Using /tmp/keystone-signing-vunJVG as cache directory for signing certificate
>2015-10-01 10:30:42.588 19764 INFO neutron.service [-] Neutron service started, listening on 0.0.0.0:9696
2015-10-01 10:30:42.589 19764 INFO oslo.messaging._drivers.impl_rabbit [-] Connecting to AMQP server on controller:5672
>2015-10-01 10:30:42.598 19764 INFO neutron.wsgi [-] (19764) wsgi starting up on http://0.0.0.0:9696/

[root@controller ~]# netstat -anp | grep 9696 tcp 0 0 0.0.0.0:9696 0.0.0.0:* LISTEN 19764/python

Can anyone suggest on how to troubleshoot it?