Ask Your Question

Nova getting "Connection to neutron failed: Maximum attempts reached" on a HA setup.

asked 2015-02-12 12:11:21 -0500

jpmethot gravatar image

updated 2015-02-12 13:06:22 -0500


We have a HA setup with haproxy that has always worked. At one point, we got issues were rsyslog was slowing down the whole server as it couldn't send it's log properly to the log server. That issue was just fixed when I switched the logging protocol to UDP. I then set all services to use rsyslog again and rebooted the server. Strange thing now though, it has become impossible to login to the dashboard. Each attempt, nova shows the following in its logs :

Connection to neutron failed: Maximum attempts reached

I have looked into keystone, in case it would be linked to authentication, but that does not appear to be the problem as there is no authentication failed message or error of any kind.

So that brings me to neutron. By default, there's no error in the neutron logs whenever nova does its request. If I enable the neutron debug log, I get several of this request :

2015-02-12 12:22:41.727 4578 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystoneclient/middleware/

2015-02-12 12:22:41.727 4578 DEBUG keystoneclient.middleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystoneclient/middleware/

2015-02-12 12:22:41.728 4578 DEBUG keystoneclient.middleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/dist-packages/keystoneclient/middleware/

2015-02-12 12:22:41.729 4578 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: ef1e3eb21eda446ebf285926a4be9b82 with project_id : 7534bf835b7142beb20a25b72c2f789f and roles: admin  _build_user_headers /usr/lib/python2.7/dist-packages/keystoneclient/middleware/

2015-02-12 12:22:41.730 4578 DEBUG routes.middleware [-] No route matched for GET /ports.json __call__ /usr/lib/python2.7/dist-packages/routes/

2015-02-12 12:22:41.731 4578 DEBUG routes.middleware [-] Matched GET /ports.json __call__ /usr/lib/python2.7/dist-packages/routes/

2015-02-12 12:22:41.731 4578 DEBUG routes.middleware [-] Route path: '/ports{.format}', defaults: {'action': u'index', 'controller': <wsgify at 140519185214800 wrapping <function resource at 0x7fcd2c0b0140>>} __call__ /usr/lib/python2.7/dist-packages/routes/

2015-02-12 12:22:41.732 4578 DEBUG routes.middleware [-] Match dict: {'action': u'index', 'controller': <wsgify at 140519185214800 wrapping <function resource at 0x7fcd2c0b0140>>, 'format': u'json'} __call__ /usr/lib/python2.7/dist-packages/routes/

2015-02-12 12:22:41.741 4578 INFO neutron.wsgi [req-c5d1517b-cb3a-487d-b5c5-1c45550790dd None] - - [12/Feb/2015 12:22:41] "GET /v2.0/ports.json?tenant_id=c09c61a2ae8746ffb5a0d7a721fd57bb&device_id=673a1d39-81a8-4d2a-9c60-372aff88867c HTTP/1.1" 200 1650 0.014931

So it looks like that neutron IS answering nova. However nova does not receive that reply. I am at a loss as to the potential causes for this, especially since both are on the same server. Could it be that nova loses the tcp connection to the neutron port after it sent this request? I must ... (more)

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2015-02-13 07:55:28 -0500

jpmethot gravatar image

It looks like this issue is caused by rsyslog running on the second controller, even though it was disabled on controller1. I thought I had the rsyslog logging issue figured out, but it appears that the main issue with rsyslog has come back. There seems to be a misconfiguration somewhere, but that's really rsyslog specific.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools



Asked: 2015-02-12 12:11:21 -0500

Seen: 194 times

Last updated: Feb 13 '15