Ask Your Question
0

Nova getting "Connection to neutron failed: Maximum attempts reached" on a HA setup.

asked 2015-02-12 12:11:21 -0600

jpmethot gravatar image

updated 2015-02-12 13:06:22 -0600

Greetings,

We have a HA setup with haproxy that has always worked. At one point, we got issues were rsyslog was slowing down the whole server as it couldn't send it's log properly to the log server. That issue was just fixed when I switched the logging protocol to UDP. I then set all services to use rsyslog again and rebooted the server. Strange thing now though, it has become impossible to login to the dashboard. Each attempt, nova shows the following in its logs :

Connection to neutron failed: Maximum attempts reached

I have looked into keystone, in case it would be linked to authentication, but that does not appear to be the problem as there is no authentication failed message or error of any kind.

So that brings me to neutron. By default, there's no error in the neutron logs whenever nova does its request. If I enable the neutron debug log, I get several of this request :

2015-02-12 12:22:41.727 4578 DEBUG keystoneclient.middleware.auth_token [-] Authenticating user token __call__ /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:569

2015-02-12 12:22:41.727 4578 DEBUG keystoneclient.middleware.auth_token [-] Removing headers from request environment: X-Identity-Status,X-Domain-Id,X-Domain-Name,X-Project-Id,X-Project-Name,X-Project-Domain-Id,X-Project-Domain-Name,X-User-Id,X-User-Name,X-User-Domain-Id,X-User-Domain-Name,X-Roles,X-Service-Catalog,X-User,X-Tenant-Id,X-Tenant-Name,X-Tenant,X-Role _remove_auth_headers /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:628

2015-02-12 12:22:41.728 4578 DEBUG keystoneclient.middleware.auth_token [-] Returning cached token _cache_get /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:1034

2015-02-12 12:22:41.729 4578 DEBUG keystoneclient.middleware.auth_token [-] Received request from user: ef1e3eb21eda446ebf285926a4be9b82 with project_id : 7534bf835b7142beb20a25b72c2f789f and roles: admin  _build_user_headers /usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py:933

2015-02-12 12:22:41.730 4578 DEBUG routes.middleware [-] No route matched for GET /ports.json __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:97

2015-02-12 12:22:41.731 4578 DEBUG routes.middleware [-] Matched GET /ports.json __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:100

2015-02-12 12:22:41.731 4578 DEBUG routes.middleware [-] Route path: '/ports{.format}', defaults: {'action': u'index', 'controller': <wsgify at 140519185214800 wrapping <function resource at 0x7fcd2c0b0140>>} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:102

2015-02-12 12:22:41.732 4578 DEBUG routes.middleware [-] Match dict: {'action': u'index', 'controller': <wsgify at 140519185214800 wrapping <function resource at 0x7fcd2c0b0140>>, 'format': u'json'} __call__ /usr/lib/python2.7/dist-packages/routes/middleware.py:103

2015-02-12 12:22:41.741 4578 INFO neutron.wsgi [req-c5d1517b-cb3a-487d-b5c5-1c45550790dd None] 10.251.0.11 - - [12/Feb/2015 12:22:41] "GET /v2.0/ports.json?tenant_id=c09c61a2ae8746ffb5a0d7a721fd57bb&device_id=673a1d39-81a8-4d2a-9c60-372aff88867c HTTP/1.1" 200 1650 0.014931

So it looks like that neutron IS answering nova. However nova does not receive that reply. I am at a loss as to the potential causes for this, especially since both are on the same server. Could it be that nova loses the tcp connection to the neutron port after it sent this request? I must ... (more)

edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2015-02-13 07:55:28 -0600

jpmethot gravatar image

It looks like this issue is caused by rsyslog running on the second controller, even though it was disabled on controller1. I thought I had the rsyslog logging issue figured out, but it appears that the main issue with rsyslog has come back. There seems to be a misconfiguration somewhere, but that's really rsyslog specific.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2015-02-12 12:11:21 -0600

Seen: 187 times

Last updated: Feb 13 '15