Ask Your Question
0

openstack commands are failing with connection abort error

asked 2019-05-09 14:43:54 -0500

arslankhan gravatar image

Hello I am trying to run openstack command from outside of openstack cluster the commands were working before but now they are failing with following error ConnectFailure: Unable to establish connection to http://{HORIZON_IP}:8774/v2.1/servers/detail: ('Connection aborted.', error(104, 'Connection reset by peer'))

i am running following command openstack --os-auth-url=http://{HORIZON_IP}:5000/v2.0 --os-project-name={PROJ_NAME} --os-username={USERNAME} --os-password={PWD} server list

The same commands is working if i run it from one of the controller nodes

in httpd/gnocchi_wsgi_error.log i have following error message ConnectionError: ('Connection aborted.', ResponseNotReady())

Any idea what could be wrong Thanks in advance

edit retag flag offensive close merge delete

Comments

Are you sure that Keystone v2 is correct, and that you are using v2 when running the command on the controller? v2 has been deprecated for a while.

I would also check the values of environment variables OS_xxxx. They might be different on the controller than the remote system.

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-09 15:21:43 -0500 )edit

The Gnocchi error is unlikely to be related to the server list command.

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-09 15:23:34 -0500 )edit

the same command was working before. Also i am providing values in the command on both machines

arslankhan gravatar imagearslankhan ( 2019-05-09 15:48:00 -0500 )edit

Which OpenStack version is that, and how was the cloud deployed?

I would check connectivity to Nova (curl http://HORIZON-IP:8774) and also check the nova-api log for relevant messages.

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-09 19:02:19 -0500 )edit

curl command succeeds. nova-api log only shows that requests are succeeding nova-conductor contains following message WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 5.72 sec

arslankhan gravatar imagearslankhan ( 2019-05-10 10:17:56 -0500 )edit

here is the message in debug logs that i get on remote GET call to http://192.168.104.15:5000/v2.0 used request id req-bb95bd6e-bfce-4781-bd84-a887ac9e2d05 Making authentication request to http://192.168.104.15:5000/v2.0/tokens Unable to establish connection to http://192.168.104.15:5000/v2.0/tokens

arslankhan gravatar imagearslankhan ( 2019-05-10 13:59:42 -0500 )edit

Which OpenStack release is this, and how was the cloud deployed?

The error message means that Keystone’s version 2 API is not reachable. Use curl http://HORIZON-IP:5000 to check which version is deployed. Try v3 instead of v2.0.

Also check that port 5000 is not blocked on server HORIZON-IP.

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-10 18:25:30 -0500 )edit

v3 is also deployed but not able to use it. 5000 isn't blocked. also getting following error now ConnectFailure: Unable to establish connection to http://192.168.104.15:8774/v2.1/servers/detail (http://192.168.104.15:8774/v2.1/serve...): ('Connection aborted.', error(104, 'Connection reset by peer')) telnet 192.168.104.15 8774 is working

arslankhan gravatar imagearslankhan ( 2019-05-11 05:41:26 -0500 )edit

There are several moving parts. On recent versions of Openstack, both Keystone and Nova API processes run as WSGI applications behind a webserver. If that is the case in your cloud, check the webserver, Keystone and Nova logs. If not, just the Keystone and Nova logs.

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-11 06:28:33 -0500 )edit

i am able to execute commands from controller nodes. Doesn't that mean that keystone is working?

arslankhan gravatar imagearslankhan ( 2019-05-13 11:24:25 -0500 )edit

That’s correct. I doubted that you were successful with Keystone API v2. If you are, the problem should be in the network infrastructure where your controller is located, or the security configuration. Still, why don’t you check those logs?

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-13 19:21:09 -0500 )edit

Do you have problems accessing other controller ports? Could it be that another server has the same IP address as the controller?

Bernd Bausch gravatar imageBernd Bausch ( 2019-05-13 19:21:52 -0500 )edit

I could not find anything in keystone logs. Some actions for galera, redis and rabiitmq are getting timed out. following error is present in multiple log files under nova InvalidRequestError: Can't reconnect until invalid transaction is rolled back.

arslankhan gravatar imagearslankhan ( 2019-05-14 04:13:46 -0500 )edit

Sometime openstack command fails to connect with http://192.168.104.15:5000/v2.0/tokens. But sometimes it connectes with token but then fails when it tries to connect with http://192.168.104.15:8774/v2.1/servers/detail (http://192.168.104.15:8774/v2.1/serve...) dashboard is accessible through browser on same remote machine

arslankhan gravatar imagearslankhan ( 2019-05-14 04:17:55 -0500 )edit

controller ip address is unique

arslankhan gravatar imagearslankhan ( 2019-05-14 04:44:15 -0500 )edit

1 answer

Sort by » oldest newest most voted
0

answered 2019-05-15 11:11:08 -0500

arslankhan gravatar image

The problem was not with openstack but with remote machine. i had 4 remote machines and it appears problem is with all of them. running the commands from docker container seems to be working. Thanks @Bernd Baush for helping me out

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2019-05-09 14:43:54 -0500

Seen: 41 times

Last updated: May 15