Ask Your Question

arslankhan's profile - activity

2019-05-16 05:51:15 -0500 received badge  Notable Question (source)
2019-05-15 16:55:04 -0500 answered a question Openstack instance rebuild is failing

The problem was with rabbitmq cluster. they were not able to communicate. Had to kill the rabbit mq instances Instead of waiting for pacemaker to restart rabbitmq i restarted the rabbitmq service which caused rabbitmq cluster to go in stopped statue To get out of it i reset the failcount for rabbitmq and then put the node in standby and reboot the nodes one by one. after reboot cluster was started again on all 3 nodes and services were back to normal

Thanks Bernad for pointing me in right direction

2019-05-15 16:55:04 -0500 answered a question openstack commands are failing with connection abort error

The problem was not with openstack but with remote machine. i had 4 remote machines and it appears problem is with all of them. running the commands from docker container seems to be working. Thanks @Bernd Baush for helping me out

2019-05-15 06:41:39 -0500 commented question openstack commands are failing with connection abort error

There is something wrong with the remote machine. I have created a docker container on the same machine and invoked open stack command from inside that docker its working fine. Most probably i have some version mismatch in my host machine

2019-05-14 04:57:06 -0500 commented question openstack commands are failing with connection abort error

controller ip address is unique

2019-05-14 04:57:06 -0500 commented question openstack commands are failing with connection abort error

Sometime openstack command fails to connect with http://192.168.104.15:5000/v2.0/tokens. But sometimes it connectes with token but then fails when it tries to connect with http://192.168.104.15:8774/v2.1/servers/detail (http://192.168.104.15:8774/v2.1/serve...) dashboard is accessible through browser on same remote machine

2019-05-14 04:57:06 -0500 commented question openstack commands are failing with connection abort error

I could not find anything in keystone logs. Some actions for galera, redis and rabiitmq are getting timed out. following error is present in multiple log files under nova InvalidRequestError: Can't reconnect until invalid transaction is rolled back.

2019-05-14 04:44:15 -0500 received badge  Commentator
2019-05-14 02:44:01 -0500 received badge  Enthusiast
2019-05-13 16:59:28 -0500 commented question openstack commands are failing with connection abort error

i am able to execute commands from controller nodes. Doesn't that mean that keystone is working?

2019-05-11 06:25:00 -0500 received badge  Popular Question (source)
2019-05-11 06:24:42 -0500 commented question openstack commands are failing with connection abort error

v3 is also deployed but not able to use it. 5000 isn't blocked. also getting following error now ConnectFailure: Unable to establish connection to http://192.168.104.15:8774/v2.1/servers/detail (http://192.168.104.15:8774/v2.1/serve...): ('Connection aborted.', error(104, 'Connection reset by peer')) telnet 192.168.104.15 8774 is working

2019-05-10 18:15:53 -0500 commented question openstack commands are failing with connection abort error

here is the message in debug logs that i get on remote GET call to http://192.168.104.15:5000/v2.0 used request id req-bb95bd6e-bfce-4781-bd84-a887ac9e2d05 Making authentication request to http://192.168.104.15:5000/v2.0/tokens Unable to establish connection to http://192.168.104.15:5000/v2.0/tokens

2019-05-10 18:15:52 -0500 commented question openstack commands are failing with connection abort error

curl command succeeds. nova-api log only shows that requests are succeeding nova-conductor contains following message WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 5.72 sec

2019-05-09 18:59:05 -0500 commented question openstack commands are failing with connection abort error

the same command was working before. Also i am providing values in the command on both machines

2019-05-09 15:09:16 -0500 asked a question openstack commands are failing with connection abort error

Hello I am trying to run openstack command from outside of openstack cluster the commands were working before but now they are failing with following error ConnectFailure: Unable to establish connection to http://{HORIZON_IP}:8774/v2.1/servers/detail: ('Connection aborted.', error(104, 'Connection reset by peer'))

i am running following command openstack --os-auth-url=http://{HORIZON_IP}:5000/v2.0 --os-project-name={PROJ_NAME} --os-username={USERNAME} --os-password={PWD} server list

The same commands is working if i run it from one of the controller nodes

in httpd/gnocchi_wsgi_error.log i have following error message ConnectionError: ('Connection aborted.', ResponseNotReady())

Any idea what could be wrong Thanks in advance

2019-04-29 04:00:08 -0500 received badge  Famous Question (source)
2019-04-25 02:55:43 -0500 received badge  Notable Question (source)
2019-04-23 06:09:43 -0500 commented question Openstack instance rebuild is failing

updated the description with different errors found in nova logs

2019-04-23 05:04:07 -0500 received badge  Popular Question (source)
2019-04-23 02:08:56 -0500 commented question Openstack instance rebuild is failing

Hello changes were made thanks

2019-04-22 06:16:57 -0500 received badge  Editor (source)
2019-04-21 16:53:59 -0500 asked a question Openstack instance rebuild is failing

Hello
Suddenly openstack is started throwing Unknown Error (HTTP 504) error on VM Rebuild. Command for rebuilding: openstack --os-auth-url=http://{URL}:5000/v2.0 --os-project-name=proj_qa --os-username={USER} --os-password=* server rebuild --image SA-3.0-27.qcow2 qa-sa-quintet-sa --debug --wait

I tried to reboot instance using nova reboot --hard qa-sa-quintet-sa it throws following error

Unknown Error (HTTP 504) ERROR (CommandError): Unable to reboot the specified server(s).

and instance is now stuck in Hard Rebooting state.

nova reset-state --active qa-sa-quintet-sa also throws following error Reset state for server qa-sa-quintet-sa failed: Unknown Error (HTTP 504) ERROR (CommandError): Unable to reset the state for the specified server(s).

nova --version 7.1.2 openstack --version openstack 3.8.1

sudo pcs status shows following errors Cluster name: tripleo_cluster

Failed Actions: * galera_monitor_10000 on controller-56 'unknown error' (1): call=444, status=Timed Out, exitreason='none', last-rc-change='Sun Apr 14 19:02:39 2019', queued=0ms, exec=0ms * rabbitmq_monitor_10000 on controller-56 'unknown error' (1): call=389, status=Timed Out, exitreason='none', last-rc-change='Sat Apr 13 19:03:00 2019', queued=0ms, exec=0ms * galera_monitor_10000 on controller--55 'unknown error' (1): call=219, status=Timed Out, exitreason='none', last-rc-change='Sun Apr 21 18:50:56 2019', queued=0ms, exec=0ms * rabbitmq_monitor_10000 on controller-55 'unknown error' (1): call=147, status=Timed Out, exitreason='none', last-rc-change='Sun Apr 21 18:50:57 2019', queued=0ms, exec=0ms * redis_monitor_60000 on controller-55 'unknown error' (1): call=172, status=Timed Out, exitreason='none', last-rc-change='Fri Apr 19 18:51:18 2019', queued=0ms, exec=0ms * galera_monitor_10000 on controller-57 'unknown error' (1): call=398, status=Timed Out, exitreason='none', last-rc-change='Sat Apr 20 15:23:35 2019', queued=0ms, exec=0ms * rabbitmq_monitor_10000 on controller-57 'unknown error' (1): call=281, status=Timed Out, exitreason='none', last-rc-change='Sat Apr 20 15:23:36 2019', queued=0ms, exec=0ms * redis_monitor_45000 on controller-57 'unknown error' (1): call=374, status=Timed Out, exitreason='none', last-rc-change='Fri Apr 19 15:22:38 2019', queued=0ms, exec=0ms

Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled

nova-conductor.log ERROR oslo_messaging.notify.messaging [req-df784075-1c86-4a48-8350-52a75b89beb3 e11e6766c40a4f1fb8dac4557bace3c2 290fad30002e46b6ad3b7fcee37645e2 - - -] Could not send notification to versioned_notifications

nova-novncproxy.log, nova-api.log and nova-conductor.log contains following error AMQP server on controller-57.internalapi.localdomain:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 1 seconds. Client port: None

nova-consoleauth.log ERROR nova.servicegroup.drivers.db DBError: Can't reconnect until invalid transaction is rolled back

nova-rowsflush.log IOError: [Errno 13] Permission denied: '/var/log/nova/nova-manage.log'

Thanks in advance