Ask Your Question
0

openstack HA: sometimes getting unable to establish connection error

asked 2016-01-18 12:34:59 -0500

Satyanarayana Patibandla gravatar image

updated 2016-01-18 12:36:48 -0500

In My 2 controller openstack HA setup ( using a galera cluster with two mysql nodes) sometimes I am getting the below errors when I try to create heat stack (which is used to create a instance with floating IP).

1) ConnectionRefused: Unable to establish connection to http://controller-vip:9696/v2.0/ports...

I verified neutron port 9696 on both controller node they are in listening state.

2) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://controller-vip:8004/v1/ca1d107...

I verified heat port 8004 on both controller node they are in listening state.

Every time I am not getting these errors. Only sometimes I am getting this error. When I looked at syslog I have observed below warnings. Not sure whether these are related to above error.

Warning] Aborted connection 102541 to db: 'keystone' user: 'keystone' host: 'controller01' (Unknown error)
[Warning] Aborted connection 103782 to db: 'nova' user: 'nova' host: 'controller01' (Unknown error)

Could you please help me in fixing this issue.

Thanks, Satya.P

edit retag flag offensive close merge delete

Comments

we need to debug this step by step... I guess the VIP that you have configured is on HAProxy... Enable the debugging logs of HA proxy and see what it has to say when the "Unable to establish connection happens"

Prateek K gravatar imagePrateek K ( 2016-01-19 00:43:53 -0500 )edit

I guess you have some timing problems here. Make sure that all database related timeouts (in OpenStack, on all Galera Nodes and in HAProxy) match. Since OpenStack uses DB pooling w/ open, idle connections per default, you need to inform HAProxy about the higher timeouts.

cheers, hauke

haukebruno gravatar imagehaukebruno ( 2016-02-28 02:22:31 -0500 )edit

Hi, I am facing the exact same problem. Any progress?

zekken gravatar imagezekken ( 2016-02-29 07:12:46 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
0

answered 2016-02-29 08:19:55 -0500

Satyanarayana Patibandla gravatar image

There was some issue with my mysql and keystone set up. I have reinstalled the entire set up then it started working fine.

edit flag offensive delete link more

Comments

Reinstalling the entire setup would be a huge task for me, and its not feasible. I am looking around for troubleshooting. Can you be little specific to the mysql/keystone issue? It would be greatly helpful. Thanks

zekken gravatar imagezekken ( 2016-03-01 06:04:34 -0500 )edit

Most OpenStack services use connection pooling, means that there are a lot of open and idle connections. If you use HAProxy you need to specify longer timeouts. Check idle_timeout in the [database] sections of your OpenStack services and tune everything (MySQL/MariaDB timeouts as well) right

haukebruno gravatar imagehaukebruno ( 2016-03-01 18:41:51 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2016-01-18 12:34:59 -0500

Seen: 1,021 times

Last updated: Feb 29 '16