openstack api fail 3-5 time out of 10
we have deployed openstack juno release on Ubuntu 14.04.3 LTS. we have three controller nodes with rabbitmq and galera installed on the controller node as well. we have setup two separate haproxy nodes and the Galera is behind the vip, as well as other openstack endpoints but not rabbitmq.
when i run keystone token-get command, about 3 to 5 times it fails out of 10 times. below is the output. i am currently confused that should put galera behind a vip or not. i am not sure where to start looking into this issue, I think its a mariadb issue but not sure. any help would be great.
Thanks.
for i in $(seq 1 10); do keystone token-get;done
Authorization Failed: An unexpected error prevented the server from fulfilling your request. (HTTP 500)
Authorization Failed: An unexpected error prevented the server from fulfilling your request. (HTTP 500)
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-10-03T22:26:06Z |
| id | fb9c8032b27e434fa3d6a4646ef8e972 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
| user_id | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-10-03T22:26:07Z |
| id | ef65d53914c24144b91e1fc573ee2e85 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
| user_id | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-10-03T22:26:07Z |
| id | 60c6d22805a749e2a96d4cc2193ec0f0 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
| user_id | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-10-03T22:26:07Z |
| id | cabc4d73bb6f4d9894c2bcc77ba3a2a1 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
| user_id | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-10-03T22:26:08Z |
| id | 0bccd20c9e3c4421bdc150c156811aee |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
| user_id | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
Authorization Failed: An unexpected error prevented the server from fulfilling your request. (HTTP 500)
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-10-03T22:26:09Z |
| id | f2c928afe7544d6392ac17c32d53dea1 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
| user_id | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-10-03T22:26:09Z |
| id | 19604e7048ae4cb68bf30de032c9fc7c |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
| user_id | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
some reason i couldnt had command output to comments so im putting here. this is the output of keystone --debug token-get
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://172.16.100.100:35357/v2.0/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.16.100.100
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 500 143
DEBUG:keystoneclient.session:Request returned failure status: 500
this is good one
DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://172.16.100.100:35357/v2.0/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.16.100.100
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 1471
+-----------+----------------------------------+
| Property | Value |
+-----------+----------------------------------+
| expires | 2015-10-05T16:37:06Z |
| id | 305f67047d1e4a27a4a7e945eb346da5 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
| user_id | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
I went to check the keystone logs and seems like keystone is disconnecting in the backend with the galera cluster. can can anyone help with this issue ?
2015-10-05 15:37:05.680 12364 ERROR keystone.common.wsgi [-] (OperationalError) (2006, 'MySQL server has gone away') 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user \nWHERE user.name = %s AND user.domain_id = %s' ('admin', 'default')
2015-10-05 15:37:05.680 12364 TRACE keystone.common.wsgi self.errorhandler(self, exc, value)
2015-10-05 15:37:05.680 12364 ...
Try to run
keystone --debug token-getand take a look at debug log. You may also paste debug log here.some reason i couldnt add code to the comment so check above, as you can see it doesnt give out much info.
Take a look at
/var/log/mysql/error.log. If this error is related to mysql, there should be errors in this log.i couldnt find any log files for mysql , i think its mainly due the configuration issue.