Ask Your Question
0

openstack api fail 3-5 time out of 10

asked 2015-10-03 16:48:13 -0500

Bindas gravatar image

updated 2015-10-05 10:53:03 -0500

we have deployed openstack juno release on Ubuntu 14.04.3 LTS. we have three controller nodes with rabbitmq and galera installed on the controller node as well. we have setup two separate haproxy nodes and the Galera is behind the vip, as well as other openstack endpoints but not rabbitmq.

when i run keystone token-get command, about 3 to 5 times it fails out of 10 times. below is the output. i am currently confused that should put galera behind a vip or not. i am not sure where to start looking into this issue, I think its a mariadb issue but not sure. any help would be great.

Thanks.

    for i in $(seq 1 10); do keystone token-get;done
Authorization Failed: An unexpected error prevented the server from fulfilling your request. (HTTP 500)
Authorization Failed: An unexpected error prevented the server from fulfilling your request. (HTTP 500)
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2015-10-03T22:26:06Z       |
|     id    | fb9c8032b27e434fa3d6a4646ef8e972 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
|  user_id  | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2015-10-03T22:26:07Z       |
|     id    | ef65d53914c24144b91e1fc573ee2e85 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
|  user_id  | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2015-10-03T22:26:07Z       |
|     id    | 60c6d22805a749e2a96d4cc2193ec0f0 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
|  user_id  | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2015-10-03T22:26:07Z       |
|     id    | cabc4d73bb6f4d9894c2bcc77ba3a2a1 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
|  user_id  | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2015-10-03T22:26:08Z       |
|     id    | 0bccd20c9e3c4421bdc150c156811aee |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
|  user_id  | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
Authorization Failed: An unexpected error prevented the server from fulfilling your request. (HTTP 500)
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2015-10-03T22:26:09Z       |
|     id    | f2c928afe7544d6392ac17c32d53dea1 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
|  user_id  | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2015-10-03T22:26:09Z       |
|     id    | 19604e7048ae4cb68bf30de032c9fc7c |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
|  user_id  | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+

some reason i couldnt had command output to comments so im putting here. this is the output of keystone --debug token-get

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://172.16.100.100:35357/v2.0/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.16.100.100
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 500 143
DEBUG:keystoneclient.session:Request returned failure status: 500

this is good one

DEBUG:keystoneclient.auth.identity.v2:Making authentication request to http://172.16.100.100:35357/v2.0/tokens
INFO:urllib3.connectionpool:Starting new HTTP connection (1): 172.16.100.100
DEBUG:urllib3.connectionpool:Setting read timeout to 600.0
DEBUG:urllib3.connectionpool:"POST /v2.0/tokens HTTP/1.1" 200 1471
+-----------+----------------------------------+
|  Property |              Value               |
+-----------+----------------------------------+
|  expires  |       2015-10-05T16:37:06Z       |
|     id    | 305f67047d1e4a27a4a7e945eb346da5 |
| tenant_id | cc5c94dbdf684551ba10b0b5c22c1908 |
|  user_id  | a60b3894adf54394a0e0bc72228789bc |
+-----------+----------------------------------+

I went to check the keystone logs and seems like keystone is disconnecting in the backend with the galera cluster. can can anyone help with this issue ?

2015-10-05 15:37:05.680 12364 ERROR keystone.common.wsgi [-] (OperationalError) (2006, 'MySQL server has gone away') 'SELECT user.id AS user_id, user.name AS user_name, user.domain_id AS user_domain_id, user.password AS user_password, user.enabled AS user_enabled, user.extra AS user_extra, user.default_project_id AS user_default_project_id \nFROM user \nWHERE user.name = %s AND user.domain_id = %s' ('admin', 'default')
2015-10-05 15:37:05.680 12364 TRACE keystone.common.wsgi     self.errorhandler(self, exc, value)
2015-10-05 15:37:05.680 12364 ...
(more)
edit retag flag offensive close merge delete

Comments

Try to run keystone --debug token-get and take a look at debug log. You may also paste debug log here.

Cheng Liu gravatar imageCheng Liu ( 2015-10-04 03:28:59 -0500 )edit

some reason i couldnt add code to the comment so check above, as you can see it doesnt give out much info.

Bindas gravatar imageBindas ( 2015-10-05 10:45:19 -0500 )edit

Take a look at /var/log/mysql/error.log. If this error is related to mysql, there should be errors in this log.

Cheng Liu gravatar imageCheng Liu ( 2015-10-06 06:13:36 -0500 )edit

i couldnt find any log files for mysql , i think its mainly due the configuration issue.

Bindas gravatar imageBindas ( 2015-10-06 14:03:06 -0500 )edit

1 answer

Sort by ยป oldest newest most voted
1

answered 2015-10-06 14:03:55 -0500

Bindas gravatar image

updated 2015-10-06 14:04:22 -0500

I was able to find the root cause for this issue, i just basically double my timeout setting for the cluster in my.cnf file and it now its not giving me http 500 error anymore.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2015-10-03 16:48:13 -0500

Seen: 178 times

Last updated: Oct 06 '15