I am running curl command on the keystone container itself and the result is coming after 40s . If I try with any other user it shows gateway time out . I have deployed openstack using openstack ansible ( tried newton and ocata ) following their reference architecture for production . It worked fine for some time and then the gateway time out syndrome started coming out intermittently . After a restart of the controller all commands are giving gateway time out. Suspecting HAproxy i have increased time out value and finally ran the API inside the keystone container itself . Any suggestion is appreciated .

In keystone container’s apache-error.log has this error , all other logs are normal

2017-05-17 15:18:37.910591 Timeout when reading response headers from daemon process 'keystone-admin': /var/www/cgi-bin/keystone/admin
2017-05-17 15:19:07.406267 mod_wsgi (pid=3139): Exception occurred processing WSGI script '/var/www/cgi-bin/keystone/admin'.
2017-05-17 15:19:07.406433 IOError: failed to write data

Thanks.

Warm Regards, Bhujay

I tried this command .

root@infra1-keystone-container-2d53b3ef:/var/log/keystone# while :; do curl -s -w "%{http_code} %{time_total}" -i -X POST -H "Content-Type: application/json" -d '{"auth":{ "identity":{"methods":["password"],"password"
> :{"user":{"name":"admin","domain":{"id": "default" },"password": "welcome@123"}}}}}' http://localhost:5000/v3/auth/tokens -o /dev/null; echo; sleep 1; done
201 41.630
201 40.248
201 40.252
201 40.906

Keystone.conf

[DEFAULT]
# Disable stderr logging
use_stderr = False
debug = False
admin_endpoint = http://10.0.0.14:35357
fatal_deprecations = False
member_role_name = _member_
max_token_size = 255


log_file = keystone.log
log_dir = /var/log/keystone

## RabbitMQ RPC
rpc_backend = rabbit
transport_url = rabbit://keystone:49ff40a5f4d312009f166acb5b5@10.0.0.105:5671,keystone:49ff40a5f4d312009f166acb5b5@10.0.0.194:5671,keystone:49ff40a5f4d312009f166acb5b5@10.0.0.253:5671//keystone
[oslo_messaging_rabbit]
rabbit_use_ssl = True

[oslo_messaging_notifications]
driver = messagingv2
transport_url = rabbit://keystone:49ff40a5f4d312009f166acb5b5@10.0.0.105:5671,keystone:49ff40a5f4d312009f166acb5b5@10.0.0.194:5671,keystone:49ff40a5f4d312009f166acb5b5@10.0.0.253:5671//keystone

[cache]
backend = dogpile.cache.memcached
backend_argument = url:10.0.0.120,10.0.0.127,10.0.0.187:11211
config_prefix = cache.keystone
distributed_lock = True
enabled = true


[revoke]
caching = true
driver = sql
expiration_buffer = 1800
cache_time = 3600


[auth]
methods = password,token


[database]
connection = mysql+pymysql://keystone:f016431489d4728b@10.0.0.14/keystone?charset=utf8
idle_timeout = 200
min_pool_size = 5
max_pool_size = 120
pool_timeout = 30


[fernet_tokens]
key_repository = /etc/keystone/fernet-keys
max_active_keys = 7


[identity]
driver = sql


[assignment]
driver = sql


[resource]
cache_time = 3600
caching = true
driver = sql


[token]
enforce_token_bind = permissive
expiration = 43200
caching = true
cache_time = 3600
provider = fernet


[eventlet_server]
admin_bind_host = 0.0.0.0
admin_port = 35357
public_port = 5000

[credential]
key_repository = /etc/keystone/credential-keys