how keystone using gunicorn

asked 2019-02-04 22:23:46 -0600

binooetomo gravatar image

updated 2019-02-05 18:21:12 -0600

Dear All. I try to run keystone by gunicorn and nginx but first, I run it without nginx .. just for a test.

Here is my results : Run keystone in debug

[root@ceph-node1 ostackwsgi]# gunicorn --bind --log-level DEBUG kspublic:application
[2019-02-05 16:54:32 +0000] [875334] [DEBUG] Current configuration:
  proxy_protocol: False
  worker_connections: 1000
  statsd_host: None
  max_requests_jitter: 0
  post_fork: <function post_fork at 0x7f3d52c24230>
  errorlog: -
  enable_stdio_inheritance: False
  worker_class: sync
  ssl_version: 2
  suppress_ragged_eofs: True
  syslog: False
  syslog_facility: user
  when_ready: <function when_ready at 0x7f3d52c1bed8>
  pre_fork: <function pre_fork at 0x7f3d52c240c8>
  cert_reqs: 0
  preload_app: False
  keepalive: 2
  accesslog: None
  group: 0
  graceful_timeout: 30
  do_handshake_on_connect: False
  spew: False
  workers: 1
  proc_name: None
  sendfile: None
  pidfile: None
  umask: 0
  on_reload: <function on_reload at 0x7f3d52c1bd70>
  pre_exec: <function pre_exec at 0x7f3d52c247d0>
  worker_tmp_dir: None
  limit_request_fields: 100
  pythonpath: None
  on_exit: <function on_exit at 0x7f3d52c28050>
  config: None
  logconfig: None
  check_config: False
  secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
  reload_engine: auto
  proxy_allow_ips: ['']
  pre_request: <function pre_request at 0x7f3d52c24938>
  post_request: <function post_request at 0x7f3d52c24a28>
  forwarded_allow_ips: ['']
  worker_int: <function worker_int at 0x7f3d52c24500>
  raw_paste_global_conf: []
  threads: 1
  max_requests: 0
  chdir: /opt/ostackwsgi
  daemon: False
  user: 0
  limit_request_line: 4094
  access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
  certfile: None
  on_starting: <function on_starting at 0x7f3d52c1bc08>
  post_worker_init: <function post_worker_init at 0x7f3d52c24398>
  child_exit: <function child_exit at 0x7f3d52c24b90>
  worker_exit: <function worker_exit at 0x7f3d52c24cf8>
  paste: None
  default_proc_name: kspublic:application
  syslog_addr: udp://localhost:514
  syslog_prefix: None
  ciphers: TLSv1
  worker_abort: <function worker_abort at 0x7f3d52c24668>
  loglevel: DEBUG
  bind: ['']
  raw_env: []
  initgroups: False
  capture_output: False
  reload: False
  limit_request_field_size: 8190
  nworkers_changed: <function nworkers_changed at 0x7f3d52c24e60>
  timeout: 30
  keyfile: None
  ca_certs: None
  tmp_upload_dir: None
  backlog: 2048
  logger_class: gunicorn.glogging.Logger
[2019-02-05 16:54:32 +0000] [875334] [INFO] Starting gunicorn 19.7.1
[2019-02-05 16:54:32 +0000] [875334] [DEBUG] Arbiter booted
[2019-02-05 16:54:32 +0000] [875334] [INFO] Listening at: (875334)
[2019-02-05 16:54:32 +0000] [875334] [INFO] Using worker: sync
[2019-02-05 16:54:32 +0000] [875343] [INFO] Booting worker with pid: 875343
[2019-02-05 16:54:32 +0000] [875334] [DEBUG] 1 workers
[2019-02-05 16:55:20 +0000] [875343] [DEBUG] POST /v3/auth/tokens
[2019-02-05 16:56:24 +0000] [875334] [CRITICAL] WORKER TIMEOUT (pid:875975)
[2019-02-05 16:56:25 +0000] [876227] [INFO] Booting worker with pid: 876227
[2019-02-05 16:56:26 +0000] [876227] [DEBUG] GET /v3
[2019-02-05 16:56:26 +0000] [876227] [DEBUG] POST /v3/auth/tokens
[2019-02-05 16:56:57 +0000] [875334] [CRITICAL] WORKER TIMEOUT (pid:876227)
[2019-02-05 16:56:58 +0000] [876505] [INFO] Booting worker with pid: 876505
[2019-02-05 16:56:59 +0000] [876505] [DEBUG] POST /v3/auth/tokens
[2019-02-05 16:57:30 +0000] [875334] [CRITICAL] WORKER TIMEOUT (pid:876505)
[2019-02-05 16:57:31 +0000] [876772] [INFO] Booting worker with pid: 876772
[2019-02-05 16:57:33 +0000] [876772] [DEBUG] POST /v3/auth/tokens

here is I run openstack cli in verbose :

[root@ceph-node1 ~]# openstack user list -vv -d
START with options: [u'user', u'list', u'-vv ...
edit retag flag offensive close merge delete


can you reformat the question so that it becomes readable?

Bernd Bausch gravatar imageBernd Bausch ( 2019-02-05 00:21:05 -0600 )edit

I try to 'edit'

But it did't saved

I'll try once more

binooetomo gravatar imagebinooetomo ( 2019-02-05 04:16:54 -0600 )edit


Kindly please check if te edit works

binooetomo gravatar imagebinooetomo ( 2019-02-05 04:19:56 -0600 )edit

Thanks, it's readable now.

The connection between Keystone and Gunicorn is not working. The information you posted doesn't show how the connection is configured.

Add the --debug option to the openstack command, and/or try curl to see what you get from the web server.

Bernd Bausch gravatar imageBernd Bausch ( 2019-02-05 05:37:22 -0600 )edit

I try to re edit my post, but got red pop up box with text 'null value in column "ip_addr" violatesnot-null constraint ...'

binooetomo gravatar imagebinooetomo ( 2019-02-05 18:34:32 -0600 )edit

but with curl, I got :

{"version": {"status": "stable", "updated": "2018-10-15T00:00:00Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v3+json"}], "id": "v3.11", "links": [{"href": "", "rel": "self"}]}}
binooetomo gravatar imagebinooetomo ( 2019-02-05 18:36:40 -0600 )edit

So there is a working connection between gunicorn and keystone.

Was this a glitch or a problem during start-up? Can you run openstack user list now? What do you get from openstack --debug user list?

Bernd Bausch gravatar imageBernd Bausch ( 2019-02-05 20:46:45 -0600 )edit

Sir. First step I changed value of wgi.debug_middleware on keystone.conf to True. Found that openstack cli will first send 'get' request to keystone followed by a 'post' request. The first request is running well. The second request is received by keystone but keystone never send response. (contin)

binooetomo gravatar imagebinooetomo ( 2019-02-06 18:45:00 -0600 )edit

(continue). Second step, I set gunicorn to run with 25 workers and 25 threads (I have 12 core).

And magicaly every request run seamless.

Till now, I do not know where is the glitch/problem.

binooetomo gravatar imagebinooetomo ( 2019-02-06 18:47:19 -0600 )edit

perhaps the problem was not solved by the number of workers (25 seems excessive for a testing environment), but by restarting your environment.

Bernd Bausch gravatar imageBernd Bausch ( 2019-02-06 19:54:03 -0600 )edit

hmmm last check ... my suspicion goes to my pgsql. When no other service run, keystone run well. But once I turn on all services (so there is 101 db connection) keystone going stuck for every 'post' request'.

binooetomo gravatar imagebinooetomo ( 2019-02-07 02:47:03 -0600 )edit