Ask Your Question

Keystone through haproxy

asked 2015-01-09 01:53:48 -0600

mathias gravatar image

Hi, I just noted some behavior I cannot explain. As code means more than 1000 words:

mathias@desktop:~$ cat token-request.json 
    "auth": {
        "identity": {
            "methods": [
            "password": {
                "user": {
                    "domain": {
                        "name": "Default"
                    "name": "admin",
                    "password": "OpenStack123"
        "scope": {
            "project": {
                "domain": {
                    "name": "Default"
                "name": "admin"

mathias@desktop:~$ export TOKEN=`curl -si -d @token-request.json -H "Content-type: application/json" | awk '/X-Subject-Token/ {print $2}'`
mathias@desktop:~$ curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json"
HTTP/1.0 400 Bad request
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.

mathias@desktop:~$ curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json"
HTTP/1.1 200 OK
Vary: X-Auth-Token
X-Distribution: Ubuntu
Content-Type: application/json
Content-Length: 518
Date: Fri, 09 Jan 2015 07:50:00 GMT

{"domains": [{"links": {"self": ""}, "enabled": true, "description": "customer.invalid", "name": "customer", "id": "38a34ccea4af48a79e1c888027089251"}, {"links": {"self": ""}, "enabled": true, "description": "Owns users and tenants (i.e. projects) available on Identity API v2.", "name": "Default", "id": "default"}], "links": {"self": "", "previous": null, "next": null}}

You can see that when I request /v3/domains via (the haproxy load balancer IP) the request fails. Running it directly to (keystones actual IP) it works.

Here my haproxy.conf:

        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        log local0
        stats socket /var/lib/haproxy/stats
        maxconn 4000

        log     global
        mode    http
        option  httplog
        option  dontlognull
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen stats
        mode http
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth admin:password

listen keystone_admin
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server identity01 check inter 10000 rise 2 fall 5

listen keystone_api
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server identity01 check inter 10000 rise 2 fall 5

Any ideas?

edit retag flag offensive close merge delete


Is the 400 Bad Request returned from HAProxy or Keystone? Sometimes Keystone's relatively lengthy token strings can have impacts that mean load balancers need default settings changed.

fifieldt gravatar imagefifieldt ( 2015-01-09 02:51:51 -0600 )edit

I ran haproxy in foreground with -d for debug. Repeating the request above gave me this: Also, I had a look at keystone.log and couldnt see and incoming GET requests. So, I assume it is haproxy who returns the bad request.

Any ideas how to solve this?

mathias gravatar imagemathias ( 2015-01-09 04:46:36 -0600 )edit

Sniffing on the load balancer shows that the request comes in with correct (in my perception) headers:

mathias gravatar imagemathias ( 2015-01-09 05:02:36 -0600 )edit

yeah, so it looks like haproxy is doing the rejection then, and this is not really related to keystone. I'd be trying things like option accept-invalid-http-request in the config and seeing what works.

fifieldt gravatar imagefifieldt ( 2015-01-09 11:13:02 -0600 )edit

2 answers

Sort by ยป oldest newest most voted

answered 2015-01-10 07:25:58 -0600

mathias gravatar image

I changed the haproxy config file to exactly what is shown in OpenStack documentation and it worked. I never figured out what the difference was.

edit flag offensive delete link more


Probably somewhere in defaults area.

  log  global
  maxconn  8000
  option  redispatch
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s
sfcloudman gravatar imagesfcloudman ( 2015-01-11 22:44:32 -0600 )edit

I would comment out one at time and restart ha-proxy to see which change did it.

sfcloudman gravatar imagesfcloudman ( 2015-01-11 22:44:59 -0600 )edit

i suggest that use diff command to find the difference between the default haproxy.cfg and yours, then focus on the most suspected options and try again.:-)

9lives gravatar image9lives ( 2015-01-12 01:49:11 -0600 )edit

answered 2016-04-13 07:59:22 -0600

I meet the exactly the same problem.

An when I comment out "mode http" in haproxy.conf.

It works OK.

It is really weird ! Maybe it is related to some http check mechanism in haproxy.

Hope someone give a better solution.

edit flag offensive delete link more

Get to know Ask OpenStack

Resources for moderators

Question Tools



Asked: 2015-01-09 01:53:48 -0600

Seen: 1,997 times

Last updated: Jan 10 '15