Ask Your Question
1

Keystone through haproxy

asked 2015-01-09 01:53:48 -0500

mathias gravatar image

Hi, I just noted some behavior I cannot explain. As code means more than 1000 words:

mathias@desktop:~$ cat token-request.json 
{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
                "user": {
                    "domain": {
                        "name": "Default"
                    },
                    "name": "admin",
                    "password": "OpenStack123"
                }
            }
        },
        "scope": {
            "project": {
                "domain": {
                    "name": "Default"
                },
                "name": "admin"
            }
        }
    }
}

mathias@desktop:~$ export TOKEN=`curl -si -d @token-request.json -H "Content-type: application/json" http://10.0.10.11:35357/v3/auth/tokens | awk '/X-Subject-Token/ {print $2}'`
mathias@desktop:~$ curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://10.0.10.11:35357/v3/domains
HTTP/1.0 400 Bad request
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html><body><h1>400 Bad request</h1>
Your browser sent an invalid request.
</body></html>

mathias@desktop:~$ curl -si -H"X-Auth-Token:$TOKEN" -H "Content-type: application/json" http://10.0.10.101:35357/v3/domains
HTTP/1.1 200 OK
Vary: X-Auth-Token
X-Distribution: Ubuntu
Content-Type: application/json
Content-Length: 518
Date: Fri, 09 Jan 2015 07:50:00 GMT

{"domains": [{"links": {"self": "http://10.0.10.101:35357/v3/domains/38a34ccea4af48a79e1c888027089251"}, "enabled": true, "description": "customer.invalid", "name": "customer", "id": "38a34ccea4af48a79e1c888027089251"}, {"links": {"self": "http://10.0.10.101:35357/v3/domains/default"}, "enabled": true, "description": "Owns users and tenants (i.e. projects) available on Identity API v2.", "name": "Default", "id": "default"}], "links": {"self": "http://10.0.10.101:35357/v3/domains", "previous": null, "next": null}}

You can see that when I request /v3/domains via 10.0.10.11 (the haproxy load balancer IP) the request fails. Running it directly to 10.0.10.101 (keystones actual IP) it works.

Here my haproxy.conf:

global  
        chroot /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        log 10.0.10.254 local0
        stats socket /var/lib/haproxy/stats
        maxconn 4000

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen stats 10.0.10.254:80
        mode http
        stats enable
        stats uri /stats
        stats realm HAProxy\ Statistics
        stats auth admin:password


listen keystone_admin 10.0.10.11:35357
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server identity01 10.0.10.101:35357 check inter 10000 rise 2 fall 5

listen keystone_api 10.0.10.11:5000
        balance source
        option tcpka
        option httpchk
        maxconn 10000
        server identity01 10.0.10.101:5000 check inter 10000 rise 2 fall 5

Any ideas?

edit retag flag offensive close merge delete

Comments

Is the 400 Bad Request returned from HAProxy or Keystone? Sometimes Keystone's relatively lengthy token strings can have impacts that mean load balancers need default settings changed.

fifieldt gravatar imagefifieldt ( 2015-01-09 02:51:51 -0500 )edit

I ran haproxy in foreground with -d for debug. Repeating the request above gave me this: http://pastebin.com/aAe3bkHp Also, I had a look at keystone.log and couldnt see and incoming GET requests. So, I assume it is haproxy who returns the bad request.

Any ideas how to solve this?

mathias gravatar imagemathias ( 2015-01-09 04:46:36 -0500 )edit

Sniffing on the load balancer shows that the request comes in with correct (in my perception) headers: http://pastebin.com/79emMjQi

mathias gravatar imagemathias ( 2015-01-09 05:02:36 -0500 )edit

yeah, so it looks like haproxy is doing the rejection then, and this is not really related to keystone. I'd be trying things like option accept-invalid-http-request in the config and seeing what works.

fifieldt gravatar imagefifieldt ( 2015-01-09 11:13:02 -0500 )edit

2 answers

Sort by ยป oldest newest most voted
1

answered 2015-01-10 07:25:58 -0500

mathias gravatar image

I changed the haproxy config file to exactly what is shown in OpenStack documentation and it worked. I never figured out what the difference was.

edit flag offensive delete link more

Comments

Probably somewhere in defaults area.

   defaults
  log  global
  maxconn  8000
  option  redispatch
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s
sfcloudman gravatar imagesfcloudman ( 2015-01-11 22:44:32 -0500 )edit
1

I would comment out one at time and restart ha-proxy to see which change did it.

sfcloudman gravatar imagesfcloudman ( 2015-01-11 22:44:59 -0500 )edit

i suggest that use diff command to find the difference between the default haproxy.cfg and yours, then focus on the most suspected options and try again.:-)

9lives gravatar image9lives ( 2015-01-12 01:49:11 -0500 )edit
0

answered 2016-04-13 07:59:22 -0500

I meet the exactly the same problem.

An when I comment out "mode http" in haproxy.conf.

It works OK.

It is really weird ! Maybe it is related to some http check mechanism in haproxy.

Hope someone give a better solution.

edit flag offensive delete link more

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

2 followers

Stats

Asked: 2015-01-09 01:53:48 -0500

Seen: 1,585 times

Last updated: Jan 10 '15