Ask Your Question

creating multiple instances

asked 2014-12-24 12:14:28 -0600

bgyako gravatar image

updated 2014-12-24 12:19:31 -0600

Hi Guys, Having weird problem I suspect is related to HAproxy.

My set up:
Running IceHouse and Ubuntu 14.04
2 Haproxy servers doing the load balancing.
2 controllers.
3 mariaDB servers clustered.

Everything is working properly except when I try to create multiple instances at once. If I create 1, 2 , or 3 instances at one time everything works perfect, but creating 5 or more keeps throwing DB lock errors.

My HaProxy Config:

        chroot  /var/lib/haproxy
        user haproxy
        group haproxy
        log local0
        stats socket /var/lib/haproxy/stats
        pidfile /var/run/
        maxconn 5000
        tune.bufsize 32000

        log     global
        mode    http
        option  httplog
        option  dontlognull
        option  redispatch
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen galera
        balance source
        mode    tcp
        option  tcpka
        option  mysql-check user haproxy
        server  cwtrdb001 check weight 1
        server  cwtrdb002 check weight 1
        server  cwtrdb003 check weight 1
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted

answered 2014-12-25 21:38:09 -0600

updated 2014-12-26 02:46:39 -0600

Running a sql cluster in active/active for Openstack behind a LB wont work very well. You will get lock errors for most operations.

Configure HA proxy to do active/backup for you sql cluster and the problems will go away.

Openstack needs some work done around DB transactions locking for you to be able to write to multiple DB nodes at the same time.

Haproxy galera config:

listen galera
        balance source
        mode    tcp
        option  tcpka
        option  mysql-check user haproxy
        server  cwtrdb001 check weight 1
        server  cwtrdb002 backup weight 1
        server  cwtrdb003 backup weight 1

Once you make that change you will be able to spin up any number of instances you want.

edit flag offensive delete link more


Sorry got side tracked, but i'm back onto this. So my question is, what if I am running 2 Ha proxy's and while nova is processing requests to create multiple instances sends one to ha1 and the next request to ha2, how would you control that?

bobyakov gravatar imagebobyakov ( 2015-03-18 12:25:26 -0600 )edit

You should run 2 HA proxies if you want high availability. Your config for the second HAproxy should be the same. That means when ever any service calls the DB it will always hit cwtrdb001 until that one fails.

sfcloudman gravatar imagesfcloudman ( 2015-03-18 16:42:32 -0600 )edit

I noticed my other environment is not facing this issue. The key difference is the working environment has 3 compute nodes, the non-working has 30 compute nodes everything else is the same.

bobyakov gravatar imagebobyakov ( 2015-03-20 13:16:27 -0600 )edit

Did you configure haproxy for Galera like I posted above?

sfcloudman gravatar imagesfcloudman ( 2015-03-20 21:40:42 -0600 )edit

No not yet was trying to re-create it in my test environment first then make change. Will try on Monday in my main environment

bobyakov gravatar imagebobyakov ( 2015-03-20 23:15:07 -0600 )edit

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower


Asked: 2014-12-24 12:14:28 -0600

Seen: 914 times

Last updated: Dec 26 '14