Ask Your Question
0

creating multiple instances

asked 2014-12-24 12:14:28 -0500

bgyako gravatar image

updated 2014-12-24 12:19:31 -0500

Hi Guys, Having weird problem I suspect is related to HAproxy.

My set up:
Running IceHouse and Ubuntu 14.04
2 Haproxy servers doing the load balancing.
2 controllers.
3 mariaDB servers clustered.

Everything is working properly except when I try to create multiple instances at once. If I create 1, 2 , or 3 instances at one time everything works perfect, but creating 5 or more keeps throwing DB lock errors.

My HaProxy Config:

global
        chroot  /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        log 10.1.0.4 local0
        stats socket /var/lib/haproxy/stats
        pidfile /var/run/haproxy.pid
        maxconn 5000
        tune.bufsize 32000

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option  redispatch
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen galera   10.1.0.2:3306
        balance source
        mode    tcp
        option  tcpka
        option  mysql-check user haproxy
        server  cwtrdb001 10.1.0.7:3306 check weight 1
        server  cwtrdb002 10.1.0.8:3306 check weight 1
        server  cwtrdb003 10.1.0.16:3306 check weight 1
edit retag flag offensive close merge delete

1 answer

Sort by ยป oldest newest most voted
0

answered 2014-12-25 21:38:09 -0500

updated 2014-12-26 02:46:39 -0500

Running a sql cluster in active/active for Openstack behind a LB wont work very well. You will get lock errors for most operations.

Configure HA proxy to do active/backup for you sql cluster and the problems will go away.

Openstack needs some work done around DB transactions locking for you to be able to write to multiple DB nodes at the same time.

Haproxy galera config:

listen galera   10.1.0.2:3306
        balance source
        mode    tcp
        option  tcpka
        option  mysql-check user haproxy
        server  cwtrdb001 10.1.0.7:3306 check weight 1
        server  cwtrdb002 10.1.0.8:3306 backup weight 1
        server  cwtrdb003 10.1.0.16:3306 backup weight 1

Once you make that change you will be able to spin up any number of instances you want.

edit flag offensive delete link more

Comments

Sorry got side tracked, but i'm back onto this. So my question is, what if I am running 2 Ha proxy's and while nova is processing requests to create multiple instances sends one to ha1 and the next request to ha2, how would you control that?

bobyakov gravatar imagebobyakov ( 2015-03-18 12:25:26 -0500 )edit

You should run 2 HA proxies if you want high availability. Your config for the second HAproxy should be the same. That means when ever any service calls the DB it will always hit cwtrdb001 until that one fails.

sfcloudman gravatar imagesfcloudman ( 2015-03-18 16:42:32 -0500 )edit

I noticed my other environment is not facing this issue. The key difference is the working environment has 3 compute nodes, the non-working has 30 compute nodes everything else is the same.

bobyakov gravatar imagebobyakov ( 2015-03-20 13:16:27 -0500 )edit

Did you configure haproxy for Galera like I posted above?

sfcloudman gravatar imagesfcloudman ( 2015-03-20 21:40:42 -0500 )edit

No not yet was trying to re-create it in my test environment first then make change. Will try on Monday in my main environment

bobyakov gravatar imagebobyakov ( 2015-03-20 23:15:07 -0500 )edit

Your Answer

Please start posting anonymously - your entry will be published after you log in or create a new account.

Add Answer

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

Stats

Asked: 2014-12-24 12:14:28 -0500

Seen: 653 times

Last updated: Dec 26 '14