Ask Your Question
0

creating multiple instances

asked 2014-12-24 12:14:28 -0600

bgyako gravatar image

updated 2014-12-24 12:19:31 -0600

Hi Guys, Having weird problem I suspect is related to HAproxy.

My set up:
Running IceHouse and Ubuntu 14.04
2 Haproxy servers doing the load balancing.
2 controllers.
3 mariaDB servers clustered.

Everything is working properly except when I try to create multiple instances at once. If I create 1, 2 , or 3 instances at one time everything works perfect, but creating 5 or more keeps throwing DB lock errors.

My HaProxy Config:

global
        chroot  /var/lib/haproxy
        user haproxy
        group haproxy
        daemon
        log 10.1.0.4 local0
        stats socket /var/lib/haproxy/stats
        pidfile /var/run/haproxy.pid
        maxconn 5000
        tune.bufsize 32000

defaults
        log     global
        mode    http
        option  httplog
        option  dontlognull
        option  redispatch
        contimeout 5000
        clitimeout 50000
        srvtimeout 50000
        errorfile 400 /etc/haproxy/errors/400.http
        errorfile 403 /etc/haproxy/errors/403.http
        errorfile 408 /etc/haproxy/errors/408.http
        errorfile 500 /etc/haproxy/errors/500.http
        errorfile 502 /etc/haproxy/errors/502.http
        errorfile 503 /etc/haproxy/errors/503.http
        errorfile 504 /etc/haproxy/errors/504.http

listen galera   10.1.0.2:3306
        balance source
        mode    tcp
        option  tcpka
        option  mysql-check user haproxy
        server  cwtrdb001 10.1.0.7:3306 check weight 1
        server  cwtrdb002 10.1.0.8:3306 check weight 1
        server  cwtrdb003 10.1.0.16:3306 check weight 1
edit retag flag offensive close merge delete
add a comment

1 answer

Sort by ยป oldest newest most voted
0

answered 2014-12-25 21:38:09 -0600

sfcloudman gravatar image

updated 2014-12-26 02:46:39 -0600

Running a sql cluster in active/active for Openstack behind a LB wont work very well. You will get lock errors for most operations.

Configure HA proxy to do active/backup for you sql cluster and the problems will go away.

Openstack needs some work done around DB transactions locking for you to be able to write to multiple DB nodes at the same time.

Haproxy galera config:

listen galera   10.1.0.2:3306
        balance source
        mode    tcp
        option  tcpka
        option  mysql-check user haproxy
        server  cwtrdb001 10.1.0.7:3306 check weight 1
        server  cwtrdb002 10.1.0.8:3306 backup weight 1
        server  cwtrdb003 10.1.0.16:3306 backup weight 1

Once you make that change you will be able to spin up any number of instances you want.

edit flag offensive delete link more

Comments

Sorry got side tracked, but i'm back onto this. So my question is, what if I am running 2 Ha proxy's and while nova is processing requests to create multiple instances sends one to ha1 and the next request to ha2, how would you control that?

bobyakov gravatar imagebobyakov ( 2015-03-18 12:25:26 -0600 )edit

You should run 2 HA proxies if you want high availability. Your config for the second HAproxy should be the same. That means when ever any service calls the DB it will always hit cwtrdb001 until that one fails.

sfcloudman gravatar imagesfcloudman ( 2015-03-18 16:42:32 -0600 )edit

I noticed my other environment is not facing this issue. The key difference is the working environment has 3 compute nodes, the non-working has 30 compute nodes everything else is the same.

bobyakov gravatar imagebobyakov ( 2015-03-20 13:16:27 -0600 )edit

Did you configure haproxy for Galera like I posted above?

sfcloudman gravatar imagesfcloudman ( 2015-03-20 21:40:42 -0600 )edit

No not yet was trying to re-create it in my test environment first then make change. Will try on Monday in my main environment

bobyakov gravatar imagebobyakov ( 2015-03-20 23:15:07 -0600 )edit
see more comments

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2014-12-24 12:14:28 -0600

Seen: 914 times

Last updated: Dec 26 '14

Related questions

Keystone through haproxy

Not able to launch Instance

Cannot connect to vm instance by ssh key or cloud-init behind http network proxy

login into mysql kolla ansible openstack

OpenStack not running after reboot. (MariaDB recovery needed)

How to increase default timeout value of LB ?

devstack: cannot launch cirros-0.3.4-x86_64-uec image: Block Device Mapping is Invalid.

Launching multiple fails cause of VolumeNotCreated: ... did not finish being created even after we waited 190 seconds or 61 attempts.

HAProxy, Keepalived and neutron.

Can't Populate the database su -s /bin/sh -c "neutron-db-manage --config-file /etc/neutron/neutron.conf \ --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade juno" neutron"