configure keystone service in pacemaker HA cluster.

asked 2015-03-07 01:13:49 -0500

Rajeshwar Mukund gravatar image

updated 2015-03-07 03:26:07 -0500

Hi I am implementing Juno in active-passive HA cluster for Juno on CentOS 7.

1) Configured Mysql & VIP in active-passive pacemaker 2 node cluster using DRBD. Mysql & Mysql VIP successfully switch-over on both nodes. 2) Configured RabbitMQ in cluster mode (queue mirrored) to prevent message loss.

3) Configured Keystone service in active-passive pacemaker 2 node cluster as per open stack HA guide; but pacemaker cluster fail to start keystone service. Steps for Keystone Implementation -

1.  Install Keystone package on both nodes.
2. Add keystone resource in pacemaker
    cd /usr/lib/ocf/resource.d/openstack
    wget https://raw.github.com/madkiss/openstack-resource-agents/master/ocf/keystone
    chmod a+rx *
3. crete VIP for keystone service (192.168.99.8)
pcs resource create KEY_VIP ocf:heartbeat:Ipaddr2 ip=192.168.99.8 cidr_netmask=24 op monitor interval=30s
4. configure keystone service on both nodes identically as per Open stack Implementation Guide.
    Add following HA specific configuration in keystone.conf so that all keystone data will be stored in DB.

    public_bind_host = 192.168.99.8
    admin_bind_host = 192.168.99.8
    ...
    [catalog]
    driver = keystone.catalog.backends.sql.Catalog
    ...
    [identity]
    driver = keystone.identity.backends.sql.Identity
5. Start keystone service manyally on one node (where DB service is running)
    systemctl start openstack-keystone.service
6. configure keystone user and tanents of keystone, keystone endpoints(over keystone VIP), & verify the operation.
7. add keystone resource in pacemaker.
        pcs resource create IDENTITY_KEY ocf:openstack:keystone config="/etc/keystone/keystone.conf" os_password="XXXXXXXXXX" \
        os_username="admin" os_tenant_name="admin" os_auth_url="http://192.168.99.8:5000/v2.0/"  \
        op monitor interval="30s" timeout="30s"
8. Restart pacemaker cluster on both nodes.
    pcs cluster stop --all
    pcs cluster start --all

Keystone service is failed to start by pacemaker, throwing following error in /var/log/messages.

Mar 7 12:30:34 node3 keystone(IDENTITY_KEY)[24106]: INFO: Old PID file found, but OpenStack Identity (Keystone) is not running Mar 7 12:30:35 node3 keystone(IDENTITY_KEY)[24106]: WARNING: /usr/lib/ocf/lib/heartbeat/ocf-shellfuncs: line 381: kill: (24185) - No such process Mar 7 12:30:35 node3 keystone(IDENTITY_KEY)[24106]: INFO: Old PID file found, but OpenStack Identity (Keystone) is not running Mar 7 12:30:35 node3 lrmd[22757]: warning: child_timeout_callback: IDENTITY_KEY_start_0 process (PID 24106) timed out Mar 7 12:30:35 node3 lrmd[22757]: warning: operation_finished: IDENTITY_KEY_start_0:24106 - timed out after 20000ms Mar 7 12:30:35 node3 crmd[22760]: error: process_lrm_event: LRM operation IDENTITY_KEY_start_0 (50) Timed Out (timeout=20000ms) Mar 7 12:30:35 node3 crmd[22760]: warning: status_from_rc: Action 80 (IDENTITY_KEY_start_0) on node3 failed (target: 0 vs. rc: 1): Error

However when Keystone service is start manually (by removing from pacemaker cluster resource) on both nodes (node on which DB service is running). Keystone starts successfully & and also able to list keystone users created.

[root@node4 openstack]# keystone service-list; +----------------------------------+----------+----------+--------------------+ | id | name | type | description | +----------------------------------+----------+----------+--------------------+ | 551b5c1e9f4144b6a4850815b865a30d | keystone | identity | OpenStack Identity | +----------------------------------+----------+----------+--------------------+ [root@node4 openstack]# keystone user-list; +----------------------------------+-------+---------+---------------+ | id | name | enabled | email | +----------------------------------+-------+---------+---------------+ | 442964dc37c84e5caaaa0fdcc4206166 | admin | True | EMAIL_ADDRESS | | 180b74694cc9437ba443a6f080be7385 | demo | True | EMAIL_ADDRESS | +----------------------------------+-------+---------+---------------+ [root@node4 openstack]#

Please Guide ... (more)

edit retag flag offensive close merge delete