Revision history [back]

click to hide/show revision 1
initial version

Database access in High Availability mode

I have been trying to setup a 3-controller based HA prototype. So far I have got the following items (appear to be) working:

  1. Galera cluster with Mariadb
  2. RabbitMQ with HA Queue
  3. HAProxy
  4. Pacemaker cluster for VIP, HAProxy, and OpenStack-keystone service

The installation is mainly based on the official OpenStack HA guide. Some adjustments for CentOS 7 are as follows:

  1. For the Galera cluster, use wsrep_provider="/usr/lib64/galera/libgalera_smm.so" # the file path in the guide does not work in CentOS 7.
  2. For the Pacemaker cluster resource, there is no openstack-keystone.service for RHEL/CentOS distro. Instead, the keystone service relies on Apache wsgi. Therefore, do pcs resource create openstack-keystone systemd:httpd --clone interleave=true

Problem:

Although the Galera database cluster appears to be working (i.e. sync works), I get following error when trying to access the database with openstack cli.

Unable to establish connection to http://192.168.10.10:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

note: 192.168.10.10 is the cluster VIP and the port is open and reachable.

The log from /var/log/keystone/keystone.log shows the following error:

2018-07-26 20:29:28.905 27063 WARNING oslo_db.sqlalchemy.engines [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

### 9 more attempt records omitted here ###

2018-07-26 20:31:09.016 27063 ERROR keystone.common.wsgi [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8): DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

I can verify that the database is running and can be accessed via mysql command line. Any changes can be successfully synchronized to other nodes with Galera. What could be wrong in this case? Where should I start to debug?

Database access in High Availability mode

I have been trying to setup a 3-controller based HA prototype. So far I have got the following items (appear to be) working:

  1. Galera cluster with Mariadb
  2. RabbitMQ with HA Queue
  3. HAProxy
  4. Pacemaker cluster for VIP, HAProxy, and OpenStack-keystone service

The installation is mainly based on the official OpenStack HA guide. Some adjustments for CentOS 7 are as follows:

  1. For the Galera cluster, use wsrep_provider="/usr/lib64/galera/libgalera_smm.so" # the file path in the guide does not work in CentOS 7.
  2. For the Pacemaker cluster resource, there is no openstack-keystone.service for RHEL/CentOS distro. Instead, the keystone service relies on Apache wsgi. Therefore, do pcs resource create openstack-keystone systemd:httpd --clone interleave=true

Problem:

Although the Galera database cluster appears to be working (i.e. sync works), I get following error when trying to access the database with openstack cli.

Unable to establish connection to http://192.168.10.10:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

note: 192.168.10.10 is the cluster VIP and the port is open and reachable.

The log from /var/log/keystone/keystone.log shows the following error:

2018-07-26 20:29:28.905 27063 WARNING oslo_db.sqlalchemy.engines [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

### 9 more attempt records omitted here ###

2018-07-26 20:31:09.016 27063 ERROR keystone.common.wsgi [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8): DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

I can verify that the database is running and can be accessed via mysql command line. Any changes can be successfully synchronized to other nodes with Galera. I have already disabled the firewall and selinux for the sake of experiment. What could be wrong in this case? Where should I start to debug? debug?

Database access in High Availability mode

I have been trying to setup a 3-controller based HA prototype. prototype based on the Queens release. So far I have got the following items (appear to be) working:

  1. Galera cluster with Mariadb
  2. RabbitMQ with HA Queue
  3. HAProxy
  4. Pacemaker cluster for VIP, HAProxy, and OpenStack-keystone service

The installation is mainly based on the official OpenStack HA guide. Some adjustments for CentOS 7 are as follows:

  1. For the Galera cluster, use wsrep_provider="/usr/lib64/galera/libgalera_smm.so" # the file path in the guide does not work in CentOS 7.
  2. For the Pacemaker cluster resource, there is no openstack-keystone.service for RHEL/CentOS distro. Instead, the keystone service relies on Apache wsgi. Therefore, do pcs resource create openstack-keystone systemd:httpd --clone interleave=true
  3. The following two settings in the guide appear to be deprecated and does not working in Queens:

    [catalog] driver = keystone.catalog.backends.sql.Catalog

    [identity] driver = keystone.identity.backends.sql.Identity

    So I used the default value "sql"

Problem:

Although the Galera database cluster appears to be working (i.e. sync works), I get following error when trying to access the database with openstack cli.

Unable to establish connection to http://192.168.10.10:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

note: 192.168.10.10 is the cluster VIP and the port is open and reachable.

The log from /var/log/keystone/keystone.log shows the following error:

2018-07-26 20:29:28.905 27063 WARNING oslo_db.sqlalchemy.engines [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

### 9 more attempt records omitted here ###

2018-07-26 20:31:09.016 27063 ERROR keystone.common.wsgi [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8): DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

I can verify that the database is running and can be accessed via mysql command line. Any changes can be successfully synchronized to other nodes with Galera. I have already disabled the firewall and selinux for the sake of experiment. What could be wrong in this case? Where should I start to debug?

Database access in High Availability mode

I have been trying to setup a 3-controller based HA prototype based on the Queens release. So far I have got the following items (appear to be) working:

  1. Galera cluster with Mariadb
  2. RabbitMQ with HA Queue
  3. HAProxy
  4. Pacemaker cluster for VIP, HAProxy, and OpenStack-keystone service

The installation is mainly based on the official OpenStack HA guide. Some adjustments for CentOS 7 are as follows:

  1. For the Galera cluster, use wsrep_provider="/usr/lib64/galera/libgalera_smm.so" # the file path in the guide does not work in CentOS 7.
  2. For the Pacemaker cluster resource, there is no openstack-keystone.service for RHEL/CentOS distro. Instead, the keystone service relies on Apache wsgi. Therefore, do pcs resource create openstack-keystone systemd:httpd --clone interleave=true
  3. The following two settings in /etc/keystone/keystone.conf from the guide appear to be deprecated and does do not working work in Queens:

    [catalog] driver = keystone.catalog.backends.sql.Catalog

    [identity] driver = keystone.identity.backends.sql.Identity

    So I used the default value "sql"

Problem:

Although the Galera database cluster appears to be working (i.e. sync works), I get following error when trying to access the database with openstack cli.

Unable to establish connection to http://192.168.10.10:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

note: 192.168.10.10 is the cluster VIP and the port is open and reachable.

The log from /var/log/keystone/keystone.log shows the following error:

2018-07-26 20:29:28.905 27063 WARNING oslo_db.sqlalchemy.engines [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

### 9 more attempt records omitted here ###

2018-07-26 20:31:09.016 27063 ERROR keystone.common.wsgi [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8): DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

I can verify that the database is running and can be accessed via mysql command line. Any changes can be successfully synchronized to other nodes with Galera. I have already disabled the firewall and selinux for the sake of experiment. What could be wrong in this case? Where should I start to debug?

Database access in High Availability mode

I have been trying to setup a 3-controller based HA prototype based on the Queens release. So far I have got the following items (appear to be) working:

  1. Galera cluster with Mariadb
  2. RabbitMQ with HA Queue
  3. HAProxy
  4. Pacemaker cluster for VIP, HAProxy, and OpenStack-keystone service

The installation is mainly based on the official OpenStack HA guide. Some adjustments for CentOS 7 are as follows:

  1. For the Galera cluster, use wsrep_provider="/usr/lib64/galera/libgalera_smm.so" # the file path in the guide does not work in CentOS 7.
  2. For the Pacemaker cluster resource, there is no openstack-keystone.service for RHEL/CentOS distro. Instead, the keystone service relies on Apache wsgi. Therefore, do pcs resource create openstack-keystone systemd:httpd --clone interleave=true
  3. The following two settings in /etc/keystone/keystone.conf from the guide appear to be deprecated and do not work in Queens:

    [catalog] driver = keystone.catalog.backends.sql.Catalog

    [identity] driver = keystone.identity.backends.sql.Identity

    So I used the default value "sql"

Problem:

Although the Galera database cluster appears to be working (i.e. sync works), I get following error when trying to access the database with openstack cli.cli (e.g. openstack project list).

Unable to establish connection to http://192.168.10.10:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

note: 192.168.10.10 is the cluster VIP and the port is open and reachable.

The log from /var/log/keystone/keystone.log shows the following error:

2018-07-26 20:29:28.905 27063 WARNING oslo_db.sqlalchemy.engines [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

### 9 more attempt records omitted here ###

2018-07-26 20:31:09.016 27063 ERROR keystone.common.wsgi [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8): DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

I can verify that the database is running and can be accessed via mysql command line. Any changes can be successfully synchronized to other nodes with Galera. I have already disabled the firewall and selinux for the sake of experiment. What could be wrong in this case? Where should I start to debug?

Database access in High Availability mode

I have been trying to setup a 3-controller based HA prototype based on the Queens release. So far I have got the following items (appear to be) working:

  1. Galera cluster with Mariadb
  2. RabbitMQ with HA Queue
  3. HAProxy
  4. Pacemaker cluster for VIP, HAProxy, and OpenStack-keystone service

The installation is mainly based on the official OpenStack HA guide. Some adjustments for CentOS 7 are as follows:

  1. For the Galera cluster, use wsrep_provider="/usr/lib64/galera/libgalera_smm.so" # the file path in the guide does not work in CentOS 7.
  2. For the Pacemaker cluster resource, there is no openstack-keystone.service for RHEL/CentOS distro. Instead, Since the keystone service relies on the Apache wsgi. Therefore, do wsgi, a work-around is to use pcs resource create openstack-keystone systemd:httpd --clone interleave=true
  3. The following two settings in /etc/keystone/keystone.conf from the guide appear to be deprecated and do not work in Queens:

    [catalog] driver = keystone.catalog.backends.sql.Catalog

    [identity] driver = keystone.identity.backends.sql.Identity

    So I used the default value "sql"

Problem:

Although the Galera database cluster appears to be working (i.e. sync works), I get a following error when trying to access the database with openstack cli (e.g. openstack project list).

Unable to establish connection to http://192.168.10.10:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

note: 192.168.10.10 is the cluster VIP and the port is open and reachable.

The log from /var/log/keystone/keystone.log shows the following error:

2018-07-26 20:29:28.905 27063 WARNING oslo_db.sqlalchemy.engines [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

### 9 more attempt records attempts... logs omitted here ###

2018-07-26 20:31:09.016 27063 ERROR keystone.common.wsgi [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8): DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

I can verify that the database is running and can be accessed via mysql command line. Any changes can be successfully synchronized to other nodes with Galera. I have already disabled the firewall and selinux for the sake of experiment. What could be wrong in this case? Where should I start to debug?

Database access in High Availability mode

I have been trying to setup a 3-controller based HA prototype based on the Queens release. So far I have got the following items (appear to be) working:

  1. Galera cluster with Mariadb
  2. RabbitMQ with HA Queue
  3. HAProxy
  4. Pacemaker cluster for VIP, HAProxy, and OpenStack-keystone service

The installation is mainly based on the official OpenStack HA guide. Some adjustments for CentOS 7 are as follows:

  1. For the Galera cluster, use wsrep_provider="/usr/lib64/galera/libgalera_smm.so" # the file path in the guide does not work in for CentOS 7.
  2. For the Pacemaker cluster resource, there is no openstack-keystone.service for RHEL/CentOS distro. Since the keystone service relies on the Apache wsgi, a work-around is to use pcs resource create openstack-keystone systemd:httpd --clone interleave=true
  3. The following two settings in /etc/keystone/keystone.conf from the guide appear to be deprecated and do not work in Queens:

    [catalog] driver = keystone.catalog.backends.sql.Catalog

    [identity] driver = keystone.identity.backends.sql.Identity

    So I used the default value "sql"

Problem:

Although the Galera database cluster appears to be working (i.e. sync works), I get a following error when trying to access the database with openstack cli (e.g. openstack project list).

Unable to establish connection to http://192.168.10.10:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

note: 192.168.10.10 is the cluster VIP and the port is open and reachable.

The log from /var/log/keystone/keystone.log shows the following error:

2018-07-26 20:29:28.905 27063 WARNING oslo_db.sqlalchemy.engines [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

### 9 more attempts... logs omitted here ###

2018-07-26 20:31:09.016 27063 ERROR keystone.common.wsgi [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8): DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

I can verify that the database is running and can be accessed via mysql command line. Any changes can be successfully synchronized to other nodes with Galera. I have already disabled the firewall and selinux for the sake of experiment. What could be wrong in this case? Where should I start to debug?

Database access in High Availability mode

I have been trying to setup a 3-controller based HA prototype based on the Queens release. So far I have got the following items (appear to be) working:

  1. Galera cluster with Mariadb
  2. RabbitMQ with HA Queue
  3. HAProxy
  4. Pacemaker cluster for VIP, HAProxy, and OpenStack-keystone service

The installation is mainly based on the official OpenStack HA guide. Some adjustments for CentOS 7 are as follows:

  1. For the Galera cluster, use wsrep_provider="/usr/lib64/galera/libgalera_smm.so" # the file path in the guide does not work for CentOS 7.
  2. For the Pacemaker cluster resource, there is no openstack-keystone.service for RHEL/CentOS distro. Since the keystone service relies on the Apache wsgi, a work-around is to use pcs resource create openstack-keystone systemd:httpd --clone interleave=true
  3. The following two settings in /etc/keystone/keystone.conf from the guide appear to be deprecated and do not work in Queens:

    [catalog] driver = keystone.catalog.backends.sql.Catalog

    [identity] driver = keystone.identity.backends.sql.Identity

    So I used the default value "sql"

Problem:

Although the Galera database cluster appears to be working (i.e. sync works), I get a following error when trying to access the database with openstack cli (e.g. openstack project list).

Unable to establish connection to http://192.168.10.10:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

note: 192.168.10.10 is the cluster VIP and the port is open and reachable.

The log from /var/log/keystone/keystone.log shows the following error:

2018-07-26 20:29:28.905 27063 WARNING oslo_db.sqlalchemy.engines [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

### 9 more attempts... logs omitted here omitted... ###

2018-07-26 20:31:09.016 27063 ERROR keystone.common.wsgi [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8): DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

I can verify that the database is running and can be accessed via mysql command line. Any changes can be successfully synchronized to other nodes with Galera. I have already disabled the firewall and selinux for the sake of experiment. What could be wrong in this case? Where should I start to debug?

Database access in High Availability mode

I have been trying to setup a 3-controller based HA prototype based on the Queens release. So far I have got the following items (appear to be) working:

  1. Galera cluster with Mariadb
  2. RabbitMQ with HA Queue
  3. HAProxy
  4. Pacemaker cluster resource management for VIP, HAProxy, and OpenStack-keystone serviceAPI

The installation is mainly based on the official OpenStack HA guide. Some adjustments changes were made for CentOS 7 are 7, as follows:

  1. For the Galera cluster, use wsrep_provider="/usr/lib64/galera/libgalera_smm.so" # the file path in the guide does not work for CentOS 7.
  2. For the Pacemaker cluster resource, there is no openstack-keystone.service for RHEL/CentOS distro. Since the keystone service relies on the Apache wsgi, a work-around is to use pcs resource create openstack-keystone systemd:httpd --clone interleave=true
  3. The following two settings in /etc/keystone/keystone.conf from the guide appear to be deprecated and do not work in Queens:

    [catalog] driver = keystone.catalog.backends.sql.Catalog

    [identity] driver = keystone.identity.backends.sql.Identity

    So I used the default value "sql"

Problem:

Although the Galera database cluster appears to be working (i.e. sync works), I get a following error when trying to access the database with openstack cli (e.g. openstack project list).

Unable to establish connection to http://192.168.10.10:35357/v3/auth/tokens: ('Connection aborted.', BadStatusLine("''",))

note: 192.168.10.10 is the cluster VIP and the port is open and reachable.

The log from /var/log/keystone/keystone.log shows the following error:

2018-07-26 20:29:28.905 27063 WARNING oslo_db.sqlalchemy.engines [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] SQL connection failed. 10 attempts left.: DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

### 9 more attempts... logs omitted... ###

2018-07-26 20:31:09.016 27063 ERROR keystone.common.wsgi [req-1ad06ad5-92a6-4a4c-9488-d805ba6f1688 - - - - -] (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8): DBConnectionError: (_mysql_exceptions.OperationalError) (2013, 'Lost connection to MySQL server at \'reading initial communication packet\', system error: 0 "Internal error/check (Not system error)"') (Background on this error at: http://sqlalche.me/e/e3q8)

I can verify that the database is running and can be accessed via mysql command line. Any changes can be successfully synchronized to other nodes with Galera. I have already disabled the firewall and selinux for the sake of experiment. What could be wrong in this case? Where should I start to debug?