Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

kolla-ansible deploy rocky: Waiting for master mariadb

I see some other issues with kolla-ansible deployments where mariadb doesn't come up, but wonder if anyone has seen this one. It's a new all-in-one rocky install on Ubuntu 16.04.6.

kolla-ansible -i ./all-in-one deploy ends with:

================== RUNNING HANDLER [mariadb : Waiting for master mariadb] ******************************************* FAILED - RETRYING: Waiting for master mariadb (10 retries left). FAILED - RETRYING: Waiting for master mariadb (9 retries left). FAILED - RETRYING: Waiting for master mariadb (8 retries left). FAILED - RETRYING: Waiting for master mariadb (7 retries left). FAILED - RETRYING: Waiting for master mariadb (6 retries left). FAILED - RETRYING: Waiting for master mariadb (5 retries left). FAILED - RETRYING: Waiting for master mariadb (4 retries left). FAILED - RETRYING: Waiting for master mariadb (3 retries left). FAILED - RETRYING: Waiting for master mariadb (2 retries left). FAILED - RETRYING: Waiting for master mariadb (1 retries left). An exception occurred during task execution. To see the full traceback, use -vvv. The error was: socket.error: [Errno 104] Connection reset by peer

fatal: [localhost]: FAILED! => {"attempts": 10, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 113, in <module>\n File \"<stdin>\", line 105, in _ansiballz_main\n File \"<stdin>\", line 48, in invoke_module\n File \"/tmp/ansible_wait_for_payload_QRG4Ei/__main__.py\", line 629, in <module>\n File \"/tmp/ansible_wait_for_payload_QRG4Ei/__main__.py\", line 558, in main\nsocket.error: [Errno 104] Connection reset by peer\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Adding -vvv I get:

================== Using module file /root/cos/venv/local/lib/python2.7/site-packages/ansible/modules/utilities/logic/wait_for.py <localhost> EXEC /bin/sh -c '/usr/bin/python && sleep 0' FAILED - RETRYING: Waiting for master mariadb (1 retries left).Result was: { "attempts": 10, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 113, in <module>\n File \"<stdin>\", line 105, in _ansiballz_main\n File \"<stdin>\", line 48, in invoke_module\n File \"/tmp/ansible_wait_for_payload_uaoycs/__main__.py\", line 629, in <module>\n File \"/tmp/ansible_wait_for_payload_uaoycs/__main__.py\", line 558, in main\nsocket.error: [Errno 104] Connection reset by peer\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1, "retries": 11 } Using module file /root/cos/venv/local/lib/python2.7/site-packages/ansible/modules/utilities/logic/wait_for.py <localhost> EXEC /bin/sh -c '/usr/bin/python && sleep 0' The full traceback is: Traceback (most recent call last): File "<stdin>", line 113, in <module> File "<stdin>", line 105, in _ansiballz_main File "<stdin>", line 48, in invoke_module File "/tmp/ansible_wait_for_payload_wFR7KE/__main__.py", line 629, in <module> File "/tmp/ansible_wait_for_payload_wFR7KE/__main__.py", line 558, in main socket.error: [Errno 104] Connection reset by peer

fatal: [localhost]: FAILED! => { "attempts": 10, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 113, in <module>\n File \"<stdin>\", line 105, in _ansiballz_main\n File \"<stdin>\", line 48, in invoke_module\n File \"/tmp/ansible_wait_for_payload_wFR7KE/__main__.py\", line 629, in <module>\n File \"/tmp/ansible_wait_for_payload_wFR7KE/__main__.py\", line 558, in main\nsocket.error: [Errno 104] Connection reset by peer\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 } META: ran handlers

NO MORE HOSTS LEFT ******************************************************* to retry, use: --limit @/root/cos/venv/share/kolla-ansible/ansible/site.retry

PLAY RECAP ********************************************************* localhost : ok=48 changed=2 unreachable=0 failed=1

Command failed ansible-playbook -i ./all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /root/cos/venv/share/kolla-ansible/ansible/site.yml --verbose --verbose --verbose

mariadb container keeps restarting:

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 31ef8c79ae34 kolla/ubuntu-source-mariadb:rocky "dumb-init --single-c" 7 minutes ago Restarting (0) 2 minutes ago mariadb 92925c8829a8 kolla/ubuntu-source-memcached:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes memcached 762d73601d68 kolla/ubuntu-source-chrony:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes chrony bc1fbdb96b2c kolla/ubuntu-source-cron:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes cron 2655781f5d33 kolla/ubuntu-source-kolla-toolbox:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes kolla_toolbox 435cb987dfb9 kolla/ubuntu-source-fluentd:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes fluentd

tail of ‘docker logs mariadb’

  • CMD=/usr/bin/mysqld_safe
  • ARGS=
  • [[ ! -n '' ]]
  • . kolla_extend_start ++ [[ ! -d /var/log/kolla/mariadb ]] +++ stat -c %a /var/log/kolla/mariadb ++ [[ 755 != \7\5\5 ]] ++ [[ -n '' ]] ++ [[ -n '' ]]
  • echo 'Running command: '\''/usr/bin/mysqld_safe'\'''
  • exec /usr/bin/mysqld_safe 190306 14:29:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ Running command: '/usr/bin/mysqld_safe' 190306 14:29:48 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'. 190306 14:29:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ Running command: '/usr/bin/mysqld_safe' 190306 14:30:46 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'. 190306 14:30:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ Running command: '/usr/bin/mysqld_safe' 190306 14:32:35 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'. 190306 14:32:35 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/

and these errors in mariadb.log seem more interesting:

tail /var/lib/docker/volumes/kolla_logs/_data/mariadb/mariadb.log 2019-03-06 14:36:14 140394066910336 [Note] InnoDB: Waiting for purge to start 2019-03-06 14:36:14 140394066910336 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1601271 2019-03-06 14:36:14 140383874938624 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-03-06 14:36:14 140394066910336 [Note] Plugin 'FEEDBACK' is disabled. 2019-03-06 14:36:14 140394066910336 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded 2019-03-06 14:36:14 140394066033408 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 2019-03-06 14:36:14 140394066910336 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist 2019-03-06 14:36:14 140394066910336 [Note] Server socket created on IP: ‘x.x.x.x’. 2019-03-06 14:36:14 140394066910336 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist 190306 14:36:14 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended

I had no problem running bootstrap-servers or prechecks. One of those was precheck for docker version, as I saw that may have been an issue in the past.

docker version Client: Version: 1.12.6 API version: 1.24 Go version: go1.6.4 Git commit: 78d1802 Built: Tue Jan 10 20:38:45 2017 OS/Arch: linux/amd64

Server: Version: 1.12.6 API version: 1.24 Go version: go1.6.4 Git commit: 78d1802 Built: Tue Jan 10 20:38:45 2017 OS/Arch: linux/amd64

Love to hear any ideas, workarounds, or if anyone has the same result :)

thank you!

kolla-ansible deploy rocky: Waiting for master mariadb

I see some other issues with kolla-ansible deployments where mariadb doesn't come up, but wonder if anyone has seen this one. It's a new all-in-one rocky install on Ubuntu 16.04.6.

kolla-ansible -i ./all-in-one deploy ends with:

================== RUNNING HANDLER [mariadb : Waiting for master mariadb] ******************************************* FAILED - RETRYING: Waiting for master mariadb (10 retries left). FAILED - RETRYING: Waiting for master mariadb (9 retries left). FAILED - RETRYING: Waiting for master mariadb (8 retries left). FAILED - RETRYING: Waiting for master mariadb (7 retries left). FAILED - RETRYING: Waiting for master mariadb (6 retries left). FAILED - RETRYING: Waiting for master mariadb (5 retries left). FAILED - RETRYING: Waiting for master mariadb (4 retries left). FAILED - RETRYING: Waiting for master mariadb (3 retries left). FAILED - RETRYING: Waiting for master mariadb (2 retries left). FAILED - RETRYING: Waiting for master mariadb (1 retries left). An exception occurred during task execution. To see the full traceback, use -vvv. The error was: socket.error: [Errno 104] Connection reset by peer

fatal: [localhost]: FAILED! => {"attempts": 10, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 113, in <module>\n File \"<stdin>\", line 105, in _ansiballz_main\n File \"<stdin>\", line 48, in invoke_module\n File \"/tmp/ansible_wait_for_payload_QRG4Ei/__main__.py\", line 629, in <module>\n File \"/tmp/ansible_wait_for_payload_QRG4Ei/__main__.py\", line 558, in main\nsocket.error: [Errno 104] Connection reset by peer\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Adding -vvv I get:

================== Using module file /root/cos/venv/local/lib/python2.7/site-packages/ansible/modules/utilities/logic/wait_for.py <localhost> EXEC /bin/sh -c '/usr/bin/python && sleep 0' FAILED - RETRYING: Waiting for master mariadb (1 retries left).Result was: { "attempts": 10, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 113, in <module>\n File \"<stdin>\", line 105, in _ansiballz_main\n File \"<stdin>\", line 48, in invoke_module\n File \"/tmp/ansible_wait_for_payload_uaoycs/__main__.py\", line 629, in <module>\n File \"/tmp/ansible_wait_for_payload_uaoycs/__main__.py\", line 558, in main\nsocket.error: [Errno 104] Connection reset by peer\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1, "retries": 11 } Using module file /root/cos/venv/local/lib/python2.7/site-packages/ansible/modules/utilities/logic/wait_for.py <localhost> EXEC /bin/sh -c '/usr/bin/python && sleep 0' The full traceback is: Traceback (most recent call last): File "<stdin>", line 113, in <module> File "<stdin>", line 105, in _ansiballz_main File "<stdin>", line 48, in invoke_module File "/tmp/ansible_wait_for_payload_wFR7KE/__main__.py", line 629, in <module> File "/tmp/ansible_wait_for_payload_wFR7KE/__main__.py", line 558, in main socket.error: [Errno 104] Connection reset by peer

fatal: [localhost]: FAILED! => { "attempts": 10, "changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 113, in <module>\n File \"<stdin>\", line 105, in _ansiballz_main\n File \"<stdin>\", line 48, in invoke_module\n File \"/tmp/ansible_wait_for_payload_wFR7KE/__main__.py\", line 629, in <module>\n File \"/tmp/ansible_wait_for_payload_wFR7KE/__main__.py\", line 558, in main\nsocket.error: [Errno 104] Connection reset by peer\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1 } META: ran handlers

NO MORE HOSTS LEFT ******************************************************* to retry, use: --limit @/root/cos/venv/share/kolla-ansible/ansible/site.retry

PLAY RECAP ********************************************************* localhost : ok=48 changed=2 unreachable=0 failed=1

Command failed ansible-playbook -i ./all-in-one -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla -e kolla_action=deploy /root/cos/venv/share/kolla-ansible/ansible/site.yml --verbose --verbose --verbose

mariadb container keeps restarting:

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 31ef8c79ae34 kolla/ubuntu-source-mariadb:rocky "dumb-init --single-c" 7 minutes ago Restarting (0) 2 minutes ago mariadb 92925c8829a8 kolla/ubuntu-source-memcached:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes memcached 762d73601d68 kolla/ubuntu-source-chrony:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes chrony bc1fbdb96b2c kolla/ubuntu-source-cron:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes cron 2655781f5d33 kolla/ubuntu-source-kolla-toolbox:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes kolla_toolbox 435cb987dfb9 kolla/ubuntu-source-fluentd:rocky "dumb-init --single-c" 7 minutes ago Up 7 minutes fluentd

tail of ‘docker logs mariadb’

  • CMD=/usr/bin/mysqld_safe
  • ARGS=
  • [[ ! -n '' ]]
  • . kolla_extend_start ++ [[ ! -d /var/log/kolla/mariadb ]] +++ stat -c %a /var/log/kolla/mariadb ++ [[ 755 != \7\5\5 ]] ++ [[ -n '' ]] ++ [[ -n '' ]]
  • echo 'Running command: '\''/usr/bin/mysqld_safe'\'''
  • exec /usr/bin/mysqld_safe 190306 14:29:15 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ Running command: '/usr/bin/mysqld_safe' 190306 14:29:48 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'. 190306 14:29:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ Running command: '/usr/bin/mysqld_safe' 190306 14:30:46 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'. 190306 14:30:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/ Running command: '/usr/bin/mysqld_safe' 190306 14:32:35 mysqld_safe Logging to '/var/log/kolla/mariadb/mariadb.log'. 190306 14:32:35 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/

and these errors in mariadb.log seem more interesting:

tail /var/lib/docker/volumes/kolla_logs/_data/mariadb/mariadb.log 2019-03-06 14:36:14 140394066910336 [Note] InnoDB: Waiting for purge to start 2019-03-06 14:36:14 140394066910336 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence number 1601271 2019-03-06 14:36:14 140383874938624 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-03-06 14:36:14 140394066910336 [Note] Plugin 'FEEDBACK' is disabled. 2019-03-06 14:36:14 140394066910336 [ERROR] Could not open mysql.plugin table. Some plugins may be not loaded 2019-03-06 14:36:14 140394066033408 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 2019-03-06 14:36:14 140394066910336 [ERROR] Can't open and lock privilege tables: Table 'mysql.servers' doesn't exist 2019-03-06 14:36:14 140394066910336 [Note] Server socket created on IP: ‘x.x.x.x’. 2019-03-06 14:36:14 140394066910336 [ERROR] Fatal error: Can't open and lock privilege tables: Table 'mysql.user' doesn't exist 190306 14:36:14 mysqld_safe mysqld from pid file /var/lib/mysql/mariadb.pid ended

I had no problem running bootstrap-servers or prechecks. One of those was precheck for docker version, as I saw that may have been an issue in the past.

docker version Client: Version: 1.12.6 API version: 1.24 Go version: go1.6.4 Git commit: 78d1802 Built: Tue Jan 10 20:38:45 2017 OS/Arch: linux/amd64

Server: Version: 1.12.6 API version: 1.24 Go version: go1.6.4 Git commit: 78d1802 Built: Tue Jan 10 20:38:45 2017 OS/Arch: linux/amd64

Love to hear any ideas, workarounds, or if anyone has the same result :)

thank you!