Ask Your Question
0

mariadb connection aborted

asked 2016-05-02 22:04:07 -0600

Bindas gravatar image

Hi

i have setup openstack liberty with 3 node HA with haproxy, pacemaker , mariadb galera cluster on ubuntu 14.01 LTS. currently iam seeing alot of warning messages in mariadb error logs every time there is traffic going through open stack. every day or two i get "mysql connection is lost" in random open stack service , mainly in neutron. below are the errors im seeing in mariadb error logs. these warning messages are created almost every time traffic goes through openstack. any help will be appreciated 

    160502 22:41:23 [Warning] Aborted connection 652 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:24 [Warning] Aborted connection 655 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:24 [Warning] Aborted connection 656 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:24 [Warning] Aborted connection 657 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:24 [Warning] Aborted connection 654 to db: 'nova' user: 'nova' host: 'haproxy02' (Unknown error)
160502 22:41:25 [Warning] Aborted connection 658 to db: 'nova' user: 'nova' host: 'haproxy02' (Unknown error)
160502 22:41:25 [Warning] Aborted connection 659 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:25 [Warning] Aborted connection 660 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:25 [Warning] Aborted connection 661 to db: 'nova' user: 'nova' host: 'haproxy02' (Unknown error)
160502 22:41:26 [Warning] Aborted connection 664 to db: 'nova' user: 'nova' host: 'haproxy02' (Unknown error)
160502 22:41:26 [Warning] Aborted connection 666 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:26 [Warning] Aborted connection 667 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:26 [Warning] Aborted connection 668 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:26 [Warning] Aborted connection 669 to db: 'nova' user: 'nova' host: 'haproxy02' (Unknown error)
160502 22:41:26 [Warning] Aborted connection 662 to db: 'nova' user: 'nova' host: 'haproxy02' (Unknown error)
160502 22:41:27 [Warning] Aborted connection 651 to db: 'keystone' user: 'keystone' host: 'haproxy02' (Unknown error)
160502 22:41:27 [Warning] Aborted connection 670 to db: 'nova' user: 'nova' host: 'haproxy02' (Unknown error)
160502 22:41:28 [Warning] Aborted connection 674 to db: 'nova' user: 'nova' host: 'haproxy02' (Unknown error)
160502 22:41:54 [Warning] Aborted connection 678 to db: 'keystone' user: 'keystone' host: 'haproxy02' (Unknown error)
160502 22:41:54 [Warning] Aborted connection 679 to db: 'keystone' user: 'keystone' host: 'haproxy02' (Unknown error)
160502 22:41:54 [Warning] Aborted connection 663 to db: 'keystone' user: 'keystone' host: 'haproxy02' (Unknown error)
160502 22:41:54 [Warning] Aborted connection 572 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:55 [Warning] Aborted connection 573 to db: 'neutron' user: 'neutron' host: 'haproxy02' (Unknown error)
160502 22:41:55 [Warning] Aborted connection 680 to db: 'nova' user: 'nova' host: 'haproxy02' (Unknown error)
160502 22:41:55 [Warning] Aborted ...
(more)
edit retag flag offensive close merge delete
add a comment

3 answers

Sort by ยป oldest newest most voted
1

answered 2018-11-14 11:37:32 -0600

vgdub gravatar image

updated 2018-11-14 11:42:48 -0600

Hello All, I am not sure if this is ever fixed or if it was how was this fixed , I have been having the same issue on my Openstack kilo installation. This has severely impacted my entire platform as keystone,Glance,Cinder,Nova is showing this issue which is literally the whole platform and I am not able to create any new VM's/Volumes etc my backend ceph is working absolutely fine.

I would be grateful if you share the resolution for the same. Also my mariadb is quite old too.

Server version: 5.5.45-MariaDB-wsrep MariaDB Server, wsrep_25.11.r4026

181114 17:12:36 [Warning] Aborted connection 272732 to db: 'keystone' user: 'keystone' host: '' (Unknown error) 181114 17:13:41 [Warning] Aborted connection 271127 to db: 'keystone' user: 'keystone' host: '' (Unknown error) 181114 17:13:53 [Warning] Aborted connection 272971 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:13:53 [Warning] Aborted connection 272972 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:13:54 [Warning] Aborted connection 272964 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:13:54 [Warning] Aborted connection 267103 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:13:54 [Warning] Aborted connection 267104 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:13:55 [Warning] Aborted connection 267686 to db: 'nova' user: 'nova' host: '' (Unknown error) 181114 17:13:58 [Warning] Aborted connection 272070 to db: 'keystone' user: 'keystone' host: '' (Unknown error) 181114 17:14:35 [Warning] Aborted connection 273102 to db: 'keystone' user: 'keystone' host: '' (Unknown error) 181114 17:17:36 [Warning] Aborted connection 270964 to db: 'keystone' user: 'keystone' host: '' (Unknown error) 181114 17:17:37 [Warning] Aborted connection 272733 to db: 'keystone' user: 'keystone' host: '' (Unknown error) 181114 17:19:58 [Warning] Aborted connection 272989 to db: 'keystone' user: 'keystone' host: '' (Unknown error) 181114 17:19:58 [Warning] Aborted connection 274142 to db: 'keystone' user: 'keystone' host: '' (Unknown error) 181114 17:20:45 [Warning] Aborted connection 274285 to db: 'keystone' user: 'keystone' host: '' (Unknown error) 181114 17:22:20 [Warning] Aborted connection 274572 to db: 'glance' user: 'glance' host: ' (Unknown error) 181114 17:22:20 [Warning] Aborted connection 274573 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:22:20 [Warning] Aborted connection 269268 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:22:20 [Warning] Aborted connection 274576 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:22:20 [Warning] Aborted connection 274577 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:22:20 [Warning] Aborted connection 274578 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:22:20 [Warning] Aborted connection 274571 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:22:21 [Warning] Aborted connection 272969 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:22:21 [Warning] Aborted connection 272970 to db: 'glance' user: 'glance' host: '' (Unknown error) 181114 17:22:22 [Warning] Aborted connection 274585 to db: 'nova' user: 'nova' host ...

(more)
edit flag offensive delete link more

Comments

Kilo is an end-of-life release. This means that this release won't see any changes anymore, including bugfixes.

Bernd Bausch gravatar imageBernd Bausch ( 2018-11-14 16:43:11 -0600 )edit
add a comment
0

answered 2016-05-03 01:46:51 -0600

amedeo-salvati gravatar image

If you ran galera under haproxy you have to configure properly haproxy, so you have to choose a Virtual IP for haproxy and put this configurations for galera stanza:

listen galera
  bind VIP:3306
  maxconn 4096
  option httpchk
  stick on dst
  stick-table type ip size 4096
  timeout client 90m
  timeout server 90m
      server db01 IP1:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2
        server db02 IP2:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2
        server db03 IP3:3306 backup check fall 5 inter 2000 on-marked-down shutdown-sessions port 9200 rise 2

but before this you have to configure clustercheck to listen on port 9200, this is configuration example for clustercheck:

MYSQL_USERNAME="monitor_user"
MYSQL_PASSWORD="yourpassword"
MYSQL_HOST="localhost"
MYSQL_PORT="3306"

after all, try to login with neutron db user to your VIP and perform some queries.

HTH

edit flag offensive delete link more

Comments

currently this is my galera configuration (see below). do you see any problems what i have currently?

Bindas gravatar imageBindas ( 2016-05-03 12:31:16 -0600 )edit
add a comment
0

answered 2016-05-03 12:33:58 -0600

Bindas gravatar image

updated 2016-05-03 12:34:53 -0600

currently this is my Galera configuration 

listen galera_cluster
  bind VIP:3306
  mode tcp
  balance  source
  option  httpchk
  option tcplog
  server mariadb01 ip1:3306 rise 2 fall 5
  server mariadb02 ip2:3306 backup rise 2 fall 5
  server mariadb03 ip3:3306 backup rise 2 fall 5
edit flag offensive delete link more

Comments

you have to configure all servers as a "backup" endpoint, not only two of three, and most important you have to configure clustercheck to avoid write sql, or DDL statements to one galera node not synced or not joined to cluster!

amedeo-salvati gravatar imageamedeo-salvati ( 2016-05-04 02:37:51 -0600 )edit

you can verify your cluster status connecting to mariadb and running this query:

MariaDB [(none)]> SHOW STATUS like 'wsrep%';

and then you can see:

| wsrep_local_state_comment | Synced | | wsrep_incoming_addresses | IP1:3306,IP2:3306,IP3:3306

amedeo-salvati gravatar imageamedeo-salvati ( 2016-05-04 02:57:06 -0600 )edit
add a comment

Get to know Ask OpenStack

Resources for moderators

Question Tools

1 follower

subscribe to rss feed

Stats

Asked: 2016-05-02 22:04:07 -0600

Seen: 1,814 times

Last updated: Nov 14 '18

Related questions

liberty Self-service bridge can't build

Customize authentication in Horizon

Plugins in openstack neutron

LBaaSv2 neutron agents missing (Liberty)

unable to connect to controller:9696/extensions

Galera breaks then all the controllers go down

Stuck in mariadb install [closed]

PROBLEM: Authentication cannot be scoped to multiple targets

is there something in Mitaka that is not backwards compatible to Liberty?

upgrade fuel and openstack nodes it deployed