Ask Your Question

Revision history [back]

click to hide/show revision 1
initial version

mysql galera wsrep_cluster_size showing 1 on node1 and 0 on node 2

Hi Experts,

I am using 2 nodes for Galera clustering on ubuntu 12.04 and receiving the following output on node 1:

mysql> SHOW STATUS LIKE 'wsrep%'
 -> ;
+----------------------------+--------------------------------------+
| Variable_name | Value |
+----------------------------+--------------------------------------+
| wsrep_local_state_uuid | 7dbb763b-0bea-11e3-0800-36c52d70d461 |
| wsrep_protocol_version | 4 |
| wsrep_last_committed | 1 |
| wsrep_replicated | 1 |
| wsrep_replicated_bytes | 203 |
| wsrep_received | 2 |
| wsrep_received_bytes | 133 |
| wsrep_local_commits | 0 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.500000 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 1.000000 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced (6) |
| wsrep_cert_index_size | 2 |
| wsrep_causal_reads | 0 |
| wsrep_cluster_conf_id | 1 |
| wsrep_cluster_size | 1 |
| wsrep_cluster_state_uuid | 7dbb763b-0bea-11e3-0800-36c52d70d461 |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_index | 0 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 23.2.1(r129) |
| wsrep_ready | ON |
+----------------------------+--------------------------------------+
39 rows in set (0.00 sec)

On 2nd node: I have the output

mysql> SHOW STATUS LIKE 'wsrep%';
+--------------------------+----------------------+
| Variable_name | Value |
+--------------------------+----------------------+
| wsrep_cluster_conf_id | 18446744073709551615 |
| wsrep_cluster_size | 0 |
| wsrep_cluster_state_uuid | |
| wsrep_cluster_status | Disconnected |
| wsrep_connected | OFF |
| wsrep_local_index | 18446744073709551615 |
| wsrep_provider_name | |
| wsrep_provider_vendor | |
| wsrep_provider_version | |
| wsrep_ready | ON |
+--------------------------+----------------------+
10 rows in set (0.00 sec)

Why wsrep cluster size is 1 on node1? shouldn't it be 2 as I am using 2 nodes for clustering? as per few blogs it should be equal to number of nodes used and if it points to 1 then split-brain is in effect. Can you feed on this?

Thanks in advance

mysql Service MySQL start fail on galera wsrep_cluster_size showing 1 on node1 and 0 on node 22nd node

Hi Experts,

I am using 2 nodes for Galera clustering on ubuntu 12.04 and receiving the following output on node 1:

mysql> SHOW STATUS LIKE 'wsrep%'
 -> ;
+----------------------------+--------------------------------------+
| Variable_name | Value |
+----------------------------+--------------------------------------+
| wsrep_local_state_uuid | 7dbb763b-0bea-11e3-0800-36c52d70d461 |
| wsrep_protocol_version | 4 |
| wsrep_last_committed | 1 |
| wsrep_replicated | 1 |
| wsrep_replicated_bytes | 203 |
| wsrep_received | 2 |
| wsrep_received_bytes | 133 |
| wsrep_local_commits | 0 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.500000 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 1.000000 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced (6) |
| wsrep_cert_index_size | 2 |
| wsrep_causal_reads | 0 |
| wsrep_cluster_conf_id | 1 |
| wsrep_cluster_size | 1 |
| wsrep_cluster_state_uuid | 7dbb763b-0bea-11e3-0800-36c52d70d461 |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_index | 0 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 23.2.1(r129) |
| wsrep_ready | ON |
+----------------------------+--------------------------------------+
39 rows in set (0.00 sec)

On 2nd node: I have the output The Service MySQL fails to start - in /var/log/syslog

mysql> SHOW STATUS LIKE 'wsrep%';
+--------------------------+----------------------+
| Variable_name | Value |
+--------------------------+----------------------+
| wsrep_cluster_conf_id | 18446744073709551615 |
| wsrep_cluster_size | 0 |
| wsrep_cluster_state_uuid | |
| wsrep_cluster_status | Disconnected |
| wsrep_connected | OFF |
| wsrep_local_index | 18446744073709551615 |
| wsrep_provider_name | |
| wsrep_provider_vendor | |
| wsrep_provider_version | |
| wsrep_ready | ON |
+--------------------------+----------------------+
10 rows Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: GMCast version 0
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: (c008d869-0c10-11e3-0800-146570776113, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: (c008d869-0c10-11e3-0800-146570776113, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: EVS version 0
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: PC version 0
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: gcomm: connecting to group 'controller_cluster', peer '10.0.0.145:'
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: declaring 130214d4-0c0e-11e3-0800-27da27176580 stable
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: view(view_id(PRIM,130214d4-0c0e-11e3-0800-27da27176580,6) memb {
Aug 23 17:26:23 control2 mysqld: #011130214d4-0c0e-11e3-0800-27da27176580,
Aug 23 17:26:23 control2 mysqld: #011c008d869-0c10-11e3-0800-146570776113,
Aug 23 17:26:23 control2 mysqld: } joined {
Aug 23 17:26:23 control2 mysqld: } left {
Aug 23 17:26:23 control2 mysqld: } partitioned {
Aug 23 17:26:23 control2 mysqld: })
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: gcomm: connected
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Opened channel 'controller_cluster'
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Waiting for SST to complete.
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: STATE EXCHANGE: sent state msg: c059b4f5-0c10-11e3-0800-90932b673ef2
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: STATE EXCHANGE: got state msg: c059b4f5-0c10-11e3-0800-90932b673ef2 from 0 (control)
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: STATE EXCHANGE: got state msg: c059b4f5-0c10-11e3-0800-90932b673ef2 from 1 (control2)
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Quorum results:
Aug 23 17:26:24 control2 mysqld: #011version    = 2,
Aug 23 17:26:24 control2 mysqld: #011component  = PRIMARY,
Aug 23 17:26:24 control2 mysqld: #011conf_id    = 5,
Aug 23 17:26:24 control2 mysqld: #011members    = 1/2 (joined/total),
Aug 23 17:26:24 control2 mysqld: #011act_id     = 0,
Aug 23 17:26:24 control2 mysqld: #011last_appl. = -1,
Aug 23 17:26:24 control2 mysqld: #011protocols  = 0/4/2 (gcs/repl/appl),
Aug 23 17:26:24 control2 mysqld: #011group UUID = 1302924c-0c0e-11e3-0800-d5900352d120
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Flow-control interval: [12, 23]
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: State transfer required:
Aug 23 17:26:24 control2 mysqld: #011Group state: 1302924c-0c0e-11e3-0800-d5900352d120:0
Aug 23 17:26:24 control2 mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: New cluster view: global state: 1302924c-0c0e-11e3-0800-d5900352d120:0, view# 6: Primary, number of nodes: 2, my index: 1, protocol version 2
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Warning] WSREP: Gap in set (0.00 sec)
state sequence. Need state transfer.
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Running: 'wsrep_sst_rsync 'joiner' '10.0.0.146' 'wsrep_sst:password' '/var/lib/mysql/' '/etc/mysql/conf.d/wsrep.cnf' '8132' 2>sst.err'
Aug 23 17:26:26 control2 rsyncd[8182]: rsyncd version 3.0.9 starting, listening on port 4444
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Prepared SST request: rsync|10.0.0.146:4444/rsync_sst
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (1302924c-0c0e-11e3-0800-d5900352d120): 1 (Operation not permitted)
Aug 23 17:26:26 control2 mysqld: #011 at galera/src/replicator_str.cpp:prepare_for_IST():439. IST will be unavailable.
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Node 1 (control2) requested state transfer from '*any*'. Selected 0 (control)(SYNCED) as donor.
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Requesting state transfer: success, donor: 0
Aug 23 17:26:26 control2 rsyncd[8194]: connect from control (10.0.0.145)
Aug 23 17:26:26 control2 rsyncd[8194]: rsync to rsync_sst/ from control (10.0.0.145)
Aug 23 17:26:26 control2 rsyncd[8194]: receiving file list
Aug 23 17:26:37 control2 /etc/init.d/mysql[8262]: 0 processes alive
Aug 23 17:26:37 control2 /etc/init.d/mysql[8262]:

Why wsrep cluster size is 1 on node1? shouldn't it be 2 as I am using 2 nodes for clustering? as per few blogs it should be equal to number of nodes used and if it points to 1 then split-brain is Please suggest. thanks in effect. Can you feed on this? advance

Thanks in advance

Service MySQL start fail on galera 2nd node

Hi Experts,

I am using 2 nodes for Galera clustering on ubuntu 12.04 and receiving the following output on node 1:

mysql> SHOW STATUS LIKE 'wsrep%'
 -> ;
+----------------------------+--------------------------------------+
| Variable_name | Value |
+----------------------------+--------------------------------------+
| wsrep_local_state_uuid | 7dbb763b-0bea-11e3-0800-36c52d70d461 |
| wsrep_protocol_version | 4 |
| wsrep_last_committed | 1 |
| wsrep_replicated | 1 |
| wsrep_replicated_bytes | 203 |
| wsrep_received | 2 |
| wsrep_received_bytes | 133 |
| wsrep_local_commits | 0 |
| wsrep_local_cert_failures | 0 |
| wsrep_local_bf_aborts | 0 |
| wsrep_local_replays | 0 |
| wsrep_local_send_queue | 0 |
| wsrep_local_send_queue_avg | 0.000000 |
| wsrep_local_recv_queue | 0 |
| wsrep_local_recv_queue_avg | 0.500000 |
| wsrep_flow_control_paused | 0.000000 |
| wsrep_flow_control_sent | 0 |
| wsrep_flow_control_recv | 0 |
| wsrep_cert_deps_distance | 1.000000 |
| wsrep_apply_oooe | 0.000000 |
| wsrep_apply_oool | 0.000000 |
| wsrep_apply_window | 1.000000 |
| wsrep_commit_oooe | 0.000000 |
| wsrep_commit_oool | 0.000000 |
| wsrep_commit_window | 1.000000 |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced (6) |
| wsrep_cert_index_size | 2 |
| wsrep_causal_reads | 0 |
| wsrep_cluster_conf_id | 1 |
| wsrep_cluster_size | 1 |
| wsrep_cluster_state_uuid | 7dbb763b-0bea-11e3-0800-36c52d70d461 |
| wsrep_cluster_status | Primary |
| wsrep_connected | ON |
| wsrep_local_index | 0 |
| wsrep_provider_name | Galera |
| wsrep_provider_vendor | Codership Oy <info@codership.com> |
| wsrep_provider_version | 23.2.1(r129) |
| wsrep_ready | ON |
+----------------------------+--------------------------------------+
39 rows in set (0.00 sec)

On 2nd node: The Service MySQL fails to start - in /var/log/syslog

Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: GMCast version 0
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: (c008d869-0c10-11e3-0800-146570776113, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: (c008d869-0c10-11e3-0800-146570776113, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: EVS version 0
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: PC version 0
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: gcomm: connecting to group 'controller_cluster', peer '10.0.0.145:'
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: declaring 130214d4-0c0e-11e3-0800-27da27176580 stable
Aug 23 17:26:23 control2 mysqld: 130823 17:26:23 [Note] WSREP: view(view_id(PRIM,130214d4-0c0e-11e3-0800-27da27176580,6) memb {
Aug 23 17:26:23 control2 mysqld: #011130214d4-0c0e-11e3-0800-27da27176580,
Aug 23 17:26:23 control2 mysqld: #011c008d869-0c10-11e3-0800-146570776113,
Aug 23 17:26:23 control2 mysqld: } joined {
Aug 23 17:26:23 control2 mysqld: } left {
Aug 23 17:26:23 control2 mysqld: } partitioned {
Aug 23 17:26:23 control2 mysqld: })
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: gcomm: connected
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Changing maximum packet size to 64500, resulting msg size: 32636
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Shifting CLOSED -> OPEN (TO: 0)
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Opened channel 'controller_cluster'
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Waiting for SST to complete.
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 2
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID.
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: STATE EXCHANGE: sent state msg: c059b4f5-0c10-11e3-0800-90932b673ef2
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: STATE EXCHANGE: got state msg: c059b4f5-0c10-11e3-0800-90932b673ef2 from 0 (control)
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: STATE EXCHANGE: got state msg: c059b4f5-0c10-11e3-0800-90932b673ef2 from 1 (control2)
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Quorum results:
Aug 23 17:26:24 control2 mysqld: #011version    = 2,
Aug 23 17:26:24 control2 mysqld: #011component  = PRIMARY,
Aug 23 17:26:24 control2 mysqld: #011conf_id    = 5,
Aug 23 17:26:24 control2 mysqld: #011members    = 1/2 (joined/total),
Aug 23 17:26:24 control2 mysqld: #011act_id     = 0,
Aug 23 17:26:24 control2 mysqld: #011last_appl. = -1,
Aug 23 17:26:24 control2 mysqld: #011protocols  = 0/4/2 (gcs/repl/appl),
Aug 23 17:26:24 control2 mysqld: #011group UUID = 1302924c-0c0e-11e3-0800-d5900352d120
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Flow-control interval: [12, 23]
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 0)
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: State transfer required:
Aug 23 17:26:24 control2 mysqld: #011Group state: 1302924c-0c0e-11e3-0800-d5900352d120:0
Aug 23 17:26:24 control2 mysqld: #011Local state: 00000000-0000-0000-0000-000000000000:-1
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Note] WSREP: New cluster view: global state: 1302924c-0c0e-11e3-0800-d5900352d120:0, view# 6: Primary, number of nodes: 2, my index: 1, protocol version 2
Aug 23 17:26:24 control2 mysqld: 130823 17:26:24 [Warning] WSREP: Gap in state sequence. Need state transfer.
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Running: 'wsrep_sst_rsync 'joiner' '10.0.0.146' 'wsrep_sst:password' '/var/lib/mysql/' '/etc/mysql/conf.d/wsrep.cnf' '8132' 2>sst.err'
Aug 23 17:26:26 control2 rsyncd[8182]: rsyncd version 3.0.9 starting, listening on port 4444
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Prepared SST request: rsync|10.0.0.146:4444/rsync_sst
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification.
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Assign initial position for certification: 0, protocol version: 2
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Warning] WSREP: Failed to prepare for incremental state transfer: Local state UUID (00000000-0000-0000-0000-000000000000) does not match group state UUID (1302924c-0c0e-11e3-0800-d5900352d120): 1 (Operation not permitted)
Aug 23 17:26:26 control2 mysqld: #011 at galera/src/replicator_str.cpp:prepare_for_IST():439. IST will be unavailable.
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Node 1 (control2) requested state transfer from '*any*'. Selected 0 (control)(SYNCED) as donor.
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 0)
Aug 23 17:26:26 control2 mysqld: 130823 17:26:26 [Note] WSREP: Requesting state transfer: success, donor: 0
Aug 23 17:26:26 control2 rsyncd[8194]: connect from control (10.0.0.145)
Aug 23 17:26:26 control2 rsyncd[8194]: rsync to rsync_sst/ from control (10.0.0.145)
Aug 23 17:26:26 control2 rsyncd[8194]: receiving file list
Aug 23 17:26:37 control2 /etc/init.d/mysql[8262]: 0 processes alive
Aug 23 17:26:37 control2 /etc/init.d/mysql[8262]:

Please suggest. thanks in advance

PS: On node 2, I am getting the following error message when I run "/usr/sbin/mysqld --wsrep_cluster_address=gcomm://10.0.0.145":

root@control2:~# /usr/sbin/mysqld --wsrep_cluster_address=gcomm://10.0.0.145
130827 11:34:55 [Note] WSREP: Read nil XID from storage engines, skipping position init
130827 11:34:55 [Note] WSREP: wsrep_load(): loading provider library '/usr/lib/galera/libgalera_smm.so'
130827 11:34:55 [Note] WSREP: wsrep_load(): Galera 23.2.1(r129) by Codership Oy <info@codership.com> loaded succesfully.
130827 11:34:55 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
130827 11:34:55 [Note] WSREP: Reusing existing '/var/lib/mysql//galera.cache'.
130827 11:34:55 [Note] WSREP: Passing config to GCS: base_host = 10.0.0.146; gcache.dir = /var/lib/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/lib/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 0.5; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
130827 11:34:55 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
130827 11:34:55 [Note] WSREP: wsrep_sst_grab()
130827 11:34:55 [Note] WSREP: Start replication
130827 11:34:55 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
130827 11:34:55 [Note] WSREP: protonet asio version 0
130827 11:34:55 [Note] WSREP: backend: asio
130827 11:34:55 [Note] WSREP: GMCast version 0
130827 11:34:55 [Note] WSREP: (503b4adc-0f04-11e3-0800-9218a36b64d2, 'tcp://0.0.0.0:4567') listening at tcp://0.0.0.0:4567
130827 11:34:55 [Note] WSREP: (503b4adc-0f04-11e3-0800-9218a36b64d2, 'tcp://0.0.0.0:4567') multicast: , ttl: 1
130827 11:34:55 [Note] WSREP: EVS version 0
130827 11:34:55 [Note] WSREP: PC version 0
130827 11:34:55 [Note] WSREP: gcomm: connecting to group 'controller_cluster', peer '10.0.0.145:'
130827 11:34:55 [ERROR] WSREP: Address already in use
130827 11:34:55 [ERROR] WSREP: failed to open gcomm backend connection: 98: error while trying to listen 'tcp://0.0.0.0:4567?socket.non_blocking=1', asio error 'Address already in use': 98 (Address already in use)
         at gcomm/src/asio_tcp.cpp:listen():755
130827 11:34:55 [ERROR] WSREP: gcs/src/gcs_core.c:gcs_core_open():195: Failed to open backend connection: -98 (Address already in use)
130827 11:34:55 [ERROR] WSREP: gcs/src/gcs.c:gcs_open():1290: Failed to open channel 'controller_cluster' at 'gcomm://10.0.0.145': -98 (Address already in use)
130827 11:34:55 [ERROR] WSREP: gcs connect failed: Address already in use
130827 11:34:55 [ERROR] WSREP: wsrep::connect() failed: 6
130827 11:34:55 [ERROR] Aborting

130827 11:34:55 [Note] WSREP: Service disconnected.
130827 11:34:56 [Note] WSREP: Some threads may fail to exit.
130827 11:34:56 [Note] /usr/sbin/mysqld: Shutdown complete