software

MySQL distributed message system

April 12, 2016 Architect, Architecture, hardware, MYSQL, replication, software No comments

Based on messages, we create mysql replication platforms , using async message to build strong distributed subscription system.

read this PDF : http://www.vmcd.org/docs/MySQL_async_message.pdf

How to configure AWR system

August 6, 2015 Architect, Architecture, mongodb, MYSQL, NoSQL, rdbms, software No comments

In this article, we introduce myawr and mongoawr system .

Read this PDF, you will learn how to configure them.

How to configure AWR system.

How to configure WEBM

July 15, 2015 Architect, mongodb, MYSQL, software No comments

Architecture of WEBM system.

Reference:

http://www.vmcd.org/2014/10/webm_v2-has-been-released/
http://www.vmcd.org/2014/09/webm-mysql-database-performance-web-monitor/

View this PDF:

http://www.vmcd.org/docs/How%20to%20configure%20WEBM.pdf

MySQL 5.7 milestone

April 22, 2015 Architect, MYSQL, software No comments

MySQL 5.7 will be a great milestone in MySQL total history.
Oracle has released many useful new features in LAB version . MySQL is becoming more similar to Oracle database 🙂

Read this presentation I post on slideshare:

MySQL 5.7 milestone

Architecture of data warehouse which is based on MQ

January 20, 2015 Architect, Architecture, MYSQL, rdbms, software No comments

Recently, we create a mysql data warehouse which is based on message queue.

Most companies must prepare for particular queries in their systems if they consider to split their databases or tables into many pieces.

some problems should be solved in this situation:

1. how to get correct results in-time
2. how to build strong data warehouse for future analyst

These policies were used by YHD

They have already deployed a middle-ware layer to support these requests (between web apps and databases). Every aggregation SQL was splited into many small SQLs and runs in every data nodes.The Final result is the aggregation of these all small SQLs. In this procedure, everything was computed in memory to get high performance.

In data warehouse layer, they use self-defined ETL tools to extract data from different databases to oracle-Exadata platform. Log-based data was put into hadoop and hbase.

I found a new solution

With Canal and Roma (visit previous PDF roma system) , we could build a data warehouse which is based on metaQ. (metaQ is the final storage of roma) , so we can put some simple queries on this data warehouse directly.

We could use MySQL to build this Data warehouse and use original replication in these databases (everything is simple, especially using multiple source feature via MariaDB).

Disadvantages of this architecture: MySQL database is not the best choice for data warehouse. So we need another analyst platform to handle other log-based data.

Most BI systems were built by very expensive commercial software . For small and medium sized companies, this architecture can save a lot of costs.

Client to aggregate messages:

etl_roma

split and merge aggregation :

split1split2

total architecture:

DW_arch

MHA with binlog server

December 31, 2014 Architect, Architecture, MYSQL, software No comments

In this post ,we mainly talk about MHA GTID behavior, we test different cases and find something is different from previous versions .

we have four machines for this test.

environment:


master server: 10.0.128.77
slave server : 10.0.128.110/113/114
port : 3306 

————————————————————————————–

we first do normal failover .

kill master server

we find MHA outputs:


Tue Dec 30 13:32:14 2014 - [warning] Got error on MySQL connect ping: DBI connect(';host=10.0.128.77;port=3306;mysql_connect_timeout=1','dbadmin',...) failed: Lost connection to MySQL server at 'reading initial communication packet', system error: 111 at /usr/share/perl5/vendor_perl/MHA/HealthCheck.pm line 97
2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Dec 30 13:32:14 2014 - [info] Executing secondary network check script: /usr/bin/masterha_secondary_check -s 10.0.128.113 -s 10.0.128.114  --user=root  --master_host=10.0.128.77  --master_ip=10.0.128.77  --master_port=3306 --master_user=dbadmin --master_password=NV7yVBpn88cg4WJCVlZd --ping_type=CONNECT
Tue Dec 30 13:32:14 2014 - [info] Executing SSH check script: exit 0
Tue Dec 30 13:32:14 2014 - [info] HealthCheck: SSH to 10.0.128.77 is reachable.
Monitoring server 10.0.128.113 is reachable, Master is not reachable from 10.0.128.113. OK.
Monitoring server 10.0.128.114 is reachable, Master is not reachable from 10.0.128.114. OK.
Tue Dec 30 13:32:14 2014 - [info] Master is not reachable from all other monitoring servers. Failover should start.
Tue Dec 30 13:32:16 2014 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Dec 30 13:32:16 2014 - [warning] Connection failed 2 time(s)..
Tue Dec 30 13:32:18 2014 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Dec 30 13:32:18 2014 - [warning] Connection failed 3 time(s)..
Tue Dec 30 13:32:20 2014 - [warning] Got error on MySQL connect: 2013 (Lost connection to MySQL server at 'reading initial communication packet', system error: 111)
Tue Dec 30 13:32:20 2014 - [warning] Connection failed 4 time(s)..
Tue Dec 30 13:32:20 2014 - [warning] Master is not reachable from health checker!
Tue Dec 30 13:32:20 2014 - [warning] Master 10.0.128.77(10.0.128.77:3306) is not reachable!
Tue Dec 30 13:32:20 2014 - [warning] SSH is reachable.
Tue Dec 30 13:32:20 2014 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/masterha/test.cnf again, and trying to connect to all servers to check server status..
Tue Dec 30 13:32:20 2014 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Dec 30 13:32:20 2014 - [info] Reading application default configuration from /etc/masterha/test.cnf..
Tue Dec 30 13:32:20 2014 - [info] Reading server configuration from /etc/masterha/test.cnf..
Tue Dec 30 13:32:20 2014 - [info] GTID failover mode = 1
Tue Dec 30 13:32:20 2014 - [info] Dead Servers:
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info] Alive Servers:
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.110(10.0.128.110:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.113(10.0.128.113:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.114(10.0.128.114:3306)
Tue Dec 30 13:32:20 2014 - [info] Alive Slaves:
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.110(10.0.128.110:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.113(10.0.128.113:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.114(10.0.128.114:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info] Checking slave configurations..
Tue Dec 30 13:32:20 2014 - [info]  read_only=1 is not set on slave 10.0.128.110(10.0.128.110:3306).
Tue Dec 30 13:32:20 2014 - [info]  read_only=1 is not set on slave 10.0.128.113(10.0.128.113:3306).
Tue Dec 30 13:32:20 2014 - [info]  read_only=1 is not set on slave 10.0.128.114(10.0.128.114:3306).
Tue Dec 30 13:32:20 2014 - [info] Checking replication filtering settings..
Tue Dec 30 13:32:20 2014 - [info]  Replication filtering check ok.
Tue Dec 30 13:32:20 2014 - [info] Master is down!
Tue Dec 30 13:32:20 2014 - [info] Terminating monitoring script.
Tue Dec 30 13:32:20 2014 - [info] Got exit code 20 (Master dead).
Tue Dec 30 13:32:20 2014 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Tue Dec 30 13:32:20 2014 - [info] Reading application default configuration from /etc/masterha/test.cnf..
Tue Dec 30 13:32:20 2014 - [info] Reading server configuration from /etc/masterha/test.cnf..
Tue Dec 30 13:32:20 2014 - [info] MHA::MasterFailover version 0.56.
Tue Dec 30 13:32:20 2014 - [info] Starting master failover.
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] * Phase 1: Configuration Check Phase..
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] GTID failover mode = 1
Tue Dec 30 13:32:20 2014 - [info] Dead Servers:
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info] Checking master reachability via MySQL(double check)...
Tue Dec 30 13:32:20 2014 - [info]  ok.
Tue Dec 30 13:32:20 2014 - [info] Alive Servers:
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.110(10.0.128.110:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.113(10.0.128.113:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.114(10.0.128.114:3306)
Tue Dec 30 13:32:20 2014 - [info] Alive Slaves:
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.110(10.0.128.110:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.113(10.0.128.113:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.114(10.0.128.114:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info] Starting GTID based failover.
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] ** Phase 1: Configuration Check Phase completed.
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] * Phase 2: Dead Master Shutdown Phase..
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] Forcing shutdown so that applications never connect to the current master..
Tue Dec 30 13:32:20 2014 - [info] Executing master IP deactivation script:
Tue Dec 30 13:32:20 2014 - [info]   /usr/local/dbadmin/mhascripts/master_ip_failover_pajk01.pl --orig_master_host=10.0.128.77 --orig_master_ip=10.0.128.77 --orig_master_port=3306 --command=stopssh --ssh_user=root  
Disabling the VIP on old master: 10.0.128.77 
RTNETLINK answers: Cannot assign requested address
Tue Dec 30 13:32:20 2014 - [info]  done.
Tue Dec 30 13:32:20 2014 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Tue Dec 30 13:32:20 2014 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] * Phase 3: Master Recovery Phase..
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] The latest binary log file/position on all slaves is master-bin.000001:3679
Tue Dec 30 13:32:20 2014 - [info] Retrieved Gtid Set: a861ddd1-8f4e-11e4-aeac-86f5fb218bef:1-15
Tue Dec 30 13:32:20 2014 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.110(10.0.128.110:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.113(10.0.128.113:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.114(10.0.128.114:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info] The oldest binary log file/position on all slaves is master-bin.000001:3679
Tue Dec 30 13:32:20 2014 - [info] Retrieved Gtid Set: a861ddd1-8f4e-11e4-aeac-86f5fb218bef:1-15
Tue Dec 30 13:32:20 2014 - [info] Oldest slaves:
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.110(10.0.128.110:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.113(10.0.128.113:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info]   10.0.128.114(10.0.128.114:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 13:32:20 2014 - [info]     GTID ON
Tue Dec 30 13:32:20 2014 - [info]     Replicating from 10.0.128.77(10.0.128.77:3306)
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] * Phase 3.3: Determining New Master Phase..
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] Searching new master from slaves..
Tue Dec 30 13:32:20 2014 - [info]  Candidate masters from the configuration file:
Tue Dec 30 13:32:20 2014 - [info]  Non-candidate masters:
Tue Dec 30 13:32:20 2014 - [info] New master is 10.0.128.110(10.0.128.110:3306)
Tue Dec 30 13:32:20 2014 - [info] Starting master failover..
Tue Dec 30 13:32:20 2014 - [info] 
From:
10.0.128.77(10.0.128.77:3306) (current master)
 +--10.0.128.110(10.0.128.110:3306)
 +--10.0.128.113(10.0.128.113:3306)
 +--10.0.128.114(10.0.128.114:3306)

To:
10.0.128.110(10.0.128.110:3306) (new master)
 +--10.0.128.113(10.0.128.113:3306)
 +--10.0.128.114(10.0.128.114:3306)
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info] * Phase 3.3: New Master Recovery Phase..
Tue Dec 30 13:32:20 2014 - [info] 
Tue Dec 30 13:32:20 2014 - [info]  Waiting all logs to be applied.. 
Tue Dec 30 13:32:20 2014 - [info]   done.
Tue Dec 30 13:32:20 2014 - [info] Getting new master's binlog name and position..
Tue Dec 30 13:32:20 2014 - [info]  mysql-bin.000001:4142
Tue Dec 30 13:32:20 2014 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.128.110', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='rep', MASTER_PASSWORD='xxx';
Tue Dec 30 13:32:20 2014 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000001, 4142, a861ddd1-8f4e-11e4-aeac-86f5fb218bef:1-15
Tue Dec 30 13:32:20 2014 - [info] Executing master IP activate script:
Tue Dec 30 13:32:20 2014 - [info]   /usr/local/dbadmin/mhascripts/master_ip_failover_pajk01.pl --command=start --ssh_user=root --orig_master_host=10.0.128.77 --orig_master_ip=10.0.128.77 --orig_master_port=3306 --new_master_host=10.0.128.110 --new_master_ip=10.0.128.110 --new_master_port=3306 --new_master_user='dbadmin' --new_master_password='NV7yVBpn88cg4WJCVlZd'  
Set read_only=0 on the new master.
Enabling the VIP - 10.0.128.35/24 on the new master - 10.0.128.110 
Tue Dec 30 13:32:23 2014 - [info]  OK.
Tue Dec 30 13:32:23 2014 - [info] ** Finished master recovery successfully.
Tue Dec 30 13:32:23 2014 - [info] * Phase 3: Master Recovery Phase completed.
Tue Dec 30 13:32:23 2014 - [info] 
Tue Dec 30 13:32:23 2014 - [info] * Phase 4: Slaves Recovery Phase..
Tue Dec 30 13:32:23 2014 - [info] 
Tue Dec 30 13:32:23 2014 - [info] 
Tue Dec 30 13:32:23 2014 - [info] * Phase 4.1: Starting Slaves in parallel..
Tue Dec 30 13:32:23 2014 - [info] 
Tue Dec 30 13:32:23 2014 - [info] -- Slave recovery on host 10.0.128.113(10.0.128.113:3306) started, pid: 31607. Check tmp log /var/log/masterha/test/10.0.128.113_3306_20141230133220.log if it takes time..
Tue Dec 30 13:32:23 2014 - [info] -- Slave recovery on host 10.0.128.114(10.0.128.114:3306) started, pid: 31608. Check tmp log /var/log/masterha/test/10.0.128.114_3306_20141230133220.log if it takes time..
Tue Dec 30 13:32:24 2014 - [info] 
Tue Dec 30 13:32:24 2014 - [info] Log messages from 10.0.128.113 ...
Tue Dec 30 13:32:24 2014 - [info] 
Tue Dec 30 13:32:23 2014 - [info]  Resetting slave 10.0.128.113(10.0.128.113:3306) and starting replication from the new master 10.0.128.110(10.0.128.110:3306)..
Tue Dec 30 13:32:24 2014 - [info]  Executed CHANGE MASTER.
Tue Dec 30 13:32:24 2014 - [info]  Slave started.
Tue Dec 30 13:32:24 2014 - [info]  gtid_wait(a861ddd1-8f4e-11e4-aeac-86f5fb218bef:1-15) completed on 10.0.128.113(10.0.128.113:3306). Executed 0 events.
Tue Dec 30 13:32:24 2014 - [info] End of log messages from 10.0.128.113.
Tue Dec 30 13:32:24 2014 - [info] -- Slave on host 10.0.128.113(10.0.128.113:3306) started.
Tue Dec 30 13:32:24 2014 - [info] 
Tue Dec 30 13:32:24 2014 - [info] Log messages from 10.0.128.114 ...
Tue Dec 30 13:32:24 2014 - [info] 
Tue Dec 30 13:32:23 2014 - [info]  Resetting slave 10.0.128.114(10.0.128.114:3306) and starting replication from the new master 10.0.128.110(10.0.128.110:3306)..
Tue Dec 30 13:32:24 2014 - [info]  Executed CHANGE MASTER.
Tue Dec 30 13:32:24 2014 - [info]  Slave started.
Tue Dec 30 13:32:24 2014 - [info]  gtid_wait(a861ddd1-8f4e-11e4-aeac-86f5fb218bef:1-15) completed on 10.0.128.114(10.0.128.114:3306). Executed 0 events.
Tue Dec 30 13:32:24 2014 - [info] End of log messages from 10.0.128.114.
Tue Dec 30 13:32:24 2014 - [info] -- Slave on host 10.0.128.114(10.0.128.114:3306) started.
Tue Dec 30 13:32:24 2014 - [info] All new slave servers recovered successfully.
Tue Dec 30 13:32:24 2014 - [info] 
Tue Dec 30 13:32:24 2014 - [info] * Phase 5: New master cleanup phase..
Tue Dec 30 13:32:24 2014 - [info] 
Tue Dec 30 13:32:24 2014 - [info] Resetting slave info on the new master..
Tue Dec 30 13:32:24 2014 - [info]  10.0.128.110: Resetting slave info succeeded.
Tue Dec 30 13:32:24 2014 - [info] Master failover to 10.0.128.110(10.0.128.110:3306) completed successfully.
Tue Dec 30 13:32:24 2014 - [info] 

----- Failover Report -----

test: MySQL Master failover 10.0.128.77(10.0.128.77:3306) to 10.0.128.110(10.0.128.110:3306) succeeded

Master 10.0.128.77(10.0.128.77:3306) is down!

Check MHA Manager logs at a3-relay00.sh for details.

Started automated(non-interactive) failover.
Invalidated master IP address on 10.0.128.77(10.0.128.77:3306)
Selected 10.0.128.110(10.0.128.110:3306) as a new master.
10.0.128.110(10.0.128.110:3306): OK: Applying all logs succeeded.
10.0.128.110(10.0.128.110:3306): OK: Activated master IP address.
10.0.128.113(10.0.128.113:3306): OK: Slave started, replicating from 10.0.128.110(10.0.128.110:3306)
10.0.128.114(10.0.128.114:3306): OK: Slave started, replicating from 10.0.128.110(10.0.128.110:3306)
10.0.128.110(10.0.128.110:3306): Resetting slave info succeeded.
Master failover to 10.0.128.110(10.0.128.110:3306) completed successfully.

failover to new master 10.0.128.110:

create new database liuyang3 on 10.0.128.110:


mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| liuyang2           |
| mysql              |
| performance_schema |
| test               |
+--------------------+
5 rows in set (0.00 sec)

mysql> create database liuyang3;
Query OK, 1 row affected (0.00 sec)



show all slaves status :

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.0.128.110
                  Master_User: rep
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 4296
               Relay_Log_File: hadoop-vm-datanode3-relay-bin.000002
                Relay_Log_Pos: 562
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 4296
              Relay_Log_Space: 780
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2
                  Master_UUID: 909ee6b1-8f51-11e4-aebf-00163f00801f
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 909ee6b1-8f51-11e4-aebf-00163f00801f:1
            Executed_Gtid_Set: 909ee6b1-8f51-11e4-aebf-00163f00801f:1,
a861ddd1-8f4e-11e4-aeac-86f5fb218bef:1-15
                Auto_Position: 1
1 row in set (0.00 sec)

start from new GTID — This GTID is new and we can also see old master GTID


#cat /data/mysql/data/auto.cnf 
[auto]
server-uuid=909ee6b1-8f51-11e4-aebf-00163f00801f

this GTID was gathered when slave has been started.

start old master 10.0.128.77 again and add it to the cluster.


Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 1
Server version: 5.6.17-log MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CHANGE MASTER TO MASTER_HOST='10.0.128.110', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='rep', MASTER_PASSWORD='rep';
Query OK, 0 rows affected, 2 warnings (0.32 sec)

mysql> start slave ;
Query OK, 0 rows affected, 1 warning (0.06 sec)


mysql> show databases;
+--------------------+
| Database           |
+--------------------+
| information_schema |
| liuyang2           |
| liuyang3           |
| mysql              |
| performance_schema |
| test               |
+--------------------+
6 rows in set (0.00 sec)

database liuyang3 appeared.

———————————————————————–

initialize cluster:

drop all database;


master: 10.0.128.110
slave: 10.0.128.77/114/113


load data into master. we keep one specific slave has lag and set “sync_binlog=1” on all slaves.


mysql>  delimiter $$
mysql> 
mysql>  CREATE PROCEDURE myprocedure()
    ->  BEGIN
    ->          DECLARE i INT DEFAULT 1;
    -> 
    ->          CREATE  TABLE test
    ->              (ascii_code int, ascii_char CHAR(1));
    -> 
    ->          WHILE (i<=10000) DO
    ->                 INSERT INTO test VALUES(i,CHAR(i));
    ->                 SET i=i+1;
    ->          END WHILE;
    -> 
    ->  END$$
Query OK, 0 rows affected (0.03 sec)

mysql>  delimiter ;
mysql> 
mysql>  call myprocedure();
Query OK, 1 row affected, 1 warning (5 min 53.61 sec)

mysql> select count(*) from liuyang1.test;
+----------+
| count(*) |
+----------+
|    10000 |
+----------+
1 row in set (0.01 sec)

10.0.128.113/114 are both consistent with master

mysql> select count(*) from liuyang1.test;
+----------+
| count(*) |
+----------+
|    10000 |
+----------+
1 row in set (0.01 sec)


           Retrieved_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-10003
            Executed_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-10003
                Auto_Position: 1

10.0.128.77 has a huge lag:

           Retrieved_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-2548
            Executed_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-1979
                Auto_Position: 1
                
  

now kill master server:

oldest slave will change master to 10.0.128.113:


Tue Dec 30 16:36:15 2014 - [info] Retrieved Gtid Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-10003
Tue Dec 30 16:36:15 2014 - [info] Latest slaves (Slaves that received relay log files to the latest):
Tue Dec 30 16:36:15 2014 - [info]   10.0.128.113(10.0.128.113:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 16:36:15 2014 - [info]     GTID ON
Tue Dec 30 16:36:15 2014 - [info]     Replicating from 10.0.128.110(10.0.128.110:3306)
Tue Dec 30 16:36:15 2014 - [info]   10.0.128.114(10.0.128.114:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 16:36:15 2014 - [info]     GTID ON
Tue Dec 30 16:36:15 2014 - [info]     Replicating from 10.0.128.110(10.0.128.110:3306)
Tue Dec 30 16:36:15 2014 - [info] The oldest binary log file/position on all slaves is mysql-bin.000001:885010
Tue Dec 30 16:36:15 2014 - [info] Retrieved Gtid Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-2864
Tue Dec 30 16:36:15 2014 - [info] Oldest slaves:
Tue Dec 30 16:36:15 2014 - [info]   10.0.128.77(10.0.128.77:3306)  Version=5.6.17-log (oldest major version between slaves) log-bin:enabled
Tue Dec 30 16:36:15 2014 - [info]     GTID ON
Tue Dec 30 16:36:15 2014 - [info]     Replicating from 10.0.128.110(10.0.128.110:3306)
Tue Dec 30 16:36:15 2014 - [info] 
Tue Dec 30 16:36:15 2014 - [info] * Phase 3.3: Determining New Master Phase..
Tue Dec 30 16:36:15 2014 - [info] 
Tue Dec 30 16:36:15 2014 - [info] Searching new master from slaves..
Tue Dec 30 16:36:15 2014 - [info]  Candidate masters from the configuration file:
Tue Dec 30 16:36:15 2014 - [info]  Non-candidate masters:
Tue Dec 30 16:36:15 2014 - [info] New master is 10.0.128.113(10.0.128.113:3306)

and MHA will control 10.0.128.77 to do this command:


SELECT WAIT_UNTIL_SQL_THREAD_AFTER_GTIDS('4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-10003',0) AS Result 

</per>

show slave status\G;


<pre class="brush: text; gutter: false; first-line: 1; highlight: []; html-script: false">
shows:

           Retrieved_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:2786-5067
            Executed_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-3929
                Auto_Position: 1


this means oldest slave 10.0.128.77 changes to 10.0.128.113 for the new master . Star from transaction id 2785:

Retrieved_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-2785 in old master
Retrieved_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:2786-5067 in current master

wait a long time ……

           Retrieved_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:2786-10003
            Executed_Gtid_Set: 4f1dfbe6-8ffc-11e4-b318-00163f00801f:1-10003
                Auto_Position: 1
                
                
mysql> select count(*) from  liuyang1.test;
+----------+
| count(*) |
+----------+
|    10000 |
+----------+

oldest slave 10.0.128.77 was OK !

—————————-

TEST more details:

We test two cases:

1. just 10.0.128.77 has lag — make 10.0.128.77 as candidate master

2. all slaves have lag — make 10.0.128.114 as candidate master

initialize environment

all slave should get new GTID:

master: 10.0.128.113

slave: 10.0.128.110/77/114

make all slaves have lag manually .

First rerun procedure to initialize data


mysql>  call myprocedure();
Query OK, 1 row affected, 1 warning (1 min 9.98 sec)

kill master MHA will choose a new candidate master and do slave recovery:

10.0.128.110/114 position:

           Retrieved_Gtid_Set: f776aa4d-908d-11e4-b6ce-00163f008023:1-10006
            Executed_Gtid_Set: f776aa4d-908d-11e4-b6ce-00163f008023:1-10006
                Auto_Position: 1
                

10.0.128.77 position:

           Retrieved_Gtid_Set: f776aa4d-908d-11e4-b6ce-00163f008023:1-1269
            Executed_Gtid_Set: f776aa4d-908d-11e4-b6ce-00163f008023:1-684
                Auto_Position: 1
  
 
  +--10.0.128.114(10.0.128.114:3306)
Wed Dec 31 10:09:50 2014 - [info] 
Wed Dec 31 10:09:50 2014 - [info] * Phase 3.3: New Master Recovery Phase..
Wed Dec 31 10:09:50 2014 - [info] 
Wed Dec 31 10:09:50 2014 - [info]  Waiting all logs to be applied.. 
Wed Dec 31 10:10:26 2014 - [info]   done.
Wed Dec 31 10:11:28 2014 - [info]  Replicating from the latest slave 10.0.128.110(10.0.128.110:3306) and waiting to apply..
Wed Dec 31 10:11:28 2014 - [info]  Waiting all logs to be applied on the latest slave.. 
Wed Dec 31 10:11:28 2014 - [info]  Resetting slave 10.0.128.77(10.0.128.77:3306) and starting replication from the new master 10.0.128.110(10.0.128.110:3306)..
Wed Dec 31 10:11:28 2014 - [info]  Executed CHANGE MASTER.
Wed Dec 31 10:11:28 2014 - [info]  Slave started.
Wed Dec 31 10:11:28 2014 - [info]  Waiting to execute all relay logs on 10.0.128.77(10.0.128.77:3306)..
 
 
 
 mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: System lock
                  Master_Host: 10.0.128.110
                  Master_User: rep
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 487380
               Relay_Log_File: a3-oracle-128-77-relay-bin.000002
                Relay_Log_Pos: 23897
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 458828
              Relay_Log_Space: 52664
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 333
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2
                  Master_UUID: 354a7e91-908e-11e4-b6d0-00163f00801f
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Reading event from the relay log
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: f776aa4d-908d-11e4-b6ce-00163f008023:1539-1722
            Executed_Gtid_Set: f776aa4d-908d-11e4-b6ce-00163f008023:1-1622
                Auto_Position: 1
 
 

MHA will control 77 to change master to 10.0.128.110 (prior is 10.0.128.113)

so 77 start from new transaction (old is 1-1538, new is 1539-1722)

we can wait until reach 10006 on lastest

 
 dbadmin     | 10.0.128.25:17331 | NULL | Query   |   60 | Waiting for the slave SQL thread to advance position | SELECT MASTER_POS_WAIT('mysql-bin.000001','2831783',0) AS Result 

in 10.0.128.110 show master status:


| mysql-bin.000001 |  2831783 |              |                  | f776aa4d-908d-11e4-b6ce-00163f008023:1-10006 |

until 77 reached transaction 10006 ,MHA will continue:


Wed Dec 31 10:36:14 2014 - [info]  master_pos_wait(mysql-bin.000001:2831783) completed on 10.0.128.77(10.0.128.77:3306). Executed 4229 events.
Wed Dec 31 10:36:14 2014 - [info]   done.
Wed Dec 31 10:36:14 2014 - [info]   done.
Wed Dec 31 10:36:14 2014 - [info] Getting new master's binlog name and position..
Wed Dec 31 10:36:14 2014 - [info]  master-bin.000001:2831783
Wed Dec 31 10:36:14 2014 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.128.77', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='rep', MASTER_PASSWORD='xxx';
Wed Dec 31 10:36:14 2014 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: master-bin.000001, 2831783, f776aa4d-908d-11e4-b6ce-00163f008023:1-10006
Wed Dec 31 10:36:14 2014 - [info] Executing master IP activate script:
Wed Dec 31 10:36:14 2014 - [info]   /usr/local/dbadmin/mhascripts/master_ip_failover_pajk01.pl --command=start --ssh_user=root --orig_master_host=10.0.128.113 --orig_master_ip=10.0.128.113 --orig_master_port=3306 --new_master_host=10.0.128.77 --new_master_ip=10.0.128.77 --new_master_port=3306 --new_master_user='dbadmin' --new_master_password='NV7yVBpn88cg4WJCVlZd'  
Set read_only=0 on the new master.
Enabling the VIP - 10.0.128.35/24 on the new master - 10.0.128.77 
sysfs read broadcast value: No such file or directory
falling back to default broadcast value
Wed Dec 31 10:36:17 2014 - [info]  OK.
Wed Dec 31 10:36:17 2014 - [info] ** Finished master recovery successfully.
Wed Dec 31 10:36:17 2014 - [info] * Phase 3: Master Recovery Phase completed.
Wed Dec 31 10:36:17 2014 - [info] 
Wed Dec 31 10:36:17 2014 - [info] * Phase 4: Slaves Recovery Phase..
Wed Dec 31 10:36:17 2014 - [info] 
Wed Dec 31 10:36:17 2014 - [info] 
Wed Dec 31 10:36:17 2014 - [info] * Phase 4.1: Starting Slaves in parallel..
Wed Dec 31 10:36:17 2014 - [info] 
Wed Dec 31 10:36:17 2014 - [info] -- Slave recovery on host 10.0.128.110(10.0.128.110:3306) started, pid: 13177. Check tmp log /var/log/masterha/test/10.0.128.110_3306_20141231100948.log if it takes time..
Wed Dec 31 10:36:17 2014 - [info] -- Slave recovery on host 10.0.128.114(10.0.128.114:3306) started, pid: 13178. Check tmp log /var/log/masterha/test/10.0.128.114_3306_20141231100948.log if it takes time..
Wed Dec 31 10:36:18 2014 - [info] 
Wed Dec 31 10:36:18 2014 - [info] Log messages from 10.0.128.114 ...
Wed Dec 31 10:36:18 2014 - [info] 
Wed Dec 31 10:36:17 2014 - [info]  Resetting slave 10.0.128.114(10.0.128.114:3306) and starting replication from the new master 10.0.128.77(10.0.128.77:3306)..
Wed Dec 31 10:36:17 2014 - [info]  Executed CHANGE MASTER.
Wed Dec 31 10:36:18 2014 - [info]  Slave started.
Wed Dec 31 10:36:18 2014 - [info]  gtid_wait(f776aa4d-908d-11e4-b6ce-00163f008023:1-10006) completed on 10.0.128.114(10.0.128.114:3306). Executed 0 events.
Wed Dec 31 10:36:18 2014 - [info] End of log messages from 10.0.128.114.
Wed Dec 31 10:36:18 2014 - [info] -- Slave on host 10.0.128.114(10.0.128.114:3306) started.
Wed Dec 31 10:36:18 2014 - [info] 
Wed Dec 31 10:36:18 2014 - [info] Log messages from 10.0.128.110 ...
Wed Dec 31 10:36:18 2014 - [info] 
Wed Dec 31 10:36:17 2014 - [info]  Resetting slave 10.0.128.110(10.0.128.110:3306) and starting replication from the new master 10.0.128.77(10.0.128.77:3306)..
Wed Dec 31 10:36:18 2014 - [info]  Executed CHANGE MASTER.
Wed Dec 31 10:36:18 2014 - [info]  Slave started.
Wed Dec 31 10:36:18 2014 - [info]  gtid_wait(f776aa4d-908d-11e4-b6ce-00163f008023:1-10006) completed on 10.0.128.110(10.0.128.110:3306). Executed 0 events.
Wed Dec 31 10:36:18 2014 - [info] End of log messages from 10.0.128.110.
Wed Dec 31 10:36:18 2014 - [info] -- Slave on host 10.0.128.110(10.0.128.110:3306) started.
Wed Dec 31 10:36:18 2014 - [info] All new slave servers recovered successfully.
Wed Dec 31 10:36:18 2014 - [info] 
Wed Dec 31 10:36:18 2014 - [info] * Phase 5: New master cleanup phase..
Wed Dec 31 10:36:18 2014 - [info] 
Wed Dec 31 10:36:18 2014 - [info] Resetting slave info on the new master..
Wed Dec 31 10:36:18 2014 - [info]  10.0.128.77: Resetting slave info succeeded.
Wed Dec 31 10:36:18 2014 - [info] Master failover to 10.0.128.77(10.0.128.77:3306) completed successfully.
Wed Dec 31 10:36:18 2014 - [info] 

try to recover other slaves but other slave are both newest. So change master to 77 directly.

——————————————————————–

2. all slaves have lag and 114 is candidate master.

slaves will do not merge master’s binlog (I don’t know why)

initialize cluster again:

master : 10.0.128.113

slave : 10.0.128.114(candidate master)/77/110

drop table test ;

Run procedure again:

we got final result .

mysql> select count(*) from liuyang1.test;
+----------+
| count(*) |
+----------+
|     5852 |
+----------+

we lost 4148 records.

Wed Dec 31 10:50:33 2014 - [info] * Phase 3.3: New Master Recovery Phase..
Wed Dec 31 10:50:33 2014 - [info] 
Wed Dec 31 10:50:33 2014 - [info]  Waiting all logs to be applied.. 
Wed Dec 31 10:50:33 2014 - [info]   done.
Wed Dec 31 10:51:35 2014 - [info] Getting new master's binlog name and position..
Wed Dec 31 10:51:35 2014 - [info]  mysql-bin.000001:4488549
Wed Dec 31 10:51:35 2014 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='10.0.128.114', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='rep', MASTER_PASSWORD='xxx';
Wed Dec 31 10:51:35 2014 - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin.000001, 4488549, 5d04dd5c-908e-11e4-b6d1-86f5fb218bef:1,
f776aa4d-908d-11e4-b6ce-00163f008023:1-15861
Wed Dec 31 10:51:35 2014 - [info] Executing master IP activate script:
Wed Dec 31 10:51:35 2014 - [info]   /usr/local/dbadmin/mhascripts/master_ip_failover_pajk01.pl --command=start --ssh_user=root --orig_master_host=10.0.128.113 --orig_master_ip=10.0.128.113 --orig_master_port=3306 --new_master_host=10.0.128.114 --new_master_ip=10.0.128.114 --new_master_port=3306 --new_master_user='dbadmin' --new_master_password='NV7yVBpn88cg4WJCVlZd'  
Set read_only=0 on the new master.
Enabling the VIP - 10.0.128.35/24 on the new master - 10.0.128.114 
Wed Dec 31 10:51:38 2014 - [info]  OK.
Wed Dec 31 10:51:38 2014 - [info] ** Finished master recovery successfully.
Wed Dec 31 10:51:38 2014 - [info] * Phase 3: Master Recovery Phase completed.
Wed Dec 31 10:51:38 2014 - [info] 
Wed Dec 31 10:51:38 2014 - [info] * Phase 4: Slaves Recovery Phase..
Wed Dec 31 10:51:38 2014 - [info] 
Wed Dec 31 10:51:38 2014 - [info] 
Wed Dec 31 10:51:38 2014 - [info] * Phase 4.1: Starting Slaves in parallel..
Wed Dec 31 10:51:38 2014 - [info] 
Wed Dec 31 10:51:38 2014 - [info] -- Slave recovery on host 10.0.128.77(10.0.128.77:3306) started, pid: 13508. Check tmp log /var/log/masterha/test/10.0.128.77_3306_20141231105020.log if it takes time..
Wed Dec 31 10:51:38 2014 - [info] -- Slave recovery on host 10.0.128.110(10.0.128.110:3306) started, pid: 13509. Check tmp log /var/log/masterha/test/10.0.128.110_3306_20141231105020.log if it takes time..
Wed Dec 31 11:01:34 2014 - [info] 
Wed Dec 31 11:01:34 2014 - [info] Log messages from 10.0.128.77 ...
Wed Dec 31 11:01:34 2014 - [info] 
Wed Dec 31 10:51:38 2014 - [info]  Resetting slave 10.0.128.77(10.0.128.77:3306) and starting replication from the new master 10.0.128.114(10.0.128.114:3306)..
Wed Dec 31 10:52:41 2014 - [info]  Executed CHANGE MASTER.
Wed Dec 31 10:52:41 2014 - [info]  Slave started.
Wed Dec 31 11:01:34 2014 - [info]  gtid_wait(5d04dd5c-908e-11e4-b6d1-86f5fb218bef:1,
f776aa4d-908d-11e4-b6ce-00163f008023:1-15861) completed on 10.0.128.77(10.0.128.77:3306). Executed 855 events.
Wed Dec 31 11:01:34 2014 - [info] End of log messages from 10.0.128.77.
Wed Dec 31 11:01:34 2014 - [info] -- Slave on host 10.0.128.77(10.0.128.77:3306) started.
Wed Dec 31 11:02:03 2014 - [info] 
Wed Dec 31 11:02:03 2014 - [info] Log messages from 10.0.128.110 ...
Wed Dec 31 11:02:03 2014 - [info] 
Wed Dec 31 10:51:38 2014 - [info]  Resetting slave 10.0.128.110(10.0.128.110:3306) and starting replication from the new master 10.0.128.114(10.0.128.114:3306)..
Wed Dec 31 10:52:41 2014 - [info]  Executed CHANGE MASTER.
Wed Dec 31 10:52:41 2014 - [info]  Slave started.
Wed Dec 31 11:02:03 2014 - [info]  gtid_wait(5d04dd5c-908e-11e4-b6d1-86f5fb218bef:1,
f776aa4d-908d-11e4-b6ce-00163f008023:1-15861) completed on 10.0.128.110(10.0.128.110:3306). Executed 142 events.
Wed Dec 31 11:02:03 2014 - [info] End of log messages from 10.0.128.110.
Wed Dec 31 11:02:03 2014 - [info] -- Slave on host 10.0.128.110(10.0.128.110:3306) started.
Wed Dec 31 11:02:03 2014 - [info] All new slave servers recovered successfully.
Wed Dec 31 11:02:03 2014 - [info] 
Wed Dec 31 11:02:03 2014 - [info] * Phase 5: New master cleanup phase..
Wed Dec 31 11:02:03 2014 - [info] 
Wed Dec 31 11:02:03 2014 - [info] Resetting slave info on the new master..
Wed Dec 31 11:02:03 2014 - [info]  10.0.128.114: Resetting slave info succeeded.
Wed Dec 31 11:02:03 2014 - [info] Master failover to 10.0.128.114(10.0.128.114:3306) completed successfully.
Wed Dec 31 11:02:03 2014 - [info] 

10.0.128.114 do not merge newest binlog from 10.0.128.113 (although it can ssh to 113 server )

————————————————————————————-

add binlog server and make 77 server as candidate master

[binlog1]
hostname=10.0.128.252

10.0.128.252 is manager node machine. we set binlog server in this machine.

#ls -l /data/mysql/data/
total 2896
-rw-rw-r– 1 mysql mysql 2960898 Dec 31 13:57 mysql-bin.000001

drop test table

run procedure again

we check lastest slave

10.0.128.114 :

           Retrieved_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-3357
            Executed_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-3356
  

10.0.128.77:

           Retrieved_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-102
            Executed_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-48
            

10.0.128.110:

           Retrieved_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-344
            Executed_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-270           

and all slaves have lag : 10.0.128.77>10.0.128.110>10.0.128.114 (lastest slave)

now we kill master server:

 Waiting to execute all relay logs on 10.0.128.77(10.0.128.77:3306)..
Wed Dec 31 13:59:28 2014 - [info]  master_pos_wait(mysql-bin.000001:1219763) completed on 10.0.128.77(10.0.128.77:3306). Executed 37 events.
Wed Dec 31 13:59:28 2014 - [info]   done.
Wed Dec 31 13:59:28 2014 - [info]   done.
Wed Dec 31 13:59:28 2014 - [info] -- Saving binlog from host 10.0.128.252 started, pid: 15893
Wed Dec 31 13:59:29 2014 - [info] 
Wed Dec 31 13:59:29 2014 - [info] Log messages from 10.0.128.252 ...
Wed Dec 31 13:59:29 2014 - [info] 
Wed Dec 31 13:59:28 2014 - [info] Fetching binary logs from binlog server 10.0.128.252..
Wed Dec 31 13:59:28 2014 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=mysql-bin.000001  --start_pos=1275770 --output_file=/var/log/masterha/test/saved_binlog_binlog1_20141231135735.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.6.17-log  --binlog_dir=/data/mysql/data 
  Creating /var/log/masterha/test if not exists..    ok.
 Concat binary/relay logs from mysql-bin.000001 pos 1275770 to mysql-bin.000001 EOF into /var/log/masterha/test/saved_binlog_binlog1_20141231135735.binlog ..
 Concat succeeded.
Wed Dec 31 13:59:29 2014 - [info] scp from root@10.0.128.252:/var/log/masterha/test/saved_binlog_binlog1_20141231135735.binlog to local:/var/log/masterha/test/saved_binlog_10.0.128.252_binlog1_20141231135735.binlog succeeded.
Wed Dec 31 13:59:29 2014 - [info] End of log messages from 10.0.128.252.
Wed Dec 31 13:59:29 2014 - [info] Saved mysqlbinlog size from 10.0.128.252 is 4885833 bytes.
Wed Dec 31 13:59:29 2014 - [info] Applying differential binlog /var/log/masterha/test/saved_binlog_10.0.128.252_binlog1_20141231135735.binlog ..
Wed Dec 31 13:59:39 2014 - [info] Differential log apply from binlog server succeeded.
Wed Dec 31 13:59:39 2014 - [info] Getting new master's binlog name and position..
Wed Dec 31 13:59:39 2014 - [info]  master-bin.000001:3485577

MHA will scp binlog server binlog and Concat binlog, all slaves are consistent in the end. (same with old behavior)

check all slaves:

           Retrieved_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:4311-10003
            Executed_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-10003
                Auto_Position: 1




mysql> select count(*) from liuyang1.test;
+----------+
| count(*) |
+----------+
|    10000 |
+----------+

no data lost. and every slaves got final transaction 10003.

—————————————————————-

add an other binlog server (master server)

[binlog1]
hostname=10.0.128.252

[binlog2]
hostname=10.0.128.113

add 113 to cluster:

master : 10.0.128.113

slave : 10.0.128.77/110/114

run procedure again:


mysql> delimiter ;

mysql> call myprocedure();

10.0.128.77:


           Retrieved_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:10004-10327
            Executed_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-10151,
cfd90b74-90af-11e4-b7ab-86f5fb218bef:1
                Auto_Position: 1
   

10.0.128.110:

           Retrieved_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:10004-10621
            Executed_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-10177,
cfd90b74-90af-11e4-b7ab-86f5fb218bef:1
                Auto_Position: 1

10.0.128.114:

           Retrieved_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:10004-12723
            Executed_Gtid_Set: 1cc021e4-90b0-11e4-b7ad-00163f008023:1-12722,
cfd90b74-90af-11e4-b7ab-86f5fb218bef:1
                Auto_Position: 1
       

now we remove binlog on 10.0.128.252 manually and kill master server :

find these messages:

Wed Dec 31 15:26:16 2014 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=mysql-bin.000003  --start_pos=1243184 --output_file=/var/log/masterha/test/saved_binlog_binlog1_20141231152512.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.6.17-log  --binlog_dir=/data/mysql/data 
Failed to save binary log: Binlog not found from /data/mysql/data! If you got this error at MHA Manager, please set "master_binlog_dir=/path/to/binlog_directory_of_the_master" correctly in the MHA Manager's configuration file and try again.
 at /usr/bin/save_binary_logs line 123
	eval {...} called at /usr/bin/save_binary_logs line 70
	main::main() called at /usr/bin/save_binary_logs line 66
Wed Dec 31 15:26:16 2014 - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln660] Failed to save binary log events from the binlog server. Maybe disks on binary logs are not accessible or binary log itself is corrupt?
Wed Dec 31 15:26:16 2014 - [info] End of log messages from 10.0.128.252.
Wed Dec 31 15:26:16 2014 - [warning] Got error from 10.0.128.252.
Wed Dec 31 15:26:18 2014 - [info] 
Wed Dec 31 15:26:18 2014 - [info] Log messages from 10.0.128.113 ...
Wed Dec 31 15:26:18 2014 - [info] 
Wed Dec 31 15:26:16 2014 - [info] Fetching binary logs from binlog server 10.0.128.113..
Wed Dec 31 15:26:16 2014 - [info] Executing binlog save command: save_binary_logs --command=save --start_file=mysql-bin.000003  --start_pos=1243184 --output_file=/var/log/masterha/test/saved_binlog_binlog2_20141231152512.binlog --handle_raw_binlog=0 --skip_filter=1 --disable_log_bin=0 --manager_version=0.56 --oldest_version=5.6.17-log  --binlog_dir=/data/mysql/data 
  Creating /var/log/masterha/test if not exists..    ok.
  
  

MHA check binlog server 1 and find error with binlog then try to recover from binlog server 2.

With this feature we can add master server as first binlog server to guarantee no binlog loss.If master server crash and can not been reached,binlog server 2 will also provide service.

Final:

 MHA has changed failover procedure when turn on GTID. 
 
 1.If you want to use binlog server you must open GTID (I don't know why Yoshi design like this)
 2.MHA will not use relay-log to reocver (apply_diff_relay_logs will not be used ,just change master to lastest slave to recover with GTID)
 3.if there's no binlog server setting. MHA will ignore master binlog (even Manager node can ssh to Master server)
 4.if you do not set binlog server on MHA, data may lost (MHA just keep all slaves consistent,so if lastest slave has lag data will be lost)
 5.you can set multiple binlog server ,MHA will check them orderly,you can even set Master server as binlog server.


MySQL message subscription system

December 9, 2014 Architect, Architecture, MYSQL, replication, software No comments

Introducing MQ system based on MySQL (canal + roma)
Using this system to subscribe message between different platforms.

View this PDF roma_system
Also download this PDF from slideshare

webm_v2 has been released

October 21, 2014 Architect, NoSQL, rdbms, software No comments

Webm_v2 was add oracle and mongodb monitor module

webm was agent uploaded mode to save statistics

we use mysql database as monitor server

we already design tables to store these data (consider for future analysis)

you can download from github : https://github.com/ylouis83/webm

NoSQL压测工具YCSB

July 16, 2014 Architect, NoSQL, software No comments

Download PDF from slideshare

基于mongodb的压力评测工具 YCSB的一些概括

YCSB 是一款基于NOSQL Benchmark的工具,提供了多种方式模拟测试nosql的负载,基于现在对nosql的一些压力测试还没有得到重视.
YCSB 的全面的测试功能可以给即将上线的nosql提供一种另类保障。

There are many new serving databases available, including:

PNUTS
BigTable
HBase
Hypertable
Azure
Cassandra
CouchDB
Voldemort
MongoDb
OrientDB
Infinispan
Dynomite
Redis
GemFire
GigaSpaces XAP
DynamoDB
Couchhase
Aerospike 

下面的数据仅供参考,测试于虚拟机服务器。

给一个简单的load data的例子:

[root@mysqlstd ycsb-0.1.4]# ./bin/ycsb load  mongodb -P workloads/workloada   -p mongodb.url=mongodb://127.0.0.1:27017 -p mongodb.database=newdb -p mongodb.writeConcern=normal  -s  >data
Loading workload...
Starting test.
 0 sec: 0 operations; 
 10 sec: 18448 operations; 1837.08 current ops/sec; [INSERT AverageLatency(us)=423.35] 
 20 sec: 42134 operations; 2366.71 current ops/sec; [INSERT AverageLatency(us)=373.44] 
 30 sec: 61185 operations; 1904.34 current ops/sec; [INSERT AverageLatency(us)=661.58] 
 40 sec: 85308 operations; 2411.09 current ops/sec; [INSERT AverageLatency(us)=324.83] 
 50 sec: 97785 operations; 1247.2 current ops/sec; [INSERT AverageLatency(us)=985.33] 
 50 sec: 100000 operations; 2662.26 current ops/sec; [INSERT AverageLatency(us)=371.24] 

load data之后 可以开始模拟压测了.YCSB 主要分为以下几种模式:

Workload A: Update heavy workload

This workload has a mix of 50/50 reads and writes. An application example is a session store recording recent actions.

Workload B: Read mostly workload

This workload has a 95/5 reads/write mix. Application example: photo tagging; add a tag is an update, but most operations are to read tags.

Workload C: Read only

This workload is 100% read. Application example: user profile cache, where profiles are constructed elsewhere (e.g., Hadoop).

Workload D: Read latest workload

In this workload, new records are inserted, and the most recently inserted records are the most popular. Application example: user status updates; people want to read the latest.

Workload E: Short ranges

In this workload, short ranges of records are queried, instead of individual records. Application example: threaded conversations, where each scan is for the posts in a given thread (assumed to be clustered by thread id).

Workload F: Read-modify-write
In this workload, the client will read a record, modify it, and write back the changes. Application example: user database, where user records are read and modified by the user or to record user activity.

其中E模式的short range存在问题,当然我们也可以自定义模式:

# Yahoo! Cloud System Benchmark
# Workload A: Update heavy workload
#   Application example: Session store recording recent actions
#                        
#   Read/update ratio: 50/50
#   Default data size: 1 KB records (10 fields, 100 bytes each, plus key)
#   Request distribution: zipfian

recordcount=100000
operationcount=100000
workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=1
updateproportion=0
scanproportion=0
insertproportion=0

修改read update insert的百分比来达到模拟真实环境的目的, 下面给个压测例子

比如压测纯读操作:

[root@mysqlstd ycsb-0.1.4]# ./bin/ycsb run  mongodb -P workloads/workloadc  -P large.dat -p mongodb.url=mongodb://127.0.0.1:27017 -p mongodb.database=newdb  -p mongodb.writeConcern=normal  -s  >data
Loading workload...
Starting test.
 0 sec: 0 operations; 
 10 sec: 49375 operations; 4922.24 current ops/sec; [READ AverageLatency(us)=192.56] 
 18 sec: 100000 operations; 6141.57 current ops/sec; [READ AverageLatency(us)=159.72] 

95%读+%5写:

[root@mysqlstd ycsb-0.1.4]# ./bin/ycsb run  mongodb -P workloads/workloadd  -P large.dat -p mongodb.url=mongodb://127.0.0.1:27017 -p mongodb.database=newdb  -p mongodb.writeConcern=normal  -s  >data
Loading workload...
Starting test.
 0 sec: 0 operations; 
 10 sec: 43497 operations; 4333.23 current ops/sec; [INSERT AverageLatency(us)=633.66] [READ AverageLatency(us)=196.33] 
 20 sec: 92795 operations; 4925.37 current ops/sec; [INSERT AverageLatency(us)=792.15] [READ AverageLatency(us)=167.74] 
 21 sec: 100000 operations; 5637.72 current ops/sec; [INSERT AverageLatency(us)=379.57] [READ 

AverageLatency(us)=163.45]

另外 thumbtack对YCSB 进行了修改,增强了一些功能 开源地址:

https://github.com/thumbtack-technology/ycsb

主要增加支持了 Aerospike and Couchbase 目前 Aerospike已经开源 针对SSD 进行了专门优化 :

http://www.aerospike.com/blog/entrepreneurs-break-all-the-rules-aerospike-goes-open-source/

同时MongoDB 的驱动程序从 2.8.0 版(在 Mongo 2.2 发布之前的版本)升级到了 2.10.1 版,并实现了对 readPreference 配置的设置

mongodb.readPreference = primary|primaryPreferred|secondary|secondaryPreferred

QQ图片20140715114647

下面结合 thumbtack修改的版本做个测试 可以得到具体数据:

[root@mysqlstd ycsb]# fab ycsb_load:db=mongodb

[10.0.32.38] Executing task 'ycsb_load'
2014-07-15 01:09:00-07:00
[10.0.32.38] run: echo "/root/ycsb/bin/ycsb load mongodb -s -p mongodb.url=mongodb://127.0.0.1:27017 -p workload=com.yahoo.ycsb.workloads.CoreWorkload -p updateretrycount=1000 -p mongodb.writeConcern=normal -p mongodb.database=ycsb -p recordcount=5000000 -p exportmeasurementsinterval=30000 -p fieldcount=10 -p timeseries.granularity=100 -p threadcount=32 -p insertretrycount=10 -p readretrycount=1000 -p ignoreinserterrors=true -p reconnectionthroughput=10 -p operationcount=2400000000 -p fieldnameprefix=f -p maxexecutiontime=2400 -p mongodb.readPreference=primaryPreferred -p measurementtype=timeseries -p reconnectiontime=1000 -p fieldlength=10 -p insertstart=0 -p insertcount=5000000 > /root/ycsb/2014-07-15_01-09_mongodb_load.out 2> /root/ycsb/2014-07-15_01-09_mongodb_load.err" | at 01:09 today
[10.0.32.38] out: job 13 at 2014-07-15 01:09

Done.
Disconnecting from 10.0.32.38... done.

[mongo@mysqlstd ~]$ /data/mongodb/mongodb/bin/mongo

MongoDB shell version: 2.6.1
connecting to: test
> show dbs
admin   0.031GB
local   0.031GB
newdb   0.500GB
newdb1  0.500GB
newdb2  0.500GB
ycsb    1.500GB
> use ycsb
switched to db ycsb
> db.usertable.c
db.usertable.clean(                  db.usertable.convertToCapped(        db.usertable.copyTo(                 db.usertable.createIndex(
db.usertable.constructor             db.usertable.convertToSingleObject(  db.usertable.count(
> db.usertable.count()
2675710

模拟压力环境:workload mode=A

[root@mysqlstd ycsb]# fab ycsb_run:db=mongodb,workload=A
[10.0.32.38] Executing task 'ycsb_run'
2014-07-15 02:13:00-07:00
[10.0.32.38] run: echo "/root/ycsb/bin/ycsb run mongodb -s -P /root/ycsb/workloads/workloada -p mongodb.url=mongodb://127.0.0.1:27017 -p workload=com.yahoo.ycsb.workloads.CoreWorkload -p updateretrycount=1000 -p mongodb.writeConcern=normal -p mongodb.database=ycsb -p recordcount=5000000 -p exportmeasurementsinterval=30000 -p fieldcount=10 -p timeseries.granularity=100 -p threadcount=32 -p insertretrycount=10 -p readretrycount=1000 -p ignoreinserterrors=true -p reconnectionthroughput=10 -p operationcount=1800000 -p fieldnameprefix=f -p maxexecutiontime=180 -p mongodb.readPreference=primaryPreferred -p measurementtype=timeseries -p reconnectiontime=1000 -p fieldlength=10 > /root/ycsb/2014-07-15_02-13_mongodb_workloada.out 2> /root/ycsb/2014-07-15_02-13_mongodb_workloada.err" | at 02:13 today
[10.0.32.38] out: job 23 at 2014-07-15 02:13
[10.0.32.38] out: 

Done.
Disconnecting from 10.0.32.38... done.

使用merge.py得到具体数值

[root@mysqlstd ycsb]#  ./bin/merge.py																													
	OVERALL	OVERALL	READ	READ	READ	READ	READ	READ	READ	READ	READ	UPDATE	UPDATE	UPDATE	UPDATE	UPDATE	UPDATE	UPDATE	UPDATE	UPDATE	CLEANUP	CLEANUP	CLEANUCLEANUP	CLEANUP	CLEANUP	CLEANUP	CLEANUP	CLEANUP	
	RunTime	Throughput	Operations	Retries	Return=0	Return=[^0].*	AverageLatency	MinLatency	MaxLatency	95thPercentileLatency	99thPercentileLatency	Operations	Retries	Return=0	Return=[^0].*	AverageLatency	MinLatency	MaxLatency	95thPercentileLatency	99thPercentileLatency	Operations	Retries	Return=0	Return=[^0].*	AverageLatency	MinLatency	MaxLatency	95thPercentileLatency	99thPercentileLatency
1	61156	28.58264111	1665	747000	918	747	927.7820691	0.132	6630.776			83	33000	50	33	2075.808675	0.504	9767.828			32	0			18.08734375	0.465	207.159		
Total	61156	28.58264111	1665	747000	918	747	927.7820691	0.132	6630.776			83	33000	50	33	2075.808675	0.504	9767.828			32	0	0	0	18.08734375	0.465	207.159		

[mongo@mysqlstd ~]$ /data/mongodb/mongodb/bin/mongostat
connected to: 127.0.0.1

insert  query update delete getmore command flushes mapped  vsize    res faults  locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn       time 
    *0   5568   5369     *0       0  5381|0       0  3.06g  6.47g   282m      9 admin:0.9%          0       0|0     0|0     1m   894k    95   02:14:05 
    *0   4298   6267     *0       0  6279|0       0  3.06g  6.47g   282m      6 admin:0.6%          0       0|0     1|0     1m   962k    96   02:14:06 
    *0   4675   6119     *0       0  6066|0       0  3.06g  6.47g   282m      2 admin:0.0%          0      95|0     1|0     1m   948k    92   02:14:07 
    *0   4137   4866     *0       0  4948|0       0  3.06g  6.47g   282m     18 admin:2.1%          0       0|0     0|0     1m   790k    91   02:14:08 
    *0   4568   5904     *0       0  5922|0       0  3.06g  6.47g   282m      4 admin:0.1%          0       0|0     0|0     1m   927k    92   02:14:09 
    *0   4727   6034     *0       0  6046|0       0  3.06g  6.47g   282m      5 admin:0.0%          0       0|0     0|0     1m   949k    90   02:14:10 
    *0   4991   5673     *0       0  5690|0       0  3.06g  6.47g   282m      3 admin:0.9%          0       0|0     0|0     1m   914k    94   02:14:11 
    *0   4740   5173     *0       0  5183|0       1  3.06g  6.47g   282m      7 admin:0.1%          0       0|0     0|0     1m   839k    94   02:14:12 
    *0   4332   5493     *0       0  5510|0       0  3.06g  6.47g   282m      8 admin:0.9%          0       0|0     0|0     1m   866k    94   02:14:13 
    *0   4980   5583     *0       0  5592|0       0  3.06g  6.47g   282m      8 admin:0.0%          0       0|0     0|0     1m   901k    97   02:14:14 
insert  query update delete getmore command flushes mapped  vsize    res faults  locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn       time 
    *0   5750   5030     *0       0  4997|0       0  3.06g  6.47g   282m     20 admin:1.8%          0      94|0     1|1     1m   853k    97   02:14:15 
    *0   4884   5509     *0       0  5578|0       0  3.06g  6.47g   282m     10 admin:0.1%          0       0|0     0|0     1m   894k    97   02:14:16 
    *0   5733   5773     *0       0  5784|0       0  3.06g  6.47g   282m      5 admin:0.0%          0       0|0     0|0     1m   952k    92   02:14:17 
    *0   5178   5202     *0       0  5219|0       0  3.06g  6.47g   282m     14 admin:0.0%          0       0|0     0|0     1m   861k    95   02:14:18 
    *0   4179   5680     *0       0  5688|0       0  3.06g  6.47g   282m      8 admin:0.0%          0       0|0     0|1     1m   884k    93   02:14:19 
    *0   4879   5695     *0       0  5707|0       0  3.06g  6.47g   282m     11 admin:0.1%          0       0|0     0|0     1m   911k    93   02:14:20 
    *0   5271   5402     *0       0  5413|0       0  3.06g  6.47g   282m     12 admin:0.0%          0       0|0     0|0     1m   887k    95   02:14:21 
    *0   4583   4852     *0       0  4867|0       1  3.06g  6.47g   282m     11 admin:0.0%          0       0|0     0|0     1m   795k    93   02:14:22 
    *0   6654   4956     *0       0  4967|0       0  3.06g  6.47g   282m     10 admin:1.5%          0       0|0     0|0     1m   881k    95   02:14:23 
    

REF:
http://www.aerospike.com/blog/entrepreneurs-break-all-the-rules-aerospike-goes-open-source/

https://github.com/thumbtack-technology/ycsb

http://www.aerospike.com/wp-content/uploads/2013/02/Ultra-High-Performance-NoSQL-Benchmarking_zh-CN.pdf

https://github.com/brianfrankcooper/YCSB/wiki

http://labs.yahoo.com/news/yahoo-cloud-serving-benchmark/

1号店架构鸟瞰

November 18, 2013 Architect, hardware, IDC, software No comments

UPDATE 12.2: you can download complete PDF from database design of YHD