MYSQL

Logcenter Project architect

August 16, 2017 Architect, Architecture, bigdata, hadoop, hive, network, NoSQL, rdbms No comments

We created a project called LC (log center) for ops department
All member of ops are using this system for analyzing in a lower layer.
We collects all types of log including db-system, crond, secutiry log , cmdlog , api log etc.
We used MQ system for log push which based on a policy center. And we created a new background system to search and management.

Click this project LC-system-design

MySQL special character rollback

August 16, 2017 Architect, Internals, MYSQL, replication No comments

Actually, Mysqlbinlog tools can’t miner mysql binlog exactly correct.

In this post ,a special environment we got may hexadecimal characters what we don’t need.

Read this PDF: Special-code-rollback.pdf

[JKDB] BACKUP AND RESTORE

May 11, 2017 Architect, backup, hardware, IDC, maintain, MYSQL, rdbms, replication, system No comments

JKDB PLATFORM – our company’s data management platform

In this post, we introduce backup-restore module,focus on automatic backup and restore in a huge MySQL database environment.

Read this PDF: JKDB:BACKUP-RESTORE

Donkey – system for MySQL automatic maintenance

February 17, 2017 Architect, Architecture, maintain, MYSQL, system No comments

We build this system based on Inception (Qunar)。

All of MySQL DDL/DML operations are based on Donkey system, with strict approval process to keep online system strong and available.

Open this file by Mockplus Donkey-system

TCP Trace

September 21, 2016 Architecture, MYSQL, redis, system No comments

I found a useful tool named sniffer to help us analyze network packages (this tool can capture packages on specific port)

I use sniffer to analyze mysql and redis packages (translate these packages to normal queries)

#./vc-redis-sniffer –help

vc-redis-sniffer is a utility from VividCortex to monitor query activity and write results to a file.
See --license for the terms governing your usage of this program.

  -binding="[::]:6379"         This is a list of comma separated bind strings as seen in /proc/net/tcp
  -help="false"                Show this usage message
  -license="false"             Print the usage terms of this program
  -output=""                   Filepath to output queries to. Defaults to stdout if none specified.
  -show-database="false"       Include a 'USE `database`' for every statement. Supersedes show-database-changes.
  -show-database-changes="false"
                               Include a 'USE `database`' every time the database is changed.
  -verbose="false"             Enable logging on program startup to stderr
  -version="false"             Show version and exit

  Flag                         Current value
--------------------------------------------
  -binding                     "[::]:6379"
  -help                        "true"
  -license                     "false"
  -output                      ""
  -show-database               "false"
  -show-database-changes       "false"
  -verbose                     "false"
  -version                     "false"

Capture packages and gather logs

[root@a1-dba-test-242-13 /tmp/vc-redis-sniffer]
#./vc-redis-sniffer -binding=”[::]:6379″ -output=/tmp/redis.log

Analyze logs using pt-tools

[root@a1-dba-test-242-13 /tmp/vc-mysql-sniffer]
#pt-query-digest /tmp/redis.log

redis_output_result.txt

Also we can analyze mysql online queries ,do above steps to get result:

mysql_output_result.txt

MySQL TCPCOPY

June 29, 2016 Architect, Architecture, hardware, MYSQL, network No comments

we use tcpcopy to make real traffic on our core systems. Many problems will be found in advance if we enlarge queries several times.

Read this PDF TCPCOPY

Infiniflash Benchmark

May 29, 2016 Architect, Architecture, hardware, MYSQL, performance No comments

Sandisk (FusionIO) and Nexenta are working together to build this SDS solution.

Infiniflash is a very large SDS production, which manages for very large DW system who requires large storage space and also high IOPS.

We test infiniflash system ,read this Infiniflash_benchmark

MySQL distributed message system

April 12, 2016 Architect, Architecture, hardware, MYSQL, replication, software No comments

Based on messages, we create mysql replication platforms , using async message to build strong distributed subscription system.

read this PDF : http://www.vmcd.org/docs/MySQL_async_message.pdf

TokuDB benchmark on PCIe

October 21, 2015 Architect, Architecture, hardware, Internals, MYSQL, performance No comments

MariaDB TokuDB benchmark on FusionIO ,Compare TokuDB and InnoDB engines.

read: TokuDB_benchmark

MySQL lost “AUTO_INCREMENT” after a long time

October 8, 2015 Internals, MYSQL, rdbms No comments

Description:

I create a empty table and then alter this table's "AUTO_INCREMENT" starts from 10000000.After a long time ,MySQL lost "AUTO_INCREMENT" definition when I show this table again. MariaDB has the same problem but Percona Server is fine.

$mysql06

Warning: Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 141028725
Server version: 5.6.17-log MySQL Community Server (GPL)

Copyright (c) 2000, 2014, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> DROP TABLE IF EXISTS `test_bug` ;
Query OK, 0 rows affected, 1 warning (0.00 sec)


mysql> CREATE TABLE test_bug (
    ->   `id` BIGINT PRIMARY KEY NOT NULL AUTO_INCREMENT,
    ->   `consult_order_id` BIGINT ,
    ->   `pay_channel` INT COMMENT ,
    ->   `price` BIGINT ,
    ->   `trade_no` VARCHAR (64) ,
    ->   `right_no` VARCHAR (64) ,
    ->   `item_id` BIGINT NOT NULL,
    ->   `source_type` INT NOT NULL ,
    ->   `gmt_create` DATETIME DEFAULT CURRENT_TIMESTAMP ,
    ->   `gmt_modified` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP 
    -> ) ENGINE = INNODB DEFAULT CHARSET = utf8 ;
Query OK, 0 rows affected (0.01 sec)

mysql> 
mysql> ALTER TABLE test_bug 
    ->   ADD INDEX index_consult_order (consult_order_id) ;
Query OK, 0 rows affected (0.01 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> 
mysql> ALTER TABLE test_bug 
    ->   ADD INDEX index_right_no (right_no) ;
Query OK, 0 rows affected (0.00 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> 
mysql> ALTER TABLE test_bug 
    ->   AUTO_INCREMENT = 100000000 ;
Query OK, 0 rows affected (0.00 sec)
Records: 0  Duplicates: 0  Warnings: 0

mysql> 
mysql> show create table  test_bug;
  
   CREATE TABLE `test_bug` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `consult_order_id` bigint(20) DEFAULT NULL ,
  `pay_channel` int(11) DEFAULT NULL COMMENT ,
  `price` bigint(20) DEFAULT NULL COMMENT ,
  `trade_no` varchar(64) DEFAULT NULL COMMENT ,
  `right_no` varchar(64) DEFAULT NULL COMMENT ,
  `item_id` bigint(20) NOT NULL COMMENT ,
  `source_type` int(11) NOT NULL COMMENT ,
  `gmt_create` datetime DEFAULT CURRENT_TIMESTAMP ,
  `gmt_modified` datetime NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP ,
  PRIMARY KEY (`id`),
  KEY `index_consult_order` (`consult_order_id`),
  KEY `index_right_no` (`right_no`)
) ENGINE=InnoDB AUTO_INCREMENT=100000000 DEFAULT CHARSET=utf8  


1 row in set (0.00 sec)


wait about 24 hours , then show create table :


mysql> show create table test_bug; 

   CREATE TABLE `test_bug` (
  `id` BIGINT (20) NOT NULL AUTO_INCREMENT,
  `consult_order_id` BIGINT (20) DEFAULT NULL,
  `pay_channel` INT (11) DEFAULT NULL,
  `price` BIGINT (20) DEFAULT NULL,
  `trade_no` VARCHAR (64) DEFAULT NULL,
  `right_no` VARCHAR (64) DEFAULT NULL,
  `item_id` BIGINT (20) NOT NULL,
  `source_type` INT (11) NOT NULL,
  `gmt_create` DATETIME DEFAULT CURRENT_TIMESTAMP,
  `gmt_modified` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`),
  KEY `index_consult_order` (`consult_order_id`),
  KEY `index_right_no` (`right_no`)
) ENGINE = INNODB DEFAULT CHARSET = utf8 

1 row in set (0.00 sec)  

AUTO_INCREMENT disappears .

Bug has been reported :https://bugs.mysql.com/bug.php?id=77743