Architect

MySQL distributed message system

April 12, 2016 Architect, Architecture, hardware, MYSQL, replication, software No comments

Based on messages, we create mysql replication platforms , using async message to build strong distributed subscription system.

read this PDF : http://www.vmcd.org/docs/MySQL_async_message.pdf

招聘帖

February 1, 2016 Architect No comments

因公司业务发展迅速,部门现在招两个mysql dba 同学,目前本人在平安健康互联网(平安好医生)数据团队,负责数据库的整体规划,想来的同学可以发简历到我的邮箱 ylouis83@gmail.com 靠谱的公司靠谱的团队 :)

关于公司:

国内最大的互联网医疗平台,提供全品类互联网医疗服务,目前国内前三大的a轮估值

How to backup remote redis instance

January 29, 2016 Architect, NoSQL, redis No comments

We wrote a python scripts to control redis backup work.

Three Steps to do this work:

1. Create meta database to store which redis instance needs to backup

2. Use scripts to connect to these redis instances with “–rdb” command (remote backup command)

3. Estimate which redis server should be transfered (only slave role redis will be transfered, wait minutes if a bgsave is running)

Backup Scripts: Redis_remote.py

TokuDB benchmark on PCIe

October 21, 2015 Architect, Architecture, hardware, Internals, MYSQL, performance No comments

MariaDB TokuDB benchmark on FusionIO ,Compare TokuDB and InnoDB engines.

read: TokuDB_benchmark

NVMFS Supports Atomic Writes

September 6, 2015 Architect, hardware, Internals, MYSQL No comments

Benchmark for NVMFS (supports atomic writes ,so we can close double-write option on specific MySQL version )

tips: some Flash-based cards could support large block map , the main idea is to avoid fractured page writes.

http://www.vmcd.org/docs/nvmfs_benchmark.pdf

How to configure AWR system

August 6, 2015 Architect, Architecture, mongodb, MYSQL, NoSQL, rdbms, software No comments

In this article, we introduce myawr and mongoawr system .

Read this PDF, you will learn how to configure them.

How to configure AWR system.

How to configure Tcpdump system

July 24, 2015 Architect, Architecture, Internals, MYSQL, network, system No comments

MySQL Tcpdump system : use percona-toolkit to analyze network packages

We can identify problem SQLs with high execution frequency.

With DBMON system and AWR system we can find problem SQLs in a special time (high frequency, occurs over a period of time)

View this PDF:

http://www.vmcd.org/docs/MySQL_TCPDUMP.pdf

How to configure WEBM

July 15, 2015 Architect, mongodb, MYSQL, software No comments

Architecture of WEBM system.

Reference:

http://www.vmcd.org/2014/10/webm_v2-has-been-released/
http://www.vmcd.org/2014/09/webm-mysql-database-performance-web-monitor/

View this PDF:

http://www.vmcd.org/docs/How%20to%20configure%20WEBM.pdf

MySQL 5.7 milestone

April 22, 2015 Architect, MYSQL, software No comments

MySQL 5.7 will be a great milestone in MySQL total history.
Oracle has released many useful new features in LAB version . MySQL is becoming more similar to Oracle database 🙂

Read this presentation I post on slideshare:

MySQL 5.7 milestone

Architecture of data warehouse which is based on MQ

January 20, 2015 Architect, Architecture, MYSQL, rdbms, software No comments

Recently, we create a mysql data warehouse which is based on message queue.

Most companies must prepare for particular queries in their systems if they consider to split their databases or tables into many pieces.

some problems should be solved in this situation:

1. how to get correct results in-time
2. how to build strong data warehouse for future analyst

These policies were used by YHD

They have already deployed a middle-ware layer to support these requests (between web apps and databases). Every aggregation SQL was splited into many small SQLs and runs in every data nodes.The Final result is the aggregation of these all small SQLs. In this procedure, everything was computed in memory to get high performance.

In data warehouse layer, they use self-defined ETL tools to extract data from different databases to oracle-Exadata platform. Log-based data was put into hadoop and hbase.

I found a new solution

With Canal and Roma (visit previous PDF roma system) , we could build a data warehouse which is based on metaQ. (metaQ is the final storage of roma) , so we can put some simple queries on this data warehouse directly.

We could use MySQL to build this Data warehouse and use original replication in these databases (everything is simple, especially using multiple source feature via MariaDB).

Disadvantages of this architecture: MySQL database is not the best choice for data warehouse. So we need another analyst platform to handle other log-based data.

Most BI systems were built by very expensive commercial software . For small and medium sized companies, this architecture can save a lot of costs.

Client to aggregate messages:

etl_roma

split and merge aggregation :

split1split2

total architecture:

DW_arch