JKDB PLATFORM – our company’s data management platform
In this post, we introduce backup-restore module,focus on automatic backup and restore in a huge MySQL database environment.
Read this PDF: JKDB:BACKUP-RESTORE
[JKDB] BACKUP AND RESTORE
May 11, 2017 Architect, backup, hardware, IDC, maintain, MYSQL, rdbms, replication, system No comments
Facebook WDT Test
We test Facebook WDT tool:https://github.com/facebook/wdt
Read this PDD: WDT-TEST
Log-Center for OPS
March 10, 2017 Architect, system No comments
We build a strong log-collection system to analyze total logs of our operation teams (including system ,database, web service ,load balance ,audit, cache system , hardware, iDrac logs). All of these logs are used to build a high streaming log system to find a particular performance point of our apps .We also analyze these logs in time to get a notification of attack or even a risk of event.
Download :log-center-arch.pdf
Donkey – system for MySQL automatic maintenance
February 17, 2017 Architect, Architecture, maintain, MYSQL, system No comments
We build this system based on Inception (Qunar)。
All of MySQL DDL/DML operations are based on Donkey system, with strict approval process to keep online system strong and available.
Open this file by Mockplus Donkey-system
TCP Trace
September 21, 2016 Architecture, MYSQL, redis, system No comments
I found a useful tool named sniffer to help us analyze network packages (this tool can capture packages on specific port)
I use sniffer to analyze mysql and redis packages (translate these packages to normal queries)
#./vc-redis-sniffer –help
vc-redis-sniffer is a utility from VividCortex to monitor query activity and write results to a file.
See --license for the terms governing your usage of this program.
-binding="[::]:6379" This is a list of comma separated bind strings as seen in /proc/net/tcp
-help="false" Show this usage message
-license="false" Print the usage terms of this program
-output="" Filepath to output queries to. Defaults to stdout if none specified.
-show-database="false" Include a 'USE `database`' for every statement. Supersedes show-database-changes.
-show-database-changes="false"
Include a 'USE `database`' every time the database is changed.
-verbose="false" Enable logging on program startup to stderr
-version="false" Show version and exit
Flag Current value
--------------------------------------------
-binding "[::]:6379"
-help "true"
-license "false"
-output ""
-show-database "false"
-show-database-changes "false"
-verbose "false"
-version "false"Capture packages and gather logs
[root@a1-dba-test-242-13 /tmp/vc-redis-sniffer]
#./vc-redis-sniffer -binding=”[::]:6379″ -output=/tmp/redis.log
Analyze logs using pt-tools
[root@a1-dba-test-242-13 /tmp/vc-mysql-sniffer]
#pt-query-digest /tmp/redis.log
Also we can analyze mysql online queries ,do above steps to get result:
Linux tools part 11 – Monitoring specified file-IO
December 28, 2015 system No comments
We could use inotify-tools to monitor file or directory IO state
yum install inotify-tools
#inotifywait -m /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd
Setting up watches. Watches established. /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY /storage/sas/mysql3306/data/diablo_im/#sql-ib4747-2643139248.ibd MODIFY
this means mysqld process is writing file #sql-ib4747-2643139248.ibd now
we could add -r option to monitor directory IO state
#inotifywait -m -r /storage/sas/mysql3306/data/diablo_im/
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
/storage/sas/mysql3306/data/diablo_im/ MODIFY #sql-ib4747-2643139248.ibd
Events:
access file or directory contents were read
modify file or directory contents were written
attrib file or directory attributes changed
close_write file or directory closed, after being opened in writeable mode
close_nowrite file or directory closed, after being opened in read-only mode
close file or directory closed, regardless of read/write mode
open file or directory opened
moved_to file or directory moved to watched directory
moved_from file or directory moved from watched directory
move file or directory moved to or from watched directory
create file or directory created within watched directory
delete file or directory deleted within watched directory
delete_self file or directory was deleted
unmount file system containing file or directory unmounted
[Reprint]Injection MySQL – Function
August 24, 2015 Internals, linux, MYSQL, rdbms No comments
Reprint a PDF for MySQL Injection Test
MySQL checksum
August 19, 2015 MYSQL, replication, system No comments
how to use percona-tookit to verify mysql data
Read this PDF : mysql_checksum
How to configure Tcpdump system
July 24, 2015 Architect, Architecture, Internals, MYSQL, network, system No comments
MySQL Tcpdump system : use percona-toolkit to analyze network packages
We can identify problem SQLs with high execution frequency.
With DBMON system and AWR system we can find problem SQLs in a special time (high frequency, occurs over a period of time)
View this PDF:
linux tools part 10– Monitoring process iostat
April 24, 2015 system No comments
We can use iopp tool to monitor process io performance
download from github : https://github.com/markwkm/iopp
pid: The process id. rchar: The number of bytes which this task has caused to be read from storage. wchar: The number of bytes which this task has caused, or shall cause to be written to disk. syscr: Count of the number of read I/O operations. syscw: Count of the number of write I/O operations. rbytes rkb rmb reads: Count of the number of bytes which this process really did cause to be fetched from the storage layer. wbytes wkb wmb writes: Count of the number of bytes which this process really did cause to be sent to the storage layer. cwbytes cwkb cwmb cwrites: The number of bytes which this process caused to not happen, by truncating pagecache. command: Filename of the executable.
There is one small bug , you’ll get different output from -i -k than you will from -i or from -i -m.
#iopp -i -k 2
#grep ^ /sys/class/block/*/dev |grep 224
#lsof |grep “8,224” |more
bash 5265 root cwd DIR 8,224 57151488 427819009 /data/yp900 redis-ser 19843 root cwd DIR 8,224 4096 469368833 /data/redis6379 sh 27579 root cwd DIR 8,224 4096 579731457 /data/mysql3306 mysqld 28324 mysql cwd DIR 8,224 4096 579731459 /data/mysql3306/data mysqld 28324 mysql 1w REG 8,224 1327901 579743077 /data/mysql3306/mysql-error.log mysqld 28324 mysql 2w REG 8,224 1327901 579743077 /data/mysql3306/mysql-error.log mysqld 28324 mysql 3u REG 8,224 23520 579743078 /data/mysql3306/binlog/mysql-bin.index mysqld 28324 mysql 4uW REG 8,224 1073741824 579743083 /data/mysql3306/data/ibdata0 mysqld 28324 mysql 5u REG 8,224 0 579731461 /data/mysql3306/mysql-tmpdir/ib1io6G9 (deleted) mysqld 28324 mysql 6u REG 8,224 0 579731462 /data/mysql3306/mysql-tmpdir/ibnkifvg (deleted) mysqld 28324 mysql 7u REG 8,224 0 579731463 /data/mysql3306/mysql-tmpdir/ibHWjojn (deleted) mysqld 28324 mysql 8u REG 8,224 0 579731464 /data/mysql3306/mysql-tmpdir/ib1o8yHC (deleted) mysqld 28324 mysql 9uW REG 8,224 1073741824 579743084 /data/mysql3306/data/ibdata1 mysqld 28324 mysql 10uW REG 8,224 1073741824 579743085 /data/mysql3306/data/ibdata2 mysqld 28324 mysql 11uW REG 8,224 27850178560 579743086 /data/mysql3306/data/ibdata3 mysqld 28324 mysql 12uW REG 8,224 536870912 579743087 /data/mysql3306/data/ib_logfile0 mysqld 28324 mysql 13uW REG 8,224 536870912 579743088 /data/mysql3306/data/ib_logfile1 mysqld 28324 mysql 14uW REG 8,224 2281701376 579733990 /data/mysql3306/data/mit/agent_info#P#p20150404.ibd mysqld 28324 mysql 15u REG 8,224 0 579731465 /data/mysql3306/mysql-tmpdir/ibBZSSPL (deleted)
A nice smart tools with a little bugs 🙂