Introduction of JKDB. JK.CN‘s data management platform.
A platform for database /cache system management
A platform for resource control management
A platform for all of data management
Download this PPT JKDB_PLATFORM
We created a project called LC (log center) for ops department
All member of ops are using this system for analyzing in a lower layer.
We collects all types of log including db-system, crond, secutiry log , cmdlog , api log etc.
We used MQ system for log push which based on a policy center. And we created a new background system to search and management.
Click this project LC-system-design
Actually, Mysqlbinlog tools can’t miner mysql binlog exactly correct.
In this post ,a special environment we got may hexadecimal characters what we don’t need.
Read this PDF: Special-code-rollback.pdf
JKDB PLATFORM – our company’s data management platform
In this post, we introduce backup-restore module,focus on automatic backup and restore in a huge MySQL database environment.
Read this PDF: JKDB:BACKUP-RESTORE
We build a strong log-collection system to analyze total logs of our operation teams (including system ,database, web service ,load balance ,audit, cache system , hardware, iDrac logs). All of these logs are used to build a high streaming log system to find a particular performance point of our apps .We also analyze these logs in time to get a notification of attack or even a risk of event.
I found a useful tool named sniffer to help us analyze network packages (this tool can capture packages on specific port)
I use sniffer to analyze mysql and redis packages (translate these packages to normal queries)
vc-redis-sniffer is a utility from VividCortex to monitor query activity and write results to a file. See --license for the terms governing your usage of this program. -binding="[::]:6379" This is a list of comma separated bind strings as seen in /proc/net/tcp -help="false" Show this usage message -license="false" Print the usage terms of this program -output="" Filepath to output queries to. Defaults to stdout if none specified. -show-database="false" Include a 'USE `database`' for every statement. Supersedes show-database-changes. -show-database-changes="false" Include a 'USE `database`' every time the database is changed. -verbose="false" Enable logging on program startup to stderr -version="false" Show version and exit Flag Current value -------------------------------------------- -binding "[::]:6379" -help "true" -license "false" -output "" -show-database "false" -show-database-changes "false" -verbose "false" -version "false"
Capture packages and gather logs
#./vc-redis-sniffer -binding=”[::]:6379″ -output=/tmp/redis.log
Analyze logs using pt-tools
Also we can analyze mysql online queries ,do above steps to get result:
we use tcpcopy to make real traffic on our core systems. Many problems will be found in advance if we enlarge queries several times.
Read this PDF TCPCOPY
Sandisk (FusionIO) and Nexenta are working together to build this SDS solution.
Infiniflash is a very large SDS production, which manages for very large DW system who requires large storage space and also high IOPS.
We test infiniflash system ,read this Infiniflash_benchmark