±à¼ÍƼö: |
±¾ÎÄÀ´×ÔÓÚÉî¶È¿ªÔ´£¬±¾ÎĽéÉÜ»ùÓÚMySQL¼°Redis´î½¨Í³Ò»µÄkv´æ´¢·þÎñ£º³£Óò¿Êð·½Ê½¼°ÆäÌØµã£¬Cluster
manager£¬MySQLºÍRedis¼¯Èº·½°¸£¬ÒÔ¼°SyncÊý¾Ýͬ²½·þÎñ¡£ |
|
Ò»¡¢MySQL+Redis ³£Óò¿Êð·½Ê½
1.1 ÍØÆË

1.2 ÌØµã
ÒµÎñ²ãͨ¹ý˫дͬʱдMySQL¼°Redis¡£¶Áͨ³£ÔÚRedis£¬Èô¶ÁÈ¡²»µ½£¬Ôò´ÓMySQL¶ÁÈ¡£¬È»ºó½«Êý¾Ýͬ²½µ½Redis£¬Redisͨ³£ÉèÖÃexpire»òÕßĬÈÏLRU½øÐÐÊý¾ÝÌÔÌ¡£
ÕâÖÖʹÓ÷½Ê½»áÓÐÈçÏÂÎÊÌ⣺
1£©MySQL¼°Redis´æÔÚÊý¾Ý²»Ò»Ö·çÏÕ£¬ÓÈÆäÊdz¤Ê±¼äÔËÐеÄϵͳ
2£©ÒµÎñ²ãÐèÒª´¦ÀíMySQL sql schemaÓëRedis kvÊý¾Ý½á¹¹ÉϵÄÂß¼²îÒì
3£©ÎÞͳһÔËά
4£©ÎÞ·¨·½±ãÀ©ÈÝ/ËõÈÝ
¶þ¡¢KV »¯µÄ´æ´¢Ê¹ÓÃÀíÄî
2.1 MySQL Is great NoSQL
ΪʲôҪÓÃMySQL:
¡°ÔÚ¿ÉÀ©Õ¹ÏµÍ³¹¹½¨Ê±£¬Ò»¸öºÜÖØÒªµÄ¿¼Á¿ÊÇʹÓõļ¼ÊõÊÇ·ñ³ÉÊ죬ѡÔñ³ÉÊìµÄ¼¼ÊõÒâζ×ųö´íʱÄܹ»Ñ¸ËÙ»Ö¸´¡£µ±È»£¬¿ª·¢ÕßÒ²¿ÉÒÔÔÚÏîÄ¿ÖÐʹÓÃ×îÐÂ×îÅ£µÄNoSQLÊý¾Ý¿â£¬¶øÕâ¸öÊý¾Ý¿âÔÚÀíÂÛÉÏÒ²¿ÉÒÔÁ¼ºÃµØÔËÐУ¬È»¶øÔÚÉú²ú»·¾³ÖгöÏÖÁËÎÊÌâ»Ö¸´ÐèÒª¶à¾Ã£¿¼¼ÊõÉÏÒÑÓеÄ֪ʶºÍ¾Ñé»ýÀÛ¶ÔÓÚÎÊÌ⻺½âÖÁ¹ØÖØÒª£¬µ±È»Õâ¸ö»ýÀÛÒ²°üÀ¨ÁËGoogle¿ÉÒÔËÑË÷µ½µÄÄÚÈÝ¡£
Ïà±È֮ϣ¬¹ØÏµÐÍÊý¾Ý¿âÒѾ´æÔÚÁ˳¬¹ýËÄÊ®Ä꣬ҵ½ç¶ÔÓÚ¹ØÏµÐÍÊý¾Ý¿âµÄά»¤Ò²»ýÀÛÁË´óÁ¿µÄ¾Ñé¡£»ùÓÚÕâЩ¿¼ÂÇ£¬ÔÚÐÂÏîÄ¿×ö¼¼ÊõÑ¡ÐÍʱͨ³£»áÑ¡ÔñMysql£¬¶ø²»ÊÇNoSQLÊý¾Ý¿â£¬³ý·ÇNoSQLÕæµÄÓзdz£·Ç³£Ã÷ÏÔµÄÓÅÊÆ¡£¡±
2.2 KVÀíÄî
¶ÔÓÚÒÚ¼¶¹æÄ£µÄÊý¾Ý´æ´¢£¬ÓÈÆäÊÇÉæ¼°µ½Ë®Æ½²ð·Ö¿ç»ú·Ö¿â·Ö±íµÄÇé¿öÏ£¬ÏßÉ϶ÔÊý¾Ý¿âµÄ·ÃÎÊÖ»ÄÜ×öµÄÔ½¼òµ¥Ô½ºÃ£¬group
by/order by/·ÖÒ³/ͨÓÃjoin/ÊÂÎñµÈµÈµÄÖ§³Ö ÔÚÕâ¸öÁ¿¼¶ÏµÄMySQLϵͳ¶¼ÊDz»ºÏÊʵġ£
»ù±¾ÉÏĿǰËùÓеÄÀàproxyµÄMySQL·½°¸ÕæÕýÉϹæÄ£ÏßÉÏÓ¦ÓÃÖ»ÄÜʹÓð´²ð·Ö¼ü½øÐжÁд²Ù×÷£¬Êµ¼ÊÉÏÒ²ÊÇÒ»¸öÓòð·Ö¼ü×öµÄÒ»¸ökvϵͳ¡£
ÈôÏëʹÓø´ÔÓµÄsql´¦Àí£¬×îºÏÀíµÄ²¿Êð·½°¸Êǽ«MysqlbinlogÁ÷ˮͬ²½·þÎñ³éÏó³öÀ´£¬Í¨¹ýʵʱͬ²½µ½OLAPÀàµÄϵͳ½øÐд¦Àí¡£
ËùÒÔÃæÏòº£Á¿´æ´¢·þÎñ£¬MySQL´ÓÒ»¿ªÊ¼¾ÍÉè¼ÆÎªÒ»¸öKVϵͳÊÇ¿ÉÐеġ£valueʹÓÃmediumblob´æ´¢xml/json/protobuf/thrift¸ñʽ»¯Êý¾ÝÐòÁл¯Ö®ºóµÄÊý¾Ý¡£
2.3 MySQL KV»¯µÄʹÓ÷½Ê½
1¡¢ÓÃMySQLÔÀ´µÄÖ÷¼ü»òÕßË÷Òý¼üµ±×ökey
2¡¢ÆäËûËùÓеķÇÖ÷¼ü·ÇË÷Òý¼ü£¬È«²¿°ü×°µ½valueÀïÃæ£¬valueʹÓÃmediumblob´æ´¢xml/json/protobuf/thrift¸ñʽ»¯Êý¾ÝÐòÁл¯Ö®ºóµÄÊý¾Ý¡£
3¡¢Êý¾Ý¶Áд²Ù×÷£¬¾ù»ùÓÚkeyÒ»ÕûÐÐÊý¾Ý×ö¶Áд£¬ÓÉÒµÎñ²ã¶ÔÀïÃævalueµÄ½á¹¹×ö½âÎö¼°¶ÔÄÚ²¿½á¹¹×öÔöɾ¸Ä²î£¬¶ø²»Óñä¸ü
MySQL ±¾ÉíµÄschema.
2.4 ²»ÊÊÓó¡¾°
1¡¢Êý¾ÝÁ¿ºÍ·ÃÎÊÁ¿²»´ó²¢ÇÒÒµÎñÂß¼ÒÀÀµ MySQL Êý¾Ý¿â½øÐд¦ÀíµÄÒµÎñ³¡¾°
2¡¢Éæ¼°µ½¶à±íjoinµÈµÄ´¦Àí
¶ÔÓÚ´ËÏÞÖÆ£¬Ò²¿ÉÒÔͨ¹ý½«¹ØÁª±í¼Ó¹¤³É»ùÓÚ¹ØÁªÌõ¼þµÄÒ»ÕÅ¿í±í½øÐÐKV»¯¡£
3¡¢Éæ¼°µ½ÊÂÎñµÈµÄ´¦Àí¡£
Èý¡¢½«MySQL+RedisÉè¼ÆÎªÍ³Ò»µÄKV´æ´¢·þÎñ
3.1 Ä¿±ê
1£©ÒµÎñ²ãͨ¹ýͳһ·½Ê½·ÃÎÊMySQL¼°Redis£¬²»ÔÙʹÓÃMySQL¿Í»§¶Ë¼°Redis¿Í»§¶Ë·ÃÎÊ
2£©MySQL¼¯Èº»¯/Redis¼¯Èº»¯²¿Êð
3£©½«ÒµÎñ˫д¸ÄΪMySQLµ½Redisµ×²ãbinlogÊý¾Ýͬ²½·½Ê½Íê³Éͬ²½
4£©Òì¹¹Êý¾Ý´æ´¢Ö§³Ö×îÖÕÒ»ÖÂÐÔÊý¾Ý¶Áд·þÎñ
5£©Ö§³Ö´æ´¢²ãÃæÀ©ÈÝËõÈÝ¡¢failoverÇÒÒµÎñÎÞ¸ÐÖª
6£©µ¥»úȺÈÕ°ÙÒÚ´ÎQPS/TPSÖ§³Ö£¨´óÀàÒµÎñÊʶȲð·Öµ½²»Í¬¼¯ÈºÖУ©
3.2 ×îÖÕʵÏÖ
»ùÓÚMySQL+RedisµÄͳһ´æ´¢·þÎñ£¨UniStore£© =
MySQL¿ç»ú·Ö¿â·Ö±í¼¯Èº
+ Redis¼¯Èº
+ MySQL->RedisʵʱÊý¾Ýͬ²½·þÎñ
+ ͳһµÄ¶ÔÍâÊý¾Ý·ÃÎʽӿÚ
+ ÄÚÔÚµÄÍêÕûÔËά֧³Öϵͳ£¨Ö§³ÖÔÚÏßÀ©ÈÝ/ËõÈÝ¡¢failoverµÈ£©
3.3 ¼Ü¹¹Í¼

3.4 ¼Ü¹¹ËµÃ÷-½«´æ´¢Éè¼ÆÎªÒ»ÖÖ·þÎñ
1¡¢½«MySQL+Redis×ö³ÉͳһKV´æ´¢·þÎñ
2¡¢Í¨¹ýacc proxyÌṩͳһµÄÊý¾Ý·ÃÎʽӿڣ¬Í¨¹ýͳһÐÒéÖ§³Ö¿çÓïÑÔÊý¾Ý·ÃÎÊ
·ÃÎÊÐÒ飨×Ô¶¨ÒåÐÒ飬protobufÐÒ飬thriftÐÒéµÈ£©
3¡¢MySQL clusterÖ§³Ö¿ç»úµÄ·Ö¿â·Ö±í£¬schemalessÉè¼Æ£¬ËùÓÐÒµÎñ±íKV»¯Éè¼Æ
4¡¢Redis clusterÖ§³Ö¿ç»úµÄʵÀý²ð·Ö
5¡¢SyncÊý¾Ýͬ²½·þÎñÌṩͳһµÄMysqlµ½Redis ¿çIDC/²»¿çIDCÊý¾Ýͬ²½·þÎñ£¬Ð¡ÓÚ100msÑÓʱ
6¡¢Õû¸öϵͳ²»Éæ¼°µ½·Ö²¼Ê½ÊÂÎñ´¦Àí
3.5 ÈýÖÖ²¿Êð·½Ê½
1¡¢´¿MySQL¼¯Èº²¿Êð
´ËÖÖ²¿Êð·½Ê½µÈͬÓÚÆäËûMySQL proxy¿ç»ú·Ö¿â·Ö±í·½°¸£¬¶Áд¾ùÔÚMySQL
2¡¢´¿Redis¼¯Èº²¿Êð
´ËÖÖ²¿Êð·½Ê½µÈͬÓÚÆäËûRedis proxy¿ç»ú·Ö¿â·Ö±í·½°¸£¬¶Áд¾ùÔÚRedis
3¡¢MySQL+RedisÒì¹¹²¿Êð
дÔÚMySQL£¬¶Á¿ÉÒÔ´ÓMySQL¶Á»òÕßRedis¶Á£¬È¡¾öÓÚÒµÎñ¶Ô×îÐÂÊý¾ÝµÄ¶ÁȡҪÇó¡£
3.6 ½Ó¿Ú˵Ã÷
1¡¢int get(int appid, string key,string& value)
Redis¶Á²Ù×÷רÓÃ
2¡¢int get_with_version(int appid,string key, string&
value, int64& version)
MySQL ¶Á²Ù×÷רÓã¬×Ô´ø°æ±¾ºÅ£¬·Àֹд¸²¸Ç
3¡¢int set(int appid, string key,string value, int64
version)
ͨ¹ýappidÇø·Ö MySQL »¹ÊÇRedis£¬¾ùÖ§³Öд²Ù×÷
4¡¢int delete(int appid, string key)
ͨ¹ýappidÇø·Ö MySQL »¹ÊÇRedis, ²»Ö§³ÖÅúÁ¿É¾³ý
5¡¢int multiget(int appid,vector<string> keys,
map<string, string>& key_value_pairs)
Ö§³ÖÅúÁ¿¶Á²Ù×÷£¬ÄÚ²¿µÄÊý¾Ý·Óɼ°Êý¾ÝºÏ²¢²»ÓùØÐÄ
6¡¢intmultiset(int appid, map<string, string>&
key_value_pairs)
²»½¨ÒéÖ§³Ö£¬Éæ¼°µ½¿ç»úÊÂÎñÎÊÌ⣬ÎÞ·¨±£Ö¤ACID
7¡¢int Redis_op(string cmd, ¡¡)
RedisÆäËûÔÉú½Ó¿Ú·â×°(incr/expire/list/setnxµÈ)
ËÄ¡¢Cluster Manager·þÎñ
4.1 Cluster Manager ÊÇÒ»¸öservice
cluster managerÖ÷ÒªÓÉÈçϼ¸ÖÖ¹¦ÄÜ
1£©MySQL/Redis·ÖƬ·ÓÉÐÅÏ¢µÄ¹ÜÀí
1¡¢ MySQL ·Ö¿â·Ö±í·ÓÉÐÅÏ¢
2¡¢Redis Slot·ÓÉÐÅÏ¢
3¡¢Â·ÓÉÐÅÏ¢±ä¸ü¹ÜÀí
2£©RedisʵÀýµÄ̽»î¼°RedisÀ©Èݼ°ËõÈÝÊý¾ÝµÄÇ¨ÒÆ
±ÈÈçÁ¬Ðø3´Î£¬Ã¿´Î¼ä¸ô30sRedis pingʧ°Ü£¬ÈÏΪʵÀý¹Òµô£¬·¢³ö±¨¾¯»òÕß×Ô¶¯Çл»
3£©Cluster manager²»½¨Òé²ÎÓëMysql groupÖ÷±¸²ãÃæµÄ¹ÜÀí
MySQL Ö÷±¸²ãÃæµÄ¼¯Èº¹ÜÀí·½°¸£º
1¡¢MHA+VIP £¨»¥ÁªÍø¹«Ë¾×î³£Óã©
2¡¢Î¢ÐÅphxsqlϵͳ£ºhttps://github.com/tencent-wechat/phxsql
½ðÈÚ¼¶¿É¿¿ÐÔ
Îå¡¢ MySQL ¼¯Èº·½°¸
5.1 ¼Ü¹¹Í¼

5.2 Éè¼ÆÔÔò
1£©Í³Ò»µÄschemaless±í½á¹¹

2£©¿ç»úµÄÊý¾Ý·Ö²¼
Ö§³Ö½«µ¥Âß¼±íˮƽ²ð·Öµ½¶à¸öMysql·þÎñÆ÷ÖÐ
3£©ÆäËû˵Ã÷
1¡¢Êý¾Ý´æ´¢¿É¿¿ÐԸߣ¬ËùÓÐÒµÎñÊý¾Ýͨ¹ýÐòÁл¯´æ´¢µ½valueÁÐ
2¡¢Ã¿ÐÐÊý¾Ý×Ô´ø°æ±¾ºÅ£¬ÒµÎñͨ¹ýcas·½Ê½·ÀÖ¹ÒµÎñ²ã¶àʵÀýͬʱдÔì³Éд¸²¸Ç
È«¾ÖΨһ°æ±¾ºÅʵÏÖ£º±¾»ú΢Ãëʱ¼ä´Á+server_id+proccess_id
3¡¢¹Ì¶¨°Ù¿â°Ù±í/°Ù¿âÊ®±íµÄÊý¾Ý²ð·Ö·½Ê½£¬¶à»ú¿çMysqlʵÀý²¿Êð
5.3 ·ÓɲßÂÔ
1) Ò»ÖÂÐÔhash

2) ·ÓɼÆËãËã·¨
crc32/md5/»ùÓÚ×Ö·û´®µÄ¸÷ÀàhashËã·¨
3) ·ÓÉÐÅÏ¢¸ñʽ
CREATETABLE
`Mysql_shard_info` (
`appid` int(32) NOT NULL,
`begin` int(32) NOT NULL,
`end` int(32) NOT NULL,
`ip` varchar(20) NOT NULL DEFAULT '',
`port` int(11) NOT NULL DEFAULT '0',
`user` varchar(50) NOT NULL DEFAULT '',
`pwd` varchar(50) NOT NULL DEFAULT '',
PRIMARY KEY (`appid`,`begin`)
)ENGINE=InnoDB DEFAULT CHARSET=utf8; |
5.4 Êý¾ÝÇ¨ÒÆ/×Ô¶¯À©Õ¹
Êý¾ÝÇ¨ÒÆ:
STEP1£ºÀûÓà MySQL Ö÷±¸¸´ÖÆ»úÖÆ½øÐÐÊý¾Ý¸´ÖÆ
STEP2£ºÊý¾Ý²îÒìСÓÚijһÁÙ½çÖµ£¬Í£Ö¹ÀÏ·ÖÆ¬Ð´²Ù×÷£¨read-only£©
STEP3£ºµÈ´ýÐÂ·ÖÆ¬Êý¾Ý¸üÐÂÍê±Ï
STEP4£º¸ü·ÓɹæÔò·ÓɹæÔò£¬Cluster ManagerÏòËùÓÐaccess proxy¸üзÓÉÐÅÏ¢
STEP5£ºÉ¾³ýÀÏ·ÖÆ¬
×Ô¶¯À©Õ¹:
¹ý³ÌÀàËÆÓÚÊý¾ÝÇ¨ÒÆ

Áù¡¢Redis ¼¯Èº·½°¸
6.1 ²¿Êð·½Ê½
1¡¢Òì¹¹¶Áд·ÖÀë-MySQLд£¬Redis¶Á
1) Êý¾Ýд²Ù×÷ÔÚ MySQL £¬¶Á²Ù×÷ÔÚRedis
2) Êý¾Ýͨ¹ýSyncϵͳ¶Ôbinlog½øÐнâÎö´ÓMysqlͬ²½µ½Redis
3) Êý¾ÝÓÐͬ²½ÑÓ³Ù£¨Ð¡ÓÚ100ms£©£¬ÊµÏÖ×îÖÕÒ»ÖÂÐÔ
ÊÊÓó¡¾°£ºÒªÇóÊý¾Ý¸ß¿É¿¿£¬ÇÒ¶ÁÁ¿±È½Ï´ó£¬ÔÊÐí¶ÁÊý¾Ý¶Ìʱ¼ä²»Ò»Ö£¬ÈôÆÚÍûÒ»Ö±¶Áµ½×îÐÂÊý¾Ý£¬ÇëʹÓÃget_with_version()½Ó¿Ú´Ó
MySQL ¶ÁÈ¡
2¡¢¶ÀÁ¢µÄRedis¼¯Èº·þÎñ
1£©¶Áд¾ùÔÚRedis£¬Ìṩ¶ÀÁ¢µÄKV´æ´¢·þÎñ
2) Óû§²»ÓùØ×¢À©ÈÝ/ËõÈÝ/¹ÊÕϻָ´µÈÎÊÌâ
3) ¼¯ÈºÄÚ¶àÒµÎñ»ì´æ£¬Ìá¸ßÄÚ´æµÄʹÓÃÂÊ
ÊÊÓó¡¾°£º¶ÀÁ¢µÄRedis¼¯Èº·þÎñ£¬ÀàËÆtwenproxy/codis
6.2 Éè¼ÆÒªµã
1¡¢Ò»ÖÂÐÔhash
Ö§³ÖÊý¾Ý¿çRedisʵÀý²ð·Ö£¬¹Ì¶¨SlotÊý½øÐвð·Ö
2¡¢µ¥»ú¶àʵÀý²¿Êð
1£©Ã¿¸öÎïÀí»úÖ§³Ö¶àRedisʵÀý
2£©Ã¿¸öRedisʵÀýÖ»·þÎñµ¥¸öÒµÎñ
3£©RedisʵÀýÄÚ´æ´óСȡ¾öÓÚÒµÎñÐèÇó£¬Í¬Ê±¿¼ÂÇÒµÎñ·ÃÎÊÁ¿ºÍÊý¾ÝÁ¿
ÒÔRedisIP+port±êʾΨһʵÀý£¬¶ÔÓÚ128GÄÚ´æ»úÆ÷£¬
¿ÉÅäÖÃ3 RedisʵÀý*ÿʵÀý30G
»ò10 RedisʵÀý*ÿʵÀý10G
»ò20 RedisʵÀý*ÿʵÀý5G
²ð·ÖÔÔò£ºµ¥ÊµÀý×î´óÄÚ´æÊ¹Óà < ±¾»úÊ£ÓàÄÚ´æ
3¡¢ÒÔSlotΪµ¥Î»µÄƽ»¬À©ÈÝ/ËõÈÝ
4¡¢ÒÔRedisʵÀýΪµ¥Î»µÄfailover´¦Àí
6.3 ƽ»¬À©ÈÝ/ËõÈÝ
Ö÷Òª²½ÖèÈçÏ£º
STEP1:È·ÈÏÀ©ÈÝ/ËõÈÝ
Cluster managerͨ¹ý¶Ôϵͳ¸ºÔغÍÊý¾ÝÁ¿½øÐи澯£¬½ø¶øÈ·ÈϽøÐÐÀ©ÈÝ»òÕßËõÈÝ
STEP2:Ð޸ķÓɱí
1)Ð޸ķÓÉ±í£¬½«¶ÔÓ¦shardµÄ״̬ÐÞ¸ÄΪmigrate״̬£¬²¢½«Ð·ÓÉÍÆË͵½ËùÓнÓÈë²ã
2)acc proxy»á½«Ð´²Ù×÷תµ½ÐµÄRedisʵÀýÖУ¬¶Á²Ù×÷ĬÈÏÏȶÁÐÂRedisʵÀý£¬key²»´æÔÚ»á¼ÌÐø´ÓÀϵÄRedisʵÀýÖжÁÈ¡
STEP3:Êý¾ÝÇ¨ÒÆ
1£©Cluster managerͨ¹ý×Ô¶¯Êý¾ÝÇ¨ÒÆ¹¤¾ß¿ªÊ¼Êý¾ÝÇ¨ÒÆ£¬¼Æ»®ÒÀÀµRedisµÄscanÃüÁÏà¹ØµÄkeyɨ³öÀ´£¬Í¨¹ýMIGRATE½øÐÐÊý¾ÝÇ¨ÒÆ
2£©¶à´ÎɨÃèÖ´Ðиùý³Ì£¬È·ÈÏSlotÖÐËùÓÐÊý¾ÝÇ¨ÒÆÍê³É
STEP4:Ð޸ķÓÉ±í£¬Ç¨ÒÆÍê³É
Cluster manager½«¶Áд¾ùÇе½ÐÂRedisʵÀý£¬²»ÔÙ´ÓÀÏRedisÖнøÐвÙ×÷
Æß¡¢ Sync Êý¾Ýͬ²½·þÎñ
7.1 ¼Ü¹¹

7.2 Ó¦Óó¡¾°
¸Ã·þÎñÍêÈ«¿ÉÒÔ³éÏó³É¶ÀÁ¢µÄÊý¾Ýͬ²½·Ö·¢·þÎñ£¬¶ÔÓÚÒòΪKV»¯¶ø¶ªÊ§µÄsql´¦ÀíÍêÈ«¿ÉÒÔͨ¹ý¸Ã·þÎñͬ²½µ½Æ«OLAPÀàµÄϵͳÖнøÐд¦Àí¡£³ýÁËͬ²½µ½Redis»¹¿ÉÒÔͬ²½µ½ElasticSearch»òÕßhbase»òÕßдhdfsÎļþ»ùÓÚhadoopÉú̬ȥʵÏÖ¸´ÔÓ¼ÆËãºÍ·ÖÎö¡£
7.3 Éè¼ÆÒªµã
1¡¢¼¯Èº¶Ô¼¯ÈºµÄʵʱÊý¾Ýͬ²½
MySQL ͳһҪÇóbinlogÈÕ־Ϊrow¸ñʽ
2¡¢²»Éæ¼°DDL´¦Àí
ÓÉÓÚ MySQL schemalessµÄÉè¼Æ£¬²»Óÿ¼ÂÇDDL´¦Àí£¬¼ò»¯Í¬²½·þÎñ£¨¿ç/²»¿çIDC£©
3¡¢»ùÓÚʱ¼ä´ÁµÄͬ²½ÑÓ³Ù¼à¿Ø
MySQL binlog row¸ñʽÈÕÖ¾×Ô´øÊ±¼ä´Á£¬»ùÓÚ´Ëʱ¼ä´Á½øÐÐͬ²½ÑÓ³Ù¼à¿Ø
4¡¢»ùÓÚbinlogÎļþÃû+offsetµÄͬ²½Î»ÖùÜÀí
¶¨Ê±¶¨Á¿³Ö¾Ã»¯±£´æµ±Ç°Í¬²½µÄbinlogÎļþÃû¼°offset,ÓÃÓÚ¸÷ÖÖ³¡¾°ÏµÄͬ²½»Ö¸´
5¡¢»ùÓÚÐеIJ¢ÐÐͬ²½
¶àÏß³Ìͬ²½Ä£Ê½£¬Ö÷Ïß³Ìͨ¹ý¶Ôtableid»òÕßkey×öhash,½«binlogeventʱ¼ä·Ö·¢µ½¶ÔÓ¦workerÏ̵߳ĶÓÁÐÖУ¬workerÏß³ÌÒÀ´Î´Ó¶ÓÁÐÖлñÈ¡binlog
eventÖ´ÐÐ
7.4 ʵÏÖÔÀí

ÔÀíÏà¶Ô±È½Ï¼òµ¥£º
1£©Syncͬ²½¹¤¾ßÄ£ÄâMysql slaveµÄ½»»¥ÐÒ飬αװ×Ô¼ºÎª MySQL slave£¬ÏòMysqlmaster·¢ËÍdumpÐÒé
2£©MysqlmasterÊÕµ½dumpÇëÇ󣬿ªÊ¼ÍÆËÍbinary log¸øslave(Ò²¾ÍÊÇͬ²½¹¤¾ß)
3£©Syncͬ²½¹¤¾ß½âÎöbinary log¶ÔÏó(ÔʼΪbyteÁ÷)£¬²¢×ª»»³ÉRedis»òÆäËû´æ´¢£¨hdfs/hbase/ESµÈÊý¾Ý¿â£©ÏàÓ¦Êý¾Ý²Ù×÷½Ó¿Ú»òÕß×÷ΪÏûÏ¢´æ´¢µ½MQÖУ¨rocketmq»òÕßkafka£©
7.5 ROW ¸ñʽevents
MySQL 5.5 BinlogµÄʼþÀàÐÍÓжàÖÖ£¬ÕâÀïÖ»½éÉÜÓëROWģʽÏà¹ØµÄʼþ
1) QUERY_EVENT£ºÓëSTATEMENTģʽ´¦ÀíÏàͬ£¬´æ´¢µÄÊÇSQL£¬Ö÷ÒªÊÇһЩÓëÊý¾ÝÎ޹صIJÙ×÷£¬eg:
begin¡¢drop table
2) TABLE_MAP_EVENT£º¼Ç¼ÁËÏÂÒ»ÌõʼþËù¶ÔÓ¦µÄ±íÐÅÏ¢£¬ÔÚÆäÖд洢ÁËÊý¾Ý¿âÃûºÍ±íÃû
3) WRITE_ROWS_EVENT£º²Ù×÷ÀàÐÍΪinsert
4) UPDATE_ROWS_EVENT£º²Ù×÷ÀàÐÍΪupdate
5) DELETE_ROWS_EVENT£º²Ù×÷ÀàÐÍΪdelete
6) XID_EVENT£¬ ÓÃÓÚ±êʶÊÂÎñÌá½»(commit)
µäÐ͵ÄinsertÓï¾äÓÐÈçÏÂ4¸öevents×é³É£º

7.6 ÆäËû¿ªÔ´Í¬²½·½°¸
1.
tungsten-replicator(JAVA)
http://code.google.com/p/tungsten-replicator/
2. linkedin databus(JAVA)
https://github.com/linkedin/databus
3. Alibaba canal(JAVA)
https://github.com/alibaba/canal /
|
|