Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Model Center   Code  
»áÔ±   
   
 
     
   
 ¶©ÔÄ
  ¾èÖú
ÈçºÎ»ùÓÚMySQL¼°Redis´î½¨Í³Ò»µÄKV´æ´¢·þÎñ
 
  4210  次浏览      28
 2017-11-24
 
±à¼­ÍƼö:
±¾ÎÄÀ´×ÔÓÚÉî¶È¿ªÔ´£¬±¾ÎĽéÉÜ»ùÓÚMySQL¼°Redis´î½¨Í³Ò»µÄkv´æ´¢·þÎñ£º³£Óò¿Êð·½Ê½¼°ÆäÌØµã£¬Cluster manager£¬MySQLºÍRedis¼¯Èº·½°¸£¬ÒÔ¼°SyncÊý¾Ýͬ²½·þÎñ¡£

 

Ò»¡¢MySQL+Redis ³£Óò¿Êð·½Ê½

1.1 ÍØÆË

1.2 ÌØµã

ÒµÎñ²ãͨ¹ý˫дͬʱдMySQL¼°Redis¡£¶Áͨ³£ÔÚRedis£¬Èô¶ÁÈ¡²»µ½£¬Ôò´ÓMySQL¶ÁÈ¡£¬È»ºó½«Êý¾Ýͬ²½µ½Redis£¬Redisͨ³£ÉèÖÃexpire»òÕßĬÈÏLRU½øÐÐÊý¾ÝÌÔÌ­¡£

ÕâÖÖʹÓ÷½Ê½»áÓÐÈçÏÂÎÊÌ⣺

1£©MySQL¼°Redis´æÔÚÊý¾Ý²»Ò»Ö·çÏÕ£¬ÓÈÆäÊdz¤Ê±¼äÔËÐеÄϵͳ

2£©ÒµÎñ²ãÐèÒª´¦ÀíMySQL sql schemaÓëRedis kvÊý¾Ý½á¹¹ÉϵÄÂß¼­²îÒì

3£©ÎÞͳһÔËά

4£©ÎÞ·¨·½±ãÀ©ÈÝ/ËõÈÝ

¶þ¡¢KV »¯µÄ´æ´¢Ê¹ÓÃÀíÄî

2.1 MySQL Is great NoSQL

ΪʲôҪÓÃMySQL:

¡°ÔÚ¿ÉÀ©Õ¹ÏµÍ³¹¹½¨Ê±£¬Ò»¸öºÜÖØÒªµÄ¿¼Á¿ÊÇʹÓõļ¼ÊõÊÇ·ñ³ÉÊ죬ѡÔñ³ÉÊìµÄ¼¼ÊõÒâζ×ųö´íʱÄܹ»Ñ¸ËÙ»Ö¸´¡£µ±È»£¬¿ª·¢ÕßÒ²¿ÉÒÔÔÚÏîÄ¿ÖÐʹÓÃ×îÐÂ×îÅ£µÄNoSQLÊý¾Ý¿â£¬¶øÕâ¸öÊý¾Ý¿âÔÚÀíÂÛÉÏÒ²¿ÉÒÔÁ¼ºÃµØÔËÐУ¬È»¶øÔÚÉú²ú»·¾³ÖгöÏÖÁËÎÊÌâ»Ö¸´ÐèÒª¶à¾Ã£¿¼¼ÊõÉÏÒÑÓеÄ֪ʶºÍ¾­Ñé»ýÀÛ¶ÔÓÚÎÊÌ⻺½âÖÁ¹ØÖØÒª£¬µ±È»Õâ¸ö»ýÀÛÒ²°üÀ¨ÁËGoogle¿ÉÒÔËÑË÷µ½µÄÄÚÈÝ¡£

Ïà±È֮ϣ¬¹ØÏµÐÍÊý¾Ý¿âÒѾ­´æÔÚÁ˳¬¹ýËÄÊ®Ä꣬ҵ½ç¶ÔÓÚ¹ØÏµÐÍÊý¾Ý¿âµÄά»¤Ò²»ýÀÛÁË´óÁ¿µÄ¾­Ñé¡£»ùÓÚÕâЩ¿¼ÂÇ£¬ÔÚÐÂÏîÄ¿×ö¼¼ÊõÑ¡ÐÍʱͨ³£»áÑ¡ÔñMysql£¬¶ø²»ÊÇNoSQLÊý¾Ý¿â£¬³ý·ÇNoSQLÕæµÄÓзdz£·Ç³£Ã÷ÏÔµÄÓÅÊÆ¡£¡±

2.2 KVÀíÄî

¶ÔÓÚÒÚ¼¶¹æÄ£µÄÊý¾Ý´æ´¢£¬ÓÈÆäÊÇÉæ¼°µ½Ë®Æ½²ð·Ö¿ç»ú·Ö¿â·Ö±íµÄÇé¿öÏ£¬ÏßÉ϶ÔÊý¾Ý¿âµÄ·ÃÎÊÖ»ÄÜ×öµÄÔ½¼òµ¥Ô½ºÃ£¬group by/order by/·ÖÒ³/ͨÓÃjoin/ÊÂÎñµÈµÈµÄÖ§³Ö ÔÚÕâ¸öÁ¿¼¶ÏµÄMySQLϵͳ¶¼ÊDz»ºÏÊʵġ£

»ù±¾ÉÏĿǰËùÓеÄÀàproxyµÄMySQL·½°¸ÕæÕýÉϹæÄ£ÏßÉÏÓ¦ÓÃÖ»ÄÜʹÓð´²ð·Ö¼ü½øÐжÁд²Ù×÷£¬Êµ¼ÊÉÏÒ²ÊÇÒ»¸öÓòð·Ö¼ü×öµÄÒ»¸ökvϵͳ¡£

ÈôÏëʹÓø´ÔÓµÄsql´¦Àí£¬×îºÏÀíµÄ²¿Êð·½°¸Êǽ«MysqlbinlogÁ÷ˮͬ²½·þÎñ³éÏó³öÀ´£¬Í¨¹ýʵʱͬ²½µ½OLAPÀàµÄϵͳ½øÐд¦Àí¡£

ËùÒÔÃæÏòº£Á¿´æ´¢·þÎñ£¬MySQL´ÓÒ»¿ªÊ¼¾ÍÉè¼ÆÎªÒ»¸öKVϵͳÊÇ¿ÉÐеġ£valueʹÓÃmediumblob´æ´¢xml/json/protobuf/thrift¸ñʽ»¯Êý¾ÝÐòÁл¯Ö®ºóµÄÊý¾Ý¡£

2.3 MySQL KV»¯µÄʹÓ÷½Ê½

1¡¢ÓÃMySQLÔ­À´µÄÖ÷¼ü»òÕßË÷Òý¼üµ±×ökey

2¡¢ÆäËûËùÓеķÇÖ÷¼ü·ÇË÷Òý¼ü£¬È«²¿°ü×°µ½valueÀïÃæ£¬valueʹÓÃmediumblob´æ´¢xml/json/protobuf/thrift¸ñʽ»¯Êý¾ÝÐòÁл¯Ö®ºóµÄÊý¾Ý¡£

3¡¢Êý¾Ý¶Áд²Ù×÷£¬¾ù»ùÓÚkeyÒ»ÕûÐÐÊý¾Ý×ö¶Áд£¬ÓÉÒµÎñ²ã¶ÔÀïÃævalueµÄ½á¹¹×ö½âÎö¼°¶ÔÄÚ²¿½á¹¹×öÔöɾ¸Ä²î£¬¶ø²»Óñä¸ü MySQL ±¾ÉíµÄschema.

2.4 ²»ÊÊÓó¡¾°

1¡¢Êý¾ÝÁ¿ºÍ·ÃÎÊÁ¿²»´ó²¢ÇÒÒµÎñÂß¼­ÒÀÀµ MySQL Êý¾Ý¿â½øÐд¦ÀíµÄÒµÎñ³¡¾°

2¡¢Éæ¼°µ½¶à±íjoinµÈµÄ´¦Àí

¶ÔÓÚ´ËÏÞÖÆ£¬Ò²¿ÉÒÔͨ¹ý½«¹ØÁª±í¼Ó¹¤³É»ùÓÚ¹ØÁªÌõ¼þµÄÒ»ÕÅ¿í±í½øÐÐKV»¯¡£

3¡¢Éæ¼°µ½ÊÂÎñµÈµÄ´¦Àí¡£

Èý¡¢½«MySQL+RedisÉè¼ÆÎªÍ³Ò»µÄKV´æ´¢·þÎñ

3.1 Ä¿±ê

1£©ÒµÎñ²ãͨ¹ýͳһ·½Ê½·ÃÎÊMySQL¼°Redis£¬²»ÔÙʹÓÃMySQL¿Í»§¶Ë¼°Redis¿Í»§¶Ë·ÃÎÊ

2£©MySQL¼¯Èº»¯/Redis¼¯Èº»¯²¿Êð

3£©½«ÒµÎñ˫д¸ÄΪMySQLµ½Redisµ×²ãbinlogÊý¾Ýͬ²½·½Ê½Íê³Éͬ²½

4£©Òì¹¹Êý¾Ý´æ´¢Ö§³Ö×îÖÕÒ»ÖÂÐÔÊý¾Ý¶Áд·þÎñ

5£©Ö§³Ö´æ´¢²ãÃæÀ©ÈÝËõÈÝ¡¢failoverÇÒÒµÎñÎÞ¸ÐÖª

6£©µ¥»úȺÈÕ°ÙÒÚ´ÎQPS/TPSÖ§³Ö£¨´óÀàÒµÎñÊʶȲð·Öµ½²»Í¬¼¯ÈºÖУ©

3.2 ×îÖÕʵÏÖ

»ùÓÚMySQL+RedisµÄͳһ´æ´¢·þÎñ£¨UniStore£© =

MySQL¿ç»ú·Ö¿â·Ö±í¼¯Èº

+ Redis¼¯Èº

+ MySQL->RedisʵʱÊý¾Ýͬ²½·þÎñ

+ ͳһµÄ¶ÔÍâÊý¾Ý·ÃÎʽӿÚ

+ ÄÚÔÚµÄÍêÕûÔËά֧³Öϵͳ£¨Ö§³ÖÔÚÏßÀ©ÈÝ/ËõÈÝ¡¢failoverµÈ£©

3.3 ¼Ü¹¹Í¼

3.4 ¼Ü¹¹ËµÃ÷-½«´æ´¢Éè¼ÆÎªÒ»ÖÖ·þÎñ

1¡¢½«MySQL+Redis×ö³ÉͳһKV´æ´¢·þÎñ

2¡¢Í¨¹ýacc proxyÌṩͳһµÄÊý¾Ý·ÃÎʽӿڣ¬Í¨¹ýͳһЭÒéÖ§³Ö¿çÓïÑÔÊý¾Ý·ÃÎÊ

·ÃÎÊЭÒ飨×Ô¶¨ÒåЭÒ飬protobufЭÒ飬thriftЭÒéµÈ£©

3¡¢MySQL clusterÖ§³Ö¿ç»úµÄ·Ö¿â·Ö±í£¬schemalessÉè¼Æ£¬ËùÓÐÒµÎñ±íKV»¯Éè¼Æ

4¡¢Redis clusterÖ§³Ö¿ç»úµÄʵÀý²ð·Ö

5¡¢SyncÊý¾Ýͬ²½·þÎñÌṩͳһµÄMysqlµ½Redis ¿çIDC/²»¿çIDCÊý¾Ýͬ²½·þÎñ£¬Ð¡ÓÚ100msÑÓʱ

6¡¢Õû¸öϵͳ²»Éæ¼°µ½·Ö²¼Ê½ÊÂÎñ´¦Àí

3.5 ÈýÖÖ²¿Êð·½Ê½

1¡¢´¿MySQL¼¯Èº²¿Êð

´ËÖÖ²¿Êð·½Ê½µÈͬÓÚÆäËûMySQL proxy¿ç»ú·Ö¿â·Ö±í·½°¸£¬¶Áд¾ùÔÚMySQL

2¡¢´¿Redis¼¯Èº²¿Êð

´ËÖÖ²¿Êð·½Ê½µÈͬÓÚÆäËûRedis proxy¿ç»ú·Ö¿â·Ö±í·½°¸£¬¶Áд¾ùÔÚRedis

3¡¢MySQL+RedisÒì¹¹²¿Êð

дÔÚMySQL£¬¶Á¿ÉÒÔ´ÓMySQL¶Á»òÕßRedis¶Á£¬È¡¾öÓÚÒµÎñ¶Ô×îÐÂÊý¾ÝµÄ¶ÁȡҪÇó¡£

3.6 ½Ó¿Ú˵Ã÷

1¡¢int get(int appid, string key,string& value)

Redis¶Á²Ù×÷רÓÃ

2¡¢int get_with_version(int appid,string key, string& value, int64& version)

MySQL ¶Á²Ù×÷רÓã¬×Ô´ø°æ±¾ºÅ£¬·Àֹд¸²¸Ç

3¡¢int set(int appid, string key,string value, int64 version)

ͨ¹ýappidÇø·Ö MySQL »¹ÊÇRedis£¬¾ùÖ§³Öд²Ù×÷

4¡¢int delete(int appid, string key)

ͨ¹ýappidÇø·Ö MySQL »¹ÊÇRedis, ²»Ö§³ÖÅúÁ¿É¾³ý

5¡¢int multiget(int appid,vector<string> keys, map<string, string>& key_value_pairs)

Ö§³ÖÅúÁ¿¶Á²Ù×÷£¬ÄÚ²¿µÄÊý¾Ý·Óɼ°Êý¾ÝºÏ²¢²»ÓùØÐÄ

6¡¢intmultiset(int appid, map<string, string>& key_value_pairs)

²»½¨ÒéÖ§³Ö£¬Éæ¼°µ½¿ç»úÊÂÎñÎÊÌ⣬ÎÞ·¨±£Ö¤ACID

7¡¢int Redis_op(string cmd, ¡­¡­)

RedisÆäËûÔ­Éú½Ó¿Ú·â×°(incr/expire/list/setnxµÈ)

ËÄ¡¢Cluster Manager·þÎñ

4.1 Cluster Manager ÊÇÒ»¸öservice

cluster managerÖ÷ÒªÓÉÈçϼ¸ÖÖ¹¦ÄÜ

1£©MySQL/Redis·ÖƬ·ÓÉÐÅÏ¢µÄ¹ÜÀí

1¡¢ MySQL ·Ö¿â·Ö±í·ÓÉÐÅÏ¢

2¡¢Redis Slot·ÓÉÐÅÏ¢

3¡¢Â·ÓÉÐÅÏ¢±ä¸ü¹ÜÀí

2£©RedisʵÀýµÄ̽»î¼°RedisÀ©Èݼ°ËõÈÝÊý¾ÝµÄÇ¨ÒÆ

±ÈÈçÁ¬Ðø3´Î£¬Ã¿´Î¼ä¸ô30sRedis pingʧ°Ü£¬ÈÏΪʵÀý¹Òµô£¬·¢³ö±¨¾¯»òÕß×Ô¶¯Çл»

3£©Cluster manager²»½¨Òé²ÎÓëMysql groupÖ÷±¸²ãÃæµÄ¹ÜÀí

MySQL Ö÷±¸²ãÃæµÄ¼¯Èº¹ÜÀí·½°¸£º

1¡¢MHA+VIP £¨»¥ÁªÍø¹«Ë¾×î³£Óã©

2¡¢Î¢ÐÅphxsqlϵͳ£ºhttps://github.com/tencent-wechat/phxsql ½ðÈÚ¼¶¿É¿¿ÐÔ

Îå¡¢ MySQL ¼¯Èº·½°¸

5.1 ¼Ü¹¹Í¼

5.2 Éè¼ÆÔ­Ôò

1£©Í³Ò»µÄschemaless±í½á¹¹

2£©¿ç»úµÄÊý¾Ý·Ö²¼

Ö§³Ö½«µ¥Âß¼­±íˮƽ²ð·Öµ½¶à¸öMysql·þÎñÆ÷ÖÐ

3£©ÆäËû˵Ã÷

1¡¢Êý¾Ý´æ´¢¿É¿¿ÐԸߣ¬ËùÓÐÒµÎñÊý¾Ýͨ¹ýÐòÁл¯´æ´¢µ½valueÁÐ

2¡¢Ã¿ÐÐÊý¾Ý×Ô´ø°æ±¾ºÅ£¬ÒµÎñͨ¹ýcas·½Ê½·ÀÖ¹ÒµÎñ²ã¶àʵÀýͬʱдÔì³Éд¸²¸Ç

È«¾ÖΨһ°æ±¾ºÅʵÏÖ£º±¾»ú΢Ãëʱ¼ä´Á+server_id+proccess_id

3¡¢¹Ì¶¨°Ù¿â°Ù±í/°Ù¿âÊ®±íµÄÊý¾Ý²ð·Ö·½Ê½£¬¶à»ú¿çMysqlʵÀý²¿Êð

5.3 ·ÓɲßÂÔ

1) Ò»ÖÂÐÔhash

2) ·ÓɼÆËãËã·¨

crc32/md5/»ùÓÚ×Ö·û´®µÄ¸÷ÀàhashËã·¨

3) ·ÓÉÐÅÏ¢¸ñʽ

CREATETABLE `Mysql_shard_info` (

`appid` int(32) NOT NULL,

`begin` int(32) NOT NULL,

`end` int(32) NOT NULL,

`ip` varchar(20) NOT NULL DEFAULT '',

`port` int(11) NOT NULL DEFAULT '0',

`user` varchar(50) NOT NULL DEFAULT '',

`pwd` varchar(50) NOT NULL DEFAULT '',

PRIMARY KEY (`appid`,`begin`)

)ENGINE=InnoDB DEFAULT CHARSET=utf8;

5.4 Êý¾ÝÇ¨ÒÆ/×Ô¶¯À©Õ¹

Êý¾ÝÇ¨ÒÆ:

STEP1£ºÀûÓà MySQL Ö÷±¸¸´ÖÆ»úÖÆ½øÐÐÊý¾Ý¸´ÖÆ

STEP2£ºÊý¾Ý²îÒìСÓÚijһÁÙ½çÖµ£¬Í£Ö¹ÀÏ·ÖÆ¬Ð´²Ù×÷£¨read-only£©

STEP3£ºµÈ´ýÐÂ·ÖÆ¬Êý¾Ý¸üÐÂÍê±Ï

STEP4£º¸ü·ÓɹæÔò·ÓɹæÔò£¬Cluster ManagerÏòËùÓÐaccess proxy¸üзÓÉÐÅÏ¢

STEP5£ºÉ¾³ýÀÏ·ÖÆ¬

×Ô¶¯À©Õ¹:

¹ý³ÌÀàËÆÓÚÊý¾ÝÇ¨ÒÆ

Áù¡¢Redis ¼¯Èº·½°¸

6.1 ²¿Êð·½Ê½

1¡¢Òì¹¹¶Áд·ÖÀë-MySQLд£¬Redis¶Á

1) Êý¾Ýд²Ù×÷ÔÚ MySQL £¬¶Á²Ù×÷ÔÚRedis

2) Êý¾Ýͨ¹ýSyncϵͳ¶Ôbinlog½øÐнâÎö´ÓMysqlͬ²½µ½Redis

3) Êý¾ÝÓÐͬ²½ÑÓ³Ù£¨Ð¡ÓÚ100ms£©£¬ÊµÏÖ×îÖÕÒ»ÖÂÐÔ

ÊÊÓó¡¾°£ºÒªÇóÊý¾Ý¸ß¿É¿¿£¬ÇÒ¶ÁÁ¿±È½Ï´ó£¬ÔÊÐí¶ÁÊý¾Ý¶Ìʱ¼ä²»Ò»Ö£¬ÈôÆÚÍûÒ»Ö±¶Áµ½×îÐÂÊý¾Ý£¬ÇëʹÓÃget_with_version()½Ó¿Ú´Ó MySQL ¶ÁÈ¡

2¡¢¶ÀÁ¢µÄRedis¼¯Èº·þÎñ

1£©¶Áд¾ùÔÚRedis£¬Ìṩ¶ÀÁ¢µÄKV´æ´¢·þÎñ

2) Óû§²»ÓùØ×¢À©ÈÝ/ËõÈÝ/¹ÊÕϻָ´µÈÎÊÌâ

3) ¼¯ÈºÄÚ¶àÒµÎñ»ì´æ£¬Ìá¸ßÄÚ´æµÄʹÓÃÂÊ

ÊÊÓó¡¾°£º¶ÀÁ¢µÄRedis¼¯Èº·þÎñ£¬ÀàËÆtwenproxy/codis

6.2 Éè¼ÆÒªµã

1¡¢Ò»ÖÂÐÔhash

Ö§³ÖÊý¾Ý¿çRedisʵÀý²ð·Ö£¬¹Ì¶¨SlotÊý½øÐвð·Ö

2¡¢µ¥»ú¶àʵÀý²¿Êð

1£©Ã¿¸öÎïÀí»úÖ§³Ö¶àRedisʵÀý

2£©Ã¿¸öRedisʵÀýÖ»·þÎñµ¥¸öÒµÎñ

3£©RedisʵÀýÄÚ´æ´óСȡ¾öÓÚÒµÎñÐèÇó£¬Í¬Ê±¿¼ÂÇÒµÎñ·ÃÎÊÁ¿ºÍÊý¾ÝÁ¿

ÒÔRedisIP+port±êʾΨһʵÀý£¬¶ÔÓÚ128GÄÚ´æ»úÆ÷£¬

¿ÉÅäÖÃ3 RedisʵÀý*ÿʵÀý30G

»ò10 RedisʵÀý*ÿʵÀý10G

»ò20 RedisʵÀý*ÿʵÀý5G

²ð·ÖÔ­Ôò£ºµ¥ÊµÀý×î´óÄÚ´æÊ¹Óà < ±¾»úÊ£ÓàÄÚ´æ

3¡¢ÒÔSlotΪµ¥Î»µÄƽ»¬À©ÈÝ/ËõÈÝ

4¡¢ÒÔRedisʵÀýΪµ¥Î»µÄfailover´¦Àí

6.3 ƽ»¬À©ÈÝ/ËõÈÝ

Ö÷Òª²½ÖèÈçÏ£º

STEP1:È·ÈÏÀ©ÈÝ/ËõÈÝ

Cluster managerͨ¹ý¶Ôϵͳ¸ºÔغÍÊý¾ÝÁ¿½øÐи澯£¬½ø¶øÈ·ÈϽøÐÐÀ©ÈÝ»òÕßËõÈÝ

STEP2:Ð޸ķÓɱí

1)Ð޸ķÓÉ±í£¬½«¶ÔÓ¦shardµÄ״̬ÐÞ¸ÄΪmigrate״̬£¬²¢½«Ð·ÓÉÍÆË͵½ËùÓнÓÈë²ã

2)acc proxy»á½«Ð´²Ù×÷תµ½ÐµÄRedisʵÀýÖУ¬¶Á²Ù×÷ĬÈÏÏȶÁÐÂRedisʵÀý£¬key²»´æÔÚ»á¼ÌÐø´ÓÀϵÄRedisʵÀýÖжÁÈ¡

STEP3:Êý¾ÝÇ¨ÒÆ

1£©Cluster managerͨ¹ý×Ô¶¯Êý¾ÝÇ¨ÒÆ¹¤¾ß¿ªÊ¼Êý¾ÝÇ¨ÒÆ£¬¼Æ»®ÒÀÀµRedisµÄscanÃüÁÏà¹ØµÄkeyɨ³öÀ´£¬Í¨¹ýMIGRATE½øÐÐÊý¾ÝÇ¨ÒÆ

2£©¶à´ÎɨÃèÖ´Ðиùý³Ì£¬È·ÈÏSlotÖÐËùÓÐÊý¾ÝÇ¨ÒÆÍê³É

STEP4:Ð޸ķÓÉ±í£¬Ç¨ÒÆÍê³É

Cluster manager½«¶Áд¾ùÇе½ÐÂRedisʵÀý£¬²»ÔÙ´ÓÀÏRedisÖнøÐвÙ×÷

Æß¡¢ Sync Êý¾Ýͬ²½·þÎñ

7.1 ¼Ü¹¹

7.2 Ó¦Óó¡¾°

¸Ã·þÎñÍêÈ«¿ÉÒÔ³éÏó³É¶ÀÁ¢µÄÊý¾Ýͬ²½·Ö·¢·þÎñ£¬¶ÔÓÚÒòΪKV»¯¶ø¶ªÊ§µÄsql´¦ÀíÍêÈ«¿ÉÒÔͨ¹ý¸Ã·þÎñͬ²½µ½Æ«OLAPÀàµÄϵͳÖнøÐд¦Àí¡£³ýÁËͬ²½µ½Redis»¹¿ÉÒÔͬ²½µ½ElasticSearch»òÕßhbase»òÕßдhdfsÎļþ»ùÓÚhadoopÉú̬ȥʵÏÖ¸´ÔÓ¼ÆËãºÍ·ÖÎö¡£

7.3 Éè¼ÆÒªµã

1¡¢¼¯Èº¶Ô¼¯ÈºµÄʵʱÊý¾Ýͬ²½

MySQL ͳһҪÇóbinlogÈÕ־Ϊrow¸ñʽ

2¡¢²»Éæ¼°DDL´¦Àí

ÓÉÓÚ MySQL schemalessµÄÉè¼Æ£¬²»Óÿ¼ÂÇDDL´¦Àí£¬¼ò»¯Í¬²½·þÎñ£¨¿ç/²»¿çIDC£©

3¡¢»ùÓÚʱ¼ä´ÁµÄͬ²½ÑÓ³Ù¼à¿Ø

MySQL binlog row¸ñʽÈÕÖ¾×Ô´øÊ±¼ä´Á£¬»ùÓÚ´Ëʱ¼ä´Á½øÐÐͬ²½ÑÓ³Ù¼à¿Ø

4¡¢»ùÓÚbinlogÎļþÃû+offsetµÄͬ²½Î»ÖùÜÀí

¶¨Ê±¶¨Á¿³Ö¾Ã»¯±£´æµ±Ç°Í¬²½µÄbinlogÎļþÃû¼°offset,ÓÃÓÚ¸÷ÖÖ³¡¾°ÏµÄͬ²½»Ö¸´

5¡¢»ùÓÚÐеIJ¢ÐÐͬ²½

¶àÏß³Ìͬ²½Ä£Ê½£¬Ö÷Ïß³Ìͨ¹ý¶Ôtableid»òÕßkey×öhash,½«binlogeventʱ¼ä·Ö·¢µ½¶ÔÓ¦workerÏ̵߳ĶÓÁÐÖУ¬workerÏß³ÌÒÀ´Î´Ó¶ÓÁÐÖлñÈ¡binlog eventÖ´ÐÐ

7.4 ʵÏÖÔ­Àí

Ô­ÀíÏà¶Ô±È½Ï¼òµ¥£º

1£©Syncͬ²½¹¤¾ßÄ£ÄâMysql slaveµÄ½»»¥Ð­Ò飬αװ×Ô¼ºÎª MySQL slave£¬ÏòMysqlmaster·¢ËÍdumpЭÒé

2£©MysqlmasterÊÕµ½dumpÇëÇ󣬿ªÊ¼ÍÆËÍbinary log¸øslave(Ò²¾ÍÊÇͬ²½¹¤¾ß)

3£©Syncͬ²½¹¤¾ß½âÎöbinary log¶ÔÏó(ԭʼΪbyteÁ÷)£¬²¢×ª»»³ÉRedis»òÆäËû´æ´¢£¨hdfs/hbase/ESµÈÊý¾Ý¿â£©ÏàÓ¦Êý¾Ý²Ù×÷½Ó¿Ú»òÕß×÷ΪÏûÏ¢´æ´¢µ½MQÖУ¨rocketmq»òÕßkafka£©

7.5 ROW ¸ñʽevents

MySQL 5.5 BinlogµÄʼþÀàÐÍÓжàÖÖ£¬ÕâÀïÖ»½éÉÜÓëROWģʽÏà¹ØµÄʼþ

1) QUERY_EVENT£ºÓëSTATEMENTģʽ´¦ÀíÏàͬ£¬´æ´¢µÄÊÇSQL£¬Ö÷ÒªÊÇһЩÓëÊý¾ÝÎ޹صIJÙ×÷£¬eg: begin¡¢drop table

2) TABLE_MAP_EVENT£º¼Ç¼ÁËÏÂÒ»ÌõʼþËù¶ÔÓ¦µÄ±íÐÅÏ¢£¬ÔÚÆäÖд洢ÁËÊý¾Ý¿âÃûºÍ±íÃû

3) WRITE_ROWS_EVENT£º²Ù×÷ÀàÐÍΪinsert

4) UPDATE_ROWS_EVENT£º²Ù×÷ÀàÐÍΪupdate

5) DELETE_ROWS_EVENT£º²Ù×÷ÀàÐÍΪdelete

6) XID_EVENT£¬ ÓÃÓÚ±êʶÊÂÎñÌá½»(commit)

µäÐ͵ÄinsertÓï¾äÓÐÈçÏÂ4¸öevents×é³É£º

7.6 ÆäËû¿ªÔ´Í¬²½·½°¸

1. tungsten-replicator(JAVA)

http://code.google.com/p/tungsten-replicator/

2. linkedin databus(JAVA)

https://github.com/linkedin/databus

3. Alibaba canal(JAVA)

https://github.com/alibaba/canal /

 
   
4210 ´Îä¯ÀÀ       28
Ïà¹ØÎÄÕÂ

»ùÓÚEAµÄÊý¾Ý¿â½¨Ä£
Êý¾ÝÁ÷½¨Ä££¨EAÖ¸ÄÏ£©
¡°Êý¾Ýºþ¡±£º¸ÅÄî¡¢ÌØÕ÷¡¢¼Ü¹¹Óë°¸Àý
ÔÚÏßÉ̳ÇÊý¾Ý¿âϵͳÉè¼Æ ˼·+Ч¹û
 
Ïà¹ØÎĵµ

GreenplumÊý¾Ý¿â»ù´¡Åàѵ
MySQL5.1ÐÔÄÜÓÅ»¯·½°¸
ijµçÉÌÊý¾ÝÖÐ̨¼Ü¹¹Êµ¼ù
MySQL¸ßÀ©Õ¹¼Ü¹¹Éè¼Æ
Ïà¹Ø¿Î³Ì

Êý¾ÝÖÎÀí¡¢Êý¾Ý¼Ü¹¹¼°Êý¾Ý±ê×¼
MongoDBʵս¿Î³Ì
²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
PostgreSQLÊý¾Ý¿âʵսÅàѵ