±à¼ÍƼö: |
±¾ÎÄÓà 7 ÕÅͼ×ܽáÁË RocketMQ µÄºËÐÄ֪ʶ£¬ÔÚ½²½âʱÁ¦Ç󾫼ò¡¢Í¨Ë×Ò×¶®£¬Í¨¹ýͼ½âÀ´¸øÕýÔÚѧϰ
RocketMQ µÄС»ï°é´øÀ´°ïÖú¡£RocketMQ Êǰ¢Àï°Í°ÍµÄ·Ö²¼Ê½ÏûÏ¢Öмä¼þ£¬ÔÚ
2012 Ä꿪Դ£¬ÔÚ 2017 Äê³ÉΪ Apache ¶¥¼¶ÏîÄ¿¡£
À´×ÔÓÚ51CTO ,ÓÉ»ðÁú¹ûÈí¼þLinda±à¼¡¢ÍƼö¡£ |
|
1 ¼¯Èº¼Ü¹¹
RocketMQ µÄ¼¯Èº¼Ü¹¹ÈçÏÂͼ£º

´ÓÉÏͼ¿ÉÒÔ¿´µ½£¬Õû¸ö¼¯ÈºÖÐÓÐËĸö½ÇÉ«£º Name Server¼¯Èº¡¢BrokerÖ÷´Ó¼¯Èº¡¢Producer¡¢Consumer¡£
1.1 Name Server ¼¯Èº
Name Server ¼¯Èº²¿Ê𣬵«ÊǽڵãÖ®¼ä²¢²»»áͬ²½Êý¾Ý£¬ÒòΪÿ¸ö½Úµã¶¼»á±£´æÍêÕûµÄÊý¾Ý¡£Òò´Ëµ¥¸ö½Úµã¹Òµô£¬²¢²»»á¶Ô¼¯Èº²úÉúÓ°Ïì¡£
1.2 Broker
Broker ²ÉÓÃÖ÷´Ó¼¯Èº£¬ÊµÏֶั±¾´æ´¢ºÍ¸ß¿ÉÓá£Ã¿¸ö Broker ½Úµã¶¼Òª¸úËùÓÐµÄ Name
Server ½Úµã½¨Á¢³¤Á¬½Ó£¬¶¨Òå×¢²á Topic ·ÓÉÐÅÏ¢ºÍ·¢ËÍÐÄÌø¡£
¸úËùÓÐ Name Server ½¨Á¢Á¬½Ó£¬¾Í²»»áÒòΪµ¥¸ö Name Server ¹ÒÁËÓ°Ïì Broker
ʹÓá£Broker Ö÷´ÓģʽÖУ¬ Slave ½ÚµãÖ÷¶¯´Ó Master ½ÚµãÀÈ¡ÏûÏ¢¡£
1.3 Producer
Producer ¸ú Name Server µÄÈÎÒâÒ»¸ö½Úµã½¨Á¢³¤Á¬½Ó£¬¶¨ÆÚ´Ó Name Server
ÀÈ¡ Topic ·ÓÉÐÅÏ¢¡£Producer ÊÇ·ñ²ÉÓü¯Èº£¬È¡¾öÓÚËüËùÔÚµÄÒµÎñϵͳ¡£
1.4 Consumer
Consumer ¸ú Name Server µÄÈÎÒâÒ»¸ö½Úµã½¨Á¢³¤Á¬½Ó£¬¶¨ÆÚ´Ó Name Server
ÀÈ¡ Topic ·ÓÉÐÅÏ¢¡£Consumer ÊÇ·ñ²ÉÓü¯Èº£¬È¡¾öÓÚËüËùÔÚµÄÒµÎñϵͳ¡£
Producer ºÍ Consumer Ö»¸úÈÎÒâÒ»¸ö Name Server ½Úµã½¨Á¢Á¬½Ó£¬ÒòΪ
Broker »áÏòËùÓÐ Name Server ×¢²á Topic ÐÅÏ¢£¬ËùÒÔÿ¸ö Name Server
±£´æµÄÊý¾ÝÆäʵÊÇÒ»Öµġ£
2 MessageQueue
Producer ·¢Ë͵ÄÏûÏ¢»áÔÚ Broker µÄ MessageQueue Öб£´æ£¬ÈçÏÂͼ£º

ÓÐÁË MessageQueue £¬Topic ¾Í¿ÉÒÔÔÚ Broker ÖÐʵÏÖ·Ö²¼Ê½´æ´¢£¬ÈçÉÏͼ£¬Broker
¼¯ÈºÖб£´æÁË 4 ¸öMessageQueue£¬ÕâЩ MessageQueue ±£´æÁË Topic1-Topic3
ÕâÈý¸ö Topic µÄÏûÏ¢¡£
MessageQueue ÀàËÆÓÚ Kafka ÖÐµÄ Partition£¬ÓÐÁË MessageQueue£¬Producer
¿ÉÒÔ²¢·¢µØÏò Broker Öз¢ËÍÏûÏ¢£¬Consumer Ò²¿ÉÒÔ²¢·¢µØÏû·ÑÏûÏ¢¡£
ĬÈÏ Topic ¿ÉÒÔ´´½¨µÄ MessageQueue ÊýÁ¿ÊÇ 4£¬Broker ¿ÉÒÔ´´½¨µÄ MessageQueue
ÊýÁ¿ÊÇ 8, RocketMQ Ñ¡Ôñ¶þÕßÖÐÊýÁ¿Ð¡µÄ£¬Ò²¾ÍÊÇ 4¡£²»¹ýÕâÁ½¸öÖµ¶¼¿ÉÒÔÅäÖá£
3 Consumer
RocketMQµÄÏû·ÑģʽÈçÏÂͼ£º

ͼÖУ¬Topic1 µÄÏûϢдÈëÁËÁ½¸ö MessageQueue£¬Á½¸ö¶ÓÁб£´æÔÚ Broker1 ºÍ
Broker2 ÉÏ¡£
RocketMQ ͨ¹ý Consumer Group ʵÏÖÏûÏ¢¹ã²¥¡£±ÈÈçÉÏͼÖÐÓÐÁ½¸öÏû·ÑÕß×飬ÿ¸öÏû·ÑÕß×éÓÐÁ½¸öÏû·ÑÕß¡£
Ò»¸öÏû·ÑÕß¿ÉÒÔÏû·Ñ¶à¸ö MessageQueue£¬µ«ÊÇͬһ¸ö MessageQueue Ö»Äܱ»Í¬Ò»¸öÏû·ÑÕß×éµÄÒ»¸öÏû·ÑÕßÏû·Ñ¡£±ÈÈç
MessageQueue0 Ö»Äܱ» Consumer Group1 ÖÐµÄ Consumer1 Ïû·Ñ£¬
²»Äܱ» Consumer2 Ïû·Ñ¡£
4 Broker ¸ß¿ÉÓü¯Èº
Broker ¼¯ÈºÈçÏÂͼ£º

Broker ͨ¹ýÖ÷´Ó¼¯ÈºÀ´ÊµÏÖÏûÏ¢¸ß¿ÉÓ᣸ú Kafka ²»Í¬µÄÊÇ£¬RocketMQ ²¢Ã»ÓÐ Master
½ÚµãÑ¡¾Ù¹¦ÄÜ£¬¶øÊDzÉÓöà Master ¶à Slave µÄ¼¯Èº¼Ü¹¹¡£Producer дÈëÏûϢʱдÈë
Master ½Úµã£¬Slave ½ÚµãÖ÷¶¯´Ó Master ½ÚµãÀÈ¡Êý¾ÝÀ´±£³Ö¸ú Master ½ÚµãµÄÊý¾ÝÒ»Ö¡£
Consumer Ïû·ÑÏûϢʱ£¬¼È¿ÉÒÔ´Ó Master ½ÚµãÀÈ¡Êý¾Ý£¬Ò²¿ÉÒÔ´Ó Slave ½ÚµãÀÈ¡Êý¾Ý¡£
µ½µ×ÊÇ´Ó Master ÀÈ¡»¹ÊÇ´Ó Slave Àȡȡ¾öÓÚ Master ½ÚµãµÄ¸ºÔØºÍ Slave
µÄͬ²½Çé¿ö ¡£Èç¹û Master ¸ºÔغܸߣ¬Master »á֪ͨ Consumer ´Ó Slave
ÀÈ¡ÏûÏ¢£¬¶øÈç¹û Slave ͬ²½ÏûÏ¢½ø¶ÈÑÓºó£¬Ôò Master »á֪ͨ Consumer ´Ó Master
ÀÈ¡Êý¾Ý¡£×ÜÖ®£¬´Ó Master ÀÈ¡»¹ÊÇ´Ó Slave ÀÈ¡ÓÉ Master À´¾ö¶¨¡£
Èç¹û Master ½Úµã·¢Éú¹ÊÕÏ£¬RocketMQ »áʹÓûùÓÚ raft ÐÒéµÄ DLedger
Ëã·¨À´½øÐÐÖ÷´ÓÇл»¡£
Broker ÿ¸ô 30s Ïò Name Server ·¢ËÍÐÄÌø£¬Name Server Èç¹û 120s
ûÓÐÊÕµ½ÐÄÌø£¬¾Í»áÅÐ¶Ï Broker å´»úÁË¡£
5 ÏûÏ¢´æ´¢
RocketMQ µÄ´æ´¢Éè¼ÆÊǷdz£Óд´ÔìÐԵġ£´æ´¢ÎļþÖ÷ÒªÓÐÈý¸ö£ºCommitLog¡¢ConsumeQueue¡¢Index¡£ÈçÏÂͼ£º

5.1 CommitLog
RocketMQ µÄÏûÏ¢±£´æÔÚ CommitLog ÖУ¬CommitLog ÿ¸öÎļþ 1G ´óС¡£ÓÐȤµÄÊÇ£¬ÎļþÃû²¢²»½Ð
CommitLog£¬¶øÊÇÓÃÏûÏ¢µÄÆ«ÒÆÁ¿À´ÃüÃû¡£±ÈÈçµÚÒ»¸öÎļþÎļþÃûÊÇ 0000000000000000000£¬µÚ¶þ¸öÎļþÎļþÃûÊÇ
00000000001073741824£¬ÒÀ´ÎÀàÍÆ¾Í¿ÉÒԵõ½ËùÓÐÎļþµÄÎļþÃû¡£
ÓÐÁËÉÏÃæµÄÃüÃû¹æÔò£¬¸ø¶¨Ò»¸öÏûÏ¢µÄÆ«ÒÆÁ¿£¬¾Í¿ÉÒÔ¸ù¾Ý¶þ·Ö²éÕÒ¿ìËÙÕÒµ½ÏûÏ¢ËùÔÚµÄÎļþ£¬²¢ÇÒÓÃÏûÏ¢Æ«ÒÆÁ¿¼õÈ¥ÎļþÃû¾Í¿ÉÒԵõ½ÏûÏ¢ÔÚÎļþÖÐµÄÆ«ÒÆÁ¿¡£
R oc ketMQ д CommitLog ʱ²ÉÓÃ˳Ðòд£¬´ó´óÌá¸ßÁËдÈëÐÔÄÜ¡£
5.2 ConsumeQueue
Èç¹ûÖ±½Ó´Ó CommitLog ÖмìË÷ Topic ÖеÄÒ»ÌõÏûÏ¢£¬Ð§ÂÊ»áºÜµÍ£¬ÒòΪÐèÒª´ÓÎļþµÄµÚÒ»ÌõÏûÏ¢¿ªÊ¼ÒÀ´Î²éÕÒ¡£ÒýÈëÁË
ConsumeQueue ×÷Ϊ CommitLog µÄË÷ÒýÎļþ£¬»áÈüìË÷ЧÂÊ´óÔö¡£
¸Õ¿ªÊ¼²»Àí½â ConsumeQueue ºÍ MessageQueue µÄÇø±ð£¬ÍøÉϲéÁËһЩ×ÊÁÏ·¢ÏÖ£¬Ã¿¸öConsumeQueue¶ÔÓ¦Ò»¸öÉÏÃæ½éÉܵÄ
MessageQueue£¬MessageQueue Ö»ÊÇÒ»¸ö¸ÅÄîÄ£ÐÍ¡£
ConsumeQueue ÖеÄÔªËØÄÚÈÝÈçÏ£º
ǰ 8 ¸ö×ֽڼǼÏûÏ¢ÔÚ CommitLog ÖÐµÄÆ«ÒÆÁ¿¡£
Öмä 4 ¸ö×ֽڼǼÏûÏ¢ÏûÏ¢´óС¡£
×îºó 8 ¸ö×ֽڼǼÏûÏ¢ÖÐ tag µÄ hashcode¡£
Õâ¸ö tag µÄ×÷Ó÷dz£ÖØÒª£¬¼ÙÈçÒ»¸ö Consumer ¶©ÔÄÁË TopicA£¬Tag1 ºÍ Tag2£¬ÄÇÕâ¸ö
Consumer µÄ¶©ÔĹØÏµÈçÏÂͼ£º

¿ÉÒÔ¿´µ½£¬Õâ¸ö¶©ÔĹØÏµÊÇÒ»¸ö hash ÀàÐ͵Ľṹ£¬key ÊÇ Topic Ãû³Æ£¬value ÊÇÒ»¸ö
SubscriptionData ÀàÐ͵ĶÔÏó£¬Õâ¸ö¶ÔÏó·â×°ÁË tag¡£
ÀÈ¡ÏûϢʱ£¬Ê×ÏÈ´Ó Name Server »ñÈ¡¶©ÔĹØÏµ£¬µÃµ½µ±Ç° Consumer ËùÓж©ÔÄ tag
µÄ hashcode ¼¯ºÏ codeSet£¬È»ºó´Ó ConsumerQueue »ñȡһÌõ¼Ç¼£¬ÅжÏ×îºó
8 ¸ö×Ö½Ú tag hashcode ÊÇ·ñÔÚ codeSet ÖУ¬ÒÔ¾ö¶¨ÊÇ·ñ½«¸ÃÏûÏ¢·¢Ë͸øConsumer¡£
5.3 Index Îļþ
RocketMQ Ö§³Ö°´ÕÕÏûÏ¢µÄÊôÐÔ²éÕÒÏûÏ¢£¬ÎªÁËÖ§³ÖÕâ¸ö¹¦ÄÜ£¬RocketMQ ÒýÈëÁË Index
Ë÷ÒýÎļþ¡£Index ÎļþÓÐÈý²¿·Ö×é³É£¬ÎļþÍ· IndexHead¡¢500Íò¸ö hash ²ÛºÍ 2000
Íò¸ö Index ÌõÄ¿×é³É¡£
5.3.1 IndexHead
×ܹ²ÓÐ 6 ¸öÔªËØ×é³É£¬Ç°Á½¸öÔªËØ±íʾµ±Ç°Õâ¸ö Index ÎļþÖеÚÒ»ÌõÏûÏ¢ºÍ×îºóÒ»ÌõÏûÏ¢µÄÂäÅÌʱ¼ä£¬µÚÈý¡¢µÚËÄÁ½¸öÔªËØ±íʾµ±Ç°Õâ¸ö
Index ÎļþÖеÚÒ»ÌõÏûÏ¢ºÍ×îºóÒ»ÌõÏûÏ¢ÔÚ CommitLog ÎļþÖеÄÎïÀíÆ«ÒÆÁ¿£¬µÚÎå¸öÔªËØ±íʾµ±Ç°Õâ¸ö
Index ÎļþÖÐ hash ²ÛµÄÊýÁ¿£¬µÚÁù¸öÔªËØ±íʾµ±Ç°Õâ¸ö Index ÎļþÖÐË÷ÒýÌõÄ¿µÄ¸öÊý¡£
²éÕÒµÄʱºò³ýÁË´«Èë key »¹ÐèÒª´«ÈëµÚÒ»ÌõÏûÏ¢ºÍ×îºóÒ»ÌõÏûÏ¢µÄÂäÅÌʱ¼ä£¬ÕâÊÇÒòΪ Index ÎļþÃûÊÇʱ¼ä´ÁÃüÃûµÄ£¬´«ÈëÂäÅÌʱ¼ä¿ÉÒÔ¸ü¼Ó¾«È·µØ¶¨Î»
Index Îļþ¡£
5.3.2 Hash ²Û
ÊìϤ Java ÖÐ HashMap µÄͬѧӦ¸Ã¶¼±È½ÏÊìϤ Hash ²ÛÕâ¸ö¸ÅÄîÁË£¬Æäʵ¾ÍÊÇ Hash
½á¹¹µÄµ×²ãÊý×é¡£Index ÎļþÖÐµÄ Hash ²ÛÓÐ 500 Íò¸öÊý×éÔªËØ£¬Ã¿¸öÔªËØÊÇ 4 ¸ö×Ö½Ú
int ÀàÐÍÔªËØ£¬±£´æµ±Ç°²ÛÏÂ×îеÄÄǸö index ÌõÄ¿µÄÐòºÅ¡£
ÕâÀï Hash ²Û½â¾ö Hash ³åÍ»µÄ·½Ê½ÊÇÁ´±í·¨£¬ÈçÏÂͼ£º

5.3.3 Index ÌõÄ¿
ÿ¸ö Index ÌõÄ¿ÖУ¬key µÄ hashcode Õ¼ 4 ¸ö×Ö½Ú£¬phyoffset ±íʾÏûÏ¢ÔÚ
CommitLog ÖеÄÎïÀíÆ«ÒÆÁ¿Õ¼ 8 ¸ö×Ö½Ú£¬timediff ±íʾÏûÏ¢µÄÂäÅÌʱ¼äÓë header
ÀïµÄ beginTimestamp µÄ²îÖµÕ¼ 4 ¸ö×Ö½Ú£¬pre index no Õ¼ 4 ¸ö×Ö½Ú¡£
pre index no ±£´æµÄÊǵ±Ç°µÄ Hash ²ÛÖÐǰһ¸ö index ÌõÄ¿µÄÐòºÅ£¬Ò»°ãÔÚ key
·¢Éú Hash ³åͻʱ²Å»áÓÐÖµ£¬·ñÔòÕâ¸öÖµ¾ÍÊÇ 0£¬±íʾµ±Ç°ÔªËØÊÇ Hash ²ÛÖеÚÒ»¸öÔªËØ¡£
In dex ÌõÄ¿Öб£´æ timediff£¬ÊÇΪÁË·ÀÖ¹ key ÖØ¸´¡£ ²éÕÒ key ʱ£¬ÔÚ key
ÏàͬµÄÇé¿öÏ£¬ Èç¹û´«ÈëµÄʱ¼ä·¶Î§¸ú timediff ²»Âú×㣬Ôò»á²éÕÒ pre index no
Õâ¸öÌõÄ¿¡£
5.3.4 ±¾½Ú×ܽá
ͨ¹ýÉÏÃæµÄ·ÖÎö£¬ÎÒÃÇ¿ÉÒÔ×ܽáÒ»¸öͨ¹ý key ÔÚ Index ÎļþÖвéÕÒÏûÏ¢µÄÁ÷³Ì£¬ÈçÏ£º
¼ÆËã key µÄ hashcode£»
¸ù¾Ý hashcode ÔÚ Hash ²ÛÖвéÕÒλÖà s£»
¼ÆËã Hash ²ÛÔÚ Index ÎļþÖÐλÖà 40+(s-1)*4£»
¶ÁÈ¡Õâ¸ö²ÛµÄÖµ£¬Ò²¾ÍÊÇIndexÌõÄ¿ÐòºÅ n£»
¼ÆËã¸Ã index ÌõÄ¿ÔÚ Index ÎļþÖеÄλÖ㬹«Ê½£º40 + 500Íò * 4 + (n-1)
* 20£»
¶ÁÈ¡Õâ¸öÌõÄ¿£¬±È½Ï key µÄ hashcode ºÍ index ÌõÄ¿ÖÐ hashcodeÊÇ·ñÏàͬ£¬ÒÔ¼°
key ´«ÈëµÄʱ¼ä·¶Î§¸ú Index ÌõÄ¿ÖÐµÄ timediff ÊÇ·ñÆ¥Åä¡£Èç¹ûÌõ¼þ²»·ûºÏ£¬Ôò²éÕÒ
pre index no Õâ¸öÌõÄ¿£¬ÕÒµ½ºó£¬´Ó CommitLog ÖÐÈ¡³öÏûÏ¢¡£
6 Ë¢Å̲ßÂÔ
Rocket MQ ²ÉÓÃÁé»îµÄË¢Å̲ßÂÔ¡£
6.1 Ò첽ˢÅÌ
ÏûϢдÈë CommitLog ʱ£¬²¢²»»áÖ±½ÓдÈë´ÅÅÌ£¬¶øÊÇÏÈдÈëPageCache »º´æÖУ¬È»ºóÓúǫ́Ïß³ÌÒì²½°ÑÏûϢˢÈë´ÅÅÌ¡£Ò첽ˢÅ̲ßÂÔ¾ÍÊÇÏûϢдÈë
PageCache ºóÁ¢¼´·µ»Ø³É¹¦£¬ÕâÑùдÈëЧÂʷdz£¸ß¡£Èç¹ûÄÜÈÝÈÌÏûÏ¢¶ªÊ§£¬Ò첽ˢÅÌÊÇ×îºÃµÄÑ¡Ôñ¡£
6.2 ͬ²½Ë¢ÅÌ
¼´Ê¹Í¬²½Ë¢ÅÌ£¬RocketMQ Ò²²»ÊÇÿÌõÏûÏ¢¶¼ÒªË¢ÅÌ£¬Ï߳̽«ÏûϢдÈëÄÚ´æºó£¬»áÇëÇóË¢ÅÌÏ߳̽øÐÐË¢ÅÌ£¬µ«ÊÇË¢ÅÌÏ̲߳¢²»»áÖ»°Ñµ±Ç°ÇëÇóµÄÏûϢˢÅÌ£¬¶øÊÇ»á°Ñ´ýË¢Å̵ÄÏûϢһͬˢÅÌ¡£Í¬²½Ë¢Å̲ßÂÔ±£Ö¤ÁËÏûÏ¢µÄ¿É¿¿ÐÔ£¬µ«ÊÇÒ²½µµÍÁËÍÌÍÂÁ¿£¬Ôö¼ÓÁËÑÓ³Ù¡£
7 ×ܽá
±¾ÎÄÓà 7 ÕÅͼ×ܽáÁË RocketMQ µÄºËÐÄ֪ʶ£¬Ï£ÍûÄÜ´øÄã¿ìËÙÈëÃÅ¡£ |