±à¼ÍƼö: |
±¾ÎĽ²ÊöKafkaÉè¼ÆÒªµã£¬KafkaÏûÏ¢´æ´¢·½Ê½£¬ÈçºÎͨ¹ýoffset²éÕÒMessage£¬Ï£Íû¶ÔÄúÓÐËù°ïÖú
±¾ÎÄÀ´×ÔÓÚ²©¿ÍÔ°£¬ÓÉ»ðÁú¹ûÈí¼þDelores±à¼¡¢ÍƼö¡£ |
|
Kafka¼Ü¹¹Èçͼ£º
Õû¸ö¼Ü¹¹ÖаüÀ¨Èý¸ö½ÇÉ«¡£
Éú²úÕߣ¨Producer£©:ÏûÏ¢ºÍÊý¾ÝÉú²úÕß
´úÀí£¨Broker£©:»º´æ´úÀí£¬KafkaµÄºËÐŦÄÜ
Ïû·ÑÕߣ¨Consumer£©:ÏûÏ¢ºÍÊý¾ÝÏû·ÑÕß
ÕûÌå¼Ü¹¹ºÜ¼òµ¥£¬Kafka¸øProducerºÍConsumerÌṩע²áµÄ½Ó¿Ú£¬Êý¾Ý´ÓProducer·¢Ë͵½Broker£¬Broker³Ðµ£Ò»¸öÖм仺´æºÍ·Ö·¢µÄ×÷Ó㬸ºÔð·Ö·¢×¢²áµ½ÏµÍ³ÖеÄConsumer¡£
¡¡Éè¼ÆÒªµã
¡¡¡¡Kafka·Ç³£¸ßЧ£¬ÏÂÃæ½éÉÜKafka¸ßЧµÄÔÒò£¬¶ÔÀí½âKafka·Ç³£ÓðïÖú¡£
Ö±½ÓʹÓÃLinuxÎļþϵͳµÄCacheÀ´¸ßЧ»º´æÊý¾Ý
²ÉÓÃLinux Zero-CopyÌá¸ß·¢ËÍÐÔÄÜ¡£´«Í³µÄÊý¾Ý·¢ËÍÐèÒª·¢ËÍ4´ÎÉÏÏÂÇл»£¬²ÉÓÃSendfileϵͳµ÷ÓÃÖ®ºó£¬Êý¾ÝÖ±½ÓÔÚÄÚºË̬½»»»£¬ÏµÍ³ÉÏÏÂÎÄÇл»¼õÉÙΪ2´Î¡£¿ÉÒÔÌá¸ß60%µÄÊý¾Ý·¢ËÍÐÔÄÜ¡£
KafkaÒÔTopicÀ´½øÐÐÏû·Ñ¹ÜÀí£¬Ã¿¸öTopic°üº¬¶à¸öPart(ition)£¬Ã¿¸öPart¶ÔÓ¦Ò»¸öÂß¼Log£¬Óɶà¸öSegment×é³É¡£Ã¿¸öSegmentÖд洢¶àÌõÏûÏ¢£¬ÏûÏ¢IDÓÉÂ߼λÖþö¶¨£¬¼´´ÓÏûÏ¢ID¿ÉÖ±½Ó¶¨Î»µ½ÏûÏ¢µÄ´æ´¢Î»Ö㬱ÜÃâIDµ½Î»ÖõĶîÍâÓ³É䡣ÿ¸öPartÔÚÄÚ´æÖжÔÓ¦Ò»¸öIndex£¬¼Ç¼ÿ¸öSegmentÖеĵÚÒ»¸öÏûÏ¢Æ«ÒÆ¡£

·¢²¼Õß·¢µ½Ä³¸öTopicµÄÏûÏ¢»á±»¾ùÔȵطֲ¼µ½¶à¸öPartÉÏ£¨Ëæ»ú»ò¸ù¾ÝÓû§Ö¸¶¨µÄ»Øµ÷º¯Êý½øÐзֲ¼£©£¬BrokerÊÕµ½·¢²¼ÏûÏ¢ºóÍù¶ÔÓ¦PartµÄ×îºóÒ»¸öSegmentÉÏÌí¼Ó¸ÃÏûÏ¢£¬µ±Ä³¸öSegmentÉϵÄÏûÏ¢ÌõÊýµ½´ïÅäÖÃÖµ»òÏûÏ¢·¢²¼Ê±¼ä³¬¹ýãÐֵʱ£¬SegmentÉϵÄÏûÏ¢±ã»á±»flushµ½´ÅÅÌÉÏ£¬Ö»ÓÐflushµ½´ÅÅÌÉϵÄÏûÏ¢¶©ÔÄÕß²ÅÄܶ©Ôĵ½£¬Segment´ïµ½Ò»¶¨µÄ´óСºó½«²»»áÔÙÍù¸ÃSegmentдÊý¾Ý£¬Broker»á´´½¨ÐµÄSegment¡£
¡¡¡¡È«ÏµÍ³·Ö²¼Ê½£¬¼´ËùÓеÄProducer¡¢BrokerºÍConsumer¶¼Ä¬ÈÏÓжà¸ö£¬¾ùΪ·Ö²¼Ê½µÄ¡£ProducerºÍBrokerÖ®¼äûÓиºÔؾùºâ»úÖÆ¡£BrokerºÍConsumer Ö®¼äÀûÓÃZooKeeper½øÐиºÔؾùºâ¡£ËùÓеÄBrokerºÍConsumer¶¼»áÔÚZookeeperÖнøÐÐ×¢²á£¬ÇÒZookeeper»á±£´æËûÃǵÄһЩԪÊý¾ÝÐÅÏ¢¡£Èç¹ûij¸öBrokerºÍConsumer·¢ÉúÁ˱仯£¬ÄÇôËùÓÐÆäËûµÄBrokerºÍConsumer¶¼»áµÃµ½Í¨Öª¡£
¡¡KafkaÏûÏ¢´æ´¢·½Ê½
Topic
¡¡¡¡TopicÊÇ·¢²¼µÄÏûÏ¢µÄÀà±ð»òÕßÖÖ×ÓFeedÃû¡£¶ÔÓÚÿ¸öTopic£¬Kafka¼¯Èº¶¼»áά»¤Õâ¸ö·ÖÇøµÄLog¡£Èçͼ

¡¡Ã¿¸ö·ÖÇø¶¼ÊÇÒ»¸ö˳ÐòµÄ¡¢²»¿É±äµÄÏûÏ¢¶ÓÁУ¬²¢ÇÒ¿ÉÒÔ³ÖÐøÌí¼Ó¡£·ÖÇøÖеÄÏûÏ¢¶¼±»·ÖÅäÁËÒ»¸öÐòÁкţ¬³ÆÖ®ÎªÆ«ÒÆÁ¿£¨offset£©£¬ÔÚÿ¸ö·ÖÇøÖÐ´ËÆ«ÒÆÁ¿¶¼ÊÇΨһµÄ¡£
¡¡¡¡Kafka¼¯Èº±£´æËùÓеÄÏûÏ¢£¬Ö±µ½ËûÃǹýÆÚ£¬ÎÞÂÛÏûÏ¢ÊÇ·ñ±»Ïû·Ñ¡£Êµ¼ÊÉÏ£¬Ïû·ÑÕßËù³ÖÓеĽöÓеÄÔªÊý¾Ý¾ÍÊÇÕâ¸öÆ«ÒÆÁ¿£¬Ò²¾ÍÊÇÏû·ÑÕßÔÚÕâ¸öLogÖеÄλÖá£ÔÚÕý³£Çé¿öÏ£¬µ±Ïû·ÑÕßÏû·ÑÏûÏ¢µÄʱºòÆ«ÒÆÁ¿Ò²ÏßÐÔÔö¼Ó¡£µ«ÊÇʵ¼ÊÆ«ÒÆÁ¿ÓÉÏû·ÑÕß¿ØÖÆ£¬Ïû·ÑÕß¿ÉÒÔÖØÖÃÆ«ÒÆÁ¿£¬ÒÔÖØÐ¶ÁÈ¡ÏûÏ¢¡£
¡¡¡¡ÕâÖÖÉè¼Æ¶ÔÏû·ÑÕßÀ´Ëµ²Ù×÷×ÔÈ磬һ¸öÏû·ÑÕߵIJÙ×÷²»»áÓ°ÏìÆäËûÏû·ÑÕß¶Ô´ËLogµÄ´¦Àí¡£
·ÖÇø
¡¡¡¡KafkaÖвÉÓ÷ÖÇøµÄÉè¼ÆÓÐÁ½¸öÄ¿µÄ£ºÒ»ÊÇ¿ÉÒÔ´¦Àí¸ü¶àµÄÏûÏ¢£¬¶ø²»Êܵ¥Ìå·þÎñÆ÷µÄÏÞÖÆ£¬TopicÓµÓжà¸ö·ÖÇø£¬Òâζ×ÅËü¿ÉÒÔ²»ÊÜÏÞÖÆµØ´¦Àí¸ü¶àÊý¾Ý£»¶þÊÇ·ÖÇø¿ÉÒÔ×÷Ϊ²¢Ðд¦ÀíµÄµ¥Ôª¡£
¡¡¡¡Kafka»áΪÿ¸ö·ÖÇø´´½¨Ò»¸öÎļþ¼Ð£¬Îļþ¼ÐµÄÃüÃû·½Ê½ÎªtopicName-·ÖÇøÐòºÅ£¬Èçͼ

¡¡¶ø·ÖÇøÊÇÓɶà¸öSegment×é³ÉµÄ£¬ÊÇΪÁË·½±ã½øÐÐÈÕÖ¾ÇåÀí¡¢»Ö¸´µÈ¹¤×÷¡£Ã¿¸öSegmentÒÔ¸ÃSegmentµÚÒ»ÌõÏûÏ¢µÄoffsetÃüÃû²¢ÒÔ¡°.log¡±×÷Ϊºó׺¡£ÁíÍ⻹ÓÐÒ»¸öË÷ÒýÎļþ£¬Ëû±êÃ÷ÁËÿ¸öSegmentϰüº¬µÄLog EntryµÄoffset·¶Î§£¬ÎļþÃüÃû·½Ê½Ò²ÊÇÈç´Ë£¬ÒÔ¡°.index¡±×÷Ϊºó׺£¬ÈçÏ£º

Ë÷ÒýºÍÈÕÖ¾ÎļþÄÚ²¿µÄ¹ØÏµ£¬Èçͼ£º

Ë÷ÒýÎļþ´æ´¢´óÁ¿ÔªÊý¾Ý£¬Êý¾ÝÎļþ´æ´¢´óÁ¿ÏûÏ¢£¨Message£©,Ë÷ÒýÎļþÖеÄÔªÊý¾ÝÖ¸Ïò¶ÔÓ¦Êý¾ÝÎļþÖÐMessageµÄÎïÀíÆ«ÒÆµØÖ·¡£ÒÔË÷ÒýÎļþÖеÄÔªÊý¾Ý3,497ΪÀý£¬ÒÀ´ÎÔÚÊý¾ÝÎļþÖбíʾµÚÈý¸öMessage£¨ÔÚÈ«¾ÖPartitionÖбíʾµÚ368772¸ömessage£©£¬ÒÔ¼°¸ÃÏûÏ¢µÄÎïÀíÆ«ÒÆµØÖ·Îª497.
¡¡ ¡¡SegmentµÄLogÎļþÓɶà¸öMessage×é³É£¬ÏÂÃæÏêϸ˵Ã÷MessageµÄÎïÀí½á¹¹£¬Èçͼ£º

²ÎÊý˵Ã÷

ÈçºÎͨ¹ýoffset²éÕÒMessage
ÀýÈ磬¶ÁÈ¡offset=368776µÄMessage£¬ÐèҪͨ¹ýÈçÏÂÁ½¸ö²½Öè¡£
µÚÒ»²½£º²éÕÒSegment File.
00000000000000000000.index±íʾ×ʼµÄÎļþ£¬ÆäÊµÆ«ÒÆÁ¿£¨offset£©Îª0£»µÚ¶þ¸öÎļþ00000000000000368769.indexµÄÆäÊµÆ«ÒÆÁ¿Îª368770£¨368769+1£©£¬ÒÀ´ÎÀàÍÆ¡£ÒÔÆäÊµÆ«ÒÆÁ¿ÃüÃû²¢ÅÅÐòÕâЩÎļþ£¬Ö»Òª¸ù¾Ýoffset**¶þ·Ö²éÕÒ**ÎļþÁÐ±í£¬¾Í¿ÉÒÔ¿ìËÙ¶¨Î»µ½¾ßÌåÎļþ¡£
¡¡ ¡¡µ±offset=368776ʱ£¬¶¨Î»µ½00000000000000368769.index|log¡£
µÚ¶þ²½£ºÍ¨¹ýSegment File ²éÕÒMessage¡£
¡¡¡¡ ͨ¹ýµÚÒ»²½¶¨Î»µ½Segment File,µ±offset=368776ʱ£¬ÒÀ´Î¶¨Î»µ½00000000000000368769.indexµÄÔªÊý¾ÝÎïÀíλÖúÍ00000000000000368769.logµÄÎïÀíÆ«ÒÆµØÖ·£¬È»ºóÔÙͨ¹ý00000000000000368769.log˳Ðò²éÕÒ£¬ÖªµÀoffset=368776Ϊֹ¡£
¡¡ ¡¡Segment Index File²ÉȡϡÊèË÷Òý´æ´¢·½Ê½£¬¿ÉÒÔ¼õÉÙË÷ÒýÎļþ´óС£¬Í¨¹ýLinux mmap½Ó¿Ú¿ÉÒÔÖ±½Ó½øÐÐÄÚ´æ²Ù×÷¡£Ï¡ÊèË÷ÒýΪÊý¾ÝÎļþµÄÿ¸ö¶ÔÓ¦MessageÉèÖÃÒ»¸öÔªÊý¾ÝÖ¸Õ룬Ëü±È³íÃÜË÷Òý½ÚÊ¡Á˸ü¶àµÄ´æ´¢¿Õ¼ä£¬µ«²éÕÒÆðÀ´ÐèÒªÏûºÄ¸ü¶àµÄʱ¼ä¡£
|