Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Model Center   Code  
»áÔ±   
   
 
     
   
 ¶©ÔÄ
  ¾èÖú
ÏûÏ¢¶ÓÁÐKafka¸ß¿É¿¿ÐÔÔ­ÀíÉî¶È½â¶Á
 
À´Ô´£º»¥ÁªÍø ·¢²¼ÓÚ£º2017-7-24
  4549  次浏览      27
 

1 ¸ÅÊö

KakfaÆð³õÊÇÓÉLinkedIn¹«Ë¾¿ª·¢µÄÒ»¸ö·Ö²¼Ê½µÄÏûϢϵͳ£¬ºó³ÉΪApacheµÄÒ»²¿·Ö£¬ËüʹÓÃScala±àд£¬ÒÔ¿ÉˮƽÀ©Õ¹ºÍ¸ßÍÌÍÂÂʶø±»¹ã·ºÊ¹Óá£Ä¿Ç°Ô½À´Ô½¶àµÄ¿ªÔ´·Ö²¼Ê½´¦ÀíϵͳÈçCloudera¡¢Apache Storm¡¢SparkµÈ¶¼Ö§³ÖÓëKafka¼¯³É¡£

Kafkaƾ½è×Å×ÔÉíµÄÓÅÊÆ£¬Ô½À´Ô½Êܵ½»¥ÁªÍøÆóÒµµÄÇàíù£¬Î¨Æ·»áÒ²²ÉÓÃKafka×÷ΪÆäÄÚ²¿ºËÐÄÏûÏ¢ÒýÇæÖ®Ò»¡£Kafka×÷Ϊһ¸öÉÌÒµ¼¶ÏûÏ¢Öмä¼þ£¬ÏûÏ¢¿É¿¿ÐÔµÄÖØÒªÐÔ¿ÉÏë¶øÖª¡£ÈçºÎÈ·±£ÏûÏ¢µÄ¾«È·´«Ê䣿ÈçºÎÈ·±£ÏûÏ¢µÄ׼ȷ´æ´¢£¿ÈçºÎÈ·±£ÏûÏ¢µÄÕýÈ·Ïû·Ñ£¿ÕâЩ¶¼ÊÇÐèÒª¿¼ÂǵÄÎÊÌâ¡£±¾ÎÄÊ×ÏÈ´ÓKafkaµÄ¼Ü¹¹×ÅÊÖ£¬ÏÈÁ˽âÏÂKafkaµÄ»ù±¾Ô­Àí£¬È»ºóͨ¹ý¶ÔkakfaµÄ´æ´¢»úÖÆ¡¢¸´ÖÆÔ­Àí¡¢Í¬²½Ô­Àí¡¢¿É¿¿ÐԺͳ־ÃÐÔ±£Ö¤µÈµÈÒ»²½²½¶ÔÆä¿É¿¿ÐÔ½øÐзÖÎö£¬×îºóͨ¹ýbenchmarkÀ´ÔöÇ¿¶ÔKafka¸ß¿É¿¿ÐÔµÄÈÏÖª¡£

2 KafkaÌåϵ¼Ü¹¹

ÈçÉÏͼËùʾ£¬Ò»¸öµäÐ͵ÄKafkaÌåϵ¼Ü¹¹°üÀ¨Èô¸ÉProducer£¨¿ÉÒÔÊÇ·þÎñÆ÷ÈÕÖ¾£¬ÒµÎñÊý¾Ý£¬Ò³ÃæÇ°¶Ë²úÉúµÄpage viewµÈµÈ£©£¬Èô¸Ébroker£¨KafkaÖ§³ÖˮƽÀ©Õ¹£¬Ò»°ãbrokerÊýÁ¿Ô½¶à£¬¼¯ÈºÍÌÍÂÂÊÔ½¸ß£©£¬Èô¸ÉConsumer (Group)£¬ÒÔ¼°Ò»¸öZookeeper¼¯Èº¡£Kafkaͨ¹ýZookeeper¹ÜÀí¼¯ÈºÅäÖã¬Ñ¡¾Ùleader£¬ÒÔ¼°ÔÚconsumer group·¢Éú±ä»¯Ê±½øÐÐrebalance¡£ProducerʹÓÃpush(ÍÆ)ģʽ½«ÏûÏ¢·¢²¼µ½broker£¬ConsumerʹÓÃpull(À­)ģʽ´Óbroker¶©ÔIJ¢Ïû·ÑÏûÏ¢¡£

Ãû´Ê½âÊÍ£º

2.1 Topic & Partition

Ò»¸ötopic¿ÉÒÔÈÏΪһ¸öÒ»ÀàÏûÏ¢£¬Ã¿¸ötopic½«±»·Ö³É¶à¸öpartition£¬Ã¿¸öpartitionÔÚ´æ´¢²ãÃæÊÇappend logÎļþ¡£Èκη¢²¼µ½´ËpartitionµÄÏûÏ¢¶¼»á±»×·¼Óµ½logÎļþµÄβ²¿£¬Ã¿ÌõÏûÏ¢ÔÚÎļþÖеÄλÖóÆÎªoffset(Æ«ÒÆÁ¿)£¬offsetΪһ¸ölongÐ͵ÄÊý×Ö£¬ËüΨһ±ê¼ÇÒ»ÌõÏûÏ¢¡£Ã¿ÌõÏûÏ¢¶¼±»appendµ½partitionÖУ¬ÊÇ˳Ðòд´ÅÅÌ£¬Òò´ËЧÂʷdz£¸ß£¨¾­ÑéÖ¤£¬Ë³Ðòд´ÅÅÌЧÂʱÈËæ»úдÄڴ滹Ҫ¸ß£¬ÕâÊÇKafka¸ßÍÌÍÂÂʵÄÒ»¸öºÜÖØÒªµÄ±£Ö¤£©¡£

ÿһÌõÏûÏ¢±»·¢Ë͵½brokerÖУ¬»á¸ù¾Ýpartition¹æÔòÑ¡Ôñ±»´æ´¢µ½ÄÄÒ»¸öpartition¡£Èç¹ûpartition¹æÔòÉèÖõĺÏÀí£¬ËùÓÐÏûÏ¢¿ÉÒÔ¾ùÔÈ·Ö²¼µ½²»Í¬µÄpartitionÀÕâÑù¾ÍʵÏÖÁËˮƽÀ©Õ¹¡££¨Èç¹ûÒ»¸ötopic¶ÔÓ¦Ò»¸öÎļþ£¬ÄÇÕâ¸öÎļþËùÔڵĻúÆ÷I/O½«»á³ÉΪÕâ¸ötopicµÄÐÔÄÜÆ¿¾±£¬¶øpartition½â¾öÁËÕâ¸öÎÊÌ⣩¡£ÔÚ´´½¨topicʱ¿ÉÒÔÔÚ$KAFKA_HOME/config/server.propertiesÖÐÖ¸¶¨Õâ¸öpartitionµÄÊýÁ¿£¨ÈçÏÂËùʾ£©£¬µ±È»¿ÉÒÔÔÚtopic´´½¨Ö®ºóÈ¥ÐÞ¸ÄpartitionµÄÊýÁ¿¡£

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=3

ÔÚ·¢ËÍÒ»ÌõÏûϢʱ£¬¿ÉÒÔÖ¸¶¨Õâ¸öÏûÏ¢µÄkey£¬producer¸ù¾ÝÕâ¸ökeyºÍpartition»úÖÆÀ´ÅжÏÕâ¸öÏûÏ¢·¢Ë͵½Äĸöpartition¡£partition»úÖÆ¿ÉÒÔͨ¹ýÖ¸¶¨producerµÄpartition.classÕâÒ»²ÎÊýÀ´Ö¸¶¨£¬¸Ãclass±ØÐëʵÏÖkafka.producer.Partitioner½Ó¿Ú¡£

ÓйØTopicÓëPartitionµÄ¸ü¶àϸ½Ú£¬¿ÉÒԲο¼ÏÂÃæµÄ¡°KafkaÎļþ´æ´¢»úÖÆ¡±ÕâÒ»½Ú¡£

3 ¸ß¿É¿¿ÐÔ´æ´¢·ÖÎö

KafkaµÄ¸ß¿É¿¿ÐԵı£ÕÏÀ´Ô´ÓÚÆä½¡×³µÄ¸±±¾£¨replication£©²ßÂÔ¡£Í¨¹ýµ÷½ÚÆä¸±±¾Ïà¹Ø²ÎÊý£¬¿ÉÒÔʹµÃKafkaÔÚÐÔÄܺͿɿ¿ÐÔÖ®¼äÔËתµÄÓÎÈÐÓÐÓà¡£Kafka´Ó0.8.x°æ±¾¿ªÊ¼Ìṩpartition¼¶±ðµÄ¸´ÖÆ,replicationµÄÊýÁ¿¿ÉÒÔÔÚ$KAFKA_HOME/config/server.propertiesÖÐÅäÖã¨default.replication.refactor£©¡£

ÕâÀïÏÈ´ÓKafkaÎļþ´æ´¢»úÖÆÈëÊÖ£¬´Ó×îµ×²ãÁ˽âKafkaµÄ´æ´¢Ï¸½Ú£¬½ø¶ø¶ÔÆäµÄ´æ´¢Óиö΢¹ÛµÄÈÏÖª¡£Ö®ºóͨ¹ýKafka¸´ÖÆÔ­ÀíºÍͬ²½·½Ê½À´²ûÊöºê¹Û²ãÃæµÄ¸ÅÄî¡£×îºó´ÓISR£¬HW£¬leaderÑ¡¾ÙÒÔ¼°Êý¾Ý¿É¿¿ÐԺͳ־ÃÐÔ±£Ö¤µÈµÈ¸÷¸öά¶ÈÀ´·á¸»¶ÔKafkaÏà¹ØÖªÊ¶µãµÄÈÏÖª¡£

3.1 KafkaÎļþ´æ´¢»úÖÆ

KafkaÖÐÏûÏ¢ÊÇÒÔtopic½øÐзÖÀàµÄ£¬Éú²úÕßͨ¹ýtopicÏòKafka broker·¢ËÍÏûÏ¢£¬Ïû·ÑÕßͨ¹ýtopic¶ÁÈ¡Êý¾Ý¡£È»¶øtopicÔÚÎïÀí²ãÃæÓÖÄÜÒÔpartitionΪ·Ö×飬һ¸ötopic¿ÉÒÔ·Ö³ÉÈô¸É¸öpartition£¬ÄÇôtopicÒÔ¼°partitionÓÖÊÇÔõô´æ´¢µÄÄØ£¿partition»¹¿ÉÒÔϸ·ÖΪsegment£¬Ò»¸öpartitionÎïÀíÉÏÓɶà¸ösegment×é³É£¬ÄÇôÕâЩsegmentÓÖÊÇÊ²Ã´ÄØ£¿ÏÂÃæÎÒÃÇÀ´Ò»Ò»½ÒÏþ¡£

ΪÁ˱ãÓÚ˵Ã÷ÎÊÌ⣬¼ÙÉèÕâÀïÖ»ÓÐÒ»¸öKafka¼¯Èº£¬ÇÒÕâ¸ö¼¯ÈºÖ»ÓÐÒ»¸öKafka broker£¬¼´Ö»ÓÐһ̨ÎïÀí»ú¡£ÔÚÕâ¸öKafka brokerÖÐÅäÖã¨$KAFKA_HOME/config /server.propertiesÖУ© log.dirs=/tmp/kafka-logs£¬ÒÔ´ËÀ´ÉèÖÃKafkaÏûÏ¢Îļþ´æ´¢Ä¿Â¼£¬Óë´Ëͬʱ´´½¨Ò»¸ötopic£ºtopic_zzh_test£¬partitionµÄÊýÁ¿Îª4£¨$KAFKA_HOME/bin/kafka-topics.sh ¨Ccreate ¨Czookeeper localhost:2181 ¨Cpartitions 4 ¨Ctopic topic_vms_test ¨Creplication-factor 4£©¡£ÄÇôÎÒÃÇ´Ëʱ¿ÉÒÔÔÚ/tmp/kafka-logsĿ¼ÖпÉÒÔ¿´µ½Éú³ÉÁË4¸öĿ¼£º

drwxr-xr-x 2 root root 4096 Apr 10 16:10 topic_zzh_test-0
drwxr-xr-x 2 root root 4096 Apr 10 16:10 topic_zzh_test-1
drwxr-xr-x 2 root root 4096 Apr 10 16:10 topic_zzh_test-2
drwxr-xr-x 2 root root 4096 Apr 10 16:10 topic_zzh_test-3

ÔÚKafkaÎļþ´æ´¢ÖУ¬Í¬Ò»¸ötopicÏÂÓжà¸ö²»Í¬µÄpartition£¬Ã¿¸öpartitonΪһ¸öĿ¼£¬partitionµÄÃû³Æ¹æÔòΪ£ºtopicÃû³Æ+ÓÐÐòÐòºÅ£¬µÚÒ»¸öÐòºÅ´Ó0¿ªÊ¼¼Æ£¬×î´óµÄÐòºÅΪpartitionÊýÁ¿¼õ1£¬partitionÊÇʵ¼ÊÎïÀíÉϵĸÅÄ¶øtopicÊÇÂß¼­ÉϵĸÅÄî¡£

ÉÏÃæÌáµ½partition»¹¿ÉÒÔϸ·ÖΪsegment£¬Õâ¸ösegmentÓÖÊÇʲô£¿Èç¹û¾ÍÒÔpartitionΪ×îС´æ´¢µ¥Î»£¬ÎÒÃÇ¿ÉÒÔÏëÏóµ±Kafka producer²»¶Ï·¢ËÍÏûÏ¢£¬±ØÈ»»áÒýÆðpartitionÎļþµÄÎÞÏÞÀ©ÕÅ£¬ÕâÑù¶ÔÓÚÏûÏ¢ÎļþµÄά»¤ÒÔ¼°ÒѾ­±»Ïû·ÑµÄÏûÏ¢µÄÇåÀí´øÀ´ÑÏÖØµÄÓ°Ï죬ËùÒÔÕâÀïÒÔsegmentΪµ¥Î»ÓÖ½«partitionϸ·Ö¡£Ã¿¸öpartition(Ŀ¼)Ï൱ÓÚÒ»¸ö¾ÞÐÍÎļþ±»Æ½¾ù·ÖÅäµ½¶à¸ö´óСÏàµÈµÄsegment(¶Î)Êý¾ÝÎļþÖУ¨Ã¿¸ösegment ÎļþÖÐÏûÏ¢ÊýÁ¿²»Ò»¶¨ÏàµÈ£©ÕâÖÖÌØÐÔÒ²·½±ãold segmentµÄɾ³ý£¬¼´·½±ãÒѱ»Ïû·ÑµÄÏûÏ¢µÄÇåÀí£¬Ìá¸ß´ÅÅ̵ÄÀûÓÃÂÊ¡£Ã¿¸öpartitionÖ»ÐèÒªÖ§³Ö˳Ðò¶Áд¾ÍÐУ¬segmentµÄÎļþÉúÃüÖÜÆÚÓÉ·þÎñ¶ËÅäÖòÎÊý£¨log.segment.bytes£¬log.roll.{ms,hours}µÈÈô¸É²ÎÊý£©¾ö¶¨¡£

segmentÎļþÓÉÁ½²¿·Ö×é³É£¬·Ö±ðΪ¡°.index¡±ÎļþºÍ¡°.log¡±Îļþ£¬·Ö±ð±íʾΪsegmentË÷ÒýÎļþºÍÊý¾ÝÎļþ¡£ÕâÁ½¸öÎļþµÄÃüÁî¹æÔòΪ£ºpartitionÈ«¾ÖµÄµÚÒ»¸ösegment´Ó0¿ªÊ¼£¬ºóÐøÃ¿¸ösegmentÎļþÃûΪÉÏÒ»¸ösegmentÎļþ×îºóÒ»ÌõÏûÏ¢µÄoffsetÖµ£¬ÊýÖµ´óСΪ64룬20λÊý×Ö×Ö·û³¤¶È£¬Ã»ÓÐÊý×ÖÓÃ0Ìî³ä£¬ÈçÏ£º

00000000000000000000.index
00000000000000000000.log
00000000000000170410.index
00000000000000170410.log
00000000000000239430.index
00000000000000239430.log

ÒÔÉÏÃæµÄsegmentÎļþΪÀý£¬Õ¹Ê¾³ösegment£º00000000000000170410µÄ¡°.index¡±ÎļþºÍ¡°.log¡±ÎļþµÄ¶ÔÓ¦µÄ¹ØÏµ£¬ÈçÏÂͼ£º

ÈçÉÏͼ£¬¡°.index¡±Ë÷ÒýÎļþ´æ´¢´óÁ¿µÄÔªÊý¾Ý£¬¡°.log¡±Êý¾ÝÎļþ´æ´¢´óÁ¿µÄÏûÏ¢£¬Ë÷ÒýÎļþÖеÄÔªÊý¾ÝÖ¸Ïò¶ÔÓ¦Êý¾ÝÎļþÖÐmessageµÄÎïÀíÆ«ÒÆµØÖ·¡£ÆäÖÐÒÔ¡°.index¡±Ë÷ÒýÎļþÖеÄÔªÊý¾Ý[3, 348]ΪÀý£¬ÔÚ¡°.log¡±Êý¾ÝÎļþ±íʾµÚ3¸öÏûÏ¢£¬¼´ÔÚÈ«¾ÖpartitionÖбíʾ170410+3=170413¸öÏûÏ¢£¬¸ÃÏûÏ¢µÄÎïÀíÆ«ÒÆµØÖ·Îª348¡£

ÄÇôÈçºÎ´ÓpartitionÖÐͨ¹ýoffset²éÕÒmessageÄØ£¿

ÒÔÉÏͼΪÀý£¬¶ÁÈ¡offset=170418µÄÏûÏ¢£¬Ê×ÏȲéÕÒsegmentÎļþ£¬ÆäÖÐ00000000000000000000.indexΪ×ʼµÄÎļþ£¬µÚ¶þ¸öÎļþΪ00000000000000170410.index£¨ÆðÊ¼Æ«ÒÆÎª170410+1=170411£©£¬¶øµÚÈý¸öÎļþΪ00000000000000239430.index£¨ÆðÊ¼Æ«ÒÆÎª239430+1=239431£©£¬ËùÒÔÕâ¸öoffset=170418¾ÍÂäµ½Á˵ڶþ¸öÎļþÖ®ÖС£ÆäËûºóÐøÎļþ¿ÉÒÔÒÀ´ÎÀàÍÆ£¬ÒÔÆäÊµÆ«ÒÆÁ¿ÃüÃû²¢ÅÅÁÐÕâЩÎļþ£¬È»ºó¸ù¾Ý¶þ·Ö²éÕÒ·¨¾Í¿ÉÒÔ¿ìËÙ¶¨Î»µ½¾ßÌåÎļþλÖá£Æä´Î¸ù¾Ý00000000000000170410.indexÎļþÖеÄ[8,1325]¶¨Î»µ½00000000000000170410.logÎļþÖеÄ1325µÄλÖýøÐжÁÈ¡¡£

ÒªÊǶÁÈ¡offset=170418µÄÏûÏ¢£¬´Ó00000000000000170410.logÎļþÖеÄ1325µÄλÖýøÐжÁÈ¡£¬ÄÇôÔõô֪µÀºÎʱ¶ÁÍê±¾ÌõÏûÏ¢£¬·ñÔò¾Í¶Áµ½ÏÂÒ»ÌõÏûÏ¢µÄÄÚÈÝÁË£¿

Õâ¸ö¾ÍÐèÒªÁªÏµµ½ÏûÏ¢µÄÎïÀí½á¹¹ÁË£¬ÏûÏ¢¶¼¾ßÓй̶¨µÄÎïÀí½á¹¹£¬°üÀ¨£ºoffset£¨8 Bytes£©¡¢ÏûÏ¢ÌåµÄ´óС£¨4 Bytes£©¡¢crc32£¨4 Bytes£©¡¢magic£¨1 Byte£©¡¢attributes£¨1 Byte£©¡¢key length£¨4 Bytes£©¡¢key£¨K Bytes£©¡¢payload(N Bytes)µÈµÈ×ֶΣ¬¿ÉÒÔÈ·¶¨Ò»ÌõÏûÏ¢µÄ´óС£¬¼´¶ÁÈ¡µ½ÄÄÀï½ØÖ¹¡£

3.2 ¸´ÖÆÔ­ÀíºÍͬ²½·½Ê½

KafkaÖÐtopicµÄÿ¸öpartitionÓÐÒ»¸öԤдʽµÄÈÕÖ¾Îļþ£¬ËäÈ»partition¿ÉÒÔ¼ÌÐøÏ¸·ÖΪÈô¸É¸ösegmentÎļþ£¬µ«ÊǶÔÓÚÉϲãÓ¦ÓÃÀ´Ëµ¿ÉÒÔ½«partition¿´³É×îСµÄ´æ´¢µ¥Ôª£¨Ò»¸öÓжà¸ösegmentÎļþÆ´½ÓµÄ¡°¾ÞÐÍ¡±Îļþ£©£¬Ã¿¸öpartition¶¼ÓÉһЩÁÐÓÐÐòµÄ¡¢²»¿É±äµÄÏûÏ¢×é³É£¬ÕâЩÏûÏ¢±»Á¬ÐøµÄ×·¼Óµ½partitionÖС£

ÉÏͼÖÐÓÐÁ½¸öÐÂÃû´Ê£ºHWºÍLEO¡£ÕâÀïÏȽéÉÜÏÂLEO£¬LogEndOffsetµÄËõд£¬±íʾÿ¸öpartitionµÄlog×îºóÒ»ÌõMessageµÄλÖá£HWÊÇHighWatermarkµÄËõд£¬ÊÇÖ¸consumerÄܹ»¿´µ½µÄ´ËpartitionµÄλÖã¬Õâ¸öÉæ¼°µ½¶à¸±±¾µÄ¸ÅÄÕâÀïÏÈÌἰһϣ¬Ï½ÚÔÙÏê±í¡£

ÑÔ¹éÕý´«£¬ÎªÁËÌá¸ßÏûÏ¢µÄ¿É¿¿ÐÔ£¬Kafkaÿ¸ötopicµÄpartitionÓÐN¸ö¸±±¾£¨replicas£©£¬ÆäÖÐN(´óÓÚµÈÓÚ1)ÊÇtopicµÄ¸´ÖÆÒò×Ó£¨replica fator£©µÄ¸öÊý¡£Kafkaͨ¹ý¶à¸±±¾»úÖÆÊµÏÖ¹ÊÕÏ×Ô¶¯×ªÒÆ£¬µ±Kafka¼¯ÈºÖÐÒ»¸öbrokerʧЧÇé¿öÏÂÈÔÈ»±£Ö¤·þÎñ¿ÉÓá£ÔÚKafkaÖз¢Éú¸´ÖÆÊ±È·±£partitionµÄÈÕÖ¾ÄÜÓÐÐòµØÐ´µ½ÆäËû½ÚµãÉÏ£¬N¸öreplicasÖУ¬ÆäÖÐÒ»¸öreplicaΪleader£¬ÆäËû¶¼Îªfollower, leader´¦ÀípartitionµÄËùÓжÁдÇëÇó£¬Óë´Ëͬʱ£¬follower»á±»¶¯¶¨ÆÚµØÈ¥¸´ÖÆleaderÉϵÄÊý¾Ý¡£

ÈçÏÂͼËùʾ£¬Kafka¼¯ÈºÖÐÓÐ4¸öbroker, ijtopicÓÐ3¸öpartition,ÇÒ¸´ÖÆÒò×Ó¼´¸±±¾¸öÊýҲΪ3£º

KafkaÌṩÁËÊý¾Ý¸´ÖÆËã·¨±£Ö¤£¬Èç¹ûleader·¢Éú¹ÊÕÏ»ò¹Òµô£¬Ò»¸öÐÂleader±»Ñ¡¾Ù²¢±»½ÓÊܿͻ§¶ËµÄÏûÏ¢³É¹¦Ð´Èë¡£KafkaÈ·±£´Óͬ²½¸±±¾ÁбíÖÐÑ¡¾ÙÒ»¸ö¸±±¾Îªleader£¬»òÕß˵follower×·¸ÏleaderÊý¾Ý¡£leader¸ºÔðά»¤ºÍ¸ú×ÙISR(In-Sync ReplicasµÄËõд£¬±íʾ¸±±¾Í¬²½¶ÓÁУ¬¾ßÌå¿É²Î¿¼Ï½Ú)ÖÐËùÓÐfollowerÖͺóµÄ״̬¡£µ±producer·¢ËÍÒ»ÌõÏûÏ¢µ½brokerºó£¬leaderдÈëÏûÏ¢²¢¸´ÖƵ½ËùÓÐfollower¡£ÏûÏ¢Ìá½»Ö®ºó²Å±»³É¹¦¸´ÖƵ½ËùÓеÄͬ²½¸±±¾¡£ÏûÏ¢¸´ÖÆÑÓ³ÙÊÜ×îÂýµÄfollowerÏÞÖÆ£¬ÖØÒªµÄÊÇ¿ìËÙ¼ì²âÂý¸±±¾£¬Èç¹ûfollower¡°Âäºó¡±Ì«¶à»òÕßʧЧ£¬leader½«»á°ÑËü´ÓISRÖÐɾ³ý¡£

3.3 ISR

ÉϽÚÎÒÃÇÉæ¼°µ½ISR (In-Sync Replicas)£¬Õâ¸öÊÇÖ¸¸±±¾Í¬²½¶ÓÁС£¸±±¾Êý¶ÔKafkaµÄÍÌÍÂÂÊÊÇÓÐÒ»¶¨µÄÓ°Ï죬µ«¼«´óµÄÔöÇ¿ÁË¿ÉÓÃÐÔ¡£Ä¬ÈÏÇé¿öÏÂKafkaµÄreplicaÊýÁ¿Îª1£¬¼´Ã¿¸öpartition¶¼ÓÐÒ»¸öΨһµÄleader£¬ÎªÁËÈ·±£ÏûÏ¢µÄ¿É¿¿ÐÔ£¬Í¨³£Ó¦ÓÃÖн«ÆäÖµ(ÓÉbrokerµÄ²ÎÊýoffsets.topic.replication.factorÖ¸¶¨)´óСÉèÖÃΪ´óÓÚ1£¬±ÈÈç3¡£ ËùÓеĸ±±¾£¨replicas£©Í³³ÆÎªAssigned Replicas£¬¼´AR¡£ISRÊÇARÖеÄÒ»¸ö×Ó¼¯£¬ÓÉleaderά»¤ISRÁÐ±í£¬follower´Óleaderͬ²½Êý¾ÝÓÐһЩÑÓ³Ù£¨°üÀ¨ÑÓ³Ùʱ¼äreplica.lag.time.max.msºÍÑÓ³ÙÌõÊýreplica.lag.max.messagesÁ½¸öά¶È, µ±Ç°×îеİ汾0.10.xÖÐÖ»Ö§³Öreplica.lag.time.max.msÕâ¸öά¶È£©£¬ÈÎÒâÒ»¸ö³¬¹ýãÐÖµ¶¼»á°ÑfollowerÌÞ³ý³öISR, ´æÈëOSR£¨Outof-Sync Replicas£©ÁÐ±í£¬Ð¼ÓÈëµÄfollowerÒ²»áÏÈ´æ·ÅÔÚOSRÖС£AR=ISR+OSR¡£

Kafka 0.10.x°æ±¾ºóÒÆ³ýÁËreplica.lag.max.messages²ÎÊý£¬Ö»±£ÁôÁËreplica.lag.time.max.ms×÷ΪISRÖи±±¾¹ÜÀíµÄ²ÎÊý¡£ÎªÊ²Ã´ÕâÑù×öÄØ£¿replica.lag.max.messages±íʾµ±Ç°Ä³¸ö¸±±¾ÂäºóleaederµÄÏûÏ¢ÊýÁ¿³¬¹ýÁËÕâ¸ö²ÎÊýµÄÖµ£¬ÄÇôleader¾Í»á°Ñfollower´ÓISRÖÐɾ³ý¡£¼ÙÉèÉèÖÃreplica.lag.max.messages=4£¬ÄÇôÈç¹ûproducerÒ»´Î´«ËÍÖÁbrokerµÄÏûÏ¢ÊýÁ¿¶¼Ð¡ÓÚ4Ìõʱ£¬ÒòΪÔÚleader½ÓÊܵ½producer·¢Ë͵ÄÏûÏ¢Ö®ºó¶øfollower¸±±¾¿ªÊ¼À­È¡ÕâЩÏûϢ֮ǰ£¬followerÂäºóleaderµÄÏûÏ¢Êý²»»á³¬¹ý4ÌõÏûÏ¢£¬¹Ê´ËûÓÐfollowerÒÆ³öISR£¬ËùÒÔÕâʱºòreplica.lag.max.messageµÄÉèÖÃËÆºõÊǺÏÀíµÄ¡£µ«ÊÇproducer·¢Æð˲ʱ¸ß·åÁ÷Á¿£¬producerÒ»´Î·¢Ë͵ÄÏûÏ¢³¬¹ý4Ìõʱ£¬Ò²¾ÍÊdz¬¹ýreplica.lag.max.messages£¬´Ëʱfollower¶¼»á±»ÈÏΪÊÇÓëleader¸±±¾²»Í¬²½ÁË£¬´Ó¶ø±»Ìß³öÁËISR¡£µ«Êµ¼ÊÉÏÕâЩfollower¶¼ÊÇ´æ»î״̬µÄÇÒûÓÐÐÔÄÜÎÊÌâ¡£ÄÇôÔÚÖ®ºó×·ÉÏleader,²¢±»ÖØÐ¼ÓÈëÁËISR¡£ÓÚÊǾͻá³öÏÖËüÃDz»¶ÏµØÌÞ³öISRÈ»ºóÖØÐ»عéISR£¬ÕâÎÞÒÉÔö¼ÓÁËÎÞνµÄÐÔÄÜËðºÄ¡£¶øÇÒÕâ¸ö²ÎÊýÊÇbrokerÈ«¾ÖµÄ¡£ÉèÖÃÌ«´óÁË£¬Ó°ÏìÕæÕý¡°Âäºó¡±followerµÄÒÆ³ý£»ÉèÖõÄ̫СÁË£¬µ¼ÖÂfollowerµÄƵ·±½ø³ö¡£ÎÞ·¨¸ø¶¨Ò»¸öºÏÊʵÄreplica.lag.max.messagesµÄÖµ£¬¹Ê´Ë£¬Ð°汾µÄKafkaÒÆ³ýÁËÕâ¸ö²ÎÊý¡£

×¢£ºISRÖаüÀ¨£ºleaderºÍfollower¡£

ÉÏÃæÒ»½Ú»¹Éæ¼°µ½Ò»¸ö¸ÅÄ¼´HW¡£HWË׳Ƹßˮ룬HighWatermarkµÄËõд£¬È¡Ò»¸öpartition¶ÔÓ¦µÄISRÖÐ×îСµÄLEO×÷ΪHW£¬consumer×î¶àÖ»ÄÜÏû·Ñµ½HWËùÔÚµÄλÖá£ÁíÍâÿ¸öreplica¶¼ÓÐHW,leaderºÍfollower¸÷×Ô¸ºÔð¸üÐÂ×Ô¼ºµÄHWµÄ״̬¡£¶ÔÓÚleaderÐÂдÈëµÄÏûÏ¢£¬consumer²»ÄÜÁ¢¿ÌÏû·Ñ£¬leader»áµÈ´ý¸ÃÏûÏ¢±»ËùÓÐISRÖеÄreplicasͬ²½ºó¸üÐÂHW£¬´ËʱÏûÏ¢²ÅÄܱ»consumerÏû·Ñ¡£ÕâÑù¾Í±£Ö¤ÁËÈç¹ûleaderËùÔÚµÄbrokerʧЧ£¬¸ÃÏûÏ¢ÈÔÈ»¿ÉÒÔ´ÓÐÂÑ¡¾ÙµÄleaderÖлñÈ¡¡£¶ÔÓÚÀ´×ÔÄÚ²¿broKerµÄ¶ÁÈ¡ÇëÇó£¬Ã»ÓÐHWµÄÏÞÖÆ¡£

ÏÂͼÏêϸµÄ˵Ã÷Á˵±producerÉú²úÏûÏ¢ÖÁbrokerºó£¬ISRÒÔ¼°HWºÍLEOµÄÁ÷ת¹ý³Ì£º

Óɴ˿ɼû£¬KafkaµÄ¸´ÖÆ»úÖÆ¼È²»ÊÇÍêÈ«µÄͬ²½¸´ÖÆ£¬Ò²²»Êǵ¥´¿µÄÒì²½¸´ÖÆ¡£ÊÂʵÉÏ£¬Í¬²½¸´ÖÆÒªÇóËùÓÐÄܹ¤×÷µÄfollower¶¼¸´ÖÆÍ꣬ÕâÌõÏûÏ¢²Å»á±»commit£¬ÕâÖÖ¸´ÖÆ·½Ê½¼«´óµÄÓ°ÏìÁËÍÌÍÂÂÊ¡£¶øÒì²½¸´ÖÆ·½Ê½Ï£¬followerÒì²½µÄ´Óleader¸´ÖÆÊý¾Ý£¬Êý¾ÝÖ»Òª±»leaderдÈëlog¾Í±»ÈÏΪÒѾ­commit£¬ÕâÖÖÇé¿öÏÂÈç¹ûfollower¶¼»¹Ã»Óи´ÖÆÍ꣬ÂäºóÓÚleaderʱ£¬Í»È»leaderå´»ú£¬Ôò»á¶ªÊ§Êý¾Ý¡£¶øKafkaµÄÕâÖÖʹÓÃISRµÄ·½Ê½ÔòºÜºÃµÄ¾ùºâÁËÈ·±£Êý¾Ý²»¶ªÊ§ÒÔ¼°ÍÌÍÂÂÊ¡£

KafkaµÄISRµÄ¹ÜÀí×îÖÕ¶¼»á·´À¡µ½Zookeeper½ÚµãÉÏ¡£¾ßÌåλÖÃΪ£º/brokers/topics/[topic] /partitions/[partition]/state¡£Ä¿Ç°ÓÐÁ½¸öµØ·½»á¶ÔÕâ¸öZookeeperµÄ½Úµã½øÐÐά»¤£º

ControllerÀ´Î¬»¤£ºKafka¼¯ÈºÖÐµÄÆäÖÐÒ»¸öBroker»á±»Ñ¡¾ÙΪController£¬Ö÷Òª¸ºÔðPartition¹ÜÀíºÍ¸±±¾×´Ì¬¹ÜÀí£¬Ò²»áÖ´ÐÐÀàËÆÓÚÖØ·ÖÅäpartitionÖ®ÀàµÄ¹ÜÀíÈÎÎñ¡£ÔÚ·ûºÏÄ³Ð©ÌØ¶¨Ìõ¼þÏ£¬ControllerϵÄLeaderSelector»áÑ¡¾ÙеÄleader£¬ISRºÍеÄleader_epoch¼°controller_epochдÈëZookeeperµÄÏà¹Ø½ÚµãÖС£Í¬Ê±·¢ÆðLeaderAndIsrRequest֪ͨËùÓеÄreplicas¡£

leaderÀ´Î¬»¤£ºleaderÓе¥¶ÀµÄÏ̶߳¨ÆÚ¼ì²âISRÖÐfollowerÊÇ·ñÍÑÀëISR, Èç¹û·¢ÏÖISR±ä»¯£¬Ôò»á½«ÐµÄISRµÄÐÅÏ¢·µ»Øµ½ZookeeperµÄÏà¹Ø½ÚµãÖС£

3.4 Êý¾Ý¿É¿¿ÐԺͳ־ÃÐÔ±£Ö¤

µ±producerÏòleader·¢ËÍÊý¾Ýʱ£¬¿ÉÒÔͨ¹ýrequest.required.acks²ÎÊýÀ´ÉèÖÃÊý¾Ý¿É¿¿ÐԵļ¶±ð£º

1£¨Ä¬ÈÏ£©£ºÕâÒâζ×ÅproducerÔÚISRÖеÄleaderÒѳɹ¦ÊÕµ½µÄÊý¾Ý²¢µÃµ½È·ÈϺó·¢ËÍÏÂÒ»Ìõmessage¡£Èç¹ûleaderå´»úÁË£¬Ôò»á¶ªÊ§Êý¾Ý¡£

0£ºÕâÒâζ×ÅproducerÎÞÐèµÈ´ýÀ´×ÔbrokerµÄÈ·È϶ø¼ÌÐø·¢ËÍÏÂÒ»ÅúÏûÏ¢¡£ÕâÖÖÇé¿öÏÂÊý¾Ý´«ÊäЧÂÊ×î¸ß£¬µ«ÊÇÊý¾Ý¿É¿¿ÐÔÈ·ÊÇ×îµÍµÄ¡£

-1£ºproducerÐèÒªµÈ´ýISRÖеÄËùÓÐfollower¶¼È·ÈϽÓÊÕµ½Êý¾Ýºó²ÅËãÒ»´Î·¢ËÍÍê³É£¬¿É¿¿ÐÔ×î¸ß¡£µ«ÊÇÕâÑùÒ²²»Äܱ£Ö¤Êý¾Ý²»¶ªÊ§£¬±ÈÈçµ±ISRÖÐÖ»ÓÐleaderʱ£¨Ç°ÃæISRÄÇÒ»½Ú½²µ½£¬ISRÖеijÉÔ±ÓÉÓÚijЩÇé¿ö»áÔö¼ÓÒ²»á¼õÉÙ£¬×îÉÙ¾Íֻʣһ¸öleader£©£¬ÕâÑù¾Í±ä³ÉÁËacks=1µÄÇé¿ö¡£

Èç¹ûÒªÌá¸ßÊý¾ÝµÄ¿É¿¿ÐÔ£¬ÔÚÉèÖÃrequest.required.acks=-1µÄͬʱ£¬Ò²Òªmin.insync.replicasÕâ¸ö²ÎÊý(¿ÉÒÔÔÚbroker»òÕßtopic²ãÃæ½øÐÐÉèÖÃ)µÄÅäºÏ£¬ÕâÑù²ÅÄÜ·¢»Ó×î´óµÄ¹¦Ð§¡£min.insync.replicasÕâ¸ö²ÎÊýÉ趨ISRÖеÄ×îС¸±±¾ÊýÊǶàÉÙ£¬Ä¬ÈÏֵΪ1£¬µ±ÇÒ½öµ±request.required.acks²ÎÊýÉèÖÃΪ-1ʱ£¬´Ë²ÎÊý²ÅÉúЧ¡£Èç¹ûISRÖеĸ±±¾ÊýÉÙÓÚmin.insync.replicasÅäÖõÄÊýÁ¿Ê±£¬¿Í»§¶Ë»á·µ»ØÒì³££ºorg.apache.kafka.common.errors .NotEnoughReplicasExceptoin: Messages are rejected since there are fewer in-sync replicas than required¡£

½ÓÏÂÀ´¶Ôacks=1ºÍ-1µÄÁ½ÖÖÇé¿ö½øÐÐÏêϸ·ÖÎö£º

1. request.required.acks=1

producer·¢ËÍÊý¾Ýµ½leader£¬leaderд±¾µØÈÕÖ¾³É¹¦£¬·µ»Ø¿Í»§¶Ë³É¹¦£»´ËʱISRÖеĸ±±¾»¹Ã»ÓÐÀ´µÃ¼°À­È¡¸ÃÏûÏ¢£¬leader¾Íå´»úÁË£¬ÄÇô´Ë´Î·¢Ë͵ÄÏûÏ¢¾Í»á¶ªÊ§¡£

2. request.required.acks=-1

ͬ²½£¨KafkaĬÈÏΪͬ²½£¬¼´producer.type=sync£©µÄ·¢ËÍģʽ£¬replication.factor>=2ÇÒmin.insync.replicas>=2µÄÇé¿öÏ£¬²»»á¶ªÊ§Êý¾Ý¡£

ÓÐÁ½ÖÖµäÐÍÇé¿ö¡£acks=-1µÄÇé¿öÏ£¨ÈçÎÞÌØÊâ˵Ã÷£¬ÒÔÏÂacks¶¼±íʾΪ²ÎÊýrequest.required.acks£©£¬Êý¾Ý·¢Ë͵½leader, ISRµÄfollowerÈ«²¿Íê³ÉÊý¾Ýͬ²½ºó£¬leader´Ëʱ¹Òµô£¬ÄÇô»áÑ¡¾Ù³öеÄleader£¬Êý¾Ý²»»á¶ªÊ§¡£

acks=-1µÄÇé¿öÏ£¬Êý¾Ý·¢Ë͵½leaderºó £¬²¿·ÖISRµÄ¸±±¾Í¬²½£¬leader´Ëʱ¹Òµô¡£±ÈÈçfollower1hºÍfollower2¶¼ÓпÉÄܱä³ÉеÄleader, producer¶Ë»áµÃµ½·µ»ØÒì³££¬producer¶Ë»áÖØÐ·¢ËÍÊý¾Ý£¬Êý¾Ý¿ÉÄÜ»áÖØ¸´¡£

µ±È»ÉÏͼÖÐÈç¹ûÔÚleader crashµÄʱºò£¬follower2»¹Ã»ÓÐͬ²½µ½ÈκÎÊý¾Ý£¬¶øÇÒfollower2±»Ñ¡¾ÙΪеÄleaderµÄ»°£¬ÕâÑùÏûÏ¢¾Í²»»áÖØ¸´¡£

   
4549 ´Îä¯ÀÀ       27
Ïà¹ØÎÄÕÂ

»ùÓÚEAµÄÊý¾Ý¿â½¨Ä£
Êý¾ÝÁ÷½¨Ä££¨EAÖ¸ÄÏ£©
¡°Êý¾Ýºþ¡±£º¸ÅÄî¡¢ÌØÕ÷¡¢¼Ü¹¹Óë°¸Àý
ÔÚÏßÉ̳ÇÊý¾Ý¿âϵͳÉè¼Æ ˼·+Ч¹û
 
Ïà¹ØÎĵµ

GreenplumÊý¾Ý¿â»ù´¡Åàѵ
MySQL5.1ÐÔÄÜÓÅ»¯·½°¸
ijµçÉÌÊý¾ÝÖÐ̨¼Ü¹¹Êµ¼ù
MySQL¸ßÀ©Õ¹¼Ü¹¹Éè¼Æ
Ïà¹Ø¿Î³Ì

Êý¾ÝÖÎÀí¡¢Êý¾Ý¼Ü¹¹¼°Êý¾Ý±ê×¼
MongoDBʵս¿Î³Ì
²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
PostgreSQLÊý¾Ý¿âʵսÅàѵ