Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Model Center   Code  
»áÔ±   
   
 
     
   
 ¶©ÔÄ
  ¾èÖú
Flink ÍêÃÀ´îµµ£ºÊý¾Ý´æ´¢²ãÉ쵀 Pravega
 
×÷ÕߣºëøêÅ
 
  2137  次浏览      27
2021-2-3
 
±à¼­ÍƼö:
±¾ÎĽ«´Ó´óÊý¾Ý¼Ü¹¹±äǨÀúÊ·£¬Pravega ¼ò½é£¬Pravega ½ø½×ÌØÐÔÒÔ¼°³µÁªÍøÊ¹Óó¡¾°ÕâËĸö·½Ãæ½éÉÜ Pravega£¬Öصã½éÉÜ DellEMC ΪºÎÒªÑз¢ Pravega£¬Pravega ½â¾öÁË´óÊý¾Ý´¦ÀíÆ½Ì¨µÄÄÄЩʹµãÒÔ¼°Óë Flink ½áºÏ»áÅöײ³öÔõÑùµÄ»ð»¨¡£
±¾ÎÄÀ´×ÔFlink ÖÐÎÄÉçÇø £¬ÓÉ»ðÁú¹ûÈí¼þLinda±à¼­¡¢ÍƼö¡£

±¾ÎĽ«´Ó´óÊý¾Ý¼Ü¹¹±äǨÀúÊ·£¬Pravega ¼ò½é£¬Pravega ½ø½×ÌØÐÔÒÔ¼°³µÁªÍøÊ¹Óó¡¾°ÕâËĸö·½Ãæ½éÉÜ Pravega£¬Öصã½éÉÜ DellEMC ΪºÎÒªÑз¢ Pravega£¬Pravega ½â¾öÁË´óÊý¾Ý´¦ÀíÆ½Ì¨µÄÄÄЩʹµãÒÔ¼°Óë Flink ½áºÏ»áÅöײ³öÔõÑùµÄ»ð»¨¡£

´óÊý¾Ý¼Ü¹¹±äǨ

Lambda ¼Ü¹¹Ö®Í´

ÈçºÎÓÐЧµØÌáÈ¡ºÍÌṩÊý¾Ý£¬ÊÇ´óÊý¾Ý´¦ÀíÓ¦Óüܹ¹ÊÇ·ñ³É¹¦µÄ¹Ø¼üÖ®´¦¡£ÓÉÓÚ´¦ÀíËÙ¶ÈºÍÆµÂʵIJ»Í¬£¬Êý¾ÝµÄÉãÈ¡ÐèҪͨ¹ýÁ½ÖÖ²ßÂÔÀ´½øÐС£ÉÏͼ¾ÍÊǵäÐ굀 Lambda¼Ü¹¹£º°Ñ´óÊý¾Ý´¦Àí¼Ü¹¹·ÖΪÅú´¦ÀíºÍʵʱÁ÷´¦ÀíÁ½Ì×¶ÀÁ¢µÄ¼ÆËã»ù´¡¼Ü¹¹¡£

¶ÔÓÚʵʱ´¦ÀíÀ´Ëµ£¬À´×Ô´«¸ÐÆ÷£¬Òƶ¯É豸»òÕßÓ¦ÓÃÈÕÖ¾µÄÊý¾Ýͨ³£Ð´ÈëÏûÏ¢¶ÓÁÐϵͳ£¨Èç Kafka), ÏûÏ¢¶ÓÁиºÔðΪÁ÷´¦ÀíÓ¦ÓÃÌṩÊý¾ÝµÄÁÙʱ»º³å¡£È»ºóÔÙʹÓà Spark Streaming ´Ó Kafka ÖжÁÈ¡Êý¾Ý×öʵʱµÄÁ÷¼ÆËã¡£µ«ÓÉÓÚ Kafka ²»»áÒ»Ö±±£´æÀúÊ·Êý¾Ý£¬Òò´ËÈç¹ûÓû§µÄÉÌÒµÂß¼­ÊǽáºÏÀúÊ·Êý¾ÝºÍʵʱÊý¾Ýͬʱ×ö·ÖÎö£¬ÄÇôÕâÌõÁ÷Ë®Ïßʵ¼ÊÉÏÊÇûÓа취Íê³ÉµÄ¡£Òò´ËΪÁ˲¹³¥£¬ÐèÒª¶îÍ⿪±ÙÒ»ÌõÅú´¦ÀíµÄÁ÷Ë®Ïߣ¬¼´Í¼ÖÐ" Batch "²¿·Ö¡£

¶ÔÓÚÅú´¦ÀíÕâÌõÁ÷Ë®ÏßÀ´Ëµ£¬¼¯ºÏÁ˷dz£¶àµÄµÄ¿ªÔ´´óÊý¾Ý×é¼þÈç ElasticSearch, Amazon S3, HDFS, Cassandra ÒÔ¼° Spark µÈ¡£Ö÷Òª¼ÆËãÂß¼­ÊÇÊÇͨ¹ý Spark À´ÊµÏÖ´ó¹æÄ£µÄ Map-Reduce ²Ù×÷£¬ÓŵãÔÚÓÚ½á¹û±È½Ï¾«È·£¬ÒòΪ¿ÉÒÔ½áºÏËùÓÐÀúÊ·Êý¾ÝÀ´½øÐмÆËã·ÖÎö£¬È±µãÔÚÓÚÑÓ³Ù»á±È½Ï´ó¡£

ÕâÌ×¾­µäµÄ´óÊý¾Ý´¦Àí¼Ü¹¹¿ÉÒÔ×ܽá³öÈý¸öÎÊÌ⣺

Á½ÌõÁ÷Ë®Ïß´¦ÀíµÄÑÓ³ÙÏà²î½Ï´ó£¬ÎÞ·¨Í¬Ê±½áºÏÁ½ÌõÁ÷Ë®Ïß½øÐÐѸËٵľۺϲÙ×÷£¬Í¬Ê±½áºÏÀúÊ·Êý¾ÝºÍʵʱÊý¾ÝµÄ´¦ÀíÐÔÄܵÍÏ¡£

Êý¾Ý´æ´¢³É±¾´ó¡£¶øÔÚÉÏͼµÄ¼Ü¹¹ÖУ¬ÏàͬµÄÊý¾Ý»áÔÚ¶à¸ö´æ´¢×é¼þÖж¼´æÔÚÒ»·Ý»ò¶à·Ý¿½±´£¬Êý¾ÝµÄÈßÓàÎÞÒÉ»á´ó´óÔö¼ÓÆóÒµ¿Í»§µÄ³É±¾¡£²¢ÇÒ¿ªÔ´´æ´¢µÄÊý¾ÝÈÝ´íºÍ³Ö¾Ã»¯¿É¿¿ÐÔÒ»Ö±Ò²ÊÇÖµµÃÉÌȶµÄµØ·½£¬¶ÔÓÚÊý¾Ý°²È«Ãô¸ÐµÄÆóÒµÓû§À´Ëµ£¬ÐèÒªÑϸñ±£Ö¤Êý¾ÝµÄ²»¶ªÊ§¡£

ÖØ¸´¿ª·¢¡£Í¬ÑùµÄ´¦ÀíÁ÷³Ì±»Á½ÌõÁ÷Ë®Ïß½øÐÐÁËÁ½´Î£¬ÏàͬµÄÊý¾Ý½ö½öÒòΪ´¦Àíʱ¼ä²»Í¬¶øÒªÔÚ²»Í¬µÄ¿ò¼ÜÄÚ·Ö±ð¼ÆËãÒ»´Î£¬ÎÞÒÉ»áÔö¼ÓÊý¾Ý¿ª·¢ÕßÖØ¸´¿ª·¢µÄ¸ºµ£¡£

Á÷ʽ´æ´¢µÄÌØµã

ÔÚÕýʽ½éÉÜ Pravega ֮ǰ£¬Ê×Ïȼòµ¥Ì¸Ì¸Á÷ʽÊý¾Ý´æ´¢µÄÒ»Ð©ÌØµã¡£

Èç¹ûÎÒÃÇÏëҪͳһÁ÷Åú´¦ÀíµÄ´óÊý¾Ý´¦Àí¼Ü¹¹£¬Æäʵ¶Ô´æ´¢ÓлìºÏµÄÒªÇó¡£

¶ÔÓÚÀ´×ÔÐòÁоɲ¿·ÖµÄÀúÊ·Êý¾Ý£¬ÐèÒªÌṩ¸ßÍÌ͵ĶÁÐÔÄÜ£¬¼´ catch-up read

¶ÔÓÚÀ´×ÔÐòÁÐв¿·ÖµÄʵʱÊý¾Ý£¬ÐèÒªÌṩµÍÑÓ³ÙµÄ append-only βд tailing write ÒÔ¼°Î²¶Á tailing read

ÖØ¹¹µÄÁ÷ʽ´æ´¢¼Ü¹¹

Ïñ Kafka£¬Cassandra µÈ·Ö²¼Ê½´æ´¢×é¼þÀ´Ëµ£¬Æä´æ´¢¼Ü¹¹¶¼´ÓÉÏÍùÏÂ×ñÑ­´ÓרÓеÄÈÕÖ¾´æ´¢£¬µ½±¾µØÎļþ£¬ÔÙµ½¼¯ÈºÉϵķֲ¼Ê½´æ´¢µÄÕâÖÖģʽ¡£

¶ø Pravega ÍŶÓÊÔÍ¼ÖØ¹¹Á÷ʽ´æ´¢µÄ¼Ü¹¹£¬ÒýÈë Pravega Stream ÕâÒ»³éÏó¸ÅÄî×÷ΪÁ÷ʽÊý¾Ý´æ´¢µÄ»ù±¾µ¥Î»¡£Stream ÊÇÃüÃûµÄ¡¢³Ö¾ÃµÄ¡¢½ö×·¼ÓµÄ¡¢ÎÞÏÞµÄ×Ö½ÚÐòÁС£

ÈçÉÏͼËùʾ£¬´æ´¢¼Ü¹¹×îµ×²ãÊÇ»ùÓÚ¿ÉÀ©Õ¹·Ö²¼Ê½ÔÆ´æ´¢£¬Öмä²ã±íʾÈÕÖ¾Êý¾Ý´æ´¢Îª Stream À´×÷Ϊ¹²ÏíµÄ´æ´¢Ô­ÓȻºó»ùÓÚ Stream ¿ÉÒÔÏòÉÏÌṩ²»Í¬¹¦ÄܵIJÙ×÷:ÈçÏûÏ¢¶ÓÁУ¬NoSQL£¬Á÷ʽÊý¾ÝµÄÈ«ÎÄËÑË÷ÒÔ¼°½áºÏ Flink À´×öʵʱºÍÅú·ÖÎö¡£»»¾ä»°Ëµ£¬Pravega ÌṩµÄ Stream Ô­Óï¿ÉÒÔ±ÜÃâÏÖÓдóÊý¾Ý¼Ü¹¹ÖÐԭʼÊý¾ÝÔÚ¶à¸ö¿ªÔ´´æ´¢ËÑË÷²úÆ·ÖÐÒÆ¶¯¶ø²úÉúµÄÊý¾ÝÈßÓàÏÖÏ󣬯äÔÚ´æ´¢²ã¾ÍÍê³ÉÁËͳһµÄÊý¾Ýºþ¡£

ÖØ¹¹µÄ´óÊý¾Ý¼Ü¹¹

ÎÒÃÇÌá³öµÄ´óÊý¾Ý¼Ü¹¹£¬ÒÔ Apache Flink ×÷Ϊ¼ÆËãÒýÇæ£¬Í¨¹ýͳһµÄÄ£ÐÍ/APIÀ´Í³Ò»Åú´¦ÀíºÍÁ÷´¦Àí¡£ÒÔ Pavega ×÷Ϊ´æ´¢ÒýÇæ£¬ÎªÁ÷ʽÊý¾Ý´æ´¢ÌṩͳһµÄ³éÏó£¬Ê¹µÃ¶ÔÀúÊ·ºÍʵʱÊý¾ÝÓÐÒ»ÖµķÃÎÊ·½Ê½¡£Á½ÕßͳһÐγÉÁË´Ó´æ´¢µ½¼ÆËãµÄ±Õ»·£¬Äܹ»Í¬Ê±Ó¦¶Ô¸ßÍÌ͵ÄÀúÊ·Êý¾ÝºÍµÍÑÓʱµÄʵʱÊý¾Ý¡£Í¬Ê± Pravega ÍŶӻ¹¿ª·¢ÁË Flink-Pravega Connector£¬Îª¼ÆËãºÍ´æ´¢µÄÕûÌ×Á÷Ë®ÏßÌṩ Exactly-Once µÄÓïÒå¡£

Pravega ¼ò½é

Pravega µÄÉè¼Æ×ÚÖ¼ÊÇΪÁ÷µÄʵʱ´æ´¢Ìṩ½â¾ö·½°¸¡£Ó¦ÓóÌÐò½«Êý¾Ý³Ö¾Ã»¯´æ´¢µ½ Pravega ÖУ¬Pravega µÄ Stream ¿ÉÒÔÓÐÎÞÏÞÖÆµÄÊýÁ¿²¢Çҳ־û¯´æ´¢ÈÎÒⳤʱ¼ä£¬Ê¹ÓÃͬÑùµÄ Reader API Ìṩβ¶Á (tail read) ºÍ×·¸Ï¶Á (catch-up read) ¹¦ÄÜ£¬Äܹ»ÓÐЧÂú×ãÀëÏß¼ÆËãºÍʵʱ¼ÆËãÁ½ÖÖ´¦Àí·½Ê½µÄͳһ¡£

Pravega »ù±¾¸ÅÄî

½áºÏÉÏͼ¼òÒª½éÉÜ Pravega µÄ»ù±¾¸ÅÄ

Stream

Pravega »á°ÑдÈëµÄÊý¾Ý×éÖ¯³É Stream,Stream ÊÇÃüÃûµÄ¡¢³Ö¾ÃµÄ¡¢½ö×·¼ÓµÄ¡¢ÎÞÏÞµÄ×Ö½ÚÐòÁС£

Stream Segments

Pravega Stream »á»®·ÖΪһ¸ö»ò¶à¸ö Segments£¬Ï൱ÓÚ Stream ÖÐÊý¾ÝµÄ·ÖƬ£¬ËüÊÇÒ»¸ö append-only µÄÊý¾Ý¿é£¬¶ø Pravega Ò²ÊÇ»ùÓÚ Segment »ù´¡ÉÏʵÏÖ×Ô¶¯µÄµ¯ÐÔÉìËõ¡£Segment µÄÊýÁ¿Ò²»á¸ù¾ÝÊý¾ÝµÄÁ÷Á¿½øÐÐ×Ô¶¯µÄÁ¬Ðø¸üС£

Event

Pravega's client API ÔÊÐíÓû§ÒÔ Event Ϊ»ù±¾µ¥Î»Ð´ÈëºÍ¶ÁÈ¡Êý¾Ý£¬Event ¾ßÌåÊÇStream ÄÚ²¿×Ö½ÚÁ÷µÄ¼¯ºÏ¡£Èç IOT ´«¸ÐÆ÷µÄÒ»´ÎζȼǼдÈë Pravega ¾Í¿ÉÒÔÀí½â³ÉΪһ¸ö Event.

Routing Key

ÿһ¸ö Event ¶¼»áÓÐÒ»¸ö Routing Key£¬ËüÊÇÓû§×Ô¶¨ÒåµÄÒ»¸ö×Ö·û´®£¬ÓÃÀ´¶ÔÏàËÆµÄ Event ½øÐзÖ×é¡£ÓµÓÐÏàͬ Routing Key µÄ Event ¶¼»á±»Ð´ÈëÏàͬµÄ Stream Segment ÖС£Pravega ͨ¹ý Routing Key À´Ìṩ¶ÁдÓïÒå¡£

Reader Group

ÓÃÓÚʵÏÖ¶ÁÈ¡Êý¾ÝµÄ¸ºÔؾùºâ¡£¿ÉÒÔͨ¹ý¶¯Ì¬Ôö¼Ó»ò¼õÉÙ Reader Group ÖÐ ReaderµÄÊýÁ¿À´¸Ä±ä¶ÁÈ¡Êý¾ÝµÄ²¢·¢¶È¡£¸üΪÏêϸµÄ½éÉÜÇë²Î¿¼ Pravega ¹Ù·½Îĵµ£º

http://pravega.io/docs/latest/pravega-concepts

Pravega ϵͳ¼Ü¹¹

ÔÚ¿ØÖƲãÃæ£¬Controller ×÷Ϊ Pravega ¼¯ÈºµÄÖ÷½Úµã¶ÔÊý¾Ý²ãÃæµÄ Segment Store×ö¹ÜÀí£¬Ìṩ¶ÔÁ÷Êý¾ÝµÄ´´½¨£¬¸üÐÂÒÔ¼°É¾³ýµÈ²Ù×÷¡£Í¬Ê±Ëü»¹³Ðµ£ÊµÊ±¼à²â¼¯Èº½¡¿µ×´Ì¬£¬»ñÈ¡Á÷Êý¾ÝÐÅÏ¢£¬ÊÕ¼¯¼à¿ØÖ¸±êµÈ¹¦ÄÜ¡£Í¨³£¼¯ÈºÖлáÓÐ3·Ý Controller À´±£Ö¤¸ß¿ÉÓá£

ÔÚÊý¾Ý²ãÃæ£¬Segment Store Ìṩ¶Áд Stream ÄÚÊý¾ÝµÄ API¡£ÔÚ Pravega ÀïÃæ£¬Êý¾ÝÊÇ·Ö²ã´æ´¢µÄ£º

Tier 1 ´æ´¢

Tier1 µÄ´æ´¢Í¨³£²¿ÊðÔÚ Pravega ¼¯ÈºÄÚ²¿£¬Ö÷ÒªÊÇÌṩ¶ÔµÍÑÓ³Ù£¬¶ÌÆÚµÄÈÈÊý¾ÝµÄ´æ´¢¡£ÔÚÿ¸ö Segment Store ½áµã¶¼ÓÐ Cache ÒÔ¼Ó¿ìÊý¾Ý¶ÁÈ¡ËÙÂÊ£¬Pravega ʹÓÃApache Bookeeper À´±£Ö¤µÍÑÓ³ÙµÄÈÕÖ¾´æ´¢·þÎñ¡£

Long-term ´æ´¢

Long-term µÄ´æ´¢Í¨³£²¿ÊðÔÚ Pravega ¼¯ÈºÍⲿ£¬Ö÷ÒªÊÇÌṩ¶ÔÁ÷Êý¾ÝµÄ³¤ÆÚ´æ´¢£¬¼´ÀäÊý¾ÝµÄ´æ´¢¡£²»½öÖ§³Ö HDFS£¬NFS£¬»¹»áÖ§³ÖÆóÒµ¼¶µÄ´æ´¢Èç Dell EMCµÄ ECS£¬Isilon µÈ²úÆ·¡£

Pravega ½ø½×ÌØÐÔ

¶Áд·ÖÀë

ÔÚ Tier1 ´æ´¢²¿·Ö£¬Ð´ÈëÊý¾ÝµÄʱºòͨ¹ý Bookkeeper ±£Ö¤ÁËÊý¾ÝÒѾ­ÔÚËùÓÐµÄ Segment Store ÖÐÂäÅÌ£¬±£Ö¤ÁËÊý¾ÝдÈë³É¹¦¡£

¶Áд·ÖÀëÓÐÖúÓÚÓÅ»¯¶ÁдÐÔÄÜ£ºÖ»´Ó Tier1 µÄ Cache ºÍ Long-term ´æ´¢È¥¶Á£¬²»È¥¶Á Tier1 ÖÐµÄ Bookkeeper¡£

ÔÚ¿Í»§¶ËÏò Pravega ·¢Æð¶ÁÊý¾ÝµÄÇëÇóµÄʱºò£¬Pravega »á¾ö¶¨Õâ¸öÊý¾Ý¾¿¾¹ÊÇ´ÓTier1 µÄ Cache ½øÐеÍÑÓʱµÄ tail-read£¬»¹ÊÇÈ¥ Long-term µÄ³¤ÆÚ´æ´¢Êý¾Ý(¶ÔÏó´æ´¢/NFS)È¥½øÐÐÒ»¸ö¸ßÍÌÍÂÁ¿µÄ catch-up read£¨Èç¹ûÊý¾Ý²»ÔÚ Cache£¬ÐèÒª°´Ðèload µ½ Cache ÖУ©¡£¶Á²Ù×÷ÊǶԿͻ§¶Ë͸Ã÷µÄ¡£

Tier1 µÄ Bookkeeper ÔÚ¼¯Èº²»³öÏÖ¹ÊÕϵÄÇé¿öÏÂÓÀÔ¶²»½øÐжÁÈ¡²Ù×÷£¬Ö»½øÐÐдÈë²Ù×÷¡£

µ¯ÐÔÉìËõ

Stream ÖÐµÄ Segment ÊýÁ¿»áËæ×Å IO ¸ºÔضø½øÐе¯ÐÔµÄ×Ô¶¯ÉìËõ¡£ÒÔÉÏͼΪÀý×Ó¼òµ¥²ûÊö£º

Êý¾ÝÁ÷ÔÚ t0 ʱ¿ÌдÈë Pravega£¬¸ù¾Ý·ÓɼüÊý¾Ý»á·Óɵ½ Segment0 ºÍSegment1 ÖУ¬Èç¹ûÊý¾ÝдÈëËٶȱ£³Öºã¶¨²»±ä£¬ÄÇô Segemnt ÊýÁ¿²»»á·¢Éú±ä»¯¡£

ÔÚ t1 ʱ¿Ìϵͳ¸ÐÖªµ½ segment1 Êý¾ÝдÈëËÙÂʼӿ죬ÓÚÊǽ«Æä»®·ÖΪÁ½¸ö²¿·Ö:Segment2 ºÍ Segment3¡£Õâʱºò Segment1 »á½øÈë Sealed ״̬£¬²»ÔÙ½ÓÊÜдÈëÊý¾Ý£¬Êý¾Ý»á¸ù¾Ý·Óɼü·Ö±ðÖØ¶¨Ïòµ½ Segment2 ºÍ Segment3.

Óë Scale-Up ²Ù×÷Ïà¶ÔÓ¦£¬ÏµÍ³Ò²¿ÉÒÔ¸ù¾ÝÊý¾ÝдÈëËٶȱäÂýºóÌṩ Scale-Down ²Ù×÷¡£ÈçÔÚ t3 ʱ¿Ìϵͳ Segment2 ºÍ Segment5 дÈëÁ÷Á¿¼õÉÙ£¬Òò´ËºÏ²¢³ÉÐ嵀 Segment6¡£

¶Ëµ½¶ËµÄµ¯ÐÔÉìËõ

Pravega ÊÇÒÔ Kubernetes Operator À´¶Ô¼¯Èº¸÷×é¼þ½øÐÐÓÐ״̬µÄÓ¦Óò¿Êð£¬Õâ¿ÉÒÔʹµÃÓ¦Óõĵ¯ÐÔÉìËõ¸üΪÁé»î·½±ã¡£

Pravega ×î½üÒ²ÔÚºÍ Ververica ½øÐÐÉî¶ÈºÏ×÷,ÖÂÁ¦ÓÚÔÚ Pravega ¶ËʵÏÖ Kubernetes Pod ¼¶±ðµÄµ¯ÐÔÉìËõͬʱÔÚ Flink ¶Ëͨ¹ý rescaling Flink µÄ Task ÊýÁ¿À´ÊµÏÖµ¯ÐÔÉìËõ¡£

ÊÂÎñÐÔдÈë

Pravega ͬÑùÌṩÊÂÎñÐÔµÄдÈë²Ù×÷¡£ÔÚÌá½»ÊÂÎñ֮ǰ£¬Êý¾Ý»á¸ù¾Ý·ÓɼüдÈëµ½²»Í¬µÄ Transaction Segment ÖУ¬Õâʱºò Segment ¶ÔÓÚ Reader À´ËµÊDz»¿É¼ûµÄ¡£Ö»ÓÐÔÚÊÂÎñÌá½»Ö®ºó£¬Transaction Segment ²Å»á¸÷×Ô×·¼Óµ½ Stream Segment µÄĩ⣬Õâʱºò Segment ¶ÔÓÚ Reader ²ÅÊǿɼûµÄ¡£Ð´ÈëÊÂÎñµÄÖ§³ÖÒ²ÊÇʵÏÖÓë Flink µÄ¶Ëµ½¶Ë Exactly-Once ÓïÒåµÄ¹Ø¼ü¡£

Pravega vs. Kafka

<

Ê×ÏÈ×î¹Ø¼üµÄ²»Í¬ÔÚÓÚÁ½ÕߵĶ¨Î»£ºKafka µÄ¶¨Î»ÊÇÏûÏ¢¶ÓÁУ¬¶ø Pravega µÄ¶¨Î»ÊÇ´æ´¢£¬»á¸ü¹Ø×¢ÓÚÊý¾ÝµÄ¶¯Ì¬ÉìËõ£¬°²È«ÐÔ£¬ÍêÕûÐÔµÈ´æ´¢ÌØÐÔ¡£

¶ÔÓÚÁ÷ʽÊý¾Ý´¦ÀíÀ´Ëµ£¬Êý¾ÝÓ¦¸Ã±»ÊÓΪÁ¬ÐøºÍÎÞÏ޵ġ£Kafka ×÷Ϊ»ùÓÚ±¾µØÎļþϵͳµÄÒ»¸öÏûÏ¢¶ÓÁУ¬Í¨¹ý²ÉÓÃÌí¼Óµ½ÈÕÖ¾ÎļþµÄĩβ²¢¸ú×ÙÆäÄÚÈÝ( offset »úÖÆ)µÄ·½Ê½À´Ä£ÄâÎÞÏÞµÄÊý¾ÝÁ÷¡£È»¶øÕâÖÖ·½Ê½±ØÈ»ÊÜÏÞÓÚ±¾µØÎļþϵͳµÄÎļþÃèÊö·ûÉÏÏÞÒÔ¼°´ÅÅÌÈÝÁ¿£¬Òò´Ë²¢·ÇÎÞÏÞ¡£

¶øÁ½ÕߵıȽÏÔÚͼÖиø³öÁ˱ȽÏÏêϸµÄ×ܽᣬ²»ÔÙ׸Êö¡£

Pravega Flink Connector

ΪÁ˸ü·½±ãÓë Flink µÄ½áºÏʹÓã¬ÎÒÃÇ»¹ÌṩÁË Pravega Flink Connector£¨https://github.com/pravega/flink-connectors), Pravega ÍŶӻ¹¼Æ»®½«¸Ã Connector ¹±Ï×µ½ Flink ÉçÇø¡£Connector ÌṩÒÔÏÂÌØÐÔ£º

¶Ô Reader ºÍ Writer ¶¼ÌṩÁË Exactly-once ÓïÒå±£Ö¤£¬È·±£ÕûÌõÁ÷Ë®Ï߶˵½¶ËµÄ Exactly-Once

Óë Flink µÄ checkpoints ºÍ savepoints »úÖÆµÄÎÞ·ìñîºÏ

Ö§³Ö¸ßÍÌ͵ÍÑӳٵIJ¢·¢¶Áд

Table API À´Í³Ò»¶Ô Pravega Sream µÄÁ÷Åúͳһ´¦Àí

³µÁªÍøÊ¹Óó¡¾°

ÒÔÎÞÈ˼ÝÊ»³µÁªÍøÕâÖÖÄܹ»²úÉúº£Á¿ PB ¼¶Êý¾ÝµÄÓ¦Ó󡾰ΪÀý£º

ÐèÒª¶Ô³µ¿ö·¿öÊý¾Ý×öʵʱµÄ´¦ÀíÒÔ¼°Ê±¶Ô·Ï߹滮×ö³ö΢¹ÛµÄÔ¤²âºÍ¹æ»®

ÐèÒª¶Ô½Ï³¤ÆÚÐÐÊ»Êý¾ÝÔËÐлúÆ÷ѧϰËã·¨À´×ö·Ïߵĺê¹ÛÔ¤²âºÍ¹æ»®£¬ÕâÊôÓÚÅú´¦Àí

ͬʱÐèÒª½áºÏʵʱ´¦ÀíºÍÅú´¦Àí£¬ÀûÓÃÀúÊ·Êý¾ÝÉú³ÉµÄ»úÆ÷ѧϰģÐͺÍʵʱÊý¾Ý·´À¡À´ÓÅ»¯¼ì²â½á¹û

¶ø¿Í»§¹Ø×¢µÄ¹Ø¼üÖ¸±êÖ÷ÒªÔÚ£º

ÈçºÎ±£Ö¤¸ßЧµØ¶Ëµ½¶Ë´¦ÀíËÙ¶È

ÈçºÎ¾¡¿ÉÄܼõÉÙ»úÆ÷ѧϰģÐ͵ÄѵÁ·Ê±¼ä

ÈçºÎ¾¡¿ÉÄܽµµÍ´æ´¢Êý¾ÝµÄÏûºÄÓë³É±¾

ÏÂÃæ¸ø³öÒýÈë Pravega ǰºóµÄ½â¾ö·½°¸±È½Ï¡£

½â¾ö·½°¸±È½Ï

Pravega µÄÒýÈëÎÞÒÉ´ó´ó¼ò½àÁË´óÊý¾Ý´¦ÀíµÄ¼Ü¹¹£º

Pravega ×÷Ϊ³éÏóµÄ´æ´¢½Ó¿Ú£¬Êý¾ÝÔÚ Pravega ²ã¾ÍʵÏÖÁËÒ»¸öÊý¾Ýºþ£ºÅú´¦Àí,ʵʱ´¦ÀíºÍÈ«ÎÄËÑË÷¶¼Ö»ÐèÒª´Ó Pravega ÖлñÈ¡Êý¾Ý¡£Êý¾ÝÖ»ÔÚ Pravega ´æ´¢Ò»·Ý£¬¶ø²»ÐèÒªÏñµÚÒ»ÖÖ·½°¸ÖÐÊý¾ÝÈßÓàµØ´æ´¢ÔÚ Kafka£¬ElasticSearch ºÍ Long Term Storage ÖУ¬Õâ¿ÉÒÔ¼«´ó¼õÉÙÁËÆóÒµÓû§Êý¾Ý´æ´¢µÄ³É±¾¡£

Pravega Äܹ»Ìṩ×Ô¶¯µÄ Tier Down£¬ÎÞÐèÒýÈë Flume µÈ×é¼þÀ´½øÐжîÍâµÄ ETL ¿ª·¢¡£

×é¼þµÃµ½¾«¼ò£¬´ÓÔ­À´µÄ Kafka+Flume+HDFS+ElasticSearch+Kibana+Spark+SparkStreaming ¾«¼òµ½ Pravega+Flink+Kibana+HDFS ,¼õÇáÔËάÈËÔ±µÄÔËάѹÁ¦¡£

Flink Äܹ»ÌṩÁ÷Åú´¦ÀíͳһµÄ¹¦ÄÜ£¬ÎÞÐèΪÏàͬµÄÊý¾ÝÌṩÁ½Ì×¶ÀÁ¢µÄ´¦Àí´úÂë¡£

×Ü ½á

Flink ٲȻÒѾ­³ÉΪÁ÷ʽ¼ÆËãÒýÇæÖеÄÒ»¿ÅÉÁÁÁµÄÃ÷ÐÇ£¬È»¶øÁ÷ʽ´æ´¢ÁìÓòÉÐÊÇһƬ¿Õ°×¡£¶ø Pravega µÄÉè¼Æ³õÖÔ¾ÍÊÇΪÁËÌîÉÏ´óÊý¾Ý´¦Àí¼Ü¹¹Õâһƴͼ×îºóµÄ¿Õ°×¡£¡°ËùÓмÆËã»úÁìÓòµÄÎÊÌ⣬¶¼¿ÉÒÔͨ¹ýÔö¼ÓÒ»¸ö¶îÍâµÄÖмä²ã³éÏó½â¾ö¡±£¬¶ø Pravega ±¾ÖʾÍÊÇÔÚ¼ÆËãÒýÇæºÍµ×²ã´æ´¢Ö®¼ä³äµ±½âñî²ã£¬Ö¼ÔÚ½â¾öÐÂÒ»´ú´óÊý¾Ýƽ̨ÔÚÊý¾Ý´æ´¢²ãÉϵÄÌôÕ½¡£

 
   
2137 ´Îä¯ÀÀ       27
Ïà¹ØÎÄÕÂ

»ùÓÚEAµÄÊý¾Ý¿â½¨Ä£
Êý¾ÝÁ÷½¨Ä££¨EAÖ¸ÄÏ£©
¡°Êý¾Ýºþ¡±£º¸ÅÄî¡¢ÌØÕ÷¡¢¼Ü¹¹Óë°¸Àý
ÔÚÏßÉ̳ÇÊý¾Ý¿âϵͳÉè¼Æ ˼·+Ч¹û
 
Ïà¹ØÎĵµ

GreenplumÊý¾Ý¿â»ù´¡Åàѵ
MySQL5.1ÐÔÄÜÓÅ»¯·½°¸
ijµçÉÌÊý¾ÝÖÐ̨¼Ü¹¹Êµ¼ù
MySQL¸ßÀ©Õ¹¼Ü¹¹Éè¼Æ
Ïà¹Ø¿Î³Ì

Êý¾ÝÖÎÀí¡¢Êý¾Ý¼Ü¹¹¼°Êý¾Ý±ê×¼
MongoDBʵս¿Î³Ì
²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
PostgreSQLÊý¾Ý¿âʵսÅàѵ
×îл¼Æ»®
DeepSeekÔÚÈí¼þ²âÊÔÓ¦ÓÃʵ¼ù 4-12[ÔÚÏß]
DeepSeek´óÄ£ÐÍÓ¦Óÿª·¢Êµ¼ù 4-19[ÔÚÏß]
UAF¼Ü¹¹ÌåϵÓëʵ¼ù 4-11[±±¾©]
AIÖÇÄÜ»¯Èí¼þ²âÊÔ·½·¨Óëʵ¼ù 5-23[ÉϺ£]
»ùÓÚ UML ºÍEA½øÐзÖÎöÉè¼Æ 4-26[±±¾©]
ÒµÎñ¼Ü¹¹Éè¼ÆÓ뽨ģ 4-18[±±¾©]
 
×îÐÂÎÄÕÂ
´óÊý¾Ýƽ̨ϵÄÊý¾ÝÖÎÀí
ÈçºÎÉè¼ÆÊµÊ±Êý¾Ýƽ̨£¨¼¼Êõƪ£©
´óÊý¾Ý×ʲú¹ÜÀí×ÜÌå¿ò¼Ü¸ÅÊö
Kafka¼Ü¹¹ºÍÔ­Àí
ELK¶àÖּܹ¹¼°ÓÅÁÓ
×îпγÌ
´óÊý¾Ýƽ̨´î½¨Óë¸ßÐÔÄܼÆËã
´óÊý¾Ýƽ̨¼Ü¹¹ÓëÓ¦ÓÃʵս
´óÊý¾ÝϵͳÔËά
´óÊý¾Ý·ÖÎöÓë¹ÜÀí
Python¼°Êý¾Ý·ÖÎö
³É¹¦°¸Àý
ijͨÐÅÉ豸ÆóÒµ PythonÊý¾Ý·ÖÎöÓëÍÚ¾ò
Ä³ÒøÐÐ È˹¤ÖÇÄÜ+Python+´óÊý¾Ý
±±¾© Python¼°Êý¾Ý·ÖÎö
ÉñÁúÆû³µ ´óÊý¾Ý¼¼Êõƽ̨-Hadoop
ÖйúµçÐÅ ´óÊý¾Ýʱ´úÓëÏÖ´úÆóÒµµÄÊý¾Ý»¯ÔËӪʵ¼ù