±à¼ÍƼö: |
±¾ÎĽ«´Ó´óÊý¾Ý¼Ü¹¹±äǨÀúÊ·£¬Pravega
¼ò½é£¬Pravega ½ø½×ÌØÐÔÒÔ¼°³µÁªÍøÊ¹Óó¡¾°ÕâËĸö·½Ãæ½éÉÜ Pravega£¬Öصã½éÉÜ
DellEMC ΪºÎÒªÑз¢ Pravega£¬Pravega ½â¾öÁË´óÊý¾Ý´¦ÀíÆ½Ì¨µÄÄÄЩʹµãÒÔ¼°Óë
Flink ½áºÏ»áÅöײ³öÔõÑùµÄ»ð»¨¡£
±¾ÎÄÀ´×ÔFlink ÖÐÎÄÉçÇø £¬ÓÉ»ðÁú¹ûÈí¼þLinda±à¼¡¢ÍƼö¡£ |
|
±¾ÎĽ«´Ó´óÊý¾Ý¼Ü¹¹±äǨÀúÊ·£¬Pravega ¼ò½é£¬Pravega
½ø½×ÌØÐÔÒÔ¼°³µÁªÍøÊ¹Óó¡¾°ÕâËĸö·½Ãæ½éÉÜ Pravega£¬Öصã½éÉÜ DellEMC ΪºÎÒªÑз¢ Pravega£¬Pravega
½â¾öÁË´óÊý¾Ý´¦ÀíÆ½Ì¨µÄÄÄЩʹµãÒÔ¼°Óë Flink ½áºÏ»áÅöײ³öÔõÑùµÄ»ð»¨¡£
´óÊý¾Ý¼Ü¹¹±äǨ
Lambda ¼Ü¹¹Ö®Í´

ÈçºÎÓÐЧµØÌáÈ¡ºÍÌṩÊý¾Ý£¬ÊÇ´óÊý¾Ý´¦ÀíÓ¦Óüܹ¹ÊÇ·ñ³É¹¦µÄ¹Ø¼üÖ®´¦¡£ÓÉÓÚ´¦ÀíËÙ¶ÈºÍÆµÂʵIJ»Í¬£¬Êý¾ÝµÄÉãÈ¡ÐèҪͨ¹ýÁ½ÖÖ²ßÂÔÀ´½øÐС£ÉÏͼ¾ÍÊǵäÐ͵Ä
Lambda¼Ü¹¹£º°Ñ´óÊý¾Ý´¦Àí¼Ü¹¹·ÖΪÅú´¦ÀíºÍʵʱÁ÷´¦ÀíÁ½Ì×¶ÀÁ¢µÄ¼ÆËã»ù´¡¼Ü¹¹¡£
¶ÔÓÚʵʱ´¦ÀíÀ´Ëµ£¬À´×Ô´«¸ÐÆ÷£¬Òƶ¯É豸»òÕßÓ¦ÓÃÈÕÖ¾µÄÊý¾Ýͨ³£Ð´ÈëÏûÏ¢¶ÓÁÐϵͳ£¨Èç Kafka),
ÏûÏ¢¶ÓÁиºÔðΪÁ÷´¦ÀíÓ¦ÓÃÌṩÊý¾ÝµÄÁÙʱ»º³å¡£È»ºóÔÙʹÓà Spark Streaming ´Ó Kafka
ÖжÁÈ¡Êý¾Ý×öʵʱµÄÁ÷¼ÆËã¡£µ«ÓÉÓÚ Kafka ²»»áÒ»Ö±±£´æÀúÊ·Êý¾Ý£¬Òò´ËÈç¹ûÓû§µÄÉÌÒµÂß¼ÊǽáºÏÀúÊ·Êý¾ÝºÍʵʱÊý¾Ýͬʱ×ö·ÖÎö£¬ÄÇôÕâÌõÁ÷Ë®Ïßʵ¼ÊÉÏÊÇûÓа취Íê³ÉµÄ¡£Òò´ËΪÁ˲¹³¥£¬ÐèÒª¶îÍ⿪±ÙÒ»ÌõÅú´¦ÀíµÄÁ÷Ë®Ïߣ¬¼´Í¼ÖÐ"
Batch "²¿·Ö¡£
¶ÔÓÚÅú´¦ÀíÕâÌõÁ÷Ë®ÏßÀ´Ëµ£¬¼¯ºÏÁ˷dz£¶àµÄµÄ¿ªÔ´´óÊý¾Ý×é¼þÈç ElasticSearch, Amazon
S3, HDFS, Cassandra ÒÔ¼° Spark µÈ¡£Ö÷Òª¼ÆËãÂß¼ÊÇÊÇͨ¹ý Spark À´ÊµÏÖ´ó¹æÄ£µÄ
Map-Reduce ²Ù×÷£¬ÓŵãÔÚÓÚ½á¹û±È½Ï¾«È·£¬ÒòΪ¿ÉÒÔ½áºÏËùÓÐÀúÊ·Êý¾ÝÀ´½øÐмÆËã·ÖÎö£¬È±µãÔÚÓÚÑÓ³Ù»á±È½Ï´ó¡£
ÕâÌ×¾µäµÄ´óÊý¾Ý´¦Àí¼Ü¹¹¿ÉÒÔ×ܽá³öÈý¸öÎÊÌ⣺
Á½ÌõÁ÷Ë®Ïß´¦ÀíµÄÑÓ³ÙÏà²î½Ï´ó£¬ÎÞ·¨Í¬Ê±½áºÏÁ½ÌõÁ÷Ë®Ïß½øÐÐѸËٵľۺϲÙ×÷£¬Í¬Ê±½áºÏÀúÊ·Êý¾ÝºÍʵʱÊý¾ÝµÄ´¦ÀíÐÔÄܵÍÏ¡£
Êý¾Ý´æ´¢³É±¾´ó¡£¶øÔÚÉÏͼµÄ¼Ü¹¹ÖУ¬ÏàͬµÄÊý¾Ý»áÔÚ¶à¸ö´æ´¢×é¼þÖж¼´æÔÚÒ»·Ý»ò¶à·Ý¿½±´£¬Êý¾ÝµÄÈßÓàÎÞÒÉ»á´ó´óÔö¼ÓÆóÒµ¿Í»§µÄ³É±¾¡£²¢ÇÒ¿ªÔ´´æ´¢µÄÊý¾ÝÈÝ´íºÍ³Ö¾Ã»¯¿É¿¿ÐÔÒ»Ö±Ò²ÊÇÖµµÃÉÌȶµÄµØ·½£¬¶ÔÓÚÊý¾Ý°²È«Ãô¸ÐµÄÆóÒµÓû§À´Ëµ£¬ÐèÒªÑϸñ±£Ö¤Êý¾ÝµÄ²»¶ªÊ§¡£
ÖØ¸´¿ª·¢¡£Í¬ÑùµÄ´¦ÀíÁ÷³Ì±»Á½ÌõÁ÷Ë®Ïß½øÐÐÁËÁ½´Î£¬ÏàͬµÄÊý¾Ý½ö½öÒòΪ´¦Àíʱ¼ä²»Í¬¶øÒªÔÚ²»Í¬µÄ¿ò¼ÜÄÚ·Ö±ð¼ÆËãÒ»´Î£¬ÎÞÒÉ»áÔö¼ÓÊý¾Ý¿ª·¢ÕßÖØ¸´¿ª·¢µÄ¸ºµ£¡£
Á÷ʽ´æ´¢µÄÌØµã
ÔÚÕýʽ½éÉÜ Pravega ֮ǰ£¬Ê×Ïȼòµ¥Ì¸Ì¸Á÷ʽÊý¾Ý´æ´¢µÄÒ»Ð©ÌØµã¡£
Èç¹ûÎÒÃÇÏëҪͳһÁ÷Åú´¦ÀíµÄ´óÊý¾Ý´¦Àí¼Ü¹¹£¬Æäʵ¶Ô´æ´¢ÓлìºÏµÄÒªÇó¡£

¶ÔÓÚÀ´×ÔÐòÁоɲ¿·ÖµÄÀúÊ·Êý¾Ý£¬ÐèÒªÌṩ¸ßÍÌ͵ĶÁÐÔÄÜ£¬¼´ catch-up read
¶ÔÓÚÀ´×ÔÐòÁÐв¿·ÖµÄʵʱÊý¾Ý£¬ÐèÒªÌṩµÍÑÓ³ÙµÄ append-only βд tailing write
ÒÔ¼°Î²¶Á tailing read
ÖØ¹¹µÄÁ÷ʽ´æ´¢¼Ü¹¹

Ïñ Kafka£¬Cassandra µÈ·Ö²¼Ê½´æ´¢×é¼þÀ´Ëµ£¬Æä´æ´¢¼Ü¹¹¶¼´ÓÉÏÍùÏÂ×ñÑ´ÓרÓеÄÈÕÖ¾´æ´¢£¬µ½±¾µØÎļþ£¬ÔÙµ½¼¯ÈºÉϵķֲ¼Ê½´æ´¢µÄÕâÖÖģʽ¡£
¶ø Pravega ÍŶÓÊÔÍ¼ÖØ¹¹Á÷ʽ´æ´¢µÄ¼Ü¹¹£¬ÒýÈë Pravega Stream ÕâÒ»³éÏó¸ÅÄî×÷ΪÁ÷ʽÊý¾Ý´æ´¢µÄ»ù±¾µ¥Î»¡£Stream
ÊÇÃüÃûµÄ¡¢³Ö¾ÃµÄ¡¢½ö×·¼ÓµÄ¡¢ÎÞÏÞµÄ×Ö½ÚÐòÁС£
ÈçÉÏͼËùʾ£¬´æ´¢¼Ü¹¹×îµ×²ãÊÇ»ùÓÚ¿ÉÀ©Õ¹·Ö²¼Ê½ÔÆ´æ´¢£¬Öмä²ã±íʾÈÕÖ¾Êý¾Ý´æ´¢Îª Stream À´×÷Ϊ¹²ÏíµÄ´æ´¢ÔÓȻºó»ùÓÚ
Stream ¿ÉÒÔÏòÉÏÌṩ²»Í¬¹¦ÄܵIJÙ×÷:ÈçÏûÏ¢¶ÓÁУ¬NoSQL£¬Á÷ʽÊý¾ÝµÄÈ«ÎÄËÑË÷ÒÔ¼°½áºÏ Flink
À´×öʵʱºÍÅú·ÖÎö¡£»»¾ä»°Ëµ£¬Pravega ÌṩµÄ Stream ÔÓï¿ÉÒÔ±ÜÃâÏÖÓдóÊý¾Ý¼Ü¹¹ÖÐÔʼÊý¾ÝÔÚ¶à¸ö¿ªÔ´´æ´¢ËÑË÷²úÆ·ÖÐÒÆ¶¯¶ø²úÉúµÄÊý¾ÝÈßÓàÏÖÏ󣬯äÔÚ´æ´¢²ã¾ÍÍê³ÉÁËͳһµÄÊý¾Ýºþ¡£
ÖØ¹¹µÄ´óÊý¾Ý¼Ü¹¹

ÎÒÃÇÌá³öµÄ´óÊý¾Ý¼Ü¹¹£¬ÒÔ Apache Flink ×÷Ϊ¼ÆËãÒýÇæ£¬Í¨¹ýͳһµÄÄ£ÐÍ/APIÀ´Í³Ò»Åú´¦ÀíºÍÁ÷´¦Àí¡£ÒÔ
Pavega ×÷Ϊ´æ´¢ÒýÇæ£¬ÎªÁ÷ʽÊý¾Ý´æ´¢ÌṩͳһµÄ³éÏó£¬Ê¹µÃ¶ÔÀúÊ·ºÍʵʱÊý¾ÝÓÐÒ»ÖµķÃÎÊ·½Ê½¡£Á½ÕßͳһÐγÉÁË´Ó´æ´¢µ½¼ÆËãµÄ±Õ»·£¬Äܹ»Í¬Ê±Ó¦¶Ô¸ßÍÌ͵ÄÀúÊ·Êý¾ÝºÍµÍÑÓʱµÄʵʱÊý¾Ý¡£Í¬Ê±
Pravega ÍŶӻ¹¿ª·¢ÁË Flink-Pravega Connector£¬Îª¼ÆËãºÍ´æ´¢µÄÕûÌ×Á÷Ë®ÏßÌṩ
Exactly-Once µÄÓïÒå¡£
Pravega ¼ò½é
Pravega µÄÉè¼Æ×ÚÖ¼ÊÇΪÁ÷µÄʵʱ´æ´¢Ìṩ½â¾ö·½°¸¡£Ó¦ÓóÌÐò½«Êý¾Ý³Ö¾Ã»¯´æ´¢µ½ Pravega
ÖУ¬Pravega µÄ Stream ¿ÉÒÔÓÐÎÞÏÞÖÆµÄÊýÁ¿²¢Çҳ־û¯´æ´¢ÈÎÒⳤʱ¼ä£¬Ê¹ÓÃͬÑùµÄ Reader
API Ìṩβ¶Á (tail read) ºÍ×·¸Ï¶Á (catch-up read) ¹¦ÄÜ£¬Äܹ»ÓÐЧÂú×ãÀëÏß¼ÆËãºÍʵʱ¼ÆËãÁ½ÖÖ´¦Àí·½Ê½µÄͳһ¡£
Pravega »ù±¾¸ÅÄî

½áºÏÉÏͼ¼òÒª½éÉÜ Pravega µÄ»ù±¾¸ÅÄ
Stream
Pravega »á°ÑдÈëµÄÊý¾Ý×éÖ¯³É Stream,Stream ÊÇÃüÃûµÄ¡¢³Ö¾ÃµÄ¡¢½ö×·¼ÓµÄ¡¢ÎÞÏÞµÄ×Ö½ÚÐòÁС£
Stream Segments
Pravega Stream »á»®·ÖΪһ¸ö»ò¶à¸ö Segments£¬Ï൱ÓÚ Stream ÖÐÊý¾ÝµÄ·ÖƬ£¬ËüÊÇÒ»¸ö
append-only µÄÊý¾Ý¿é£¬¶ø Pravega Ò²ÊÇ»ùÓÚ Segment »ù´¡ÉÏʵÏÖ×Ô¶¯µÄµ¯ÐÔÉìËõ¡£Segment
µÄÊýÁ¿Ò²»á¸ù¾ÝÊý¾ÝµÄÁ÷Á¿½øÐÐ×Ô¶¯µÄÁ¬Ðø¸üС£
Event
Pravega's client API ÔÊÐíÓû§ÒÔ Event Ϊ»ù±¾µ¥Î»Ð´ÈëºÍ¶ÁÈ¡Êý¾Ý£¬Event
¾ßÌåÊÇStream ÄÚ²¿×Ö½ÚÁ÷µÄ¼¯ºÏ¡£Èç IOT ´«¸ÐÆ÷µÄÒ»´ÎζȼǼдÈë Pravega ¾Í¿ÉÒÔÀí½â³ÉΪһ¸ö
Event.
Routing Key
ÿһ¸ö Event ¶¼»áÓÐÒ»¸ö Routing Key£¬ËüÊÇÓû§×Ô¶¨ÒåµÄÒ»¸ö×Ö·û´®£¬ÓÃÀ´¶ÔÏàËÆµÄ
Event ½øÐзÖ×é¡£ÓµÓÐÏàͬ Routing Key µÄ Event ¶¼»á±»Ð´ÈëÏàͬµÄ Stream
Segment ÖС£Pravega ͨ¹ý Routing Key À´Ìṩ¶ÁдÓïÒå¡£
Reader Group
ÓÃÓÚʵÏÖ¶ÁÈ¡Êý¾ÝµÄ¸ºÔؾùºâ¡£¿ÉÒÔͨ¹ý¶¯Ì¬Ôö¼Ó»ò¼õÉÙ Reader Group ÖÐ ReaderµÄÊýÁ¿À´¸Ä±ä¶ÁÈ¡Êý¾ÝµÄ²¢·¢¶È¡£¸üΪÏêϸµÄ½éÉÜÇë²Î¿¼
Pravega ¹Ù·½Îĵµ£º
http://pravega.io/docs/latest/pravega-concepts
Pravega ϵͳ¼Ü¹¹

ÔÚ¿ØÖƲãÃæ£¬Controller ×÷Ϊ Pravega ¼¯ÈºµÄÖ÷½Úµã¶ÔÊý¾Ý²ãÃæµÄ Segment
Store×ö¹ÜÀí£¬Ìṩ¶ÔÁ÷Êý¾ÝµÄ´´½¨£¬¸üÐÂÒÔ¼°É¾³ýµÈ²Ù×÷¡£Í¬Ê±Ëü»¹³Ðµ£ÊµÊ±¼à²â¼¯Èº½¡¿µ×´Ì¬£¬»ñÈ¡Á÷Êý¾ÝÐÅÏ¢£¬ÊÕ¼¯¼à¿ØÖ¸±êµÈ¹¦ÄÜ¡£Í¨³£¼¯ÈºÖлáÓÐ3·Ý
Controller À´±£Ö¤¸ß¿ÉÓá£
ÔÚÊý¾Ý²ãÃæ£¬Segment Store Ìṩ¶Áд Stream ÄÚÊý¾ÝµÄ API¡£ÔÚ Pravega
ÀïÃæ£¬Êý¾ÝÊÇ·Ö²ã´æ´¢µÄ£º
Tier 1 ´æ´¢
Tier1 µÄ´æ´¢Í¨³£²¿ÊðÔÚ Pravega ¼¯ÈºÄÚ²¿£¬Ö÷ÒªÊÇÌṩ¶ÔµÍÑÓ³Ù£¬¶ÌÆÚµÄÈÈÊý¾ÝµÄ´æ´¢¡£ÔÚÿ¸ö
Segment Store ½áµã¶¼ÓÐ Cache ÒÔ¼Ó¿ìÊý¾Ý¶ÁÈ¡ËÙÂÊ£¬Pravega ʹÓÃApache
Bookeeper À´±£Ö¤µÍÑÓ³ÙµÄÈÕÖ¾´æ´¢·þÎñ¡£
Long-term ´æ´¢
Long-term µÄ´æ´¢Í¨³£²¿ÊðÔÚ Pravega ¼¯ÈºÍⲿ£¬Ö÷ÒªÊÇÌṩ¶ÔÁ÷Êý¾ÝµÄ³¤ÆÚ´æ´¢£¬¼´ÀäÊý¾ÝµÄ´æ´¢¡£²»½öÖ§³Ö
HDFS£¬NFS£¬»¹»áÖ§³ÖÆóÒµ¼¶µÄ´æ´¢Èç Dell EMCµÄ ECS£¬Isilon µÈ²úÆ·¡£
Pravega ½ø½×ÌØÐÔ
¶Áд·ÖÀë

ÔÚ Tier1 ´æ´¢²¿·Ö£¬Ð´ÈëÊý¾ÝµÄʱºòͨ¹ý Bookkeeper ±£Ö¤ÁËÊý¾ÝÒѾÔÚËùÓÐµÄ Segment
Store ÖÐÂäÅÌ£¬±£Ö¤ÁËÊý¾ÝдÈë³É¹¦¡£
¶Áд·ÖÀëÓÐÖúÓÚÓÅ»¯¶ÁдÐÔÄÜ£ºÖ»´Ó Tier1 µÄ Cache ºÍ Long-term ´æ´¢È¥¶Á£¬²»È¥¶Á
Tier1 ÖÐµÄ Bookkeeper¡£
ÔÚ¿Í»§¶ËÏò Pravega ·¢Æð¶ÁÊý¾ÝµÄÇëÇóµÄʱºò£¬Pravega »á¾ö¶¨Õâ¸öÊý¾Ý¾¿¾¹ÊÇ´ÓTier1
µÄ Cache ½øÐеÍÑÓʱµÄ tail-read£¬»¹ÊÇÈ¥ Long-term µÄ³¤ÆÚ´æ´¢Êý¾Ý(¶ÔÏó´æ´¢/NFS)È¥½øÐÐÒ»¸ö¸ßÍÌÍÂÁ¿µÄ
catch-up read£¨Èç¹ûÊý¾Ý²»ÔÚ Cache£¬ÐèÒª°´Ðèload µ½ Cache ÖУ©¡£¶Á²Ù×÷ÊǶԿͻ§¶Ë͸Ã÷µÄ¡£
Tier1 µÄ Bookkeeper ÔÚ¼¯Èº²»³öÏÖ¹ÊÕϵÄÇé¿öÏÂÓÀÔ¶²»½øÐжÁÈ¡²Ù×÷£¬Ö»½øÐÐдÈë²Ù×÷¡£
µ¯ÐÔÉìËõ

Stream ÖÐµÄ Segment ÊýÁ¿»áËæ×Å IO ¸ºÔضø½øÐе¯ÐÔµÄ×Ô¶¯ÉìËõ¡£ÒÔÉÏͼΪÀý×Ó¼òµ¥²ûÊö£º
Êý¾ÝÁ÷ÔÚ t0 ʱ¿ÌдÈë Pravega£¬¸ù¾Ý·ÓɼüÊý¾Ý»á·Óɵ½ Segment0 ºÍSegment1
ÖУ¬Èç¹ûÊý¾ÝдÈëËٶȱ£³Öºã¶¨²»±ä£¬ÄÇô Segemnt ÊýÁ¿²»»á·¢Éú±ä»¯¡£
ÔÚ t1 ʱ¿Ìϵͳ¸ÐÖªµ½ segment1 Êý¾ÝдÈëËÙÂʼӿ죬ÓÚÊǽ«Æä»®·ÖΪÁ½¸ö²¿·Ö:Segment2
ºÍ Segment3¡£Õâʱºò Segment1 »á½øÈë Sealed ״̬£¬²»ÔÙ½ÓÊÜдÈëÊý¾Ý£¬Êý¾Ý»á¸ù¾Ý·Óɼü·Ö±ðÖØ¶¨Ïòµ½
Segment2 ºÍ Segment3.
Óë Scale-Up ²Ù×÷Ïà¶ÔÓ¦£¬ÏµÍ³Ò²¿ÉÒÔ¸ù¾ÝÊý¾ÝдÈëËٶȱäÂýºóÌṩ Scale-Down ²Ù×÷¡£ÈçÔÚ
t3 ʱ¿Ìϵͳ Segment2 ºÍ Segment5 дÈëÁ÷Á¿¼õÉÙ£¬Òò´ËºÏ²¢³ÉÐ嵀 Segment6¡£
¶Ëµ½¶ËµÄµ¯ÐÔÉìËõ

Pravega ÊÇÒÔ Kubernetes Operator À´¶Ô¼¯Èº¸÷×é¼þ½øÐÐÓÐ״̬µÄÓ¦Óò¿Êð£¬Õâ¿ÉÒÔʹµÃÓ¦Óõĵ¯ÐÔÉìËõ¸üΪÁé»î·½±ã¡£
Pravega ×î½üÒ²ÔÚºÍ Ververica ½øÐÐÉî¶ÈºÏ×÷,ÖÂÁ¦ÓÚÔÚ Pravega ¶ËʵÏÖ Kubernetes
Pod ¼¶±ðµÄµ¯ÐÔÉìËõͬʱÔÚ Flink ¶Ëͨ¹ý rescaling Flink µÄ Task ÊýÁ¿À´ÊµÏÖµ¯ÐÔÉìËõ¡£
ÊÂÎñÐÔдÈë

Pravega ͬÑùÌṩÊÂÎñÐÔµÄдÈë²Ù×÷¡£ÔÚÌá½»ÊÂÎñ֮ǰ£¬Êý¾Ý»á¸ù¾Ý·ÓɼüдÈëµ½²»Í¬µÄ Transaction
Segment ÖУ¬Õâʱºò Segment ¶ÔÓÚ Reader À´ËµÊDz»¿É¼ûµÄ¡£Ö»ÓÐÔÚÊÂÎñÌá½»Ö®ºó£¬Transaction
Segment ²Å»á¸÷×Ô×·¼Óµ½ Stream Segment µÄĩ⣬Õâʱºò Segment ¶ÔÓÚ
Reader ²ÅÊǿɼûµÄ¡£Ð´ÈëÊÂÎñµÄÖ§³ÖÒ²ÊÇʵÏÖÓë Flink µÄ¶Ëµ½¶Ë Exactly-Once
ÓïÒåµÄ¹Ø¼ü¡£
Pravega vs. Kafka
< 
Ê×ÏÈ×î¹Ø¼üµÄ²»Í¬ÔÚÓÚÁ½ÕߵĶ¨Î»£ºKafka µÄ¶¨Î»ÊÇÏûÏ¢¶ÓÁУ¬¶ø Pravega µÄ¶¨Î»ÊÇ´æ´¢£¬»á¸ü¹Ø×¢ÓÚÊý¾ÝµÄ¶¯Ì¬ÉìËõ£¬°²È«ÐÔ£¬ÍêÕûÐÔµÈ´æ´¢ÌØÐÔ¡£
¶ÔÓÚÁ÷ʽÊý¾Ý´¦ÀíÀ´Ëµ£¬Êý¾ÝÓ¦¸Ã±»ÊÓΪÁ¬ÐøºÍÎÞÏ޵ġ£Kafka ×÷Ϊ»ùÓÚ±¾µØÎļþϵͳµÄÒ»¸öÏûÏ¢¶ÓÁУ¬Í¨¹ý²ÉÓÃÌí¼Óµ½ÈÕÖ¾ÎļþµÄĩβ²¢¸ú×ÙÆäÄÚÈÝ(
offset »úÖÆ)µÄ·½Ê½À´Ä£ÄâÎÞÏÞµÄÊý¾ÝÁ÷¡£È»¶øÕâÖÖ·½Ê½±ØÈ»ÊÜÏÞÓÚ±¾µØÎļþϵͳµÄÎļþÃèÊö·ûÉÏÏÞÒÔ¼°´ÅÅÌÈÝÁ¿£¬Òò´Ë²¢·ÇÎÞÏÞ¡£
¶øÁ½ÕߵıȽÏÔÚͼÖиø³öÁ˱ȽÏÏêϸµÄ×ܽᣬ²»ÔÙ׸Êö¡£
Pravega Flink Connector
ΪÁ˸ü·½±ãÓë Flink µÄ½áºÏʹÓã¬ÎÒÃÇ»¹ÌṩÁË Pravega Flink Connector£¨https://github.com/pravega/flink-connectors),
Pravega ÍŶӻ¹¼Æ»®½«¸Ã Connector ¹±Ï×µ½ Flink ÉçÇø¡£Connector ÌṩÒÔÏÂÌØÐÔ£º
¶Ô Reader ºÍ Writer ¶¼ÌṩÁË Exactly-once ÓïÒå±£Ö¤£¬È·±£ÕûÌõÁ÷Ë®Ï߶˵½¶ËµÄ
Exactly-Once
Óë Flink µÄ checkpoints ºÍ savepoints »úÖÆµÄÎÞ·ìñîºÏ
Ö§³Ö¸ßÍÌ͵ÍÑӳٵIJ¢·¢¶Áд
Table API À´Í³Ò»¶Ô Pravega Sream µÄÁ÷Åúͳһ´¦Àí
³µÁªÍøÊ¹Óó¡¾°

ÒÔÎÞÈ˼ÝÊ»³µÁªÍøÕâÖÖÄܹ»²úÉúº£Á¿ PB ¼¶Êý¾ÝµÄÓ¦Ó󡾰ΪÀý£º
ÐèÒª¶Ô³µ¿ö·¿öÊý¾Ý×öʵʱµÄ´¦ÀíÒÔ¼°Ê±¶Ô·Ï߹滮×ö³ö΢¹ÛµÄÔ¤²âºÍ¹æ»®
ÐèÒª¶Ô½Ï³¤ÆÚÐÐÊ»Êý¾ÝÔËÐлúÆ÷ѧϰËã·¨À´×ö·Ïߵĺê¹ÛÔ¤²âºÍ¹æ»®£¬ÕâÊôÓÚÅú´¦Àí
ͬʱÐèÒª½áºÏʵʱ´¦ÀíºÍÅú´¦Àí£¬ÀûÓÃÀúÊ·Êý¾ÝÉú³ÉµÄ»úÆ÷ѧϰģÐͺÍʵʱÊý¾Ý·´À¡À´ÓÅ»¯¼ì²â½á¹û
¶ø¿Í»§¹Ø×¢µÄ¹Ø¼üÖ¸±êÖ÷ÒªÔÚ£º
ÈçºÎ±£Ö¤¸ßЧµØ¶Ëµ½¶Ë´¦ÀíËÙ¶È
ÈçºÎ¾¡¿ÉÄܼõÉÙ»úÆ÷ѧϰģÐ͵ÄѵÁ·Ê±¼ä
ÈçºÎ¾¡¿ÉÄܽµµÍ´æ´¢Êý¾ÝµÄÏûºÄÓë³É±¾
ÏÂÃæ¸ø³öÒýÈë Pravega ǰºóµÄ½â¾ö·½°¸±È½Ï¡£
½â¾ö·½°¸±È½Ï

Pravega µÄÒýÈëÎÞÒÉ´ó´ó¼ò½àÁË´óÊý¾Ý´¦ÀíµÄ¼Ü¹¹£º
Pravega ×÷Ϊ³éÏóµÄ´æ´¢½Ó¿Ú£¬Êý¾ÝÔÚ Pravega ²ã¾ÍʵÏÖÁËÒ»¸öÊý¾Ýºþ£ºÅú´¦Àí,ʵʱ´¦ÀíºÍÈ«ÎÄËÑË÷¶¼Ö»ÐèÒª´Ó
Pravega ÖлñÈ¡Êý¾Ý¡£Êý¾ÝÖ»ÔÚ Pravega ´æ´¢Ò»·Ý£¬¶ø²»ÐèÒªÏñµÚÒ»ÖÖ·½°¸ÖÐÊý¾ÝÈßÓàµØ´æ´¢ÔÚ
Kafka£¬ElasticSearch ºÍ Long Term Storage ÖУ¬Õâ¿ÉÒÔ¼«´ó¼õÉÙÁËÆóÒµÓû§Êý¾Ý´æ´¢µÄ³É±¾¡£
Pravega Äܹ»Ìṩ×Ô¶¯µÄ Tier Down£¬ÎÞÐèÒýÈë Flume µÈ×é¼þÀ´½øÐжîÍâµÄ ETL
¿ª·¢¡£
×é¼þµÃµ½¾«¼ò£¬´ÓÔÀ´µÄ Kafka+Flume+HDFS+ElasticSearch+Kibana+Spark+SparkStreaming
¾«¼òµ½ Pravega+Flink+Kibana+HDFS ,¼õÇáÔËάÈËÔ±µÄÔËάѹÁ¦¡£
Flink Äܹ»ÌṩÁ÷Åú´¦ÀíͳһµÄ¹¦ÄÜ£¬ÎÞÐèΪÏàͬµÄÊý¾ÝÌṩÁ½Ì×¶ÀÁ¢µÄ´¦Àí´úÂë¡£
×Ü ½á
Flink ٲȻÒѾ³ÉΪÁ÷ʽ¼ÆËãÒýÇæÖеÄÒ»¿ÅÉÁÁÁµÄÃ÷ÐÇ£¬È»¶øÁ÷ʽ´æ´¢ÁìÓòÉÐÊÇһƬ¿Õ°×¡£¶ø Pravega
µÄÉè¼Æ³õÖÔ¾ÍÊÇΪÁËÌîÉÏ´óÊý¾Ý´¦Àí¼Ü¹¹Õâһƴͼ×îºóµÄ¿Õ°×¡£¡°ËùÓмÆËã»úÁìÓòµÄÎÊÌ⣬¶¼¿ÉÒÔͨ¹ýÔö¼ÓÒ»¸ö¶îÍâµÄÖмä²ã³éÏó½â¾ö¡±£¬¶ø
Pravega ±¾ÖʾÍÊÇÔÚ¼ÆËãÒýÇæºÍµ×²ã´æ´¢Ö®¼ä³äµ±½âñî²ã£¬Ö¼ÔÚ½â¾öÐÂÒ»´ú´óÊý¾Ýƽ̨ÔÚÊý¾Ý´æ´¢²ãÉϵÄÌôÕ½¡£
|