±à¼ÍƼö: |
±¾ÎÄÊ×ÏȽéÉÜʵʱÊý²ÖµÄ¸ÅÄȻºó½éÉÜ
Flink+TiDB ʵʱÊý²ÖµÄ¼Ü¹¹ÓëÓÅÊÆ£¬½Óןø³öһЩÒѾÔÚʹÓÃÖеÄÓû§³¡¾°£¬×îºó¸ø³öÔÚ
docker-compose »·¾³Ï嵀 Demo£¬ÓÃÓÚ¶ÁÕß½øÐг¢ÊÔ¡£
±¾ÎÄÀ´×ÔFlink ÖÐÎÄÉçÇø£¬ÓÉ»ðÁú¹ûÈí¼þLinda±à¼¡¢ÍƼö¡£ |
|
Ëæ×Å»¥ÁªÍø·ÉËÙ·¢Õ¹£¬ÆóÒµÒµÎñÖÖÀà»áÔ½À´Ô½¶à£¬ÒµÎñÊý¾ÝÁ¿»áÔ½À´Ô½´ó£¬µ±·¢Õ¹µ½Ò»¶¨¹æÄ£Ê±£¬´«Í³µÄÊý¾Ý´æ´¢½á¹¹Öð½¥ÎÞ·¨Âú×ãÆóÒµÐèÇó£¬ÊµÊ±Êý¾Ý²Ö¿â¾Í±ä³ÉÁËÒ»¸ö±ØÒªµÄ»ù´¡·þÎñ¡£ÒÔά±í
Join ΪÀý£¬Êý¾ÝÔÚÒµÎñÊý¾ÝÔ´ÖÐÒÔ·¶Ê½±íµÄÐÎʽ´æ´¢£¬ÔÚ·ÖÎöʱÐèÒª×ö´óÁ¿µÄ Join ²Ù×÷£¬½µµÍÐÔÄÜ¡£Èç¹ûÔÚÊý¾ÝÇåÏ´µ¼Èë¹ý³ÌÖоÍÄÜÁ÷ʽµÄÍê³É
Join£¬ÄÇô·ÖÎöʱ¾ÍÎÞÐèÔÙ´Î Join£¬´Ó¶øÌáÉý²éѯÐÔÄÜ¡£
ÀûÓÃʵʱÊý²Ö£¬ÆóÒµ¿ÉÒÔʵÏÖʵʱ OLAP ·ÖÎö¡¢ÊµÊ±Êý¾Ý¿´°å¡¢ÊµÊ±ÒµÎñ¼à¿Ø¡¢ÊµÊ±Êý¾Ý½Ó¿Ú·þÎñµÈÓÃ;¡£µ«Ï뵽ʵʱÊý²Ö£¬ºÜ¶àÈ˵ĵÚÒ»Ó¡Ïó¾ÍÊǼܹ¹¸´ÔÓ£¬ÄÑÒÔ²Ù×÷Óëά»¤¡£¶øµÃÒæÓÚаæ
Flink ¶Ô SQL µÄÖ§³Ö£¬ÒÔ¼° TiDB HTAP µÄÌØÐÔ£¬ÎÒÃÇ̽Ë÷ÁËÒ»¸ö¸ßЧ¡¢Ò×ÓÃµÄ Flink+TiDB
ʵʱÊý²Ö½â¾ö·½°¸¡£
±¾ÎĽ«Ê×ÏȽéÉÜʵʱÊý²ÖµÄ¸ÅÄȻºó½éÉÜ Flink+TiDB ʵʱÊý²ÖµÄ¼Ü¹¹ÓëÓÅÊÆ£¬½Óןø³öһЩÒѾÔÚʹÓÃÖеÄÓû§³¡¾°£¬×îºó¸ø³öÔÚ
docker-compose »·¾³Ï嵀 Demo£¬ÓÃÓÚ¶ÁÕß½øÐг¢ÊÔ¡£
ʵʱÊý²ÖµÄ¸ÅÄî
Êý¾Ý²Ö¿âµÄ¸ÅÄîÔÚ 90 Äê´úÓÉ Bill Inmon Ìá³ö£¬ÊÇÖ¸Ò»¸öÃæÏòÖ÷ÌâµÄ¡¢¼¯³ÉµÄ¡¢Ïà¶ÔÎȶ¨µÄ¡¢·´Ó³ÀúÊ·±ä»¯µÄ¼¯ºÏ£¬ÓÃÓÚÖ§³Ö¹ÜÀí¾ö²ß¡£µ±Ê±µÄÊý¾Ý²Ö¿âͨ¹ýÏûÏ¢¶ÓÁÐÊÕ¼¯À´×ÔÊý¾ÝÔ´µÄÊý¾Ý£¬Í¨¹ýÿÌì»òÿÖܽøÐÐÒ»´Î¼ÆËãÒÔ¹©±¨±íʹÓã¬Ò²³ÆÎªÀëÏßÊý²Ö¡£

ÀëÏßÊý²Ö¼Ü¹¹
½øÈë 21 ÊÀ¼Í£¬Ëæ×żÆËã¼¼ÊõµÄ·¢Õ¹¡¢ÒÔ¼°ÕûÌåËãÁ¦µÄÌáÉý£¬¾ö²ßµÄÖ÷ÌåÖð½¥´ÓÈ˹¤¿ØÖÆ×ª±äΪ¼ÆËã»úËã·¨£¬³öÏÖÁËÊµÊ±ÍÆ¼ö¡¢ÊµÊ±¼à¿Ø·ÖÎöµÈÐèÇ󣬶ÔÓ¦µÄ¾ö²ßÖÜÆÚʱ¼äÓÉÌì¼¶Öð²½±äΪÃë¼¶£¬ÔÚÕâЩ³¡¾°Ï£¬ÊµÊ±Êý²ÖÓ¦Ô˶øÉú¡£
µ±Ç°µÄʵʱÊý²ÖÖ÷ÒªÓÐÈýÖּܹ¹£ºLambda¼Ü¹¹¡¢Kappa ¼Ü¹¹ÒÔ¼°ÊµÊ± OLAP ±äÌå¼Ü¹¹£º
Lambda ¼Ü¹¹ÊÇÖ¸ÔÚÀëÏßÊý²ÖµÄ»ù´¡Éϵþ¼ÓÁËʵʱÊý²Ö²¿·Ö£¬Ê¹ÓÃÁ÷ʽÒýÇæ´¦ÀíʵʱÐԽϸߵÄÊý¾Ý£¬×îºó½«ÀëÏߺÍÔÚÏߵĽá¹ûͳһ¹©Ó¦ÓÃʹÓá£

ʵʱÊý²ÖµÄ Lambda ¼Ü¹¹
Kappa ¼Ü¹¹ÔòÒÆ³ýÁËÀëÏßÊý²Ö²¿·Ö£¬È«²¿Ê¹ÓÃʵʱÊý¾ÝÉú²ú¡£ÕâÖּܹ¹Í³Ò»Á˼ÆËãÒýÇæ£¬½µµÍÁË¿ª·¢³É±¾¡£

ʵʱÊý²ÖµÄ Kappa ¼Ü¹¹
Ëæ×Åʵʱ OLAP ¼¼ÊõµÄÌáÉý£¬Ò»¸öеÄʵʱ¼Ü¹¹±»Ìá³ö£¬ÔÝʱ±»³ÆÎª¡°ÊµÊ± OLAP ±äÌ塱¡£¼òµ¥À´Ëµ£¬¾ÍÊǽ«Ò»²¿·Ö¼ÆËãѹÁ¦´ÓÁ÷ʽ¼ÆËãÒýÇæ×ª¼Þµ½ÊµÊ±
OLAP ·ÖÎöÒýÇæÉÏ£¬ÒԴ˽øÐиü¼ÓÁé»îµÄʵʱÊý²Ö¼ÆËã¡£

×ܽáһϣ¬¶ÔÓÚʵʱÊý²Ö£¬Lambda ¼Ü¹¹ÐèҪά»¤Á÷ÅúÁ½Ì×ÒýÇæ£¬¿ª·¢³É±¾Ïà½ÏÆäËüÁ½Õ߸ü¸ß¡£Ïà±ÈÓÚ
Kappa ¼Ü¹¹£¬ÊµÊ± OLAP ±äÌå¼Ü¹¹¿ÉÒÔÖ´Ðиü¼ÓÁé»îµÄ¼ÆË㣬µ«ÐèÒªÒÀÀµ¶îÍâµÄʵʱ OLAP ËãÁ¦×ÊÔ´¡£½ÓÏÂÀ´ÎÒÃǽ«½éÉܵÄ
Flink + TiDB ʵʱÊý²Ö·½°¸£¬¾ÍÊôÓÚʵʱ OLAP ±äÌå¼Ü¹¹¡£
¹ØÓÚʵʱÊý²Ö¼°ÕâЩ¼Ü¹¹¸ü¼ÓÏêϸµÄ¶Ô±È˵Ã÷£¬ÓÐÐËȤµÄ¶ÁÕß¿ÉÒԲο¼ Flink ÖÐÎÄÉçÇøµÄÕâÆªÎÄÕ£º»ùÓÚ
Flink µÄµäÐÍ ETL ³¡¾°ÊµÏÖ·½°¸¡£
Flink+ TiDB ʵʱÊý²Ö
Flink ÊÇÒ»¸öµÍÑÓ³Ù¡¢¸ßÍÌÍ¡¢Á÷ÅúͳһµÄ´óÊý¾Ý¼ÆËãÒýÇæ£¬±»ÆÕ±éÓÃÓÚ¸ßʵʱÐÔ³¡¾°ÏµÄʵʱ¼ÆË㣬¾ßÓÐÖ§³Ö
exactly-once µÈÖØÒªÌØÐÔ¡£
ÔÚ¼¯³ÉÁË TiFlash Ö®ºó£¬TiDB ÒѾ³ÉΪÁËÕæÕýµÄ HTAP£¨ÔÚÏßÊÂÎñ´¦Àí OLTP + ÔÚÏß·ÖÎö´¦Àí
OLAP£©Êý¾Ý¿â¡£»»¾ä»°Ëµ£¬ÔÚʵʱÊý²Ö¼Ü¹¹ÖУ¬TiDB ¼È¿ÉÒÔ×÷ΪÊý¾ÝÔ´µÄÒµÎñÊý¾Ý¿â£¬½øÐÐÒµÎñ²éѯµÄ´¦Àí£»ÓÖ¿ÉÒÔ×÷Ϊʵʱ
OLAP ÒýÇæ£¬½øÐзÖÎöÐͳ¡¾°µÄ¼ÆËã¡£
½áºÏÁË Flink Óë TiDB Á½ÕßµÄÌØÐÔ£¬Flink+ TiDB µÄ·½°¸µÄÓÅÊÆÒ²ÌåÏÖÁ˳öÀ´£ºÊ×ÏÈÊÇËÙ¶ÈÓб£ÕÏ£¬Á½Õß¶¼¿ÉÒÔͨ¹ýˮƽÀ©Õ¹½ÚµãÀ´Ôö¼ÓËãÁ¦£»Æä´Î£¬Ñ§Ï°ºÍÅäÖóɱ¾Ïà¶Ô½ÏµÍ£¬ÒòΪ
TiDB ¼æÈÝ MySQL 5.7 ÐÒ飬¶ø×îа汾µÄ Flink Ò²¿ÉÒÔÍêȫͨ¹ý Flink SQL
ºÍÇ¿´óµÄÁ¬½ÓÆ÷£¨connector£©À´±àдÌá½»ÈÎÎñ£¬½ÚÊ¡ÁËÓû§µÄѧϰ³É±¾¡£
¶ÔÓÚ Flink + TiDB ʵʱÊý²Ö£¬ÏÂÃæÊǼ¸ÖÖ³£ÓõĴÔÐÍ£¬¿ÉÒÔÓÃÀ´Âú×㲻ͬµÄÐèÇó£¬Ò²¿ÉÒÔÔÚʵ¼ÊʹÓÃÖÐ×ÔÐÐÀ©Õ¹¡£
ÒÔ MySQL ×÷ΪÊý¾ÝÔ´
ͨ¹ýʹÓà Ververica ¹Ù·½ÌṩµÄ flink-connector-mysql-cdc[1]£¬Flink
¿ÉÒÔ¼È×÷Ϊ²É¼¯²ã²É¼¯ MySQL µÄ binlog Éú³É¶¯Ì¬±í£¬Ò²×÷ΪÁ÷¼ÆËã²ãʵÏÖÁ÷ʽ¼ÆË㣬ÈçÁ÷ʽ
Join¡¢Ô¤¾ÛºÏµÈ¡£×îºó£¬Flink ͨ¹ý JDBC Á¬½ÓÆ÷½«¼ÆËãÍê³ÉµÄÊý¾ÝдÈë TiDB ÖС£

ÒÔ MySQL ×÷ΪÊý¾ÝÔ´µÄ¼ò±ã¼Ü¹¹
Õâ¸ö¼Ü¹¹µÄÓŵãÊǷdz£¼ò½à·½±ã£¬ÔÚ MySQL ºÍ TiDB ¶¼×¼±¸ºÃ¶ÔÓ¦Êý¾Ý¿âºÍ±íµÄÇé¿öÏ£¬¿ÉÒÔͨ¹ýÖ»±àд
Flink SQL À´Íê³ÉÈÎÎñµÄ×¢²áÓëÌá½»¡£¶ÁÕß¿ÉÒÔÔÚ±¾ÎÄĩβµÄ¡¾ÔÚdocker-compose ÖнøÐг¢ÊÔ¡¿Ò»½ÚÖг¢ÊԴ˼ܹ¹¡£
ÒÔ Kafka ¶Ô½Ó Flink
Èç¹ûÊý¾ÝÒѾ´ÓÆäËü;¾¶´æ·Åµ½ÁËKafka ÖУ¬¿ÉÒÔ·½±ãµØÍ¨¹ý Flink Kafka Connector[2]
ʹ Flink ´Ó Kafka ÖлñµÃÊý¾Ý¡£
ÔÚÕâÀïÐèÒªÌáһϵÄÊÇ£¬Èç¹ûÏëÒª½« MySQL »òÆäËüÊý¾ÝÔ´µÄ±ä¸üÈÕÖ¾´æ·ÅÔÚ Kafka ÖкóÐø¹© Flink
´¦Àí£¬ÄÇÃ´ÍÆ¼öʹÓà Canal »ò Debezium ²É¼¯Êý¾ÝÔ´±ä¸üÈÕÖ¾£¬ÒòΪ Flink 1.11
ÔÉúÖ§³Ö½âÎöÕâÁ½ÖÖ¹¤¾ß¸ñʽµÄ changelog£¬ÎÞÐèÔÙ¶îÍâʵÏÖ½âÎöÆ÷¡£

ÒÔ TiDB ×÷ΪÊý¾ÝÔ´
TiCDC[3] ÊÇÒ»¿îͨ¹ýÀÈ¡ TiKV ±ä¸üÈÕ־ʵÏÖµÄ TiDB ÔöÁ¿Êý¾Ýͬ²½¹¤¾ß£¬¿ÉÒÔÀûÓÃÆä½«
TiDB µÄ±ä¸üÊý¾ÝÊä³öµ½ÏûÏ¢¶ÓÁÐÖУ¬ÔÙÓÉ Flink ÌáÈ¡¡£
ÔÚ 4.0.7 °æ±¾£¬¿ÉÒÔͨ¹ý TiCDC Open Protocol[4] À´Íê³ÉÓë Flink
µÄ¶Ô½Ó¡£ÔÚÖ®ºóµÄ°æ±¾£¬TiCDC ½«Ö§³ÖÖ±½ÓÊä³öΪ canal-json ÐÎʽ£¬ÒÔ¹© Flink ʹÓá£
°¸ÀýÓëʵ¼ù
Éϸö²¿·Ö½éÉÜÁËһЩ»ù´¡µÄ¼Ü¹¹£¬Êµ¼ùÖеÄ̽Ë÷ÍùÍù¸ü¼Ó¸´ÔÓºÍÓÐȤ£¬ÕâÒ»²¿·Ö½«½éÉÜһЩ¾ßÓдú±íÐÔºÍÆô·¢ÐÔµÄÓû§°¸Àý¡£
СºìÊé
СºìÊéÊÇÄêÇáÈ˵ÄÉú»î·½Ê½Æ½Ì¨£¬Óû§¿ÉÒÔͨ¹ý¶ÌÊÓÆµ¡¢Í¼ÎĵÈÐÎʽ¼Ç¼Éú»îµãµÎ£¬·ÖÏíÉú»î·½Ê½£¬²¢»ùÓÚÐËȤÐγɻ¥¶¯¡£½ØÖÁµ½
2019 Äê 10 Ô£¬Ð¡ºìÊéÔ»îÔ¾Óû§ÊýÒѾ¹ýÒÚ£¬²¢³ÖÐø¿ìËÙÔö³¤¡£
ÔÚСºìÊéµÄÒµÎñ¼Ü¹¹ÖУ¬Flink µÄÊý¾ÝÀ´Ô´ºÍÊý¾Ý»ã×Ü´¦¶¼ÊÇ TiDB£¬ÒÔ´ïµ½ÀàËÆÓÚ¡°ÎﻯÊÓͼ¡±µÄЧ¹û£º
×óÉϽǵÄÏßÉÏÒµÎñ±íÖ´ÐÐÕý³£µÄ OLTP ÈÎÎñ¡£
Ï·½µÄ TiCDC ¼¯Èº³éÈ¡ TiDB µÄʵʱ±ä¸üÊý¾Ý£¬ÒÔ changelog ÐÎʽ´«µÝµ½ Kafka
ÖС£
Flink ¶ÁÈ¡ Kafka ÖÐµÄ changelog£¬½øÐмÆË㣬ÈçÆ´ºÃ¿í±í»ò¾ÛºÏ±í¡£
Flink ½«½á¹ûд»Øµ½ TiDB µÄ¿í±íÖУ¬ÓÃÓÚºóÐø·ÖÎöʹÓá£

Õû¸ö¹ý³ÌÐγÉÁË TiDB µÄ±Õ»·£¬½«ºóÐø·ÖÎöÈÎÎñµÄ Join ¹¤×÷×ªÒÆµ½ÁË Flink ÉÏ£¬²¢Í¨¹ýÁ÷ʽ¼ÆËãÀ´»º½âѹÁ¦¡£Ä¿Ç°ÕâÌ×·½°¸ÒѾ֧³ÖÆðÁËСºìÊéµÄÄÚÈÝÉóºË¡¢±Ê¼Ç±êÇ©ÍÆ¼ö¡¢Ôö³¤É󼯵ÈÒµÎñ£¬¾ÀúÁË´óÍÌÍÂÁ¿µÄÏßÉÏÒµÎñ¿¼ÑéÇÒ³ÖÐøÔËÐÐÎȶ¨¡£
±´¿Ç½ð·þ
±´¿Ç½ð·þ³ÖÐø¶àÄêÉî¸û¾Óס³¡¾°£¬»ýÀÛÁ˷ḻµÄÖйú·¿²ú´óÊý¾Ý¡£±´¿Ç½ð·þÒÔ½ðÈڿƼ¼ÎªÇý¶¯£¬ÀûÓà AI Ëã·¨¸ßЧӦÓöàάº£Á¿Êý¾ÝÒÔÌáÉý²úÆ·ÌåÑ飬ΪÓû§Ìṩ·á¸»¡¢¶¨ÖÆ»¯µÄ½ðÈÚ·þÎñ¡£
ÔÚ±´¿ÇÊý¾Ý×éµÄÊý¾Ý·þÎñÖУ¬Flink ʵʱ¼ÆËãÓÃÓÚµäÐ͵Äά±í Join£º
Ê×ÏÈ£¬Ê¹Óà Syncer £¨MySQL µ½ TiDB µÄÒ»¸öÇáÁ¿¼¶Í¬²½¹¤¾ß£©²É¼¯ÒµÎñÊý¾ÝÔ´ÉϵÄά±íÊý¾Ýͬ²½µ½
TiDB ÖС£
È»ºó£¬ÒµÎñÊý¾ÝÔ´ÉϵÄÁ÷±íÊý¾ÝÔòͨ¹ý Canal ²É¼¯ binlog ´æÈë kafka ÏûÏ¢¶ÓÁÐÖС£
Flink ¶ÁÈ¡ Kafka ÖÐÁ÷±íµÄ±ä¸üÈÕÖ¾£¬³¢ÊÔ½øÐÐÁ÷ʽ Join£¬Ã¿µ±ÐèҪά±íÖеÄÊý¾Ýʱ£¬¾ÍÈ¥
TiDB ÖвéÕÒ¡£
×îºó£¬Flink ½«Æ´ºÏ¶ø³ÉµÄ¿í±íдÈëµ½ TiDB ÖУ¬ÓÃÓÚÊý¾Ý·ÖÎö·þÎñ¡£

ÀûÓÃÒÔÉϵĽṹ£¬¿ÉÒÔ½«Êý¾Ý·þÎñÖеÄÖ÷±í½øÐÐʵʱ Join Â䵨£¬È»ºó·þÎñ·½Ö»ÐèÒª²éѯµ¥±í¡£ÕâÌ×ϵͳÔÚ±´¿Ç½ð·þÒѾÉîÈë¸÷¸öºËÐÄÒµÎñϵͳ£¬¿çϵͳµÄÊý¾Ý»ñȡͳһ×ßÊý¾Ý×éµÄÊý¾Ý·þÎñ£¬Ê¡È¥ÁËÒµÎñϵͳ¿ª·¢
API ºÍÄÚ´æ¾ÛºÏÊý¾Ý´úÂëµÄ¿ª·¢¹¤×÷¡£
ÖÇ»ÛÑ¿
PatSnap£¨ÖÇ»ÛÑ¿£©ÊÇÒ»¿îÈ«ÇòרÀû¼ìË÷Êý¾Ý¿â£¬ÕûºÏÁË 1790 ÄêÖÁ½ñµÄÈ«Çò 116 ¸ö¹ú¼ÒµØÇø
1.3 ÒÚרÀûÊý¾ÝºÍ 1.7 ÒÚ»¯Ñ§½á¹¹Êý¾Ý¡£¿É¼ìË÷¡¢ä¯ÀÀ¡¢·ÒëרÀû£¬Éú³É Insights רÀû·ÖÎö±¨¸æ£¬ÓÃÓÚרÀû¼ÛÖµ·ÖÎö¡¢ÒýÓ÷ÖÎö¡¢·¨ÂÉËÑË÷£¬²é¿´
3D רÀûµØÍ¼¡£
ÖÇ»ÛѿʹÓà Flink + TiDB Ìæ»»ÁËÔÓÐµÄ Segment + Redshift ¼Ü¹¹¡£
ÔÓÐµÄ Segment + Redshift ¼Ü¹¹£¬½ö¹¹½¨³öÁË ODS ²ã£¬Êý¾ÝдÈëµÄ¹æÔòºÍ schema
²»ÊÜ¿ØÖÆ¡£ÇÒÐèÒªÕë¶Ô ODS ±àд¸´Ô ETL À´°´ÕÕÒµÎñÐèÇó½øÐи÷ÀàÖ¸±êµÄ¼ÆËãÀ´Íê³ÉÉϲãÐèÇó¡£Redshift
ÖÐÂä¿âÊý¾ÝÁ¿´ó£¬¼ÆËãÂý£¨T+1 ʱЧ£©£¬²¢Ó°Ïì¶ÔÍâ·þÎñÐÔÄÜ¡£
Ìæ»»Îª»ùÓÚ Kinesis +Flink + TiDB ¹¹½¨µÄʵʱÊý²Ö¼Ü¹¹ºó£¬²»ÔÙÐèÒª¹¹½¨ ODS
²ã¡£Flink ×÷ΪǰÖüÆËãµ¥Ôª£¬Ö±½Ó´ÓÒµÎñ³ö·¢¹¹½¨³ö Flink Job ETL£¬ÍêÈ«¿ØÖÆÁËÂä¿â¹æÔò²¢×Ô¶¨Òå
schema£»¼´½ö°ÑÒµÎñ¹Ø×¢µÄÖ¸±ê½øÐÐÇåÏ´²¢Ð´Èë TiDB À´½øÐкóÐøµÄ·ÖÎö²éѯ£¬Ð´ÈëÊý¾ÝÁ¿´ó´ó¼õÉÙ¡£°´Óû§/×â»§¡¢µØÇø¡¢ÒµÎñ¶¯×÷µÈ¹Ø×¢µÄÖ¸±ê£¬½áºÏ·ÖÖÓ¡¢Ð¡Ê±¡¢ÌìµÈ²»Í¬Á£¶ÈµÄʱ¼ä´°¿ÚµÈ£¬ÔÚ
TiDB ÉϹ¹½¨³ö DWD/DWS/ADS ²ã£¬Ö±½Ó·þÎñÒµÎñÉϵÄͳ¼Æ¡¢Çåµ¥µÈÐèÇó£¬ÉϲãÓ¦ÓÿÉÖ±½ÓʹÓù¹½¨ºÃµÄÊý¾Ý£¬ÇÒ»ñµÃÁËÃë¼¶µÄʵʱÄÜÁ¦¡£

Óû§ÌåÑ飺ÔÚʹÓÃÁËмܹ¹ºó£¬Èë¿âÊý¾ÝÁ¿¡¢Èë¿â¹æÔòºÍ¼ÆË㸴ÔӶȶ¼´ó´óϽµ£¬Êý¾ÝÔÚ Flink Job
ÖÐÒѾ°´ÕÕÒµÎñÐèÇó´¦ÀíÍê³É²¢Ð´Èë TiDB£¬²»ÔÙÐèÒª»ùÓÚ Redshift µÄ È«Á¿ ODS ²ã½øÐÐ
T+1 ETL¡£»ùÓÚ TiDB ¹¹½¨µÄʵʱÊý²Ö£¬Í¨¹ýºÏÀíµÄÊý¾Ý·Ö²ã£¬¼Ü¹¹ÉÏ»ñµÃÁ˼«´óµÄ¾«¼ò£¬¿ª·¢Î¬»¤Ò²±äµÃ¸ü¼Ó¼òµ¥£»ÔÚÊý¾Ý²éѯ¡¢¸üС¢Ð´ÈëÐÔÄÜÉ϶¼»ñµÃ´ó·ù¶ÈÌáÉý£»ÔÚÂú×㲻ͬµÄadhoc
·ÖÎöÐèÇóʱ£¬²»ÔÙÐèÒªµÈ´ýÀàËÆ Redshift Ô¤±àÒëµÄ¹ý³Ì£»À©ÈÝ·½±ã¼òµ¥Ò×ÓÚ¿ª·¢¡£
ĿǰÕâÌ׼ܹ¹ÕýÔÚÉÏÏߣ¬ÔÚÖÇ»ÛÑ¿ÄÚ²¿ÓÃÀ´½øÐÐÓû§ÐÐΪ·ÖÎöºÍ×·×Ù£¬²¢»ã×ܳö¹«Ë¾ÔËÓª´óÅÌ¡¢Óû§ÐÐΪ·ÖÎö¡¢×â»§ÐÐΪ·ÖÎöµÈ¹¦ÄÜ¡£
ÍøÒ×»¥Óé
ÍøÒ× 2001 ÄêÕýʽ³ÉÁ¢ÔÚÏßÓÎÏ·ÊÂÒµ²¿£¬¾¹ý½ü 20 ÄêµÄ·¢Õ¹£¬ÒÑõÒÉíÈ«ÇòÆß´óÓÎÏ·¹«Ë¾Ö®Ò»¡£ÔÚ
App Annie ·¢²¼µÄ¡°2020 Äê¶ÈÈ«Çò·¢ÐÐÉÌ 52 Ç¿¡±°ñµ¥ÖУ¬ÍøÒ×λÁеڶþ¡£

ÔÚÍøÒ×»¥Ó鼯·Ñ×éµÄÓ¦Óüܹ¹ÖУ¬Ò»·½ÃæÊ¹Óà Flink Íê³ÉÒµÎñÊý¾ÝÔ´µ½ TiDB µÄʵʱдÈ룻ÁíÒ»·½Ã棬ÒÔ
TiDB ×÷Ϊ·ÖÎöÊý¾ÝÔ´£¬ÔÚºóÐøµÄ Flink ¼¯ÈºÖнøÐÐʵʱÁ÷¼ÆË㣬Éú³É·ÖÎö±¨±í¡£´ËÍâ£¬ÍøÒ×»¥ÓéÏÖÔÚÄÚ²¿¿ª·¢ÁË
Flink ×÷Òµ¹ÜÀíÆ½Ì¨£¬ÓÃÓÚ¹ÜÀí×÷ÒµµÄÕû¸öÉúÃüÖÜÆÚ¡£
Öªºõ
ÖªºõÊÇÖÐÎÄ»¥ÁªÍø×ÛºÏÐÔÄÚÈÝÆ½Ì¨£¬ÒÔ¡°ÈÃÿ¸öÈ˸ßЧ»ñµÃ¿ÉÐÅÀµµÄ½â´ð¡±ÎªÆ·ÅÆÊ¹ÃüºÍ±±¼«ÐÇ¡£½ØÖÁ 2019
Äê 1 Ô£¬ÖªºõÒÑÓµÓг¬¹ý 2.2 ÒÚÓû§£¬¹²²ú³ö 1.3 ÒÚ¸ö»Ø´ð¡£
Öªºõ×÷Ϊ PingCAP µÄºÏ×÷»ï°é£¬Í¬Ê±Ò²ÊÇ Flink µÄÉî¶ÈÓû§£¬ÔÚ×Ô¼ºµÄʵ¼ù¹ý³ÌÖпª·¢ÁËÒ»Ì×
TiDB Óë Flink ½»»¥¹¤¾ß²¢¹±Ï׸øÁË¿ªÔ´ÉçÇø£ºpingcap-incubator/TiBigData[5]£¬Ö÷Òª°üÀ¨ÁËÈçϹ¦ÄÜ£º
TiDB ×÷Ϊ Flink Source Connector£¬ÓÃÓÚÅúʽͬ²½Êý¾Ý¡£
TiDB ×÷Ϊ Flink Sink Connector£¬»ùÓÚ JDBC ʵÏÖ¡£
Flink TiDB Catalog£¬¿ÉÒÔÔÚ Flink SQL ÖÐÖ±½ÓʹÓà TiDB µÄ±í£¬ÎÞÐèÔٴδ´½¨¡£
ÔÚ docker-compose ÖнøÐг¢ÊÔ
ΪÁË·½±ã¶ÁÕ߸üºÃµÄÀí½â£¬ÎÒÃÇÔÚ ht tps://github.com/LittleFall/flink-tidb-rdw
ÖÐÌṩÁËÒ»¸ö»ùÓÚ docker-compose µÄ MySQL-Flink-TiDB ²âÊÔ»·¾³£¬¹©´ó¼Ò²âÊÔʹÓá£
Flink TiDB ʵʱÊý²Ö Slides[6] ÖÐÌṩÁ˸ó¡¾°ÏÂÒ»¸ö¼òµ¥µÄ½Ì³Ì£¬°üÀ¨¸ÅÄî½âÊÍ¡¢´úÂëʾÀý¡¢¼òµ¥ÔÀíÒÔ¼°Ò»Ð©×¢ÒâÊÂÏÆäÖÐʾÀý°üÀ¨£º
Flink SQL ¼òµ¥³¢ÊÔ
ÀûÓà Flink ½øÐÐ´Ó MySQL µ½ TiDB µÄÊý¾Ýµ¼Èë
Ë«Á÷ Join
ά±í Join
ÔÚÆô¶¯ docker-compose ºó£¬¿ÉÒÔͨ¹ý Flink SQL Client À´±àд²¢Ìá½»
Flink ÈÎÎñ£¬²¢Í¨¹ý localhost:8081 À´¹Û²ìÈÎÎñÖ´ÐÐÇé¿ö¡£
|