Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Model Center   Code  
»áÔ±   
   
 
     
   
 ¶©ÔÄ
  ¾èÖú
TiDB / TiSpark ÔÚÒ×¹û¼¯ÍÅʵʱÊý²ÖÖеĴ´ÐÂʵ¼ù
 
×÷Õߣº ÂÞÈðÐÇ
  2252  次浏览      29
 2020-7-23 
 
±à¼­ÍƼö:
±¾ÎÄͨ¹ý Ò»¸öÏîÄ¿½éÉÜÁËTiDB ´Ó»ù´¡ÔÚʵʱÊý²ÖÖеÄÓ¦Óã¬Ï£Íû±¾ÎĶԴó¼ÒÓаïÖú¡£
±¾ÎÄÀ´×ÔÖªºõ£¬ÓÉ»ðÁú¹ûÈí¼þAnna±à¼­¡¢ÍƼö¡£

ÏîÄ¿±³¾°

ĿǰÆóÒµ´ó¶àÊýµÄÊý¾Ý·ÖÎö³¡¾°µÄ½â¾ö·½°¸µ×²ã¶¼ÊÇÎ§ÈÆ Hadoop ´óÊý¾ÝÉú̬չ¿ªµÄ£¬³£¼ûµÄÈç HDFS + Hive + Spark + Presto + Kylin£¬ÔÚÒ×¹û¼¯ÍÅ£¬ÎÒÃdzõÆÚÒ²ÊDzÉÈ¡ÕâÖÖ˼·£¬µ«ÊÇËæ×ÅÒµÎñ¹æÄ£µÄ¿ìËÙÔö³¤ºÍÐèÇóµÄ²»¶Ï±ä»¯£¬Ò»Ð©ÊµÊ±»òÕß׼ʵʱµÄÐèÇó±äµÃÔ½À´Ô½¶à£¬ÕâÀàÒµÎñ³ýÁËÓÐʵʱµÄ OLTP ÐèÇ󣬻¹°éËæ×ÅһЩÓÐÒ»¶¨¸´ÔÓ¶ÈµÄ OLAP µÄÐèÇ󣬵¥´¿µØÊ¹Óà Hadoop ÒѾ­ÎÞ·¨Âú×ãÐèÇó¡£

ÏÖÓеÄ׼ʵʱϵͳÔËÐÐÔÚ SQL Server Ö®ÉÏ£¬Í¨¹ý¿ª·¢ÈËÔ±±àдºÍά»¤ÏàÓ¦µÄ´æ´¢¹ý³ÌÀ´ÊµÏÖ¡£ÓÉÓÚÊý¾ÝÁ¿²»´ó£¬SQL Server Äܹ»Âú×ãÐèÇ󣬵«ÊÇËæ×ÅÒµÎñµÄ·¢Õ¹£¬Êý¾ÝÁ¿ËæÖ®Ôö³¤£¬SQL Server Ô½À´Ô½²»ÄÜÂú×ãÐèÇ󣬵±Êý¾ÝÁ¿µ½´ïÒ»¶¨µÄ½×¶Î£¬ÐÔÄܱã»á³öÏֹյ㡣Õâ¸öʱºò£¬ÕâÌ×·½°¸ÒÑÍêÈ«ÎÞ·¨Ö§³ÅÒµÎñ£¬²»µÃ²»ÖØÐÂÉè¼ÆÐµķ½°¸¡£

Ñ¡ÐÍÆÀ¹À

ÔÚÆÀ¹À³õÆÚ£¬Greenplum¡¢Kudu¡¢TiDB ¶¼½øÈëÁËÎÒÃǵÄÊÓÒ°£¬¶ÔÓÚеÄʵʱϵͳ£¬ÎÒÃÇÓÐÖ÷Òª¿¼Âǵ㣺

Ê×ÏÈ£¬ÏµÍ³¼ÈÒªÂú×ã OLAP »¹ÒªÂú×ã OLTP µÄ»ù±¾ÐèÇó£»

Æä´Î£¬ÐÂϵͳҪ¾¡Á¿½µµÍÒµÎñµÄʹÓÃÒªÇó£»

×îºó£¬ÐÂϵͳ×îºÃÄܹ»ÓëÏÖÓÐµÄ Hadoop ÌåϵÏà½áºÏ¡£

Greenplum ÊÇÒ»Ì×»ùÓÚ PostgreSQL ·ÖÎöΪÖ÷µÄ MPP ÒýÇæ£¬´ó¶àÓÃÔÚ²¢·¢¶È²»¸ßµÄÀëÏß·ÖÎö³¡¾°£¬µ«ÔÚ OLTP ·½Ã棬ÎÒÃǵijõ²½²âÊÔ·¢ÏÖÆä¶Ô±È TiDB µÄÐÔÄܲîºÜ¶à¡£

ÔÙ˵˵ Kudu¡£Kudu ÊÇ CDH 2015Äê·¢²¼µÄÒ»Ì×½éÓÚ Hbase ºÍ HDFS ÖмäµÄÒ»Ì״洢ϵͳ£¬Ä¿Ç°ÔÚ¹úÄÚÖ÷ÒªÊÇСÃ×¹«Ë¾Ó¦ÓõĽ϶࣬ÔÚ²âÊÔÖУ¬ÎÒÃÇ·¢ÏÖÆäÔÚ OLTP ±íÏÖ´óÖÂÓë TiDB Ï൱£¬µ«ÊÇһЩÖеÈÊý¾ÝÁ¿Ï£¬Æä·ÖÎöÐÔÄÜÏà±È TiDB ÓÐÒ»¶¨²î¾à¡£ÁíÍâÎÒÃǵIJéѯĿǰÖ÷ÒªÒÔ Presto ΪÖ÷£¬Presto ¶Ô½Ó Kudu ºÍ PostgreSQL ¶¼ÊÇÐèÒª¿¼ÂǼæÈÝÐÔµÄÎÊÌ⣬¶ø TiDB ¼æÈÝ MySQL ЭÒ飬ÔÚÓ¦ÓóõÆÚ¿ÉÒÔÖ±½ÓʹÓà Presto-MySQL ½øÐÐͳһ²éѯ£¬ÏÂÒ»²½ÔÙ¿¼ÂÇרÃÅ¿ª·¢ Presto-TiDB¡£

ÁíÍ⣬ÎÒÃÇÏ£ÍûδÀ´µÄʵʱϵͳºÍÀëÏßϵͳÄܹ»Í¨Óã¬Ò»Ì×´úÂëÔÚÁ½¸öϵͳÖж¼Äܹ»ÍêÈ«¼æÈÝ£¬Ä¿Ç° Tispark ºÍ SparkSQL ÒѾ­ºÜ´ó³Ì¶ÈÉÏʵÏÖÁËÕâµã£¬ÕâÖ§³ÖÎÒÃÇÔÚÒÔºóÀëÏßÉϵÄСʱ¼¶ÈÎÎñ¿ÉÒÔÖ±½ÓÇл»µ½ TiDBÉÏ£¬ÔÚ TiDB ÉÏʵÏÖʵʱҵÎñµÄͬʱ£¬Èç¹ûÓÐ T+1 µÄÐèÇóÒ²Äܹ»Ö±½ÓÖ¸ HDFS ¼´¿É£¬²»Óöþ´Î¿ª·¢£¬ÕâÊÇ Kudu ºÍ GP ÔÝʱʵÏÖ²»Á˵ġ£

×îºó£¬TiSpark Êǽ¨Á¢ÔÚ Spark ÒýÇæÖ®ÉÏ£¬Spark ÔÚ»úÆ÷ѧϰÁìÓòÀïÓÐÖîÈç Mllib µÈÖî¶à³ÉÊìµÄÏîÄ¿£¬¶Ô±È GP ºÍ Kudu£¬Ëã·¨¹¤³ÌʦÃÇʹÓà TiSpark È¥²Ù×÷ TiDB µÄÃż÷·Ç³£µÍ£¬Í¬Ê±Ò²»á´ó´óÌáÉýËã·¨¹¤³ÌʦÃǵÄЧÂÊ¡£

¾­¹ý×ۺϵĿ¼ÂÇ£¬ÎÒÃÇ×îÖÕ¾ö¶¨Ê¹Óà TiDB ×÷ΪеÄʵʱϵͳ¡£Í¬Ê±£¬Ä¿Ç° TiDB µÄÉçÇø»îÔ¾¶È·Ç³£ºÃ£¬ÕâÒ²ÊÇÎÒÃÇ¿¼ÂǵÄÒ»¸öºÜÖØÒªµÄ·½Ãæ¡£

TiDB ¼ò½é

ÔÚÕâÀï½éÉÜһϠTiDB µÄÏà¹ØÌØÐÔ£ºTiDB ÊÇ»ùÓÚ Google Spanner/F1 ÂÛÎÄÆô·¢¿ªÔ´µÄÒ»Ì× NewSQL Êý¾Ý¿â£¬Ëü¾ß±¸ÈçÏ NewSQL ºËÐÄÌØÐÔ£º

SQLÖ§³Ö £¨TiDB ÊÇ MySQL ¼æÈݵģ©

ˮƽÏßÐÔµ¯ÐÔÀ©Õ¹

·Ö²¼Ê½ÊÂÎñ

Êý¾ÝǿһÖÂÐÔ±£Ö¤

¹ÊÕÏ×Ô»Ö¸´µÄ¸ß¿ÉÓÃ

ͬʱ£¬TiDB »¹ÓÐÒ»Ì׷ḻµÄÉú̬¹¤¾ß£¬ÀýÈ磺¿ìËÙ²¿ÊðµÄ TiDB-Ansible¡¢ÎÞ·ìÇ¨ÒÆ MySQL µÄ Syncer¡¢Òì¹¹Êý¾ÝÇ¨ÒÆ¹¤¾ß Wormhole¡¢ÒÔ¼° TiDB-Binlog¡¢Backup & Recovery µÈ¡£

SQL Server Ç¨ÒÆµ½ TiDB

ÓÉÓÚÎÒÃǹ«Ë¾µÄ¼Ü¹¹ÊÇ .NET + SQL Server ¼Ü¹¹£¬ËùÒÔÎÒÃÇÎÞ·¨Ïñ´ó¶àÊý¹«Ë¾Ò»ÑùȥʹÓà MySQL Binlog È¥×öÊý¾Ýͬ²½£¬µ±È»Ò²¾ÍÎÞ·¨Ê¹Óà TiDB ¹Ù·½ÌṩµÄ Syncer ¹¤¾ßÁË¡£Òò´ËÎÒÃDzÉÓÃÁË Flume + Kafka µÄ¼Ü¹¹£¬ÎÒÃÇ×Ô¼º¿ª·¢ÁË»ùÓÚ Flume µÄ SQL Server Source ȥʵʱ¼à¿Ø SQL Server Êý¾Ý±ä»¯£¬½øÐв¶×½²¢Ð´Èë Kafka ÖУ¬Í¬Ê±£¬ÎÒÃÇʹÓà Spark Streaming È¥¶ÁÈ¡ Kafka ÖеÄÊý¾Ý²¢Ð´Èë TiDB£¬Í¬Ê±ÎÒÃǽ«Ö®Ç° SQL Server µÄ´æ´¢¹ý³Ì¸ÄÔì³É¶¨Ê±µ÷¶ÈµÄ MySQL ½Å±¾¡£

ͼ£ºSQL Server Êý¾ÝÇ¨ÒÆµ½ TiDB

TiDB ǰÆÚ²âÊÔ

ÔÚ²âÊÔ³õÆÚ£¬ÎÒÃDzÉÓà TiDB µÄ°æ±¾Îª RC4£¬ÔÚ²âÊÔ¹ý³ÌÖÐÔø¾­ÔÚͬʱ¶ÔÒ»ÕÅ±í½øÐжÁдʱ£¬³öÏÖ Region is stale µÄ´íÎó£¬ÔÚ GitHub ÉÏÌá³ö Issue ºó£¬TiDB ¹Ù·½ºÜ¿ìÔÚ Pre-GA °æ±¾ÖнøÐÐÁËÐÞ¸´¡£ÔÚ²âÊÔ»·¾³£¬ÎÒÃÇÊÇÊÖ¶¯Í¨¹ý¶þ½øÖưüµÄÐÎʽÀ´²¿Êð TiDB £¬ËäÈ»±È½Ï¼òµ¥£¬µ«Êǵ± TiDB ·¢²¼ GA °æ±¾Ö®ºó£¬°æ±¾Éý¼¶È´ÊÇÒ»¸ö±È½Ï´óµÄÎÊÌ⣬ÓÉÓÚÔçÆÚûÓÐʹÓà TiDB-ansible °²×°£¬¹Ù·½ÖÆ×÷µÄÉý¼¶½Å±¾ÎÞ·¨Ê¹Ó㬶øÊÖ¶¯½øÐйö¶¯Éý¼¶µÈ²Ù×÷·Ç³£Âé·³¡£ÓÉÓÚµ±Ê±ÊDzâÊÔ»·¾³£¬ÔÚÌýÈ¡ÁË TiDB ¹Ù·½µÄ½¨ÒéÖ®ºó£¬ÎÒÃÇÖØÐÂÀûÓà TiDB ¹Ù·½ÌṩµÄ TiDB-ansible ²¿ÊðÁË TiDB µÄ GA °æ±¾¡£Ö»ÐèÒªÏÂÔØ¹Ù·½ÌṩµÄ°ü£¬ÐÞ¸ÄÏàÓ¦µÄÅäÖ㬾ÍÄÜÍê³É°²×°ºÍ²¿Êð¡£¹Ù·½Ò²ÌṩÁËÉý¼¶½Å±¾£¬Äܹ»ÔÚÏàÁÚµÄ TiDB °æ±¾Ö®Ç°Íê³ÉÎÞ·ì¹ö¶¯Éý¼¶¡£Í¬Ê± TiDB-ansible ĬÈÏ»áÌṩ Prometheus + Grafana µÄ¼à¿Ø°²×°£¬¹Ù·½ÌṩÁ˷dz£·á¸»ÍêÉÆµÄ Grafana Ä£°å£¬Ê¡È¥ÁËÔËάºÜ¶à¼à¿ØÅäÖõŤ×÷Á¿£¬½è×Å TiDB ²¿Êð¼à¿ØµÄÆõ»ú£¬ÎÒÃÇÒ²Íê³ÉÁËÖîÈç Redis£¬RabbitMQ£¬Elasticsearch µÈºÜ¶àÓ¦ÓóÌÐòµÄ¼à¿ØÓÉ Zabbix Íù Prometheus µÄÇ¨ÒÆ¡£ÕâÀïÐèҪעÒâµÄÊÇ£¬Èç¹ûÊÇÓùٷ½ÌṩµÄ²¿Ê𹤾߲¿Êð Prometheus ºÍ Grafana£¬ÔÚÖ´Ðйٷ½µÄÍ£Ö¹½Å±¾Ê±ÇмÇÌø¹ýÏàÓ¦µÄ×é¼þ£¬ÒÔÃâ¸ÉÈÅÆäËû³ÌÐòµÄ¼à¿Ø¡£

TiDB ÉÏÏß¹ý³Ì

ÔÚ10ÔÂÖÐÑ®£¬Ëæ×ÅлúÆ÷µÄ²É¹ºµ½Î»£¬ÎÒÃÇÕýʽ½« TiDB ²¿Êðµ½Éú²ú»·¾³½øÐвâÊÔ£¬Õû¸ö¼Ü¹¹Îª 3 ̨»úÆ÷£¬3TiKV£«3PD£«2TiDB µÄ¼Ü¹¹¡£ÔÚÉú²ú»·¾³ÖеĴóÊý¾ÝÁ¿³¡¾°Ï£¬Óöµ½ÁËһЩеÄÎÊÌâ¡£

Ê×ÏÈÓöµ½µÄÎÊÌâÊÇ OLTP ·½Ã棬Spark Streaming ³ÌÐòÉèÖÃµÄ 5 ÃëÒ»¸ö´°¿Ú£¬µ± 5 ÃëÖ®ÄÚ²»ÄÜ´¦ÀíÍ굱ǰÅú´ÎµÄÊý¾Ý£¬¾Í»á²úÉúÑÓ³Ù£¬Í¬Ê± Streaming ÔÚÕâ¸öÅú´Î½áÊøºó»áÂíÉÏÆô¶¯ÏÂÒ»¸öÅú´Î£¬µ«ÊÇËæ×Åʱ¼äµÄ»ýÀÛ£¬ÑÓ³ÙµÄÊý¾Ý¾Í»áÔ½À´Ô½¶à£¬×îºóÉõÖÁÑÓ³ÙÁË 8 Сʱ֮¾Ã£»ÁíÒ»·½Ã棬ÓÉÓÚÎÒÃÇʹÓõÄÊÇ»úеӲÅÌ£¬Òò´ËдÈëµÄЧÂÊÊ®·Ö²»Îȶ¨£¬ÕâÒ²ÊÇÔì³ÉдÈëÑÓ³ÙµÄÒ»¸öºÜÖ÷ÒªµÄÒòËØ¡£

³öÏÖÎÊÌâÖ®ºóÎÒÃÇÁ¢¼´Óë TiDB ¹Ù·½È¡µÃÁªÏµ£¬È·ÈÏ TiDB ÕûÌå¼Ü¹¹Ö÷Òª»ùÓÚ SSD ´æ´¢ÐÔÄÜÖ®ÉϽøÐÐÉè¼ÆµÄ¡£ÎÒÃǽ« 3 ̨»úÆ÷µÄÓ²Å̶¼»»³ÉÁË SSD£»Óë´Ëͬʱ£¬ÎÒÃǵŤ³ÌʦҲ¿ª·¢ÁËÏàÓ¦µÄͬ²½³ÌÐòÀ´Ìæ´ú Spark Streaming£¬Ëæ×ÅÓ²¼þµÄ¸üÐÂÒÔ¼°³ÌÐòµÄÌæ»»£¬Ð´Èë·½ÃæÖð½¥Îȶ¨£¬³ÌÐòÔËÐеķ½Ê½Ò²ºÍ Streaming ³ÌÐòÀàËÆ£¬¶à³ÌÐòͬʱָ¶¨Ò»¸ö Kafka µÄ Group ID£¬Í¬Ê±Á¬½Ó²»Í¬»úÆ÷µÄ TiDB ÒԴﵽдÈëЧÂÊ×î´ó»¯£¬Í¬Ê±Ò²ÊµÏÖÁË HA£¬±£Ö¤Á˼´Ê¹Ò»¸ö½ø³Ì¹ÒµôÒ²²»Ó°ÏìÕûÌåÊý¾ÝµÄдÈë¡£

ÔÚ OLTP ÓÅ»¯½áÊøÖ®ºó£¬ËæÖ®¶øÀ´µÄÊÇ·ÖÎö·½ÃæµÄÐèÇó¡£ÓÉÓÚÎÒÃÇ¶Ô TiDB µÄ¶¨Î»ÊÇʵʱÊý¾Ý²Ö¿â£¬ÕâÑù¾Í»áÏñ Hadoop Ò»Ñù´æÔںܶà ETL µÄÁ÷³Ì£¬ÔÚ Hadoop µÄÁ÷³ÌÖУ¬ÒÔ T+1 ΪÖ÷µÄÈÎÎñÕ¼¾ÝÁ˾ø´ó¶àÊý£¬¶øÕâЩÈÎÎñÆÕ±éÔÚÁ賿Æô¶¯Ö´ÐУ¬Òò´ËÖ»ÄÜÓÃÓÚ¶Ôʱ¼äÑӳٱȽϴóµÄ³¡¾°£¬¶ÔʵʱÐÔÒªÇó±È½Ï¸ßµÄ³¡¾°Ôò²»Êʺϣ¬¶ø TiDB ÔòÄܺܺõÄÂú×ãʵʱ»òÕß׼ʵʱµÄÐèÇó£¬ÔÚÎÒÃǵÄÒµÎñ³¡¾°Ï£¬ºÜ¶àÈÎÎñÒÔ 5-10 ·ÖÖÓΪִÐÐÖÜÆÚ£¬Òò´Ë£¬±ØÐëÈ·±£ÈÎÎñµÄÖ´ÐÐʱ³¤ÔÚ¼ä¸ôÖÜÆÚÄÚÍê³É¡£

ÎÒÃÇÈ¡ÁËÁ½¸öÔÚ SQL Server ÉÏÅܵıȽÏÂýµÄÖØÒª½Å±¾×öÁËÇ¨ÒÆ£¬Ïà±ÈÓÚ SQL Server£¯MySQL Ç¨ÒÆÖÁ Hadoop£¬´Ó SQL Server Ç¨ÒÆÖÁ TiDB µÄ¸Ä¶¯·Ç³£Ð¡£¬SQL Server µÄ Merge ²Ù×÷ÔÚ TiDB ÀïҲͨ¹ý replace into Äܹ»Íê³É£¬ÆäÓàһЩ SQL Server µÄÌØÐÔ£¬Ò²Äܹ»Í¨¹ý TiDB µÄ¶àÐÐÊÂÎñµÃÒÔʵÏÖ£¬ÔÚÕâÒ»·½Ã棬TiDB µÄ GA °æ±¾ÒѾ­×öµÄ·Ç³£ÍêÉÆ£¬¸ß¶È¼æÈÝ MySQL£¬Òò´ËÇ¨ÒÆµÄ³É±¾·Ç³£Ð¡£¬´Ó¶øÊ¹ÎÒÃÇÄܹ»½«´ó²¿·Ö¾«Á¦·ÅÔÚÁ˵÷ÓÅ·½Ãæ¡£

Ôڽű¾Ç¨ÒÆÍê±ÏÖ®ºó£¬Ò»Ð©¼òµ¥µÄ½Å±¾Äܹ»ÔÚÃë¼¶Íê³É´ïµ½ÁËÎÒÃǵÄÔ¤ÆÚ¡£µ«ÊÇһЩ¸´ÔӵĽű¾µÄ±íÏÖÔÚ³õÆÚ²¢Ã»±íÏÖ³öÓÅÊÆ£¬Ò»Ð©½Å±¾Óë SQL Server ³ÖƽÉõÖÁ¸üÂý£¬ÆäÖÐ×î´óµÄ½Å±¾ SQL ´úÂëÁ¿Ò»¹² 1000 ¶àÐУ¬Éæ¼°½«½ü 20 ÕÅÖмä±í¡£ÔÚ֮ǰµÄ SQL Server ÉÏ£¬Ëæ×ÅÊý¾ÝÁ¿ÂýÂýÔö´ó£¬Ã¿ÌìµÄÖ´ÐÐʱ³¤Öð½¥ÓÉ 1-2 ·ÖÖÓÔö³¤µ½ 5-6 ·ÖÖÓÉõÖÁ¸ü¾Ã£¬ÔÚË«11µ±ÌìÁ賿£¬Ëæ×ŵ¥Á¿µÄÓ¿ÈëºÍÆäËûÈÎÎñµÄ¸ÉÈÅÑÓ³Ùµ½ 20 ·ÖÖÓÉõÖÁÒÔÉÏ¡£ÔÚÇ¨ÒÆÖÁ TiDB ³õÆÚ£¬ÔÚ°ëÌìµÄÊý¾ÝÁ¿Ï TiDB µÄÖ´ÐÐʱ³¤´óÖÂΪ 15 ·ÖÖÓ×óÓÒ£¬Óë SQL Server ´óÖÂÏàͬ£¬µ«ÊDz¢²»ÄÜÂú×ãÎÒÃǵÄÔ¤ÆÚ¡£ÎÒÃDzο¼ÁË TiDB µÄÏà¹ØÎĵµ¶Ô²éѯ²ÎÊý×öÁËһЩµ÷ÓÅ£¬¼¸¸öÖØÒª²ÎÊýΪ£ºtidb_distsql_scan_concurrency£¬tidb_index_serial_scan_concurrency£¬tidb_index_join_batch_size£¨TiDB ÌṩÁ˺ܺõIJ¢ÐмÆËãÄÜÁ¦£©¡£¾­¹ýÑéÖ¤£¬µ÷Õû²ÎÊýºó£¬Ò»Ð© SQL Äܹ»Ëõ¶ÌÒ»±¶µÄÖ´ÐÐʱ¼ä£¬µ«ÕâÀïÒÀ¾É²»ÄÜÍêÈ«Âú×ãÎÒÃǵÄÐèÇó¡£

ÒýÈë TiSpark

Ëæºó£¬ÎÒÃǰÑÄ¿¹âתÏòÁË TiDB µÄÒ»¸ö×ÓÏîÄ¿ TiSpark£¨https://github.com/pingcap/tispark£©£¬ÓùÙÍøµÄ½éÉÜÀ´½² TiSpark ¾ÍÊǽèÖú Spark ƽ̨£¬Í¬Ê±ÈÚºÏ TiKV ·Ö²¼Ê½¼¯ÈºµÄÓÅÊÆ£¬ºÍ TiDB Ò»Æð½â¾ö HTAP µÄÐèÇó¡£TiDB-ansible ÖÐÒ²´øÓÐ TiSpark µÄÅäÖã¬ÓÉÓÚÎÒÃÇÒѾ­ÓµÓÐÁË Spark ¼¯Èº£¬ËùÒÔÖ±½ÓÔÚÏÖÓÐµÄ Spark ¼¯ÈºÖм¯³ÉÁË TiSpark¡£ËäÈ»¸ÃÏîÄ¿¿ª·¢²»¾Ã£¬µ«ÊǾ­¹ý²âÊÔ£¬ÊÕÒæ·Ç³£Ã÷ÏÔ¡£

TiSpark µÄÅäÖ÷dz£¼òµ¥£¬Ö»ÐèÒª°Ñ TiSprak Ïà¹ØµÄ jar °ü·ÅÈë Spark ¼¯ÈºÖÐµÄ jars Îļþ¼ÐÖоÍÄÜÒýÈë TiSpark£¬Í¬Ê±¹Ù·½Ò²ÌṩÁË 3 ¸ö½Å±¾£¬ÆäÖÐÁ½¸öÊÇÆô¶¯ºÍÍ£Ö¹ TiSpark µÄ Thrift Server£¬ÁíÒ»¸öÊÇÌṩµÄ TiSpark µÄ cli ¿Í»§¶Ë£¬ÕâÑùÎÒÃǾÍÄÜÏñʹÓà Hive Ò»ÑùʹÓà TiSpark È¥×ö²éѯ¡£

ÔÚ³õ²½Ê¹ÓÃÖ®ºó£¬ÎÒÃÇ·¢ÏÖһЩÖîÈç select count(*) from table µÈ SQL Ïà±ÈÓÚ TiDB Óзdz£Ã÷ÏÔµÄÌáÉý£¬Ò»Ð©¼òµ¥µÄ OLAP µÄ²éѯ»ù±¾É϶¼Äܹ»ÔÚ 5 ÃëÖ®ÄÚ·µ»Ø½á¹û¡£¾­¹ý³õ²½²âÊÔ£¬´óÖÂÔÚ OLAP µÄ½áÂÛÈçÏ£ºÒ»Ð©¼òµ¥µÄ²éѯ SQL£¬ÔÚÊý¾ÝÁ¿°ÙÍò¼¶×óÓÒ£¬TiDB µÄÖ´ÐÐЧÂÊ¿ÉÄÜ»á±È TiSpark ¸üºÃ£¬ÔÚÊý¾ÝÁ¿Ôö¶àÖ®ºó TiSpark µÄÖ´ÐÐЧÂʻᳬ¹ý TiDB£¬µ±È»ÕâÒ²¿´ TiKV µÄÅäÖᢱí½á¹¹µÈ¡£ÔÚ TiSpark µÄʹÓùý³ÌÖУ¬ÎÒÃÇ·¢ÏÖ TiSpark µÄ²éѯ½á¹ûÔÚ°ÙÍò¼¶Ê±£¬Ö´ÐÐʱ¼ä¶¼·Ç³£Îȶ¨£¬¶ø TiDB µÄ²éѯʱ¼äÔò»áËæ×ÅÊý¾ÝÁ¿µÄÔö³¤¶øÔö³¤£¨¾­¹ýÓë TiDB ¹Ù·½¹µÍ¨£¬Õâ¸öÇé¿öÖ÷ÒªÊÇÒòΪûÓбȽϺõÄË÷Òý½øÐÐÊý¾Ýɸѡ£©¡£Õë¶ÔÎÒÃǵĶ©µ¥±í×ö²âÊÔ£¬ÔÚÊý¾ÝÁ¿Îª½ü°ÙÍò¼¶Ê±£¬TiDB µÄÖ´ÐÐʱ¼äΪ 2 Ãë×óÓÒ£¬TiSpark µÄÖ´ÐÐʱ¼äΪ 7 Ã룻µ±Êý¾ÝÁ¿Ôö³¤Îª½üǧÍò¼¶Ê±£¬TiDB µÄÖ´ÐÐʱ¼ä´óÖÂΪ 12 Ã루²»¿¼ÂÇ»º´æ£©£¬TiSpark ÒÀ¾ÉΪ 7 Ã룬·Ç³£Îȶ¨¡£

Òò´Ë£¬ÎÒÃǾö¶¨½«Ò»Ð©¸´Ô ETL ½Å±¾Óà TiSpark À´ÊµÏÖ£¬¶ÔÉÏÊöµÄ¸´Ôӽű¾½øÐзÖÎöºó£¬ÎÒÃÇ·¢ÏÖ£¬´ó¶àÊý½Å±¾Öмä±íºÜ¶à£¬ÔÚ SQL Server ÖÐÊÇͨ¹ý SQL Server ÄÚ´æ±íʵÏÖ£¬¶øÇ¨ÒÆÖÁ TiDB£¬Ã¿ÕÅÖмä±í¶¼ÒªÉ¾³ýºÍ²åÈëÂ䵨£¬ÕâЩ¿ªÏú´ó´óÔö¼ÓÁËÖ´ÐÐʱ³¤£¨¾Ý¹Ù·½´ð¸´ TiDB ºÜ¿ìÒ²»áÖ§³Ö View¡¢ÄÚ´æ±í£©¡£ÔÚÓÐÁË TiSpark Ö®ºó£¬ÎÒÃDZãÀûÓà TiSpark ½«Öмä±í»º´æÎª Spark µÄÄÚ´æ±í£¬Ö»ÐèÒª½«×îºóµÄÊý¾ÝÂ䵨»Ø TiDB£¬ÔÙÖ´ÐÐ Merge ²Ù×÷¼´¿É£¬ÕâÑùÊ¡µôÁ˺ܶàÖмäÊý¾ÝµÄÂ䵨£¬´ó´ó½ÚÊ¡Á˺ܶà½Å±¾Ö´ÐеÄʱ¼ä¡£

ÔÚ²éѯËٶȽâ¾öÖ®ºó£¬ÎÒÃÇ·¢Ïֽű¾ÖлáÓкܶàÕë¶ÔÖмä±í update ºÍ delete µÄÓï¾ä¡£Ä¿Ç° TiSpark ÔÝʱ²»Ö§³Ö update ºÍ delete µÄ²Ù×÷£¨ºÍ TiSpark ×÷Õß¹µÍ¨£¬ºóÐø»á¿¼ÂÇÖ§³ÖÕâÁ½¸ö²Ù×÷£©£¬ÎÒÃDZ㳢ÊÔÁËÁ½ÖÖ·½°¸£¬Ò»²¿·ÖÖ´ÐÐÀàËÆÓÚ Hive£¬²ÉÓà insert into Ò»ÕÅбíµÄ·½Ê½À´½â¾ö£»ÁíÍâÒ»²¿·Ö£¬ÎÒÃÇÒýÈëÁË Spark ÖÐµÄ Snappydata ×÷Ϊһ²¿·ÖÄÚ´æ±í´æ´¢£¬ÔÚ Snappydata ÖнøÐÐ update ºÍ delete£¬ÒÔ´ïµ½ÏëÒªµÄÄ¿µÄ¡£ÒòΪ¶¼ÊÇ Spark µÄÏîÄ¿£¬Òò´ËÔÚÈÚºÏÁ½¸öÏîÄ¿µÄʱºò»¹ÊDZȽÏÇáËɵġ£

×îºó£¬¹ØÓÚʵʱµÄµ÷¶È¹¤¾ß£¬Ä¿Ç°ÎÒÃÇÊǺÍÀëÏßµ÷¶ÈÒ»Æð½øÐе÷¶È£¬ÕâÒ²´øÀ´ÁËһЩÎÊÌ⣬ÿ´Î½Å±¾¶¼»á³õʼ»¯Ò»Ð© Spark ²ÎÊýµÈ£¬ÕâÒ²Ï൱ºÄʱ¡£ÔÚδÀ´£¬ÎÒÃÇ´òËã²ÉÓà Spark Streaming ×÷Ϊµ÷¶È¹¤¾ß£¬Ã¿´ÎÖ´ÐÐÍê³ÉÖ®ºó¼Ç¼ʱ¼ä´Á£¬Spark Streaming Ö»Ðè¼à¿ØÊ±¼ä´Á±ä»¯¼´¿É£¬Äܹ»±ÜÃâ¶à´Î³õʼ»¯µÄºÄʱ£¬Í¨¹ý Spark ¼à¿Ø£¬ÎÒÃÇÒ²Äܹ»Çå³þµÄ¿´µ½ÈÎÎñµÄÑÓ³ÙºÍһЩ״̬£¬ÕâÒ»²¿·Ö½«ÔÚδÀ´½øÐвâÊÔ¡£

TiDB ¹Ù·½Ö§³Ö

ÔÚÇ¨ÒÆ¹ý³ÌÖУ¬ÎÒÃǵõ½ÁË TiDB ¹Ù·½ºÜºÃµÄÖ§³Ö£¬ÆäÖÐÒ²°üÀ¨ TiSpark Ïà¹ØµÄ¼¼Êõ¸ºÔðÈË£¬Ò»Ð© TiSpark µÄ Corner Case ¼°Ê¹ÓÃÎÊÌ⣬ÎÒÃǶ¼»áÔÚȺÀïÅ׳ö£¬TiDB µÄ¹Ù·½ÈËÔ±»á·Ç³£¼°Ê±µÄ°ïÖúÎÒÃǽâ¾öÎÊÌ⣬ÔÚ¹Ù·½Ö§³ÖÏ£¬ÎÒÃÇÇ¨ÒÆÖÁ TiSpark µÄ¹ý³ÌºÜ˳Àû£¬Ã»ÓÐÊܵ½Ê²Ã´Ì«´óµÄ¼¼Êõ×è°­¡£

ʵʱÊý²Ö TiDB / TiSpark

ÔÚÇ¨ÒÆÍê³ÉÖ®ºó£¬ÆäÖÐÒ»Ìõ¸´Ô SQL£¬Ò»¹² Join ÁË 12 ÕÅ±í£¨×î´ó±íÊýÁ¿ÒÚ¼¶£¬²¿·Ö±í°ÙÍò¼¶£©£¬ÔÚÆ½Ê±Ð¡ÅúÁ¿µÄÇé¿öÏ£¬Ö´ÐÐʱ¼ä»áÔÚ 5 ·ÖÖÓ×óÓÒ£¬ÎÒÃÇÒ²ÄÃÁËË«11È«Á¿µÄÊý¾Ý½øÐÐÁ˲âÊÔ£¬Ö´ÐÐʱ¼äÔÚ 9 ·ÖÖÓÒÔÉÏ£¬¶ø²ÉÓÃÁË TiSpark µÄ·½Ê½È¥Ö´ÐУ¬Ë«11È«Á¿µÄÊý¾ÝÒ²½ö½ö»¨ÁË 1 ·ÖÖÓ£¬ÐÔÄÜÌáÉýÁË 9 ±¶¡£Õû¸ö´ó½Å±¾ÔÚ SQL Server ÉÏÔËÐÐË«11µÄÈ«Á¿Êý¾ÝÒÔǰÖÁÉÙÒªÏûºÄ 30 ·ÖÖÓ£¬ÀûÓà TiDB È¥Ö´ÐдóÖÂÐèÒª 20 ·ÖÖÓ×óÓÒ£¬ÀûÓà TiSpark Ö»ÐèÒª 8 ·ÖÖÓ×óÓÒ£¬Ïà¶Ô SQL Server ÐÔÄÜÌáÉý 4 ±¶£¬Ò²¾ÍÊÇ˵£¬Ã¿ÄêÊý¾ÝÁ¿×î¸ß·åµÄ´¦ÀíÄÜÁ¦´ïµ½ÁË·ÖÖÓ¼¶£¬ºÜºÃµÄÂú×ãÁËÎÒÃǵÄÐèÇó¡£

×îºó£¬²»¹ÜÊÇÓà TiDB »¹ÊÇÓà TiSpark ¶¼»áÓÐÒ»²¿·ÖÖмä±íÒÔ¼°ÓëÔ­±í½øÐÐ Merge µÄ²Ù×÷£¬ÕâÀïÓÉÓÚ TiDB ¶ÔÊÂÎñ½øÐеÄÏÞÖÆ£¬ÎÒÃÇÒ²²ÉÓÃÒÔÍòÌõΪµ¥Åú´Î½øÐÐÅúÁ¿µÄ²åÈëºÍ Merge£¬¼È±ÜÃâÁ˳¬¹ýÊÂÎñµÄ±¨´íÓÖ·ûºÏ TiDB µÄÉè¼ÆÀíÄÄܹ»´ïµ½×î¼Ñʵ¼ù¡£

ÓÐÁË TiSpark Õâ¸öÏîÄ¿£¬TiDB Óë Hadoop µÄÉú̬ÌåϵµÃµ½½øÒ»²½µÄÈںϣ¬ÔÚûÓÐ TiSpark ֮ǰ£¬ÎÒÃǵÄϵͳÉè¼ÆÈçÏ£º

ͼ£º¶àÌ×Êý²Ö²¢´æ

¿ÉÒÔ·¢ÏÖ£¬ÊµÊ±Êý²ÖÓë T+1 Òì²½Êý²ÖÊÇÁ½¸öÏà¶Ô¶ÀÁ¢µÄϵͳ£¬²¢Ã»ÓÐÈκν»¼¯£¬ÎÒÃÇÐèÒª½øÐÐÊý¾ÝʵʱµÄͬ²½£¬Í¬Ê±Ò²»áÔÚÒ¹Íí×öÒ»´ÎÒ첽ͬ²½£¬²»¹ÜÊÇ Datax »¹ÊÇ Sqoop ¶ÁÈ¡¹ØÏµÐÍÊý¾Ý¿âµÄЧÂʶ¼Ô¶Ô¶´ï²»µ½ TiSpark µÄËÙ¶È£¬¶øÔÚÓÐÁË TiSpark Ö®ºó£¬ÎÒÃÇ¿ÉÒÔ¶Ô T+1 Òì²½Êý²Ö½øÐÐÕûºÏ£¬ÓÚÊÇÎÒÃǵļܹ¹½ø»¯ÎªÈçÏ£º

ͼ£ºTiDB / TiSpark ʵʱÊý²Öƽ̨

ÕâÑù¾ÍÄܹ»ÀûÓà TiSpark ½« TiDB ºÍ Hadoop ºÜºÃµÄ´®ÁªÆðÀ´£¬»¥Îª²¹³ä£¬TiDB µÄ¹¦ÄÜÒ²Óɵ¥´¿µÄʵʱÊý²Ö±ä³ÉÄܹ»ÌṩÈçϼ¸¸ö¹¦ÄÜ»ìºÏÊý¾Ý¿â£º

1. ʵʱÊý²Ö£¬ÉÏÓÎ OLTP µÄÊý¾Ýͨ¹ý TiDB ʵʱдÈ룬ÏÂÓÎ OLAP µÄÒµÎñͨ¹ý TiDB / TiSpark ʵʱ·ÖÎö¡£

2. T+1 µÄ³éÈ¡Äܹ»´Ó TiDB ÖÐÀûÓà TiSpark ½øÐгéÈ¡¡£

TiSpark ËÙ¶ÈÔ¶Ô¶³¬¹ý Datax ºÍ Sqoop ¶ÁÈ¡¹ØÏµÐÍÊý¾Ý¿âµÄËÙ¶È£»

³éÈ¡¹¤¾ßÒ²²»ÓÃά»¤¶à¸öϵͳ¿â£¬Ö»ÐèҪά»¤Ò»¸ö TiDB ¼´¿É£¬´ó´ó·½±ãÁËÒµÎñµÄͳһʹÓ㬻¹½ÚÊ¡Á˶à´Îά»¤³É±¾¡£

TiDB ÌìÈ»·Ö²¼Ê½µÄÉè¼ÆÒ²±£Ö¤ÁËϵͳµÄÎȶ¨¡¢¸ß¿ÉÓá£

3. TiDB ·Ö²¼Ê½ÌØÐÔ¿ÉÒÔºÜºÃµÄÆ½ºâÈȵãÊý¾Ý£¬¿ÉÒÔÓÃËü×÷ΪҵÎñ¿âÈȵãÊý¾ÝµÄÒ»¸ö±¸·Ý¿â£¬»òÕßÖ±½ÓǨÈë TiDB ¡£

ÉÏÃæÕâÈýµãÒ²ÊÇÎÒÃǽñºóȥŬÁ¦µÄ·½Ïò£¬Óɴ˿ɼû£¬TiSpark ²»½ö¶ÔÓÚ ETL ½Å±¾Æðµ½Á˺ÜÖØÒªµÄ×÷Óã¬ÔÚÎÒÃǽñºóµÄ¼Ü¹¹ÖÐÒ²Æðµ½Á˾Ù×ãÇáÖØµÄ×÷Óã¬ÎªÎÒÃÇ´´½¨Ò»¸öʵʱµÄͳһµÄ»ìºÏÊý¾Ý¿âÌṩÁË¿ÉÄÜ¡£

Óë´Ëͬʱ£¬ÎÒÃÇÒ²µÃµ½ TiDB ¹Ù·½ÈËÔ±µÄÈ·ÈÏ£¬TiDB ½«ÓÚ½üÆÚÖ§³ÖÊÓͼ¡¢·ÖÇø±í£¬²¢»á³ÖÐøÔöÇ¿ SQL ÓÅ»¯Æ÷£¬Í¬Ê±Ò²»áÌṩһ¿îÃûΪ TiDB Wormhole µÄÒ칹ƽ̨Êý¾ÝÊµÊ±Ç¨ÒÆ¹¤¾ßÀ´±ã½ÝµÄÖ§³ÖÓû§µÄ¶àÔª»¯Ç¨ÒÆÐèÇó¡£ÎÒÃÇÒ²¼Æ»®½«¸ü¶àµÄ²úÆ·ÏßÖð²½Ç¨Èë TiDB¡£

×ܽá

ͬʱ½â¾ö OLAP ºÍ OLTP ÊÇÒ»¼þÏ൱À§ÄѵÄÊÂÇ飬TiDB ºÍ TiSpark ËäÈ»ÍÆ³ö²»¾Ã£¬µ«ÊÇÒѾ­Âú×ãºÜ¶àÓ¦Óó¡¾°£¬Í¬Ê±ÔÚÒ×ÓÃÐԺͼ¼ÊõÖ§³ÖÉÏÒ²·Ç³£ÖµµÃ³ÆÔÞ£¬ÏàÐÅ TiDB Ò»¶¨Äܹ»ÔÚÔ½À´Ô½¶àµÄÆóÒµÖеõ½¹ã·ºÓ¦Óá£

   
2252 ´Îä¯ÀÀ       29
Ïà¹ØÎÄÕÂ

»ùÓÚEAµÄÊý¾Ý¿â½¨Ä£
Êý¾ÝÁ÷½¨Ä££¨EAÖ¸ÄÏ£©
¡°Êý¾Ýºþ¡±£º¸ÅÄî¡¢ÌØÕ÷¡¢¼Ü¹¹Óë°¸Àý
ÔÚÏßÉ̳ÇÊý¾Ý¿âϵͳÉè¼Æ ˼·+Ч¹û
 
Ïà¹ØÎĵµ

GreenplumÊý¾Ý¿â»ù´¡Åàѵ
MySQL5.1ÐÔÄÜÓÅ»¯·½°¸
ijµçÉÌÊý¾ÝÖÐ̨¼Ü¹¹Êµ¼ù
MySQL¸ßÀ©Õ¹¼Ü¹¹Éè¼Æ
Ïà¹Ø¿Î³Ì

Êý¾ÝÖÎÀí¡¢Êý¾Ý¼Ü¹¹¼°Êý¾Ý±ê×¼
MongoDBʵս¿Î³Ì
²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
PostgreSQLÊý¾Ý¿âʵսÅàѵ
×îл¼Æ»®
DeepSeek´óÄ£ÐÍÓ¦Óÿª·¢ 6-12[ÏÃÃÅ]
È˹¤ÖÇÄÜ.»úÆ÷ѧϰTensorFlow 6-22[Ö±²¥]
»ùÓÚ UML ºÍEA½øÐзÖÎöÉè¼Æ 6-30[±±¾©]
ǶÈëʽÈí¼þ¼Ü¹¹-¸ß¼¶Êµ¼ù 7-9[±±¾©]
Óû§ÌåÑé¡¢Ò×ÓÃÐÔ²âÊÔÓëÆÀ¹À 7-25[Î÷°²]
ͼÊý¾Ý¿âÓë֪ʶͼÆ× 8-23[±±¾©]
 
×îÐÂÎÄÕÂ
InfluxDB¸ÅÄîºÍ»ù±¾²Ù×÷
InfluxDB TSM´æ´¢ÒýÇæÖ®Êý¾ÝдÈë
Éî¶ÈÂþ̸Êý¾Ýϵͳ¼Ü¹¹¡ª¡ªLambda architecture
Lambda¼Ü¹¹Êµ¼ù
InfluxDB TSM´æ´¢ÒýÇæÖ®Êý¾Ý¶ÁÈ¡
×îпγÌ
OracleÊý¾Ý¿âÐÔÄÜÓÅ»¯¡¢¼Ü¹¹Éè¼ÆºÍÔËÐÐά»¤
²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
NoSQLÊý¾Ý¿â£¨Ô­Àí¡¢Ó¦Óá¢×î¼Ñʵ¼ù£©
ÆóÒµ¼¶Hadoop´óÊý¾Ý´¦Àí×î¼Ñʵ¼ù
OracleÊý¾Ý¿âÐÔÄÜÓÅ»¯×î¼Ñʵ¼ù
³É¹¦°¸Àý
ij½ðÈÚ¹«Ë¾ Mysql¼¯ÈºÓëÐÔÄÜÓÅ»¯
±±¾© ²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
ÖªÃûijÐÅϢͨÐŹ«Ë¾ NoSQL»º´æÊý¾Ý¿â¼¼Êõ
±±¾© oracleÊý¾Ý¿âSQLÓÅ»¯
ÖйúÒÆ¶¯ IaaSÔÆÆ½Ì¨-Ö÷Á÷Êý¾Ý¿â¼°´æ´¢¼¼Êõ