±à¼ÍƼö: |
±¾ÎÄͨ¹ý Ò»¸öÏîÄ¿½éÉÜÁËTiDB ´Ó»ù´¡ÔÚʵʱÊý²ÖÖеÄÓ¦Óã¬Ï£Íû±¾ÎĶԴó¼ÒÓаïÖú¡£
±¾ÎÄÀ´×ÔÖªºõ£¬ÓÉ»ðÁú¹ûÈí¼þAnna±à¼¡¢ÍƼö¡£ |
|
ÏîÄ¿±³¾°
ĿǰÆóÒµ´ó¶àÊýµÄÊý¾Ý·ÖÎö³¡¾°µÄ½â¾ö·½°¸µ×²ã¶¼ÊÇÎ§ÈÆ Hadoop ´óÊý¾ÝÉú̬չ¿ªµÄ£¬³£¼ûµÄÈç HDFS
+ Hive + Spark + Presto + Kylin£¬ÔÚÒ×¹û¼¯ÍÅ£¬ÎÒÃdzõÆÚÒ²ÊDzÉÈ¡ÕâÖÖ˼·£¬µ«ÊÇËæ×ÅÒµÎñ¹æÄ£µÄ¿ìËÙÔö³¤ºÍÐèÇóµÄ²»¶Ï±ä»¯£¬Ò»Ð©ÊµÊ±»òÕß׼ʵʱµÄÐèÇó±äµÃÔ½À´Ô½¶à£¬ÕâÀàÒµÎñ³ýÁËÓÐʵʱµÄ
OLTP ÐèÇ󣬻¹°éËæ×ÅһЩÓÐÒ»¶¨¸´ÔÓ¶ÈµÄ OLAP µÄÐèÇ󣬵¥´¿µØÊ¹Óà Hadoop ÒѾÎÞ·¨Âú×ãÐèÇó¡£
ÏÖÓеÄ׼ʵʱϵͳÔËÐÐÔÚ SQL Server Ö®ÉÏ£¬Í¨¹ý¿ª·¢ÈËÔ±±àдºÍά»¤ÏàÓ¦µÄ´æ´¢¹ý³ÌÀ´ÊµÏÖ¡£ÓÉÓÚÊý¾ÝÁ¿²»´ó£¬SQL
Server Äܹ»Âú×ãÐèÇ󣬵«ÊÇËæ×ÅÒµÎñµÄ·¢Õ¹£¬Êý¾ÝÁ¿ËæÖ®Ôö³¤£¬SQL Server Ô½À´Ô½²»ÄÜÂú×ãÐèÇ󣬵±Êý¾ÝÁ¿µ½´ïÒ»¶¨µÄ½×¶Î£¬ÐÔÄܱã»á³öÏֹյ㡣Õâ¸öʱºò£¬ÕâÌ×·½°¸ÒÑÍêÈ«ÎÞ·¨Ö§³ÅÒµÎñ£¬²»µÃ²»ÖØÐÂÉè¼ÆÐµķ½°¸¡£
Ñ¡ÐÍÆÀ¹À
ÔÚÆÀ¹À³õÆÚ£¬Greenplum¡¢Kudu¡¢TiDB ¶¼½øÈëÁËÎÒÃǵÄÊÓÒ°£¬¶ÔÓÚеÄʵʱϵͳ£¬ÎÒÃÇÓÐÖ÷Òª¿¼Âǵ㣺
Ê×ÏÈ£¬ÏµÍ³¼ÈÒªÂú×ã OLAP »¹ÒªÂú×ã OLTP µÄ»ù±¾ÐèÇó£»
Æä´Î£¬ÐÂϵͳҪ¾¡Á¿½µµÍÒµÎñµÄʹÓÃÒªÇó£»
×îºó£¬ÐÂϵͳ×îºÃÄܹ»ÓëÏÖÓÐµÄ Hadoop ÌåϵÏà½áºÏ¡£
Greenplum ÊÇÒ»Ì×»ùÓÚ PostgreSQL ·ÖÎöΪÖ÷µÄ MPP ÒýÇæ£¬´ó¶àÓÃÔÚ²¢·¢¶È²»¸ßµÄÀëÏß·ÖÎö³¡¾°£¬µ«ÔÚ
OLTP ·½Ã棬ÎÒÃǵijõ²½²âÊÔ·¢ÏÖÆä¶Ô±È TiDB µÄÐÔÄܲîºÜ¶à¡£
ÔÙ˵˵ Kudu¡£Kudu ÊÇ CDH 2015Äê·¢²¼µÄÒ»Ì×½éÓÚ Hbase ºÍ HDFS ÖмäµÄÒ»Ì״洢ϵͳ£¬Ä¿Ç°ÔÚ¹úÄÚÖ÷ÒªÊÇСÃ×¹«Ë¾Ó¦ÓõĽ϶࣬ÔÚ²âÊÔÖУ¬ÎÒÃÇ·¢ÏÖÆäÔÚ
OLTP ±íÏÖ´óÖÂÓë TiDB Ï൱£¬µ«ÊÇһЩÖеÈÊý¾ÝÁ¿Ï£¬Æä·ÖÎöÐÔÄÜÏà±È TiDB ÓÐÒ»¶¨²î¾à¡£ÁíÍâÎÒÃǵIJéѯĿǰÖ÷ÒªÒÔ
Presto ΪÖ÷£¬Presto ¶Ô½Ó Kudu ºÍ PostgreSQL ¶¼ÊÇÐèÒª¿¼ÂǼæÈÝÐÔµÄÎÊÌ⣬¶ø
TiDB ¼æÈÝ MySQL ÐÒ飬ÔÚÓ¦ÓóõÆÚ¿ÉÒÔÖ±½ÓʹÓà Presto-MySQL ½øÐÐͳһ²éѯ£¬ÏÂÒ»²½ÔÙ¿¼ÂÇרÃÅ¿ª·¢
Presto-TiDB¡£
ÁíÍ⣬ÎÒÃÇÏ£ÍûδÀ´µÄʵʱϵͳºÍÀëÏßϵͳÄܹ»Í¨Óã¬Ò»Ì×´úÂëÔÚÁ½¸öϵͳÖж¼Äܹ»ÍêÈ«¼æÈÝ£¬Ä¿Ç° Tispark
ºÍ SparkSQL ÒѾºÜ´ó³Ì¶ÈÉÏʵÏÖÁËÕâµã£¬ÕâÖ§³ÖÎÒÃÇÔÚÒÔºóÀëÏßÉϵÄСʱ¼¶ÈÎÎñ¿ÉÒÔÖ±½ÓÇл»µ½ TiDBÉÏ£¬ÔÚ
TiDB ÉÏʵÏÖʵʱҵÎñµÄͬʱ£¬Èç¹ûÓÐ T+1 µÄÐèÇóÒ²Äܹ»Ö±½ÓÖ¸ HDFS ¼´¿É£¬²»Óöþ´Î¿ª·¢£¬ÕâÊÇ
Kudu ºÍ GP ÔÝʱʵÏÖ²»Á˵ġ£
×îºó£¬TiSpark Êǽ¨Á¢ÔÚ Spark ÒýÇæÖ®ÉÏ£¬Spark ÔÚ»úÆ÷ѧϰÁìÓòÀïÓÐÖîÈç Mllib
µÈÖî¶à³ÉÊìµÄÏîÄ¿£¬¶Ô±È GP ºÍ Kudu£¬Ëã·¨¹¤³ÌʦÃÇʹÓà TiSpark È¥²Ù×÷ TiDB µÄÃż÷·Ç³£µÍ£¬Í¬Ê±Ò²»á´ó´óÌáÉýËã·¨¹¤³ÌʦÃǵÄЧÂÊ¡£
¾¹ý×ۺϵĿ¼ÂÇ£¬ÎÒÃÇ×îÖÕ¾ö¶¨Ê¹Óà TiDB ×÷ΪеÄʵʱϵͳ¡£Í¬Ê±£¬Ä¿Ç° TiDB µÄÉçÇø»îÔ¾¶È·Ç³£ºÃ£¬ÕâÒ²ÊÇÎÒÃÇ¿¼ÂǵÄÒ»¸öºÜÖØÒªµÄ·½Ãæ¡£
TiDB ¼ò½é
ÔÚÕâÀï½éÉÜһϠTiDB µÄÏà¹ØÌØÐÔ£ºTiDB ÊÇ»ùÓÚ Google Spanner/F1 ÂÛÎÄÆô·¢¿ªÔ´µÄÒ»Ì×
NewSQL Êý¾Ý¿â£¬Ëü¾ß±¸ÈçÏÂ
NewSQL ºËÐÄÌØÐÔ£º
SQLÖ§³Ö £¨TiDB ÊÇ MySQL ¼æÈݵģ©
ˮƽÏßÐÔµ¯ÐÔÀ©Õ¹
·Ö²¼Ê½ÊÂÎñ
Êý¾ÝǿһÖÂÐÔ±£Ö¤
¹ÊÕÏ×Ô»Ö¸´µÄ¸ß¿ÉÓÃ
ͬʱ£¬TiDB »¹ÓÐÒ»Ì׷ḻµÄÉú̬¹¤¾ß£¬ÀýÈ磺¿ìËÙ²¿ÊðµÄ TiDB-Ansible¡¢ÎÞ·ìÇ¨ÒÆ MySQL
µÄ Syncer¡¢Òì¹¹Êý¾ÝÇ¨ÒÆ¹¤¾ß Wormhole¡¢ÒÔ¼° TiDB-Binlog¡¢Backup &
Recovery µÈ¡£
SQL Server Ç¨ÒÆµ½ TiDB
ÓÉÓÚÎÒÃǹ«Ë¾µÄ¼Ü¹¹ÊÇ .NET + SQL Server ¼Ü¹¹£¬ËùÒÔÎÒÃÇÎÞ·¨Ïñ´ó¶àÊý¹«Ë¾Ò»ÑùȥʹÓÃ
MySQL Binlog È¥×öÊý¾Ýͬ²½£¬µ±È»Ò²¾ÍÎÞ·¨Ê¹Óà TiDB ¹Ù·½ÌṩµÄ Syncer ¹¤¾ßÁË¡£Òò´ËÎÒÃDzÉÓÃÁË
Flume + Kafka µÄ¼Ü¹¹£¬ÎÒÃÇ×Ô¼º¿ª·¢ÁË»ùÓÚ Flume µÄ SQL Server Source
ȥʵʱ¼à¿Ø SQL Server Êý¾Ý±ä»¯£¬½øÐв¶×½²¢Ð´Èë Kafka ÖУ¬Í¬Ê±£¬ÎÒÃÇʹÓà Spark
Streaming È¥¶ÁÈ¡ Kafka ÖеÄÊý¾Ý²¢Ð´Èë TiDB£¬Í¬Ê±ÎÒÃǽ«Ö®Ç° SQL Server
µÄ´æ´¢¹ý³Ì¸ÄÔì³É¶¨Ê±µ÷¶ÈµÄ MySQL ½Å±¾¡£

ͼ£ºSQL Server Êý¾ÝÇ¨ÒÆµ½ TiDB
TiDB ǰÆÚ²âÊÔ
ÔÚ²âÊÔ³õÆÚ£¬ÎÒÃDzÉÓà TiDB µÄ°æ±¾Îª RC4£¬ÔÚ²âÊÔ¹ý³ÌÖÐÔø¾ÔÚͬʱ¶ÔÒ»ÕÅ±í½øÐжÁдʱ£¬³öÏÖ Region
is stale µÄ´íÎó£¬ÔÚ GitHub ÉÏÌá³ö Issue ºó£¬TiDB ¹Ù·½ºÜ¿ìÔÚ Pre-GA
°æ±¾ÖнøÐÐÁËÐÞ¸´¡£ÔÚ²âÊÔ»·¾³£¬ÎÒÃÇÊÇÊÖ¶¯Í¨¹ý¶þ½øÖưüµÄÐÎʽÀ´²¿Êð TiDB £¬ËäÈ»±È½Ï¼òµ¥£¬µ«Êǵ±
TiDB ·¢²¼ GA °æ±¾Ö®ºó£¬°æ±¾Éý¼¶È´ÊÇÒ»¸ö±È½Ï´óµÄÎÊÌ⣬ÓÉÓÚÔçÆÚûÓÐʹÓà TiDB-ansible
°²×°£¬¹Ù·½ÖÆ×÷µÄÉý¼¶½Å±¾ÎÞ·¨Ê¹Ó㬶øÊÖ¶¯½øÐйö¶¯Éý¼¶µÈ²Ù×÷·Ç³£Âé·³¡£ÓÉÓÚµ±Ê±ÊDzâÊÔ»·¾³£¬ÔÚÌýÈ¡ÁË TiDB
¹Ù·½µÄ½¨ÒéÖ®ºó£¬ÎÒÃÇÖØÐÂÀûÓà TiDB ¹Ù·½ÌṩµÄ TiDB-ansible ²¿ÊðÁË TiDB µÄ
GA °æ±¾¡£Ö»ÐèÒªÏÂÔØ¹Ù·½ÌṩµÄ°ü£¬ÐÞ¸ÄÏàÓ¦µÄÅäÖ㬾ÍÄÜÍê³É°²×°ºÍ²¿Êð¡£¹Ù·½Ò²ÌṩÁËÉý¼¶½Å±¾£¬Äܹ»ÔÚÏàÁÚµÄ
TiDB °æ±¾Ö®Ç°Íê³ÉÎÞ·ì¹ö¶¯Éý¼¶¡£Í¬Ê± TiDB-ansible ĬÈÏ»áÌṩ Prometheus
+ Grafana µÄ¼à¿Ø°²×°£¬¹Ù·½ÌṩÁ˷dz£·á¸»ÍêÉÆµÄ Grafana Ä£°å£¬Ê¡È¥ÁËÔËάºÜ¶à¼à¿ØÅäÖõŤ×÷Á¿£¬½è×Å
TiDB ²¿Êð¼à¿ØµÄÆõ»ú£¬ÎÒÃÇÒ²Íê³ÉÁËÖîÈç Redis£¬RabbitMQ£¬Elasticsearch
µÈºÜ¶àÓ¦ÓóÌÐòµÄ¼à¿ØÓÉ Zabbix Íù Prometheus µÄÇ¨ÒÆ¡£ÕâÀïÐèҪעÒâµÄÊÇ£¬Èç¹ûÊÇÓùٷ½ÌṩµÄ²¿Ê𹤾߲¿Êð
Prometheus ºÍ Grafana£¬ÔÚÖ´Ðйٷ½µÄÍ£Ö¹½Å±¾Ê±ÇмÇÌø¹ýÏàÓ¦µÄ×é¼þ£¬ÒÔÃâ¸ÉÈÅÆäËû³ÌÐòµÄ¼à¿Ø¡£
TiDB ÉÏÏß¹ý³Ì
ÔÚ10ÔÂÖÐÑ®£¬Ëæ×ÅлúÆ÷µÄ²É¹ºµ½Î»£¬ÎÒÃÇÕýʽ½« TiDB ²¿Êðµ½Éú²ú»·¾³½øÐвâÊÔ£¬Õû¸ö¼Ü¹¹Îª 3 ̨»úÆ÷£¬3TiKV£«3PD£«2TiDB
µÄ¼Ü¹¹¡£ÔÚÉú²ú»·¾³ÖеĴóÊý¾ÝÁ¿³¡¾°Ï£¬Óöµ½ÁËһЩеÄÎÊÌâ¡£
Ê×ÏÈÓöµ½µÄÎÊÌâÊÇ OLTP ·½Ã棬Spark Streaming ³ÌÐòÉèÖÃµÄ 5 ÃëÒ»¸ö´°¿Ú£¬µ±
5 ÃëÖ®ÄÚ²»ÄÜ´¦ÀíÍ굱ǰÅú´ÎµÄÊý¾Ý£¬¾Í»á²úÉúÑÓ³Ù£¬Í¬Ê± Streaming ÔÚÕâ¸öÅú´Î½áÊøºó»áÂíÉÏÆô¶¯ÏÂÒ»¸öÅú´Î£¬µ«ÊÇËæ×Åʱ¼äµÄ»ýÀÛ£¬ÑÓ³ÙµÄÊý¾Ý¾Í»áÔ½À´Ô½¶à£¬×îºóÉõÖÁÑÓ³ÙÁË
8 Сʱ֮¾Ã£»ÁíÒ»·½Ã棬ÓÉÓÚÎÒÃÇʹÓõÄÊÇ»úеӲÅÌ£¬Òò´ËдÈëµÄЧÂÊÊ®·Ö²»Îȶ¨£¬ÕâÒ²ÊÇÔì³ÉдÈëÑÓ³ÙµÄÒ»¸öºÜÖ÷ÒªµÄÒòËØ¡£
³öÏÖÎÊÌâÖ®ºóÎÒÃÇÁ¢¼´Óë TiDB ¹Ù·½È¡µÃÁªÏµ£¬È·ÈÏ TiDB ÕûÌå¼Ü¹¹Ö÷Òª»ùÓÚ SSD ´æ´¢ÐÔÄÜÖ®ÉϽøÐÐÉè¼ÆµÄ¡£ÎÒÃǽ«
3 ̨»úÆ÷µÄÓ²Å̶¼»»³ÉÁË SSD£»Óë´Ëͬʱ£¬ÎÒÃǵŤ³ÌʦҲ¿ª·¢ÁËÏàÓ¦µÄͬ²½³ÌÐòÀ´Ìæ´ú Spark Streaming£¬Ëæ×ÅÓ²¼þµÄ¸üÐÂÒÔ¼°³ÌÐòµÄÌæ»»£¬Ð´Èë·½ÃæÖð½¥Îȶ¨£¬³ÌÐòÔËÐеķ½Ê½Ò²ºÍ
Streaming ³ÌÐòÀàËÆ£¬¶à³ÌÐòͬʱָ¶¨Ò»¸ö Kafka µÄ Group ID£¬Í¬Ê±Á¬½Ó²»Í¬»úÆ÷µÄ
TiDB ÒԴﵽдÈëЧÂÊ×î´ó»¯£¬Í¬Ê±Ò²ÊµÏÖÁË HA£¬±£Ö¤Á˼´Ê¹Ò»¸ö½ø³Ì¹ÒµôÒ²²»Ó°ÏìÕûÌåÊý¾ÝµÄдÈë¡£
ÔÚ OLTP ÓÅ»¯½áÊøÖ®ºó£¬ËæÖ®¶øÀ´µÄÊÇ·ÖÎö·½ÃæµÄÐèÇó¡£ÓÉÓÚÎÒÃÇ¶Ô TiDB µÄ¶¨Î»ÊÇʵʱÊý¾Ý²Ö¿â£¬ÕâÑù¾Í»áÏñ
Hadoop Ò»Ñù´æÔںܶà ETL µÄÁ÷³Ì£¬ÔÚ Hadoop µÄÁ÷³ÌÖУ¬ÒÔ T+1 ΪÖ÷µÄÈÎÎñÕ¼¾ÝÁ˾ø´ó¶àÊý£¬¶øÕâЩÈÎÎñÆÕ±éÔÚÁ賿Æô¶¯Ö´ÐУ¬Òò´ËÖ»ÄÜÓÃÓÚ¶Ôʱ¼äÑӳٱȽϴóµÄ³¡¾°£¬¶ÔʵʱÐÔÒªÇó±È½Ï¸ßµÄ³¡¾°Ôò²»Êʺϣ¬¶ø
TiDB ÔòÄܺܺõÄÂú×ãʵʱ»òÕß׼ʵʱµÄÐèÇó£¬ÔÚÎÒÃǵÄÒµÎñ³¡¾°Ï£¬ºÜ¶àÈÎÎñÒÔ 5-10 ·ÖÖÓΪִÐÐÖÜÆÚ£¬Òò´Ë£¬±ØÐëÈ·±£ÈÎÎñµÄÖ´ÐÐʱ³¤ÔÚ¼ä¸ôÖÜÆÚÄÚÍê³É¡£
ÎÒÃÇÈ¡ÁËÁ½¸öÔÚ SQL Server ÉÏÅܵıȽÏÂýµÄÖØÒª½Å±¾×öÁËÇ¨ÒÆ£¬Ïà±ÈÓÚ SQL Server£¯MySQL
Ç¨ÒÆÖÁ Hadoop£¬´Ó SQL Server Ç¨ÒÆÖÁ TiDB µÄ¸Ä¶¯·Ç³£Ð¡£¬SQL Server
µÄ Merge ²Ù×÷ÔÚ TiDB ÀïҲͨ¹ý replace into Äܹ»Íê³É£¬ÆäÓàһЩ SQL Server
µÄÌØÐÔ£¬Ò²Äܹ»Í¨¹ý TiDB µÄ¶àÐÐÊÂÎñµÃÒÔʵÏÖ£¬ÔÚÕâÒ»·½Ã棬TiDB µÄ GA °æ±¾ÒѾ×öµÄ·Ç³£ÍêÉÆ£¬¸ß¶È¼æÈÝ
MySQL£¬Òò´ËÇ¨ÒÆµÄ³É±¾·Ç³£Ð¡£¬´Ó¶øÊ¹ÎÒÃÇÄܹ»½«´ó²¿·Ö¾«Á¦·ÅÔÚÁ˵÷ÓÅ·½Ãæ¡£
Ôڽű¾Ç¨ÒÆÍê±ÏÖ®ºó£¬Ò»Ð©¼òµ¥µÄ½Å±¾Äܹ»ÔÚÃë¼¶Íê³É´ïµ½ÁËÎÒÃǵÄÔ¤ÆÚ¡£µ«ÊÇһЩ¸´ÔӵĽű¾µÄ±íÏÖÔÚ³õÆÚ²¢Ã»±íÏÖ³öÓÅÊÆ£¬Ò»Ð©½Å±¾Óë
SQL Server ³ÖƽÉõÖÁ¸üÂý£¬ÆäÖÐ×î´óµÄ½Å±¾ SQL ´úÂëÁ¿Ò»¹² 1000 ¶àÐУ¬Éæ¼°½«½ü 20
ÕÅÖмä±í¡£ÔÚ֮ǰµÄ SQL Server ÉÏ£¬Ëæ×ÅÊý¾ÝÁ¿ÂýÂýÔö´ó£¬Ã¿ÌìµÄÖ´ÐÐʱ³¤Öð½¥ÓÉ 1-2 ·ÖÖÓÔö³¤µ½
5-6 ·ÖÖÓÉõÖÁ¸ü¾Ã£¬ÔÚË«11µ±ÌìÁ賿£¬Ëæ×ŵ¥Á¿µÄÓ¿ÈëºÍÆäËûÈÎÎñµÄ¸ÉÈÅÑÓ³Ùµ½ 20 ·ÖÖÓÉõÖÁÒÔÉÏ¡£ÔÚÇ¨ÒÆÖÁ
TiDB ³õÆÚ£¬ÔÚ°ëÌìµÄÊý¾ÝÁ¿Ï TiDB µÄÖ´ÐÐʱ³¤´óÖÂΪ 15 ·ÖÖÓ×óÓÒ£¬Óë SQL Server
´óÖÂÏàͬ£¬µ«ÊDz¢²»ÄÜÂú×ãÎÒÃǵÄÔ¤ÆÚ¡£ÎÒÃDzο¼ÁË TiDB µÄÏà¹ØÎĵµ¶Ô²éѯ²ÎÊý×öÁËһЩµ÷ÓÅ£¬¼¸¸öÖØÒª²ÎÊýΪ£ºtidb_distsql_scan_concurrency£¬tidb_index_serial_scan_concurrency£¬tidb_index_join_batch_size£¨TiDB
ÌṩÁ˺ܺõIJ¢ÐмÆËãÄÜÁ¦£©¡£¾¹ýÑéÖ¤£¬µ÷Õû²ÎÊýºó£¬Ò»Ð© SQL Äܹ»Ëõ¶ÌÒ»±¶µÄÖ´ÐÐʱ¼ä£¬µ«ÕâÀïÒÀ¾É²»ÄÜÍêÈ«Âú×ãÎÒÃǵÄÐèÇó¡£
ÒýÈë TiSpark
Ëæºó£¬ÎÒÃǰÑÄ¿¹âתÏòÁË TiDB µÄÒ»¸ö×ÓÏîÄ¿ TiSpark£¨https://github.com/pingcap/tispark£©£¬ÓùÙÍøµÄ½éÉÜÀ´½²
TiSpark ¾ÍÊǽèÖú Spark ƽ̨£¬Í¬Ê±ÈÚºÏ TiKV ·Ö²¼Ê½¼¯ÈºµÄÓÅÊÆ£¬ºÍ TiDB Ò»Æð½â¾ö
HTAP µÄÐèÇó¡£TiDB-ansible ÖÐÒ²´øÓÐ TiSpark µÄÅäÖã¬ÓÉÓÚÎÒÃÇÒѾӵÓÐÁË Spark
¼¯Èº£¬ËùÒÔÖ±½ÓÔÚÏÖÓÐµÄ Spark ¼¯ÈºÖм¯³ÉÁË TiSpark¡£ËäÈ»¸ÃÏîÄ¿¿ª·¢²»¾Ã£¬µ«ÊǾ¹ý²âÊÔ£¬ÊÕÒæ·Ç³£Ã÷ÏÔ¡£
TiSpark µÄÅäÖ÷dz£¼òµ¥£¬Ö»ÐèÒª°Ñ TiSprak Ïà¹ØµÄ jar °ü·ÅÈë Spark ¼¯ÈºÖеÄ
jars Îļþ¼ÐÖоÍÄÜÒýÈë TiSpark£¬Í¬Ê±¹Ù·½Ò²ÌṩÁË 3 ¸ö½Å±¾£¬ÆäÖÐÁ½¸öÊÇÆô¶¯ºÍÍ£Ö¹ TiSpark
µÄ Thrift Server£¬ÁíÒ»¸öÊÇÌṩµÄ TiSpark µÄ cli ¿Í»§¶Ë£¬ÕâÑùÎÒÃǾÍÄÜÏñʹÓÃ
Hive Ò»ÑùʹÓà TiSpark È¥×ö²éѯ¡£
ÔÚ³õ²½Ê¹ÓÃÖ®ºó£¬ÎÒÃÇ·¢ÏÖһЩÖîÈç select count(*) from table µÈ SQL
Ïà±ÈÓÚ TiDB Óзdz£Ã÷ÏÔµÄÌáÉý£¬Ò»Ð©¼òµ¥µÄ OLAP µÄ²éѯ»ù±¾É϶¼Äܹ»ÔÚ 5 ÃëÖ®ÄÚ·µ»Ø½á¹û¡£¾¹ý³õ²½²âÊÔ£¬´óÖÂÔÚ
OLAP µÄ½áÂÛÈçÏ£ºÒ»Ð©¼òµ¥µÄ²éѯ SQL£¬ÔÚÊý¾ÝÁ¿°ÙÍò¼¶×óÓÒ£¬TiDB µÄÖ´ÐÐЧÂÊ¿ÉÄÜ»á±È TiSpark
¸üºÃ£¬ÔÚÊý¾ÝÁ¿Ôö¶àÖ®ºó TiSpark µÄÖ´ÐÐЧÂʻᳬ¹ý TiDB£¬µ±È»ÕâÒ²¿´ TiKV µÄÅäÖᢱí½á¹¹µÈ¡£ÔÚ
TiSpark µÄʹÓùý³ÌÖУ¬ÎÒÃÇ·¢ÏÖ TiSpark µÄ²éѯ½á¹ûÔÚ°ÙÍò¼¶Ê±£¬Ö´ÐÐʱ¼ä¶¼·Ç³£Îȶ¨£¬¶ø
TiDB µÄ²éѯʱ¼äÔò»áËæ×ÅÊý¾ÝÁ¿µÄÔö³¤¶øÔö³¤£¨¾¹ýÓë TiDB ¹Ù·½¹µÍ¨£¬Õâ¸öÇé¿öÖ÷ÒªÊÇÒòΪûÓбȽϺõÄË÷Òý½øÐÐÊý¾Ýɸѡ£©¡£Õë¶ÔÎÒÃǵĶ©µ¥±í×ö²âÊÔ£¬ÔÚÊý¾ÝÁ¿Îª½ü°ÙÍò¼¶Ê±£¬TiDB
µÄÖ´ÐÐʱ¼äΪ 2 Ãë×óÓÒ£¬TiSpark µÄÖ´ÐÐʱ¼äΪ 7 Ã룻µ±Êý¾ÝÁ¿Ôö³¤Îª½üǧÍò¼¶Ê±£¬TiDB
µÄÖ´ÐÐʱ¼ä´óÖÂΪ 12 Ã루²»¿¼ÂÇ»º´æ£©£¬TiSpark ÒÀ¾ÉΪ 7 Ã룬·Ç³£Îȶ¨¡£
Òò´Ë£¬ÎÒÃǾö¶¨½«Ò»Ð©¸´Ô ETL ½Å±¾Óà TiSpark À´ÊµÏÖ£¬¶ÔÉÏÊöµÄ¸´Ôӽű¾½øÐзÖÎöºó£¬ÎÒÃÇ·¢ÏÖ£¬´ó¶àÊý½Å±¾Öмä±íºÜ¶à£¬ÔÚ
SQL Server ÖÐÊÇͨ¹ý SQL Server ÄÚ´æ±íʵÏÖ£¬¶øÇ¨ÒÆÖÁ TiDB£¬Ã¿ÕÅÖмä±í¶¼ÒªÉ¾³ýºÍ²åÈëÂ䵨£¬ÕâЩ¿ªÏú´ó´óÔö¼ÓÁËÖ´ÐÐʱ³¤£¨¾Ý¹Ù·½´ð¸´
TiDB ºÜ¿ìÒ²»áÖ§³Ö View¡¢ÄÚ´æ±í£©¡£ÔÚÓÐÁË TiSpark Ö®ºó£¬ÎÒÃDZãÀûÓà TiSpark
½«Öмä±í»º´æÎª Spark µÄÄÚ´æ±í£¬Ö»ÐèÒª½«×îºóµÄÊý¾ÝÂ䵨»Ø TiDB£¬ÔÙÖ´ÐÐ Merge ²Ù×÷¼´¿É£¬ÕâÑùÊ¡µôÁ˺ܶàÖмäÊý¾ÝµÄÂ䵨£¬´ó´ó½ÚÊ¡Á˺ܶà½Å±¾Ö´ÐеÄʱ¼ä¡£
ÔÚ²éѯËٶȽâ¾öÖ®ºó£¬ÎÒÃÇ·¢Ïֽű¾ÖлáÓкܶàÕë¶ÔÖмä±í update ºÍ delete µÄÓï¾ä¡£Ä¿Ç°
TiSpark ÔÝʱ²»Ö§³Ö update ºÍ delete µÄ²Ù×÷£¨ºÍ TiSpark ×÷Õß¹µÍ¨£¬ºóÐø»á¿¼ÂÇÖ§³ÖÕâÁ½¸ö²Ù×÷£©£¬ÎÒÃDZ㳢ÊÔÁËÁ½ÖÖ·½°¸£¬Ò»²¿·ÖÖ´ÐÐÀàËÆÓÚ
Hive£¬²ÉÓà insert into Ò»ÕÅбíµÄ·½Ê½À´½â¾ö£»ÁíÍâÒ»²¿·Ö£¬ÎÒÃÇÒýÈëÁË Spark ÖеÄ
Snappydata ×÷Ϊһ²¿·ÖÄÚ´æ±í´æ´¢£¬ÔÚ Snappydata ÖнøÐÐ update ºÍ delete£¬ÒÔ´ïµ½ÏëÒªµÄÄ¿µÄ¡£ÒòΪ¶¼ÊÇ
Spark µÄÏîÄ¿£¬Òò´ËÔÚÈÚºÏÁ½¸öÏîÄ¿µÄʱºò»¹ÊDZȽÏÇáËɵġ£
×îºó£¬¹ØÓÚʵʱµÄµ÷¶È¹¤¾ß£¬Ä¿Ç°ÎÒÃÇÊǺÍÀëÏßµ÷¶ÈÒ»Æð½øÐе÷¶È£¬ÕâÒ²´øÀ´ÁËһЩÎÊÌ⣬ÿ´Î½Å±¾¶¼»á³õʼ»¯Ò»Ð©
Spark ²ÎÊýµÈ£¬ÕâÒ²Ï൱ºÄʱ¡£ÔÚδÀ´£¬ÎÒÃÇ´òËã²ÉÓà Spark Streaming ×÷Ϊµ÷¶È¹¤¾ß£¬Ã¿´ÎÖ´ÐÐÍê³ÉÖ®ºó¼Ç¼ʱ¼ä´Á£¬Spark
Streaming Ö»Ðè¼à¿ØÊ±¼ä´Á±ä»¯¼´¿É£¬Äܹ»±ÜÃâ¶à´Î³õʼ»¯µÄºÄʱ£¬Í¨¹ý Spark ¼à¿Ø£¬ÎÒÃÇÒ²Äܹ»Çå³þµÄ¿´µ½ÈÎÎñµÄÑÓ³ÙºÍһЩ״̬£¬ÕâÒ»²¿·Ö½«ÔÚδÀ´½øÐвâÊÔ¡£
TiDB ¹Ù·½Ö§³Ö
ÔÚÇ¨ÒÆ¹ý³ÌÖУ¬ÎÒÃǵõ½ÁË TiDB ¹Ù·½ºÜºÃµÄÖ§³Ö£¬ÆäÖÐÒ²°üÀ¨ TiSpark Ïà¹ØµÄ¼¼Êõ¸ºÔðÈË£¬Ò»Ð©
TiSpark µÄ Corner Case ¼°Ê¹ÓÃÎÊÌ⣬ÎÒÃǶ¼»áÔÚȺÀïÅ׳ö£¬TiDB µÄ¹Ù·½ÈËÔ±»á·Ç³£¼°Ê±µÄ°ïÖúÎÒÃǽâ¾öÎÊÌ⣬ÔÚ¹Ù·½Ö§³ÖÏ£¬ÎÒÃÇÇ¨ÒÆÖÁ
TiSpark µÄ¹ý³ÌºÜ˳Àû£¬Ã»ÓÐÊܵ½Ê²Ã´Ì«´óµÄ¼¼Êõ×è°¡£
ʵʱÊý²Ö TiDB / TiSpark
ÔÚÇ¨ÒÆÍê³ÉÖ®ºó£¬ÆäÖÐÒ»Ìõ¸´Ô SQL£¬Ò»¹² Join ÁË 12 ÕÅ±í£¨×î´ó±íÊýÁ¿ÒÚ¼¶£¬²¿·Ö±í°ÙÍò¼¶£©£¬ÔÚÆ½Ê±Ð¡ÅúÁ¿µÄÇé¿öÏ£¬Ö´ÐÐʱ¼ä»áÔÚ
5 ·ÖÖÓ×óÓÒ£¬ÎÒÃÇÒ²ÄÃÁËË«11È«Á¿µÄÊý¾Ý½øÐÐÁ˲âÊÔ£¬Ö´ÐÐʱ¼äÔÚ 9 ·ÖÖÓÒÔÉÏ£¬¶ø²ÉÓÃÁË TiSpark
µÄ·½Ê½È¥Ö´ÐУ¬Ë«11È«Á¿µÄÊý¾ÝÒ²½ö½ö»¨ÁË 1 ·ÖÖÓ£¬ÐÔÄÜÌáÉýÁË 9 ±¶¡£Õû¸ö´ó½Å±¾ÔÚ SQL Server
ÉÏÔËÐÐË«11µÄÈ«Á¿Êý¾ÝÒÔǰÖÁÉÙÒªÏûºÄ 30 ·ÖÖÓ£¬ÀûÓà TiDB È¥Ö´ÐдóÖÂÐèÒª 20 ·ÖÖÓ×óÓÒ£¬ÀûÓÃ
TiSpark Ö»ÐèÒª 8 ·ÖÖÓ×óÓÒ£¬Ïà¶Ô SQL Server ÐÔÄÜÌáÉý 4 ±¶£¬Ò²¾ÍÊÇ˵£¬Ã¿ÄêÊý¾ÝÁ¿×î¸ß·åµÄ´¦ÀíÄÜÁ¦´ïµ½ÁË·ÖÖÓ¼¶£¬ºÜºÃµÄÂú×ãÁËÎÒÃǵÄÐèÇó¡£
×îºó£¬²»¹ÜÊÇÓà TiDB »¹ÊÇÓà TiSpark ¶¼»áÓÐÒ»²¿·ÖÖмä±íÒÔ¼°ÓëÔ±í½øÐÐ Merge µÄ²Ù×÷£¬ÕâÀïÓÉÓÚ
TiDB ¶ÔÊÂÎñ½øÐеÄÏÞÖÆ£¬ÎÒÃÇÒ²²ÉÓÃÒÔÍòÌõΪµ¥Åú´Î½øÐÐÅúÁ¿µÄ²åÈëºÍ Merge£¬¼È±ÜÃâÁ˳¬¹ýÊÂÎñµÄ±¨´íÓÖ·ûºÏ
TiDB µÄÉè¼ÆÀíÄÄܹ»´ïµ½×î¼Ñʵ¼ù¡£
ÓÐÁË TiSpark Õâ¸öÏîÄ¿£¬TiDB Óë Hadoop µÄÉú̬ÌåϵµÃµ½½øÒ»²½µÄÈںϣ¬ÔÚûÓÐ TiSpark
֮ǰ£¬ÎÒÃǵÄϵͳÉè¼ÆÈçÏ£º

ͼ£º¶àÌ×Êý²Ö²¢´æ
¿ÉÒÔ·¢ÏÖ£¬ÊµÊ±Êý²ÖÓë T+1 Òì²½Êý²ÖÊÇÁ½¸öÏà¶Ô¶ÀÁ¢µÄϵͳ£¬²¢Ã»ÓÐÈκν»¼¯£¬ÎÒÃÇÐèÒª½øÐÐÊý¾ÝʵʱµÄͬ²½£¬Í¬Ê±Ò²»áÔÚÒ¹Íí×öÒ»´ÎÒ첽ͬ²½£¬²»¹ÜÊÇ
Datax »¹ÊÇ Sqoop ¶ÁÈ¡¹ØÏµÐÍÊý¾Ý¿âµÄЧÂʶ¼Ô¶Ô¶´ï²»µ½ TiSpark µÄËÙ¶È£¬¶øÔÚÓÐÁË
TiSpark Ö®ºó£¬ÎÒÃÇ¿ÉÒÔ¶Ô T+1 Òì²½Êý²Ö½øÐÐÕûºÏ£¬ÓÚÊÇÎÒÃǵļܹ¹½ø»¯ÎªÈçÏ£º

ͼ£ºTiDB / TiSpark ʵʱÊý²Öƽ̨
ÕâÑù¾ÍÄܹ»ÀûÓà TiSpark ½« TiDB ºÍ Hadoop ºÜºÃµÄ´®ÁªÆðÀ´£¬»¥Îª²¹³ä£¬TiDB
µÄ¹¦ÄÜÒ²Óɵ¥´¿µÄʵʱÊý²Ö±ä³ÉÄܹ»ÌṩÈçϼ¸¸ö¹¦ÄÜ»ìºÏÊý¾Ý¿â£º
1. ʵʱÊý²Ö£¬ÉÏÓÎ OLTP µÄÊý¾Ýͨ¹ý TiDB ʵʱдÈ룬ÏÂÓÎ OLAP µÄÒµÎñͨ¹ý TiDB
/ TiSpark ʵʱ·ÖÎö¡£
2. T+1 µÄ³éÈ¡Äܹ»´Ó TiDB ÖÐÀûÓà TiSpark ½øÐгéÈ¡¡£
TiSpark ËÙ¶ÈÔ¶Ô¶³¬¹ý Datax ºÍ Sqoop ¶ÁÈ¡¹ØÏµÐÍÊý¾Ý¿âµÄËÙ¶È£»
³éÈ¡¹¤¾ßÒ²²»ÓÃά»¤¶à¸öϵͳ¿â£¬Ö»ÐèҪά»¤Ò»¸ö TiDB ¼´¿É£¬´ó´ó·½±ãÁËÒµÎñµÄͳһʹÓ㬻¹½ÚÊ¡Á˶à´Îά»¤³É±¾¡£
TiDB ÌìÈ»·Ö²¼Ê½µÄÉè¼ÆÒ²±£Ö¤ÁËϵͳµÄÎȶ¨¡¢¸ß¿ÉÓá£
3. TiDB ·Ö²¼Ê½ÌØÐÔ¿ÉÒÔºÜºÃµÄÆ½ºâÈȵãÊý¾Ý£¬¿ÉÒÔÓÃËü×÷ΪҵÎñ¿âÈȵãÊý¾ÝµÄÒ»¸ö±¸·Ý¿â£¬»òÕßÖ±½ÓǨÈë
TiDB ¡£
ÉÏÃæÕâÈýµãÒ²ÊÇÎÒÃǽñºóȥŬÁ¦µÄ·½Ïò£¬Óɴ˿ɼû£¬TiSpark ²»½ö¶ÔÓÚ ETL ½Å±¾Æðµ½Á˺ÜÖØÒªµÄ×÷Óã¬ÔÚÎÒÃǽñºóµÄ¼Ü¹¹ÖÐÒ²Æðµ½Á˾Ù×ãÇáÖØµÄ×÷Óã¬ÎªÎÒÃÇ´´½¨Ò»¸öʵʱµÄͳһµÄ»ìºÏÊý¾Ý¿âÌṩÁË¿ÉÄÜ¡£
Óë´Ëͬʱ£¬ÎÒÃÇÒ²µÃµ½ TiDB ¹Ù·½ÈËÔ±µÄÈ·ÈÏ£¬TiDB ½«ÓÚ½üÆÚÖ§³ÖÊÓͼ¡¢·ÖÇø±í£¬²¢»á³ÖÐøÔöÇ¿
SQL ÓÅ»¯Æ÷£¬Í¬Ê±Ò²»áÌṩһ¿îÃûΪ TiDB Wormhole µÄÒ칹ƽ̨Êý¾ÝÊµÊ±Ç¨ÒÆ¹¤¾ßÀ´±ã½ÝµÄÖ§³ÖÓû§µÄ¶àÔª»¯Ç¨ÒÆÐèÇó¡£ÎÒÃÇÒ²¼Æ»®½«¸ü¶àµÄ²úÆ·ÏßÖð²½Ç¨Èë
TiDB¡£
×ܽá
ͬʱ½â¾ö OLAP ºÍ OLTP ÊÇÒ»¼þÏ൱À§ÄѵÄÊÂÇ飬TiDB ºÍ TiSpark ËäÈ»ÍÆ³ö²»¾Ã£¬µ«ÊÇÒѾÂú×ãºÜ¶àÓ¦Óó¡¾°£¬Í¬Ê±ÔÚÒ×ÓÃÐԺͼ¼ÊõÖ§³ÖÉÏÒ²·Ç³£ÖµµÃ³ÆÔÞ£¬ÏàÐÅ
TiDB Ò»¶¨Äܹ»ÔÚÔ½À´Ô½¶àµÄÆóÒµÖеõ½¹ã·ºÓ¦Óᣠ|