±à¼ÍƼö: |
±¾ÎÄÀ´×ÔÓÚweixin£¬±¾ÎÄÖ÷Òª½éÉÜÁ˸ßÐÔÄÜʱ¼äÐòÁÐÊý¾Ý¿â
(High-Performance Time Series Database ,
¼ò³Æ HiTSDB) ÊÇÒ»ÖÖ¸ßÐÔÄÜ£¬µÍ³É±¾£¬Îȶ¨¿É¿¿µÄÔÚÏßʱÐòÊý¾Ý¿â·þÎñ,Ï£ÍûÄܶÔÄúÓÐËù°ïÖú¡£ |
|

±³¾°
HiTSDBʱÐòÊý¾Ý¿âÒýÇæÔÚ·þÎñÓÚ°¢Àï°Í°Í¼¯ÍÅÄڵĿͻ§Ê±£¬ ¸ù¾Ý¼¯ÍÅÒµÎñÌØÐÔ×öÁ˺ܶàÕë¶ÔÐÔµÄÓÅ»¯¡£ È»¶øÔÚHiTSDBÔÆ²úÆ·µÄ´òÄ¥¹ý³ÌÖÐÖð½¥·¢ÏÖ£¬ºÜ¶àÕë¶ÔÐÔµÄÓÅ»¯ºÜÄÑÔÚ¹«ÓÐÔÆÉÏÕë¶ÔÌØ¶¨Óû§È¥ÊµÊ©¡£
ÓÚ´Ëͬʱ£¬ ÔÚ¹«ÓÐÔÆ¿Í»§Ê¹ÓÃHiTSDBµÄ¹ý³ÌÖУ¬·¢ÏÖÁËÔ½À´Ô½¶àÓÉÓھۺϲéѯµ¼ÖµÄÎÊÌ⣬±ÈÈ磺
·µ»ØÊý¾Ýµã¹ý¶à»á³öÏÖÕ»Òç³öµÈ´íÎ󣬾ۺϵã¹ý¶àµ¼ÖÂOOM£¬ »òÕßÎÞ·¨Íê³É¾ÛºÏ£¬ÊµÀýÍêÈ«¿¨ËÀµÈµÈÎÊÌâ¡£ÕâЩÎÊÌâÖ÷ÒªÓÉÓÚÔʼµÄ¾ÛºÏÒýÇæ¼Ü¹¹ÉϵÄȱÏݵ¼Ö¡£
Òò´ËHiTSDB¿ª·¢ÍŶӯÀ¹Àºó¾ö¶¨Î§ÈÆÐµľۺÏÒýÇæ¼Ü¹¹¶ÔHiTSDBÒýÇæ½øÐÐÉý¼¶£¬°üº¬£º ´æ´¢Ä£Ð͵ĸÄÔ죬Ë÷Òý·½Ê½µÄÉý¼¶£¬ÊµÏÖȫеÄÁ÷ʽ¾ÛºÏ£¬Êý¾ÝÇ¨ÒÆ£¬ÐÔÄÜÆÀ²â¡£ ±¾ÎÄÖ÷ÒªÎ§ÈÆÕâ5¸ö·½Ãæ½øÐÐÊáÀí£¬ÖØµãÔÚ¡°È«ÐµÄÁ÷ʽ¾ÛºÏ²¿·Ö¡±¡£
1. ʱÐòÊý¾Ý´æ´¢Ä£ÐÍ£º
1.1 ʱÐòµÄÊý¾Ý´æ´¢¸ñʽ¡£
Ò»¸öµäÐ͵ÄʱÐòÊý¾ÝÓÉÁ½¸öά¶ÈÀ´±íʾ£¬Ò»¸öά¶È±íʾʱ¼äÖá£¬Ëæ×Åʱ¼äµÄ²»¶ÏÁ÷È룬Êý¾Ý»á²»¶ÏµØ×·¼Ó¡£ ÁíÍâÒ»¸öά¶ÈÊÇʱ¼äÏߣ¬ÓÉÖ¸±êºÍÊý¾ÝÔ´×é³É£¬Êý¾ÝÔ´¾ÍÊÇÓÉһϵÁеıêÇ©±êʾµÄΨһÊý¾Ý²É¼¯µã¡£ÀýÈçÖ¸±êcpu.usageµÄÊý¾ÝÀ´×ÔÓÚ»ú·¿£¬Ó¦Óã¬ÊµÀýµÈά¶È×éºÏ³ÉµÄ²É¼¯µã¡£ ÕâÑù´ó¼ÒÂß¼ÉϾͿÉÒÔ³éÏó³öÀ´Ò»¸öid+{timestamp, value}µÄʱÐòÊý¾ÝÄ£ÐÍ¡£ÕâÖÖÊý¾ÝÄ£Ð͵Ĵ洢ÊÇÈçºÎÄØ¡£Ò»°ãÓÐÁ½ÖÖµäÐ͵ÄÊý¾Ý´æ´¢Ë¼Â·£º
Ò»ÖÖ°´ÕÕʱ¼ä´°¿Úά¶È»®·ÖÊý¾Ý¿é£¬Í¬Ò»¶Î×ÔȻʱ¼ä´°¿ÚÄÚµÄÁ¬ÐøÊý¾Ý·Åµ½ÏàÁÚµÄλÖ㬱ÈÈç{1:00,
2:00}->(id1, id2, id3, ... ... ,idN)¡£ ²ÉÓÃÕâÖÖ·½Ê½µÄµäÐÍʱÐòÊý¾Ý¿â°üº¬InfluxDB,
PromethuesµÈµÈTSMT½á¹¹µÄÊý¾Ý¿â¡£OpenTSDBÓÐÐ©ÌØÊ⣬ÒòΪOpenTSDBÊǵ¥ÖµÄ£ÐÍ£¬Ö¸±êÕâ¸öά¶ÈÔÚ²éѯµÄʱºòÊDZشøµÄ¡£
ËùÒÔ¿ÉÒÔÏȰ´ÕÕÖ¸±ê×öÁËÒ»¼¶»®·Ö£¬ÔÙ¸ù¾Ýʱ¼ä´°¿Ú×ö¶þ¼¶µÄ»®·Ö£¬±¾ÖÊÉÏ»¹ÊÇͬһʱ¼ä´°¿ÚÄÚµÄÁ¬ÐøÊý¾Ý¡£ °´ÕÕʱ¼ä´°¿ÚÇзֵķ½Ê½£¬
ÓÅÊÆÊÇдÈëµÄʱºò¿ÉÒÔºÜÌìÈ»µÄ°´ÕÕ´°¿ÚÈ¥ÂäÅÌ£¬¶ÔÓÚ¸ßγ¶ÈµÄ±êÇ©²éѯ»ù±¾ÉÏÊÇһЩÁ¬ÐøScan. ÕâÖÖ·½Ê½Óиö±È½ÏÄѽâµÄÎÊÌâ¾ÍÊÇ"out
of order"ÂÒÐòÎÊÌ⣬¶ÔÓÚʱ¼ä´°¿Ú¹ýÆÚºóÔÙÀ´µÄʱ¼äµã£¬ PromethuesÖ±½Ó²ÉÓöªÆúµÄ·½Ê½£¬InfluxDBÔÚÕâÖÖÇé¿öÏÂÐÔÄÜ»áÓÐËðºÄ¡£
ÁíÍâÒ»ÖÖ°´ÕÕʱ¼äÏßά¶È»®·ÖÊý¾Ý¿é£¬Í¬Ò»Ê±¼äÏßµÄÊý¾Ý·Åµ½ÏàÁÚµÄλÖ㬱ÈÈç(id1)->(1:00, 2:00, 3:00,
... ... , 23:00)¡£ HiTSDB²ÉÓÃʱ¼äÏßά¶È»®·ÖµÄ·½Ê½£ºÄ¿Ç°ÂäÅÌÊý¾Ý´æ´¢ÓÚHBASE£¬
µ×²ãRowkeyÓÉÖ¸±ê+±êÇ©+×ÔÈ»´°¿ÚµÄ·½Ê½×éºÏ¶ø³É. Rowkey°´ÕÕ´óС˳ÐòºÏ²¢Ä³¸öʱ¼äÏßµÄÊý¾ÝµãÊÇÁ¬ÐøÏàÁڵġ£
Òò´Ë¶ÔÓÚһЩµÍάµÄ²éѯЧÂÊÊǷdz£¸ßЧµÄ¡£¸ù¾ÝĿǰ½Ó´¥µÄһЩÎïÁªÍø·þÎñ£¬¸ü¶àµÄÊÇһЩµÍάµÄ·ÃÎÊ¡£¶ÔÓÚÖеÈά¶ÈµÄ²éѯ²ÉÓÃÁ÷ʽscan¡£¶ÔÓÚ¼«¸ßγ¶È±êÇ©µÄ²éѯHiTSDB²ÉÓÃÔ¤¾ÛºÏµÄ·þÎñ(²»ÔÚ±¾ÎÄÌÖÂÛ·¶Î§ÄÚ)¡£

1.2 ʱÐòÄ£Ð͵ÄÈȵãÎÊÌâ´¦Àí
Éú²ú»·¾³ÖÐÒµÎñ·½²É¼¯µÄÖ¸±êÀàÐͶàÖÖ¶àÑù£¬¶ÔÖ¸±êµÄ²É¼¯ÖÜÆÚ¸÷²»Ïàͬ¡£±ÈÈçcpu.usageÕâ¸öÖ¸±êµÄ±ä»¯ÆµÂʱȽϿ죬ҵÎñ·½¹Ø×¢¶È¸ß£¬²É¼¯ÖÜÆÚͨ³£ºÜ¶Ì£¬1Ã룬5Ã룬10ÃëµÈµÈ¡£ È»¶øÖ¸±êdisk.usageÕâ¸öÖ¸±ê±ä»¯Ç÷ÊÆÏà¶Ôƽ»¬£¬²É¼¯ÖÜÆÚͨ³£Îª1·ÖÖÓ£¬5·ÖÖÓ£¬ 10·ÖÖӵȡ£ÕâÖÖÇé¿öÏ£¬Êý¾ÝµÄ´æ´¢Èç¹ûÕë¶Ôͬһ¸öÖ¸±ê²»×öÌØÊâ´¦Àí£¬ÈÝÒ×ÐγÉÈȵãÎÊÌâ¡£ ¼ÙÉè°´ÕÕÖ¸±êÀàÐͽøÐд洢×ÊÔ´µÄ·ÖƬ£¬ÏëÏóÒ»ÏÂÈç¹ûÓÐ20¸öÒµÎñ£¬Ã¿¸öÒµÎñ10¸ö¼¯Èº£¬Ã¿¸ö¼¯Èº500̨Ö÷»ú£¬²É¼¯ÖÜÆÚÊÇ1ÃëµÄ»°£¬Ã¿Ãë¾Í»áÓÐ10Íò¸öcpu.usageµÄÖ¸±êÊý¾ÝµãÂ䵽ͬһ¸ö´æ´¢×ÊԴʵÀýÖУ¬ ¶ødisk.usage²É¼¯ÖÜÆÚΪ1·ÖÖÓ£¬ËùÒÔ´óÔ¼Ö»ÓÐ1666¸öÖ¸±êÊý¾ÝµãÂäµ½ÁíÍâÒ»¸ö´æ´¢×ÊÔ´ÉÏ£¬ÕâÑùÊý¾ÝÇãбµÄÏÖÏó·Ç³£ÑÏÖØ¡£
1.2.1 ·ÖͰ
ÕâÀàÎÊÌâµÄ¾µä½â·¨¾ÍÊÇ·ÖͰ¡£±ÈÈç³ýÁËÖ¸±êÀàÐÍÍ⣬ͬʱ½«ÒµÎñÃûºÍÖ÷»úÃû×÷Ϊά¶È±êʶtags£¬°ÑÖ¸±êcpu.usage»®·Öµ½²»Í¬µÄͰÀïÃæ¡£ дÈëʱ¸ù¾Ýʱ¼äÏß¹þÏ£Öµ·ÖɢдÈëµ½²»Í¬µÄͰÀïÃæ¡£ OpenTSDBÔÚ´¦ÀíÈȵãÎÊÌâÒ²ÊDzÉÓÃÁË·ÖͰģʽ£¬µ«ÊÇÐèÒª¹ã²¥¶ÁÈ¡£¬¸ù±¾ÔÒòÔÚÓÚ²éѯ·½Ê½ÐèÒªÔÚij¸öʱ¼ä´°¿ÚÄÚµÄÈ«¾ÖɨÃè¡£ ËùÒÔÉèÖÃOpenTSDBµÄ·ÖͰÊýÁ¿ÐèÒªÒ»¸öƽºâ²ßÂÔ£¬Èç¹ûÊýÁ¿Ì«ÉÙ£¬Èȵ㻹ÊÇÓоֲ¿ÐÔµÄÎÊÌ⣬Èç¹ûÌ«¶à£¬²éѯʱ¹ã²¥¶Á´øÀ´µÄ¿ªÏú»á·Ç³£´ó¡£
ÓëÆäÏà±È½Ï£¬HiTSDB±ÜÃâÁ˹㲥¶Á£¬Ìá¸ßÁ˲éѯЧÂÊ¡£ÓÉÓÚHiTSDBÔÚ²éѯʱ£¬Ï·¢µ½µ×²ã´æ´¢É¨ÃèÊý¾Ý֮ǰ£¬Ê×ÏÈ»á¸ù¾Ý²éѯÓï¾äµÃµ½¾«È·ÃüÖеÄʱ¼äÏß¡£ ÓÐÁ˾ßÌåµÄʱ¼äÏ߾ͿÉÒÔÈ·¶¨Í°µÄλÖã¬È»ºóµ½ÏàÓ¦µÄ¿éÇøÓòÈ¡Êý¾Ý£¬²»´æÔڹ㲥¶Á¡£ ¹ØÓÚHiTSDBÈçºÎÔÚ²éѯÊý¾ÝµÄʱºò»ñÈ¡ÃüÖеÄʱ¼äÏߣ¬ÏàÐŶÁÕßÕâ¸öÒÉÎÊ»áÔÚ¶ÁÈ¡Íêµ¹ÅÅÕâÒ»½ÚµÄʱºòÏûÊÍ¡£
1.2.2 Region Pre-Split
µ±Ò»¸ö±í¸Õ±»´´½¨µÄʱºò£¬HBaseĬÈÏ·ÖÅäÒ»¸öRegion¸øÐÂ±í¡£ËùÓеĶÁдÇëÇó¶¼»á·ÃÎʵ½Í¬Ò»¸öregionServerµÄͬһ¸öregionÖС£ ´Ëʱ¼¯ÈºÖÐµÄÆäËûregionServer»á´¦ÓڱȽϿÕÏеÄ״̬£¬Õâ¸öʱºò¾Í´ï²»µ½¸ºÔؾùºâµÄЧ¹ûÁË¡£ ½â¾öÕâ¸öÎÊÌâʹÓÃpre-split,ÔÚ´´½¨Ð±íµÄʱºò¸ù¾Ý·ÖͰ¸öÊý²ÉÓÃ×Ô¶¨ÒåµÄpre-splitµÄËã·¨£¬Éú³É¶à¸öregion¡£ byte[][] splitKeys =new byte[bucketNumber-1][]; splitKeys[bucketIndex-1] = (bucketIndex&0xFF);

2. µ¹ÅÅË÷Òý£º
2.1 ʱÐòÊý¾ÝÖеĶàάʱ¼äÏß
¶àά֧³Ö¶ÔÓÚÈκÎÐÂÒ»´úʱÐòÊý¾Ý¿â¶¼ÊǼ«ÆäÖØÒªµÄ¡£ ʱÐòÊý¾ÝµÄÀàÐͶàÖÖ¶àÑù,À´Ô´¸üÊǷdz£¸´ÔÓ,²»Ö¹Óе¥Ò»Î¬¶ÈÉÏ»ùÓÚʱ¼äµÄÓÐÐòÊýÖµ£¬»¹Óжàάʱ¼äÏßÏà¹ØµÄ´óÁ¿×éºÏ¡£ ¾Ù¸ö¼òµ¥Àý×Ó£¬cpuµÄload¿ÉÒÔÓÐÈý¸öά¶ÈÃèÊöcpu core£¬ host£¬ appÓ¦Óã¬Ã¿¸öά¶È¿ÉÒÔÓаټ¶±ðÉõÖÁÍò¼¶±ðµÄ±êǩֵ¡£ sys.cpu.load cpu=1 host=ipA app=hitsdb£¬¸÷¸öά¶È×éºÏºóʱ¼äÏß¿ÉÒÔÇáËÉ´ïµ½°ÙÍò¼¶±ð¡£ ÈçºÎ¹ÜÀíÕâЩʱ¼äÏߣ¬½¨Á¢Ë÷Òý²¢ÇÒÌṩ¸ßЧµÄ²éѯÊÇʱÐòÊý¾Ý¿âÀïÃæÐèÒª½â¾öµÄÖØÒªÎÊÌâ¡£ ĿǰʱÐòÁìÓò±È½ÏÖ÷Á÷µÄ×ö·¨ÊDzÉÓõ¹ÅÅË÷ÒýµÄ·½Ê½¡£
2.2 µ¹ÅÅË÷Òý»ù±¾×éºÏ
»ù±¾µÄʱ¼äÏßÔÚµ¹ÅÅÖеÄ×éºÏ˼·ÈçÏ£º
ʱ¼äÏßµÄÔʼÊäÈëÖµ£º

µ¹Åʹ½¨ºó£º

²éѯʱ¼äÏß cpu=3 and host=ipB£º

È¡½»¼¯ºó²éѯ½á¹ûΪ7£º

2.3 µ¹ÅÅÃæÁÙµÄÎÊÌâÒÔ¼°ÓÅ»¯Ë¼Â·
µ¹ÅÅÖ÷ÒªÃæÁÙµÄÊÇÄÚ´æÅòÕ͵ÄÎÊÌ⣺
posting list¹ý³¤£¬ ¶ÔÓÚ¸ßγ¶ÈµÄtag£¬±ÈÈç¡°»ú·¿=º¼ÖÝ¡±£¬º¼ÖÝ¿ÉÄÜ»áÓÐǧ¼¶±ðÉõÖÁÍò¼¶±ðµÄ»úÆ÷£¬Õâ¾ÍÒâζ×Åposting listÐèÒª´æ´¢³ÉǧÉÏÍò¸ö64-bitµÄid¡£ ½â¾öÕâ¸öÎÊÌâµÄ˼·ÊDzÉÓÃѹËõposting listµÄ·½Ê½£¬ ÔÚ¹¹½¨posting listµÄʱºò¶ÔÊý×éÀïÃæµÄid½øÐÐÅÅÐò£¬È»ºó²ÉÓÃdelta±àÂëµÄ·½Ê½Ñ¹Ëõ¡£
Èç¹ûTag¼üÖµ¶ÔÖ±½Ó×÷ΪtermʹÓã¬ÄÚ´æÕ¼ÓÃÈ¡¾öÓÚ×Ö·û´®µÄ´óС£¬²ÉÓÃ×Ö·û´®×ֵ仯£¬Ò²¿É´ó´ó¼õÉÙÄڴ濪Ïú¡£
3. Á÷ʽ¾ÛºÏÒýÇæ
3.1 HiTSDB¾ÛºÏÒýÇæµÄ¼¼ÊõÍ´µã
HiTSDBÏÖÓоۺÏÒýÇæ¹«ÓÐÔÆ¹«²âÒÔ¼°¼¯ÌåÄÚ²¿ÒµÎñÔËÐÐÖУ¬±©Â¶·¢ÏÖÁËÒÔÏÂÎÊÌ⣺
3.1.1 MaterializationÖ´ÐÐģʽÔì³ÉHeapÄÚ´æÒ×´ò±¬
ÏÂͼÏÔʾÁËÔ²éѯÒýÇæµÄ¼Ü¹¹Í¼¡£HiTSDBÒÔHBase×÷Ϊ´æ´¢£¬ÔÒýÇæÍ¨¹ýAsync HBase client ´ÓHBase»ñȡʱÐòÊý¾Ý¡£ÓÉÓÚHBaseµÄÊý¾Ý¶ÁÈ¡ÊÇÒ»¸öºÄʱµÄ¹ý³Ì£¬Í¨³£µÄ½â·¨ÊDzÉÓÃÒì²½HBase clientµÄAPI£¬´Ó¶øÓÐЧÌá¸ßϵͳµÄ²¢ÐÐÐÔ¡£µ«Ô¾ÛºÏÒýÇæ²ÉÓÃÁËÒ»ÖÖµäÐ͵ÄmaterializationµÄÖ´Ðз½Ê½£º1£©Æô¶¯¶à¸öÒì²½HBase APIÆôHBase¶Á£¬2£©Ö»Óе±²éѯËùÉæ¼°µÄÈ«²¿Ê±ÐòÊý¾Ý¶ÁÈëµ½ÄÚ´æÖк󣬾ۺÏÔËËã²Å¿ªÊ¼Æô¶¯¡£ÕâÖÖ°ÑHBase Scan½á¹ûÏÈÔÚÄÚ´æÖÐmaterializedÔپۺϵķ½Ê½Ê¹µÃHiTSDBÈÝÒ×·¢ÉúHeapÄÚ´æ´ò±¬µÄÏÖÏó¡£ÓÈÆäµ±Óû§½øÐдóʱ¼ä·¶Î§²éѯ£¬»òÕß²éѯµÄʱ¼äÏßµÄÊý¾Ý·Ç³£¶àµÄʱºò£¬ÒòÎªÉæ¼°µÄʱÐòÊý¾Ý¶à£¬HiTSDB»á·¢ÉúHeap OOM¶øµ¼Ö²éѯʧ°Ü¡£

3.1.2 ´ó²éѯ´ò±¬HBaseµÄÎÊÌâ
Á½¸öÔÒòÔì³ÉHiTSDB´¦Àí¾ÛºÏ²éѯµÄʱºò£¬ÈÝÒ×·¢Éú½«µ×²ãHBase´ò±¬¡£
HBase ¿ÉÄܶÁÈ¡¶àÓàʱ¼äÏßÊý¾Ý¡£HiTSDBµÄʱ¼äÏß²ÉÓÃÖ¸±ê+ʱ¼ä´°¿Ú+±êÇ©µÄ±àÂ뷽ʽ´æ´¢ÔÚHBase¡£µäÐ͵IJéѯÊÇÓû§Ö¸¶¨Ò»¸öÖ¸±ê£¬Ê±¼ä·¶Î§£¬ÒÔ¼°¿Õ¼äά¶ÈÉϱêǩҪѰÕ񵀮¥ÅäÖµ¡£¿Õ¼äά¶ÈµÄ±êÇ©²éѯÌõ¼þ²¢²»¶¼ÊÇÔÚ±êÇ©±àÂëǰ׺¡£µ±ÕâÖÖÇé¿ö·¢Éúʱ£¬HiTSDBµ¹ÅÅË÷Òý²»Äܸù¾Ý¿Õ¼äά¶ÈµÄ²éѯÌõ¼þ£¬¾«È·¶¨Î»µ½¾ßÌåµÄHBaseµÄ²éѯÌõ¼þ£¬¶øÊDzÉÓÃÏȶÁÈ¡ÔÙ¹ýÂ˵ķ½Ê½¡£ÕâÒâζ×ÅHBaseÓпÉÄܶÁÈ¡ºÜ¶àÈßÓàÊý¾Ý£¬´Ó¶ø¼ÓÖØHBaseµÄ¸ºÔØ¡£
HiTSDBÓпÉÄÜÔÚ¶Ìʱ¼äÄÚÏ·¢Ì«¶àHBase¶ÁÇëÇó¡£Ò»·½Ã棬HiTSDBÔÚHBase²ÉÓÃ·ÖÆ¬´æ´¢·½Ê½£¬¶Ôÿһ¸ö·ÖƬ£¬¶¼ÖÁÉÙÆô¶¯Ò»¸ö¶ÁÇëÇó£¬ÁíÒ»·½Ã棬ÒòΪÉÏÃæÌáµ½µÄmaterializationµÄÖ´Ðз½Ê½£¬Ò»¸ö²éÑ¯Éæ¼°µ½µÄHBase¶ÁÇëÇóͬʱÒì²½Ìá½»£¬ÓпÉÄÜÔںܶÌʱ¼äÄÚÏòHBaseÏ·¢´óÁ¿µÄ¶ÁÇëÇó¡£ÕâÑù£¬Ò»¸ö´ó²éѯ¾ÍÓпÉÄܰѵײãµÄHBase´ò±¬¡£
µ±ÕâÖÖÇé¿ö·¢Éúʱ£¬¸üÔã¸âµÄ³¡¾°ÊÇHiTSDBÎÞ·¨´¦ÀíʱÐòÊý¾ÝµÄдÈëÇëÇó£¬Ôì³ÉºóÐøÐÂÊý¾ÝµÄ¶ªÊ§¡£
3.1.3 Ö´Ðмܹ¹¸ß¶ÈñîºÏ£¬Ð޸ĻòÔö¼Ó¹¦ÄÜÀ§ÄÑ
¾ÛºÏÒýÇæÖ÷ÒªÕë¶ÔÓ¦Óó¡¾°ÊÇÐÔÄÜ¼à¿Ø£¬²éѯģʽ¹Ì¶¨£¬ËùÒÔÒýÇæ¼Ü¹¹²ÉÓõ¥Ò»Ä£Ê½£¬°Ñ²éѯ£¬¹ýÂË£¬ÌîÖµ/²åÖµ£¬ºÍ¾ÛºÏÔËËãµÄÂß¼¸ß¶ÈñîºÏÔÚÒ»Æð¡£ÕâÖÖÒýÇæ¼Ü¹¹¶ÔÓÚ¼à¿ØÓ¦ÓõĹ̶¨²éѯûÓÐÌ«¶àÎÊÌ⣬µ«HiTSDBÄ¿±ê²»½ö½öÊÇ¼à¿Ø³¡¾°Ïµļòµ¥²éѯ£¬¶øÊÇ×ÅÑÛÓÚ¸ü¶àÓ¦Ó󡾰ϵĸ´ÔÓ²éѯ¡£
ÎÒÃÇ·¢ÏÖ²ÉÓÃÔÓÐÒýÇæµÄ¼Ü¹¹£¬ºÜÄÑÔÚÔÓлù´¡ÉϽøÐÐÔö¼Ó¹¦ÄÜ£¬»òÐÞ¸ÄÔÀ´µÄʵÏÖ¡£±¾ÖÊÉϵÄÔÒòÔÚÓÚÔÓоۺÏÒýÇæÃ»ÓвÉÓô«Í³Êý¾Ý¿âËùͨ³£²ÉÓõÄÖ´Ðмܹ¹£¬Ö´ÐвãÓɿɶ¨ÖƵĶà¸öÖ´ÐÐËã×Ó×é³É£¬²éѯÓïÒå¿ÉÒÔÓɲ»Í¬µÄÖ´ÐÐËã×Ó×éºÏ¶øÍê³É¡£Õâ¸öÎÊÌâÔÚ²úÆ·¿ª·¢¿ªÊ¼½×¶Î²¢²»¸ÐÊܺÜÉµ«È·ÊÇÑÏÖØÓ°ÏìHiTSDBÍØ¿íÓ¦Óó¡¾°£¬Ôö¼Óй¦ÄܵÄÒ»¸öÖØÒªÒòËØ¡£
3.1.4 ¾ÛºÏÔËËãЧÂÊÓдýÌá¸ß
ÔÓÐÒýÇæÔÚÖ´ÐоۺÏÔËËãµÄʱºò£¬Ò²ºÍ´«Í³Êý¾Ý¿âËùͨ³£²ÉÓõÄiterativeÖ´ÐÐģʽһÑù£¬µü´úÖ´ÐоۺÏÔËËã¡£ÎÊÌâÔÚÓÚÿ´ÎiterationÖ´ÐУ¬·µ»ØµÄÊÇÒ»¸öʱ¼äµã¡£Iterative Ö´ÐÐÿ´Î·µ»ØÒ»Ìõʱ¼äµã£¬»òÕßÒ»Ìõ¼Ç¼£¬³£¼ûÓÚOLTPÕâÑùµÄ³¡¾°£¬ÒòΪOLTPµÄ²éѯËùÐèÒª·ÃÎʵļǼÊýºÜС¡£µ«¶ÔHiTSDB²éѯÓпÉÄÜÐèÒª·ÃÎÊ´óÁ¿Ê±¼äÏßÊý¾Ý£¬ÕâÑùµÄÖ´Ðз½Ê½Ð§ÂÊÉϲ¢²»¿ÉÈ¡¡£
ÔÒò1£©Ã¿´Î´¦ÀíÒ»¸öʱ¼äµã£¬¶¼ÐèҪһϵÁеĺ¯Êýµ÷Óã¬ÐÔÄÜÉÏÓÐÓ°Ï죬2£©iterativeÑ»·µü´úËùÉæ¼°µ½µÄº¯Êýµ÷Óã¬ÎÞ·¨ÀûÓÃÐÂÓ²¼þËùÖ§³ÖµÄSIMD²¢ÐÐÖ´ÐÐÓÅ»¯£¬Ò²ÎÞ·¨½«º¯Êý´úÂëͨ¹ýinlineµÈJVM³£ÓõÄhotspotµÄÓÅ»¯·½Ê½¡£ÔÚ´óÊý¾ÝÁ¿µÄ³¡¾°Ï£¬Ä¿Ç°Á÷ÐеÄͨÓÃ×ö·¨ÊÇÒýÈëVectorization processing, Ò²¾ÍÊÇÿ´Îiteration·µ»ØµÄ²»ÔÙÊÇÒ»Ìõ¼Ç¼£¬¶øÊÇÒ»¸ö¼Ç¼¼¯(batch of rows)£¬±ÈÈçGoogle Spanner ÓÃbatch-at-a-time ´úÌæÁËrow-at-a-time, Spark SQLͬÑùÒ²ÔÚÆäÖ´Ðвã²ÉÓÃÁËVectorizationµÄÖ´ÐÐģʽ¡£
3.2 Á÷ʽ¾ÛºÏÒýÇæÉè¼ÆË¼Â·
Õë¶ÔHiTSDBÔÓоۺÏÔËËãÒýÇæÉϵÄÎÊÌ⣬ΪÁËÓÅ»¯HiTSDB,Ö§³ÖHiTSDBÉÌÒµ»¯ÔËÓª£¬ÎÒÃǾö¶¨¸ÄÔìHiTSDB¾ÛºÏÔËËãÒýÇæ¡£ÏÂͼ¸ø³öÁËоۺϲéѯÒýÇæµÄ»ù±¾¼Ü¹¹¡£

3.2.1 pipelineÖ´ÐÐģʽ
½è¼ø´«Í³Êý¾Ý¿âÖ´ÐÐģʽ£¬ÒýÈëpipelineµÄÖ´ÐÐģʽ£¨aka Volcano / Iterator Ö´ÐÐģʽ£©¡£Pipeline°üº¬²»Í¬µÄÖ´ÐмÆËãËã×Ó(operator), Ò»¸ö²éѯ±»ÎïÀí¼Æ»®Éú³ÉÆ÷½âÎö·Ö½â³ÉÒ»¸öDAG»òÕßoperator tree£¬ Óɲ»Í¬µÄÖ´ÐÐËã×Ó×é³É£¬DAGÉϵÄroot operator¸ºÔðÇý¶¯²éѯµÄÖ´ÐУ¬²¢½«²éѯ½á¹û·µ»Øµ÷ÓÃÕß¡£ÔÚÖ´ÐвãÃæ£¬²ÉÓõÄÊÇtop-downÐèÇóÇý¶¯ (demand-driven)µÄ·½Ê½£¬´Óroot operatorÇý¶¯ÏÂÃæoperatorµÄÖ´ÐС£ÕâÑùµÄÖ´ÐÐÒýÇæ¼Ü¹¹¾ßÓÐÓŵ㣺
ÕâÖּܹ¹·½Ê½±»ºÜ¶àÊý¾Ý¿âϵͳ²ÉÓò¢Ö¤Ã÷ÊÇÓÐЧ;
½Ó¿Ú¶¨ÒåÇåÎú£¬²»Í¬µÄÖ´ÐмÆËãËã×Ó¿ÉÒÔ¶ÀÁ¢ÓÅ»¯£¬¶ø²»Ó°ÏìÆäËûËã×Ó;
Ò×ÓÚÀ©Õ¹£ºÍ¨¹ýÔö¼ÓеļÆËãËã×Ó£¬ºÜÈÝÒ×ʵÏÖÀ©Õ¹¹¦ÄÜ¡£±ÈÈçĿǰ²éѯÐÒéÀïÖ»¶¨ÒåÁËtagÉϵIJéѯÌõ¼þ¡£Èç¹ûÒªÖ§³ÖÖ¸±êÖµÉϵIJéѯÌõ¼þ(cpu.usage >= 70% and cpu.usage <=90%),¿ÉÒÔͨ¹ýÔö¼ÓÒ»¸öеÄFieldFilterOpÀ´ÊµÏÖ¡£
ÿ¸öoperator,ʵÏÖÈçϽӿڣº
Open : ³õʼ»¯²¢ÉèÖÃ×ÊÔ´
Next : µ÷ÓÃÊäÈëoperatorµÄnext()»ñµÃÒ»¸öbatch of time series, ´¦ÀíÊäÈ룬Êä³öbatch of time series
Close : ¹Ø±Õ²¢ÊÍ·Å×ÊÔ´
ÎÒÃÇÔÚHiTSDBÖÐʵÏÖÁËÒÔÏÂËã×Ó£º
ScanOp: ÓÃÓÚ´ÓHBaseÒì²½¶Áȡʱ¼äÏßÊý¾Ý
DsAggOp: ÓÃÓÚ½øÐнµ²ÉÑù¼ÆË㣬²¢´¦ÀíÌîÖµ
AggOp£ºÓÃÓÚ½øÐзÖ×é¾ÛºÏÔËË㣬·Ö³ÉPipeAggOp, MTAggOp
RateOp: ÓÃÓÚ¼ÆËãʱ¼äÏßÖµµÄ±ä»¯ÂÊ
3.2.2 Ö´ÐмÆËãËã×ÓÒ»¸öbatchµÄʱ¼äÏßÊý¾ÝΪÔËË㵥λ
ÔÚ¼ÆËãËã×ÓÖ®¼äÒÔÒ»¸öbatchµÄʱ¼äÏßÊý¾ÝΪµ¥Î»£¬Ìá¸ß¼ÆËãÒýÇæµÄÖ´ÐÐÐÔÄÜ¡£Æä˼Ïë½è¼øÓÚOLAPϵͳËù²ÉÓõÄVectorizationµÄ´¦Àíģʽ¡£ÕâÑùOperatorÔÚ´¦ÀíÒ»¸öbatchµÄ¶àÌõʱ¼äÏߣ¬ÒÔ¼°Ã¿Ìõʱ¼äÏߵĶà¸öʱ¼äµã£¬Äܹ»¼õÉÙº¯Êýµ÷ÓõĴú¼Û£¬Ìá¸ßloopµÄÖ´ÐÐЧÂÊ¡£
ÿ¸öOperatorÒÔÁ÷ʽÏߵķ½Ê½£¬´ÓÊäÈë»ñµÃʱ¼äÏßbatch, ¾¹ý´¦ÀíÔÙÊä³öʱ¼äÏßbatch, ²»Óô洢ÊäÈëµÄʱ¼äÏßbatch£¬´Ó¶ø½µµÍ¶ÔÄÚ´æµÄÒªÇó¡£Ö»Óе±OperatorµÄÓïÒåÒªÇó±ØÐ뽫ÊäÈëmaterialize£¬²Å½øÐÐÕâÑùµÄ²Ù×÷£¨²Î¼ûÏÂÃæÌáµ½µÄ¾ÛºÏËã×ӵIJ»Í¬ÊµÏÖ£©¡£
3.2.3. Çø·Ö²»Í¬²éѯ³¡¾°£¬²ÉÓò»Í¬¾ÛºÏËã×Ó·Ö±ðÓÅ»¯
HiTSDBÔÀ´µÄ¾ÛºÏÒýÇæ²ÉÓÃmaterializationµÄÖ´ÐÐģʽ£¬ºÜÖØÒªµÄÒ»¸öÔÒòÔÚÓÚ´¦ÀíʱÐòÊý¾ÝµÄ²åÖµÔËË㣬ÕâÖ÷ÒªÊÇÒòΪʱÐòÊý¾ÝµÄÒ»¸öµäÐÍÌØµãÊÇʱ¼äÏßÉϲ»¶ÔÆë£º²»Í¬µÄʱ¼äÏßÔÚ²»Í¬µÄʱ¼ä´ÁÉÏÓÐÊý¾Ý¡£HiTSDB¼æÈÝOpenTSDBµÄÐÒ飬ÒýÈëÁ˲åÖµ(interpolation)µÄ¸ÅÄĿµÄÔÚÓÚ¾ÛºÏÔËËãʱͨ¹ýÖ¸¶¨µÄ²åÖµ·½Ê½£¬ÔÚ²»¶ÔÆëµÄʱ¼ä´ÁÉϲåÈë¼ÆËã³öÀ´µÄÖµ£¬´Ó¶ø½«²»¶ÔÆëµÄʱ¼äÏßÊý¾Ýת»»³É¶ÔÆëµÄʱ¼äÏß¡£²åÖµÊÇÔÚͬһ¸ögroupµÄËùÓÐʱ¼äÏßÖ®¼ä±È½Ï£¬À´¾ö¶¨ÔÚÄĸöʱ¼ä´ÁÉÏÐèÒª½øÐвåÖµ £¨²Î¼ûOpenTSDB Îĵµ£©¡£
ΪÁËÓÅ»¯¾ÛºÏ²éѯµÄÐÔÄÜ£¬ÎÒÃÇÒýÈëÁ˲»Í¬µÄ¾ÛºÏÔËËãËã×Ó¡£Ä¿µÄÔÚÓÚÕë¶Ô²»Í¬µÄ²éѯµÄÓïÒ壬½øÐв»Í¬µÄÓÅ»¯¡£ÓÐЩ¾ÛºÏ²éѯÐèÒª²åÖµ£¬¶øÓÐЩ²éѯ²¢²»ÒªÇó²åÖµ£»¼´Ê¹ÐèÒª²åÖµ£¬Ö»ÐèÒª°Ñͬһ¾ÛºÏ×éµÄʱ¼äÏßÊý¾Ý¶ÁÈëÄڴ棬¾Í¿ÉÒÔ½øÐвåÖµÔËËã¡£
PipeAggOp: µ±¾ÛºÏ²éѯÂú×ãÒÔÏÂÌõ¼þʱ£¬
1£©²»ÐèÒª²åÖµ£º ²éѯʹÓÃÁ˽µ²ÉÑù(downsample),²¢ÇÒ½µ²ÉÑùµÄÌîÖµ²ÉÓÃÁË·Çnull/NaNµÄ²ßÂÔ¡£ÕâÑùµÄ²éѯ£¬¾¹ý½µ²ÉÑùºó£¬Ê±¼äÏßµÄÊý¾Ý¶¼ÊÇ¶ÔÆë²¹ÆëµÄ£¬Ò²¾ÍÊǾۺϺ¯ÊýËùÓõ½µÄ²åÖµ²»ÔÙÐèÒª¡£
2£©¾ÛºÏº¯Êý¿ÉÒÔÖ§³Ö½¥½øÊ½µü´ú¼ÆËãģʽ (Incremental iterative aggregation), ±ÈÈçsum, count ,avg, min, max, zerosum, mimmim, mimmax£¬ÎÒÃÇ¿ÉÒÔ²ÉÓÃincremental¾ÛºÏµÄ·½Ê½£¬¶ø²»ÐèÒª°ÑÈ«²¿ÊäÈëÊý¾Ý¶ÁÈëÄÚ´æ¡£Õâ¸öÖ´ÐÐËã×Ó²ÉÓÃÁËÁ÷Ë®Ïߵķ½Ê½£¬Ã¿´Î´ÓÊäÈëµÄoperator»ñµÃһϵÁÐʱ¼äÏߣ¬¼ÆËã·Ö×é²¢¸üоۺϺ¯ÊýµÄ²¿·ÖÖµ£¬Íê³Éºó¿ÉÒÔÇåÀíÊäÈëµÄʱ¼äÏߣ¬Æä×ÔÉíÖ»Óñ£Áôÿ¸ö·Ö×éµÄ¾ÛºÏº¯ÊýµÄÖµ¡£
MTAgOp: ÐèÒª²åÖµ£¬²¢ÇÒÊäÈëËã×ÓÎÞ·¨°ïÖú½«Ê±¼äÏßIDÔ¤ÏÈ·Ö×飬ÕâÖÖ·½Ê½»ØÍ˵½ÔÀ´¾ÛºÏÒýÇæËù²ÉÓõÄÖ´ÐÐģʽ¡£
¶ÔÓÚMTAggOp, ÎÒÃÇ¿ÉÒÔÒýÈë·Ö×é¾ÛºÏµÄ·½·¨½øÐÐÓÅ»¯£º
GroupedAggOp: ÐèÒª²åÖµ£¬µ«ÊÇÊäÈëËã×ÓÄܹ»±£Ö¤ÒѾ½«Ê±¼äÏßµÄID¸ù¾Ý±êʶ(tags)½øÐÐÅÅÐò·Ö×飬ÕâÑùÔÚÁ÷Ë®Ïß´¦ÀíÖУ¬Ö»Òªmaterialize×î¶àÒ»¸ö×éµÄÊý¾Ý£¬ÕâÑùµÄËã×Ó±ÈÆðÄÚ´æ±£ÁôËùÓзÖ×éʱ¼äÏߣ¬ÄÚ´æÒªÇóÒªµÍ£¬Í¬Ê±Ö§³Ö²»Í¬×éÖ®¼äµÄ²¢ÐоۺÏÔËËã¡£
3.2.4 ²éѯÓÅ»¯Æ÷ºÍÖ´ÐÐÆ÷
ÒýÈëÖ´ÐÐËã×ÓºÍpipelineÖ´ÐÐģʽºó£¬ÎÒÃÇ¿ÉÒÔÔÚHiTSDB·Ö³ÉÁ½´óÄ£¿é£¬²éѯÓÅ»¯Æ÷ºÍÖ´ÐÐÆ÷¡£ÓÅ»¯Æ÷¸ù¾Ý²éѯÓïÒåºÍÖ´ÐÐËã×ӵIJ»Í¬Ìص㣬²úÉú²»Í¬µÄÖ´Ðмƻ®£¬ÓÅ»¯²éѯ´¦Àí¡£ÀýÈçHiTSDB¿ÉÒÔÀûÓÃÉÏÃæÌÖÂÛµÄÈý¸ö¾ÛºÏÔËËãËã×Ó£¬ÔÚ²»Í¬µÄ³¡¾°Ï£¬Ê¹Óò»Í¬µÄÖ´ÐÐËã×Ó£¬ÒÔ½µµÍ²éѯִÐÐʱµÄÄڴ濪ÏúºÍÌá¸ßÖ´ÐÐЧÂÊΪĿµÄ¡£ÕâÑùµÄ´¦Àí·½Ê½Ïà±ÈÓÚÔÀ´¾ÛºÏÒýÇæµ¥Ò»µÄÖ´ÐÐģʽ£¬¸ü¼ÓÓÅ»¯¡£
4. Êý¾ÝÇ¨ÒÆ
HiTSDBеľۺÏÒýÇæ²ÉÓÃµÄµ×²ã´æ´¢¸ñʽÓëÒÔǰµÄ°æ±¾²¢²»¼æÈÝ¡£ ¹«ÓÐÔÆ¹«²âÆÚ¼äÔËÐÐÔھɰ汾ʵÀýµÄÊý¾Ý£¬ÐèÒªÇ¨ÒÆÖÁеľۺÏÒýÇæ¡£ ͬʱÈÈÉý¼¶³öÏÖÁËÎÊÌ⣬Êý¾ÝÇ¨ÒÆ»¹Ó¦»Ø¹ö¹¦ÄÜ£¬½«Ð°汾µÄÊý¾Ýµãת»»³É¾ÉµÄÊý¾Ý½á¹¹£¬ÊµÏÖ°æ±¾»Ø¹ö¡£ ÕûÌå·½°¸¶ÔÓÚÓû§µÄÓ°Ïì×öµ½£ºÐ´ÈëÎÞ¸ÐÖª£¬Éý¼¶¹ý³ÌÖУ¬ÀúÊ·Êý¾Ý²»¿É¶Á¡£
4.1 Êý¾ÝÇ¨ÒÆ¼Ü¹¹

²¢·¢×ª»»ºÍÇ¨ÒÆÊý¾Ý: ÔÓеÄHiTSDBÊý¾ÝµãÒѾÔÚдÈëµÄʱºò½øÐÐÁË·ÖÆ¬¡£Ä¬ÈÏÓÐ20¸öSalts¡£Êý¾ÝÇ¨ÒÆ¹¤¾ß»á¶Ôÿ¸öSaltµÄÊý¾Ýµã½øÐв¢·¢´¦Àí¡£ ÿ¸ö¡°Salt¡±¶¼ÓÐÒ»¸öProducerºÍÒ»¸öConsumer¡£Producer¸ºÔð¿ªÆôHBase Scanner»ñÈ¡Êý¾Ýµã¡£ ÿ¸öScannerÒì²½¶ÔHBase½øÐÐɨÃ裬ÿ´Î»ñÈ¡HBASE_MAX_SCAN_SIZEÐÐÊýµÄÊý¾Ýµã¡£È»ºó½«HBaseµÄRow Keyת»»³ÉеĽṹ¡£
×îºó½«¸ÃRow·Åµ½ËùÓеÄÒ»¸öQueueÉϵȴýConsumerÏû·Ñ¡£ Consumerÿ´Î»á´¦ÀíHBASE_PUT_BATCHSIZE»òÕßHBASE_PUT_MIN_DATAPOINTSµÄÊý¾ÝÁ¿¡£ ÿ´ÎConsumer˳ÀûдÈë¸ÃBatchµÄʱºò£¬ÎÒÃÇ»áÔÚUID±íÖмǼ¶ÔÓ¦¡°Salt¡±µÄÊý¾Ý´¦ÀíλÖᣠÕâÑù±ãÓÚ¹ÊÕÏÖØÆôʱProducer´Ó×îºóÒ»´Î³É¹¦µÄµØ·½ÖØÐ¿ªÊ¼»ñÈ¡Êý¾Ýµã½øÐÐת»»¡£ Êý¾ÝÇ¨ÒÆ¹¤¾ß¶ÔHBaseµÄ²Ù×÷¶¼²ÉÓÃÒì²½µÄ¶Áд¡£µ±É¨ÃèÊý¾Ý»òÕßдÈëÊý¾Ýʧ°ÜµÄʱºò£¬ÎÒÃÇ»á½øÐÐÓÐÏÞÖÆµÄ³¢ÊÔ¡£ Èç¹û³¬³ö³¢ÊÔ´ÎÊý£¬ÎÒÃǾÍÖÕÖ¹¸Ã¡°Salt¡±µÄÊý¾ÝÇ¨ÒÆ¹¤×÷£¬ÆäËû¡±Salt¡°µÄ¹¤×÷²»Êܵ½ÈκÎÓ°Ïì¡£ µ±Ï´ι¤¾ß×Ô¶¯ÖØÆôʱ£¬ÎÒÃÇ»á³öÏÖÎÊÌâµÄ¡±Salt¡°Êý¾Ý¼ÌÐø½øÐÐÇ¨ÒÆ£¬Ö±µ½ËùÓÐÊý¾ÝÈ«²¿Ë³Àûת»»Íê³É¡£
Á÷¿ØÏÞÖÆ: ´ó²¿·ÖÇé¿öÏ£¬Producer¶ÔHBaseµÄɨÃèÊý¾ÝÒª¿ìÓÚConsumer¶ÔHBaseµÄдÈë¡£ ΪÁË·ÀÖ¹QueueµÄÊý¾Ý»ýѹ¶ÔÄÚ´æÔì³ÉѹÁ¦Í¬Ê±ÎªÁ˼õÉÙProducerɨÃèÊý¾Ýʱ¶ÔHBaseµÄѹÁ¦£¬ÎÒÃÇÉèÖÃÁËÁ÷¿Ø¡£ µ±QueueµÄ´óС´ïµ½HBASE_MAX_REQUEST_QUEUE_SIZEʱºò£¬Producer»áÔÝʱֹͣ¶ÔHBaseµÄÊý¾ÝɨÃèµÈ´ýConsumerÏû·Ñ¡£ µ±QueueµÄ´óС¼õÉÙµ½HBASE_RESUME_SCANNING_REQUEST_QUEUE_SIZEʱºò£¬Producer»áÖØÐ»ָ´¡£
ProducerºÍConsumer½ø³ÌµÄÍ˳ö
˳ÀûÍê³ÉʱºòÈçºÎÍ˳ö: µ±Ò»ÇнøÕ¹Ë³Àûʱºò£¬µ±ProducerÍê³ÉÊý¾ÝɨÃèÖ®ºó£¬»áÔÚQueueÉÏ·ÅÒ»¸öEOS£¨End of Scan£©£¬È»ºóÍ˳ö¡£ ConsumerÓöµ½EOS¾Í»áÖªµÀ¸ÃBatchΪ×îºóÒ»Åú£¬³É¹¦´¦ÀíÍê¸ÃBatchÖ®ºó¾Í»á×Ô¶¯Í˳ö¡£
ʧ°ÜºóÈçºÎ¹Ø±Õ: ConsumerÓöµ½ÎÊÌâʱ£ºµ±ConsumerдÈëHBaseʧ°ÜÖ®ºó£¬consumer»áÉèÖÃÒ»¸öFlag£¬È»ºóÍ˳öÏ̡߳£ ÿµ±Producer×¼±¸½øÐÐÏÂÒ»¸öHBASE_MAX_SCAN_SIZEµÄɨÃèʱºò£¬Ëû»áÏȼì²é¸ÃFlag¡£ Èç¹û±»ÉèÖã¬Ëû»áÖªµÀ¶ÔÓ¦µÄConsumerÏß³ÌÒѾʧ°Ü²¢ÇÒÍ˳ö¡£ProducerÒ²»áֹͣɨÃè²¢ÇÒÍ˳ö¡£ ProducerÓöµ½ÎÊÌâʱ£ºµ±ProducerɨÃèÊý¾Ýʧ°Üʱ£¬´¦Àí·½Ê½ºÍ˳ÀûÍê³ÉʱºòÀàËÆ¡£¶¼ÊÇͨ¹ýÍùQueueÉÏEOSÀ´Íê³É֪ͨ¡£ Ï´ÎÖØÆôʱ£¬Producer»á´ÓÉϴμǼµÄÊý¾Ý´¦ÀíλÖÿªÊ¼ÖØÐÂɨÃè¡£
4.2 Êý¾ÝÇ¨ÒÆµÄÒ»ÖÂÐÔ
ÓÉÓÚÄ¿Ç°ÔÆÉϰ汾HiTSDBΪ˫½Úµã£¬ÔÚ½áµãÉý¼¶½áÊøºó»á×Ô¶¯ÖØÆôHiTSDB¡£×Ô¶¯Æô¶¯½Å±¾»á×Ô¶¯ÔËÐÐÊý¾ÝÇ¨ÒÆ¹¤¾ß¡£ Èç¹ûûÓÐÈκÎÔ¤·À´ëÊ©£¬´ËʱÁ½¸öHiTSDB½Úµã»áͬʱ½øÐÐÊý¾ÝÇ¨ÒÆ¡£ËäÈ»Êý¾ÝÉϲ»»áÔì³ÉÈκζªÊ§»òÕßË𻵣¬ µ«ÊÇ»á¶ÔHBaseÔì³É´óÁ¿µÄдÈëºÍ¶ÁȡѹÁ¦´Ó¶øÑÏÖØÓ°ÏìÓû§µÄÕý³£µÄдÈëºÍ²éѯÐÔÄÜ¡£
ΪÁË·ÀÖ¹ÕâÑùµÄÊÂÇé·¢Éú£¬ÎÒÃÇͨ¹ýHBaseµÄZoo KeeperʵÏÖÁËÀàËÆFileLockËø£¬ÎÒÃdzÆÎªDataLock£¬µÄ»úÖÆ±£Ö¤Ö»ÓÐÒ»¸ö½áµãÆô¶¯Êý¾ÝÇ¨ÒÆ½ø³Ì¡£ ÔÚÊý¾ÝÇ¨ÒÆ½ø³ÌÆô¶¯Ê±£¬Ëû»áͨ¹ýÀàËÆ·Ç×èÈûµÄtryLock()µÄÐÎʽÔÚZoo KeeperµÄÌØ¶¨Â·¾¶´´½¨Ò»¸öÔÝʱµÄ½Úµã¡£ Èç¹û³É¹¦´´½¨½ÚµãÔò´ú±í³É¹û»ñµÃDataLock¡£Èç¹û¸Ã½ÚµãÒѾ´æÔÚ£¬¼´±»ÁíÒ»¸öHiTSDB´´½¨£¬ÎÒÃÇ»áÊÕµ½KeeperException¡£ÕâÑù´ú±íδ»ñµÃËø£¬ÂíÉÏ·µ»ØÊ§°Ü¡£ Èç¹ûδ³É¹¦»ñµÃDataLock£¬¸Ã½ÚµãÉϵÄÊý¾ÝÇ¨ÒÆ½ø³Ì¾Í»á×Ô¶¯Í˳ö¡£³É¹û»ñµÃDataLockµÄ½ÚµãÔò¿ªÊ¼½øÐÐÊý¾ÝÇ¨ÒÆ¡£

4.3 Êý¾ÝÇ¨ÒÆÖеÄ"Ö´ÐÐÒ»´Î"
µ±ËùÓС°Salt¡±µÄÊý¾ÝµãÈ«²¿Ë³ÀûÍê³ÉÇ¨ÒÆÖ®ºó£¬ÎÒÃÇ»áÔÚHBaseµÄ¾É±íÖвåÈëÒ»ÐÐÐÂÊý¾Ý£¬data_conversion_completed¡£ ´ËÐдú±íÁËÊý¾ÝÇ¨ÒÆ¹¤³ÌÈ«²¿Ë³ÀûÍê³É¡£Í¬Ê±×Ô¶¯½Å±¾»áÿ¸ô12¸öСʱÆô¶¯Êý¾ÝÇ¨ÒÆ¹¤¾ß£¬ÕâÑùÊÇΪÁË·ÀÖ¹ÉÏ´ÎÊý¾ÝÇ¨ÒÆÃ»ÓÐÈ«²¿Íê³É¡£ ÿ´ÎÆô¶¯Ê±£¬ÎÒÃǶ¼»áÏȼì²é¡°data_conversion_completed¡±±êÖ¾¡£Èç¹û±êÖ¾´æÔÚ£¬¹¤¾ß¾Í»áÂíÉÏÍ˳ö¡£ ´ËÏî²Ù×÷Ö»»á½øÐÐÒ»´ÎHBaseµÄ²éѯ£¬±ÈÕý³£µÄ½¡¿µ¼ì²é³É±¾»¹ÒªµÍ¡£ËùÒÔÖÜÆÚÐÔµÄÆô¶¯Êý¾ÝÇ¨ÒÆ¹¤¾ß²¢²»»á¶ÔHiTSDB»òÕßHBase²úÉúÓ°Ïì¡£
4.4. Êý¾ÝÇ¨ÒÆµÄÆÀ²â
²âÊÔ»úÐÍ: 4core,8G,SSD

Ч¹û£ºÉÏÏߺóÎÞ¹ÊÕÏÍê³É100+ʵÀýÊý¾ÝµÄÇ¨ÒÆ£¬ÈÈÉý¼¶¡£
5. ²éѯÐÔÄÜÆÀ²â
²âÊÔ»·¾³ÅäÖÃ
192.168.12.3 2.1.5°æ±¾
192.168.12.4 2.2.0°æ±¾£¨Pipelined Engine£©
²âÊÔÊý¾Ý - 1ÍòÌõʱ¼ä£¬²»Í¬µÄ²É¼¯ÆµÂʺÍʱ¼ä´°¿Ú£¬»¹ÓвéѯÃüÖеÄʱ¼äÏßÊýÁ¿¡£
Case 1: Êý¾Ý²É¼¯ÆµÂÊ5s, ²éѯÃüÖÐ1000Ìõ£¬Ê±¼ä´°¿Ú3600s

Case 2: Êý¾Ý²É¼¯ÆµÂÊ1s£¬²éѯÃüÖÐ1Ìõ£¬Ê±¼ä´°¿Ú36000s

×ܽ᣺ еIJéѯ¾ÛºÏÒýÇæ½«²éѯËÙ¶ÈÌá¸ßÁË10±¶ÒÔÉÏ¡£

ÆäËû
±¾ÎĽéÉÜÁ˸ßÐÔÄÜʱ¼äÐòÁÐÊý¾Ý¿âHiTSDBÒýÇæÔÚÉÌÒµ»¯ÔËӪ֮ǰ½øÐеÄÓÅ»¯Éý¼¶£¬Ä¿µÄÊÇÌá¸ßHiTSDBÒýÇæµÄÎȶ¨ÐÔ£¬Êý¾ÝдÈëºÍ²éѯÐÔÄÜÒÔ¼°Ð¹¦ÄܵÄÀ©Õ¹ÐÔ¡£HiTSDBÒѾÔÚ°¢ÀïÔÆÕýʽÉÌÒµ»¯ÔËÓª£¬ÎÒÃǽ«¸ù¾ÝÓû§·´À¡£¬½øÒ»²½Ìá¸ßHiTSDBÒýÇæ£¬¸üºÃ·þÎñÓÚHiTSDBµÄ¿Í»§¡£
|