Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Modeler   Code  
»áÔ±   
 
   
 
 
     
   
 ¶©ÔÄ
  ¾èÖú
¹È¸è¼¼ÊõÖ®MapReduceºÍBigTable
 
×÷Õß OpenNaive²©¿Í£¬»ðÁú¹ûÈí¼þ    ·¢²¼ÓÚ 2014-11-24
  5146  次浏览      27
 

¹È¸è¼¼Êõ"Èý±¦"Ö®MapReduce

1. MapReduceÊǸÉɶµÄ

ÒòΪûÕÒµ½¹È¸èµÄʾÒâͼ£¬ËùÒÔÎÒÏë½èÓÃÒ»ÕÅHadoopÏîÄ¿µÄ½á¹¹Í¼À´ËµÃ÷ÏÂMapReduceËù´¦µÄλÖã¬ÈçÏÂͼ¡£

Hadoopʵ¼ÊÉϾÍÊǹȸèÈý±¦µÄ¿ªÔ´ÊµÏÖ£¬Hadoop MapReduce¶ÔÓ¦Google MapReduce£¬HBase¶ÔÓ¦BigTable£¬HDFS¶ÔÓ¦GFS¡£HDFS£¨»òGFS£©ÎªÉϲãÌṩ¸ßЧµÄ·Ç½á¹¹»¯´æ´¢·þÎñ£¬HBase£¨»òBigTable£©ÊÇÌṩ½á¹¹»¯Êý¾Ý·þÎñµÄ·Ö²¼Ê½Êý¾Ý¿â£¬Hadoop MapReduce£¨»òGoogle MapReduce£©ÊÇÒ»ÖÖ²¢ÐмÆËãµÄ±à³ÌÄ£ÐÍ£¬ÓÃÓÚ×÷Òµµ÷¶È¡£

GFSºÍBigTableÒѾ­ÎªÎÒÃÇÌṩÁ˸ßÐÔÄÜ¡¢¸ß²¢·¢µÄ·þÎñ£¬µ«ÊDz¢Ðбà³Ì¿É²»ÊÇËùÓгÌÐòÔ±¶¼ÍæµÃתµÄ»î¶ù£¬Èç¹ûÎÒÃǵÄÓ¦Óñ¾Éí²»Äܲ¢·¢£¬ÄÇGFS¡¢BigTableÒ²¶¼ÊÇûÓÐÒâÒåµÄ¡£MapReduceµÄΰ´óÖ®´¦¾ÍÔÚÓÚÈò»ÊìϤ²¢Ðбà³ÌµÄ³ÌÐòÔ±Ò²Äܳä·Ö·¢»Ó·Ö²¼Ê½ÏµÍ³µÄÍþÁ¦¡£

¼òµ¥¸ÅÀ¨µÄ˵£¬MapReduceÊǽ«Ò»¸ö´ó×÷Òµ²ð·ÖΪ¶à¸öС×÷ÒµµÄ¿ò¼Ü£¨´ó×÷ÒµºÍС×÷ÒµÓ¦¸Ã±¾ÖÊÊÇÒ»ÑùµÄ£¬Ö»ÊǹæÄ£²»Í¬£©£¬Óû§ÐèÒª×öµÄ¾ÍÊǾö¶¨²ð³É¶àÉÙ·Ý£¬ÒÔ¼°¶¨Òå×÷Òµ±¾Éí¡£

ÏÂÃæÓÃÒ»¸ö¹á´©È«ÎĵÄÀý×ÓÀ´½âÊÍMapReduceÊÇÈçºÎ¹¤×÷µÄ¡£

2. Àý×Ó£ºÍ³¼Æ´ÊƵ

Èç¹ûÎÒÏëͳ¼ÆÏ¹ýÈ¥10Äê¼ÆËã»úÂÛÎijöÏÖ×î¶àµÄ¼¸¸öµ¥´Ê£¬¿´¿´´ó¼Ò¶¼ÔÚÑо¿Ð©Ê²Ã´£¬ÄÇÎÒÊÕ¼¯ºÃÂÛÎĺ󣬸ÃÔõô°ìÄØ£¿

·½·¨Ò»£ºÎÒ¿ÉÒÔдһ¸öС³ÌÐò£¬°ÑËùÓÐÂÛÎݴ˳Ðò±éÀúÒ»±é£¬Í³¼ÆÃ¿Ò»¸öÓöµ½µÄµ¥´ÊµÄ³öÏÖ´ÎÊý£¬×îºó¾Í¿ÉÒÔÖªµÀÄöµ¥´Ê×îÈÈÃÅÁË¡£

ÕâÖÖ·½·¨ÔÚÊý¾Ý¼¯±È½ÏСʱ£¬ÊǷdz£ÓÐЧµÄ£¬¶øÇÒʵÏÖ×î¼òµ¥£¬ÓÃÀ´½â¾öÕâ¸öÎÊÌâºÜºÏÊÊ¡£

·½·¨¶þ£ºÐ´Ò»¸ö¶àÏ̳߳ÌÐò£¬²¢·¢±éÀúÂÛÎÄ¡£

Õâ¸öÎÊÌâÀíÂÛÉÏÊÇ¿ÉÒԸ߶Ȳ¢·¢µÄ£¬ÒòΪͳ¼ÆÒ»¸öÎļþʱ²»»áÓ°Ïìͳ¼ÆÁíÒ»¸öÎļþ¡£µ±ÎÒÃǵĻúÆ÷ÊǶàºË»òÕß¶à´¦ÀíÆ÷£¬·½·¨¶þ¿Ï¶¨±È·½·¨Ò»¸ßЧ¡£µ«ÊÇдһ¸ö¶àÏ̳߳ÌÐòÒª±È·½·¨Ò»À§ÄѶàÁË£¬ÎÒÃDZØÐë×Ô¼ºÍ¬²½¹²ÏíÊý¾Ý£¬±ÈÈçÒª·ÀÖ¹Á½¸öÏß³ÌÖØ¸´Í³¼ÆÎļþ¡£

·½·¨Èý£º°Ñ×÷Òµ½»¸ø¶à¸ö¼ÆËã»úÈ¥Íê³É¡£

ÎÒÃÇ¿ÉÒÔʹÓ÷½·¨Ò»µÄ³ÌÐò£¬²¿Êðµ½N̨»úÆ÷ÉÏÈ¥£¬È»ºó°ÑÂÛÎֳÉN·Ý£¬Ò»Ì¨»úÆ÷ÅÜÒ»¸ö×÷Òµ¡£Õâ¸ö·½·¨ÅܵÃ×ã¹»¿ì£¬µ«ÊDz¿ÊðÆðÀ´ºÜÂé·³£¬ÎÒÃÇÒªÈ˹¤°Ñ³ÌÐòcopyµ½±ðµÄ»úÆ÷£¬ÒªÈ˹¤°ÑÂÛÎļ¯·Ö¿ª£¬×îÍ´¿àµÄÊÇ»¹Òª°ÑN¸öÔËÐнá¹û½øÐÐÕûºÏ£¨µ±È»ÎÒÃÇÒ²¿ÉÒÔÔÙдһ¸ö³ÌÐò£©¡£

·½·¨ËÄ£ºÈÃMapReduceÀ´°ï°ïÎÒÃǰɣ¡

MapReduce±¾ÖÊÉϾÍÊÇ·½·¨Èý£¬µ«ÊÇÈçºÎ²ð·ÖÎļþ¼¯£¬ÈçºÎcopy³ÌÐò£¬ÈçºÎÕûºÏ½á¹ûÕâЩ¶¼ÊÇ¿ò¼Ü¶¨ÒåºÃµÄ¡£ÎÒÃÇÖ»Òª¶¨ÒåºÃÕâ¸öÈÎÎñ£¨Óû§³ÌÐò£©£¬ÆäËü¶¼½»¸øMapReduce¡£

ÔÚ½éÉÜMapReduceÈçºÎ¹¤×÷֮ǰ£¬ÏȽ²½²Á½¸öºËÐĺ¯ÊýmapºÍreduceÒÔ¼°MapReduceµÄα´úÂë¡£

3. mapº¯ÊýºÍreduceº¯Êý

mapº¯ÊýºÍreduceº¯ÊýÊǽ»¸øÓû§ÊµÏֵģ¬ÕâÁ½¸öº¯Êý¶¨ÒåÁËÈÎÎñ±¾Éí¡£

1.mapº¯Êý£º½ÓÊÜÒ»¸ö¼üÖµ¶Ô£¨key-value pair£©£¬²úÉúÒ»×éÖмä¼üÖµ¶Ô¡£MapReduce¿ò¼Ü»á½«mapº¯Êý²úÉúµÄÖмä¼üÖµ¶ÔÀï¼üÏàͬµÄÖµ´«µÝ¸øÒ»¸öreduceº¯Êý¡£

2.reduceº¯Êý£º½ÓÊÜÒ»¸ö¼ü£¬ÒÔ¼°Ïà¹ØµÄÒ»×éÖµ£¬½«Õâ×éÖµ½øÐкϲ¢²úÉúÒ»×鹿ģ¸üСµÄÖµ£¨Í¨³£Ö»ÓÐÒ»¸ö»òÁã¸öÖµ£©¡£

ͳ¼Æ´ÊƵµÄMapReduceº¯ÊýµÄºËÐÄ´úÂë·Ç³£¼ò¶Ì£¬Ö÷Òª¾ÍÊÇʵÏÖÕâÁ½¸öº¯Êý¡£

map(String key, String value):  
02. // key: document name
03. // value: document contents
04. for each word w in value:
05. EmitIntermediate(w, "1");
06.
07.reduce(String key, Iterator values):
08. // key: a word
09. // values: a list of counts
10. int result = 0;
11. for each v in values:
12. result += ParseInt(v);
13. Emit(AsString(result));

ÔÚͳ¼Æ´ÊƵµÄÀý×ÓÀmapº¯Êý½ÓÊܵļüÊÇÎļþÃû£¬ÖµÊÇÎļþµÄÄÚÈÝ£¬mapÖð¸ö±éÀúµ¥´Ê£¬Ã¿Óöµ½Ò»¸öµ¥´Êw£¬¾Í²úÉúÒ»¸öÖмä¼üÖµ¶Ô<w, "1">£¬Õâ±íʾµ¥´ÊwÔÛÓÖÕÒµ½ÁËÒ»¸ö£»MapReduce½«¼üÏàͬ£¨¶¼Êǵ¥´Êw£©µÄ¼üÖµ¶Ô´«¸øreduceº¯Êý£¬ÕâÑùreduceº¯Êý½ÓÊܵļü¾ÍÊǵ¥´Êw£¬ÖµÊÇÒ»´®"1"£¨×î»ù±¾µÄʵÏÖÊÇÕâÑù£¬µ«¿ÉÒÔÓÅ»¯£©£¬¸öÊýµÈÓÚ¼üΪwµÄ¼üÖµ¶ÔµÄ¸öÊý£¬È»ºó½«ÕâЩ¡°1¡±ÀۼӾ͵õ½µ¥´ÊwµÄ³öÏÖ´ÎÊý¡£×îºóÕâЩµ¥´ÊµÄ³öÏÖ´ÎÊý»á±»Ð´µ½Óû§¶¨ÒåµÄλÖ㬴洢ÔڵײãµÄ·Ö²¼Ê½´æ´¢ÏµÍ³£¨GFS»òHDFS£©¡£

4. MapReduceÊÇÈçºÎ¹¤×÷µÄ

ÉÏͼÊÇÂÛÎÄÀï¸ø³öµÄÁ÷³Ìͼ¡£Ò»Çж¼ÊÇ´Ó×îÉÏ·½µÄuser program¿ªÊ¼µÄ£¬user programÁ´½ÓÁËMapReduce¿â£¬ÊµÏÖÁË×î»ù±¾µÄMapº¯ÊýºÍReduceº¯Êý¡£Í¼ÖÐÖ´ÐеÄ˳Ðò¶¼ÓÃÊý×Ö±ê¼ÇÁË¡£

1.MapReduce¿âÏȰÑuser programµÄÊäÈëÎļþ»®·ÖΪM·Ý£¨MΪÓû§¶¨Ò壩£¬Ã¿Ò»·Ýͨ³£ÓÐ16MBµ½64MB£¬Èçͼ×ó·½Ëùʾ·Ö³ÉÁËsplit0~4£»È»ºóʹÓÃfork½«Óû§½ø³Ì¿½±´µ½¼¯ÈºÄÚÆäËü»úÆ÷ÉÏ¡£

2.user programµÄ¸±±¾ÖÐÓÐÒ»¸ö³ÆÎªmaster£¬ÆäÓà³ÆÎªworker£¬masterÊǸºÔðµ÷¶ÈµÄ£¬Îª¿ÕÏÐworker·ÖÅä×÷Òµ£¨Map×÷Òµ»òÕßReduce×÷Òµ£©£¬workerµÄÊýÁ¿Ò²ÊÇ¿ÉÒÔÓÉÓû§Ö¸¶¨µÄ¡£

3.±»·ÖÅäÁËMap×÷ÒµµÄworker£¬¿ªÊ¼¶ÁÈ¡¶ÔÓ¦·ÖƬµÄÊäÈëÊý¾Ý£¬Map×÷ÒµÊýÁ¿ÊÇÓÉM¾ö¶¨µÄ£¬ºÍsplitÒ»Ò»¶ÔÓ¦£»Map×÷Òµ´ÓÊäÈëÊý¾ÝÖгéÈ¡³ö¼üÖµ¶Ô£¬Ã¿Ò»¸ö¼üÖµ¶Ô¶¼×÷Ϊ²ÎÊý´«µÝ¸ømapº¯Êý£¬mapº¯Êý²úÉúµÄÖмä¼üÖµ¶Ô±»»º´æÔÚÄÚ´æÖС£

4.»º´æµÄÖмä¼üÖµ¶Ô»á±»¶¨ÆÚдÈë±¾µØ´ÅÅÌ£¬¶øÇÒ±»·ÖΪR¸öÇø£¬RµÄ´óСÊÇÓÉÓû§¶¨ÒåµÄ£¬½«À´Ã¿¸öÇø»á¶ÔÓ¦Ò»¸öReduce×÷Òµ£»ÕâЩÖмä¼üÖµ¶ÔµÄλÖûᱻͨ±¨¸ømaster£¬master¸ºÔð½«ÐÅϢת·¢¸øReduce worker¡£

5.master֪ͨ·ÖÅäÁËReduce×÷ÒµµÄworkerËü¸ºÔðµÄ·ÖÇøÔÚʲôλÖ㨿϶¨²»Ö¹Ò»¸öµØ·½£¬Ã¿¸öMap×÷Òµ²úÉúµÄÖмä¼üÖµ¶Ô¶¼¿ÉÄÜÓ³Éäµ½ËùÓÐR¸ö²»Í¬·ÖÇø£©£¬µ±Reduce worker°ÑËùÓÐËü¸ºÔðµÄÖмä¼üÖµ¶Ô¶¼¶Á¹ýÀ´ºó£¬ÏȶÔËüÃǽøÐÐÅÅÐò£¬Ê¹µÃÏàͬ¼üµÄ¼üÖµ¶Ô¾Û¼¯ÔÚÒ»Æð¡£ÒòΪ²»Í¬µÄ¼ü¿ÉÄÜ»áÓ³É䵽ͬһ¸ö·ÖÇøÒ²¾ÍÊÇͬһ¸öReduce×÷Òµ£¨Ë­È÷ÖÇøÉÙÄØ£©£¬ËùÒÔÅÅÐòÊDZØÐëµÄ¡£

6.reduce worker±éÀúÅÅÐòºóµÄÖмä¼üÖµ¶Ô£¬¶ÔÓÚÿ¸öΨһµÄ¼ü£¬¶¼½«¼üÓë¹ØÁªµÄÖµ´«µÝ¸øreduceº¯Êý£¬reduceº¯Êý²úÉúµÄÊä³ö»áÌí¼Óµ½Õâ¸ö·ÖÇøµÄÊä³öÎļþÖС£

7.µ±ËùÓеÄMapºÍReduce×÷Òµ¶¼Íê³ÉÁË£¬master»½ÐÑÕý°æµÄuser program£¬MapReduceº¯Êýµ÷Ó÷µ»Øuser programµÄ´úÂë¡£

ËùÓÐÖ´ÐÐÍê±Ïºó£¬MapReduceÊä³ö·ÅÔÚÁËR¸ö·ÖÇøµÄÊä³öÎļþÖУ¨·Ö±ð¶ÔÓ¦Ò»¸öReduce×÷Òµ£©¡£Óû§Í¨³£²¢²»ÐèÒªºÏ²¢ÕâR¸öÎļþ£¬¶øÊǽ«Æä×÷ΪÊäÈë½»¸øÁíÒ»¸öMapReduce³ÌÐò´¦Àí¡£Õû¸ö¹ý³ÌÖУ¬ÊäÈëÊý¾ÝÊÇÀ´×Եײã·Ö²¼Ê½Îļþϵͳ£¨GFS£©µÄ£¬ÖмäÊý¾ÝÊÇ·ÅÔÚ±¾µØÎļþϵͳµÄ£¬×îÖÕÊä³öÊý¾ÝÊÇдÈëµ×²ã·Ö²¼Ê½Îļþϵͳ£¨GFS£©µÄ¡£¶øÇÒÎÒÃÇҪעÒâMap/Reduce×÷ÒµºÍmap/reduceº¯ÊýµÄÇø±ð£ºMap×÷Òµ´¦ÀíÒ»¸öÊäÈëÊý¾ÝµÄ·ÖƬ£¬¿ÉÄÜÐèÒªµ÷Óöà´Îmapº¯ÊýÀ´´¦Àíÿ¸öÊäÈë¼üÖµ¶Ô£»Reduce×÷Òµ´¦ÀíÒ»¸ö·ÖÇøµÄÖмä¼üÖµ¶Ô£¬ÆÚ¼äÒª¶Ôÿ¸ö²»Í¬µÄ¼üµ÷ÓÃÒ»´Îreduceº¯Êý£¬Reduce×÷Òµ×îÖÕÒ²¶ÔÓ¦Ò»¸öÊä³öÎļþ¡£

ÎÒ¸üϲ»¶°ÑÁ÷³Ì·ÖΪÈý¸ö½×¶Î¡£µÚÒ»½×¶ÎÊÇ×¼±¸½×¶Î£¬°üÀ¨1¡¢2£¬Ö÷½ÇÊÇMapReduce¿â£¬Íê³É²ð·Ö×÷ÒµºÍ¿½±´Óû§³ÌÐòµÈÈÎÎñ£»µÚ¶þ½×¶ÎÊÇÔËÐн׶Σ¬°üÀ¨3¡¢4¡¢5¡¢6£¬Ö÷½ÇÊÇÓû§¶¨ÒåµÄmapºÍreduceº¯Êý£¬Ã¿¸öС×÷Òµ¶¼¶ÀÁ¢ÔËÐÐ×Å£»µÚÈý½×¶ÎÊÇɨβ½×¶Î£¬Õâʱ×÷ÒµÒѾ­Íê³É£¬×÷Òµ½á¹û±»·ÅÔÚÊä³öÎļþÀ¾Í¿´Óû§ÏëÔõô´¦ÀíÕâЩÊä³öÁË¡£

5. ´ÊƵÊÇÔõôͳ¼Æ³öÀ´µÄ

½áºÏµÚËĽڣ¬ÎÒÃǾͿÉÒÔÖªµÀµÚÈý½ÚµÄ´úÂëÊÇÈçºÎ¹¤×÷µÄÁË¡£¼ÙÉèÔÛÃǶ¨ÒåM=5£¬R=3£¬²¢ÇÒÓÐ6̨»úÆ÷£¬Ò»Ì¨master¡£

Õâ·ùͼÃèÊöÁËMapReduceÈçºÎ´¦Àí´ÊƵͳ¼Æ¡£ÓÉÓÚmap workerÊýÁ¿²»¹»£¬Ê×ÏÈ´¦ÀíÁË·ÖÆ¬1¡¢3¡¢4£¬²¢²úÉúÖмä¼üÖµ¶Ô£»µ±ËùÓÐÖмäÖµ¶¼×¼±¸ºÃÁË£¬Reduce×÷Òµ¾Í¿ªÊ¼¶ÁÈ¡¶ÔÓ¦·ÖÇø£¬²¢Êä³öͳ¼Æ½á¹û¡£

6. Óû§µÄȨÀû

Óû§×îÖ÷ÒªµÄÈÎÎñÊÇʵÏÖmapºÍreduce½Ó¿Ú£¬µ«»¹ÓÐһЩÓÐÓõĽӿÚÊÇÏòÓû§¿ª·ÅµÄ¡£

1.an input reader¡£Õâ¸öº¯Êý»á½«ÊäÈë·ÖΪM¸ö²¿·Ö£¬²¢ÇÒ¶¨ÒåÁËÈçºÎ´ÓÊý¾ÝÖгéÈ¡×î³õµÄ¼üÖµ¶Ô£¬±ÈÈç´ÊƵµÄÀý×ÓÖж¨ÒåÎļþÃûºÍÎļþÄÚÈÝÊǼüÖµ¶Ô¡£

2.a partition function¡£Õâ¸öº¯ÊýÓÃÓÚ½«mapº¯Êý²úÉúµÄÖмä¼üÖµ¶ÔÓ³Éäµ½Ò»¸ö·ÖÇøÀïÈ¥£¬×î¼òµ¥µÄʵÏÖ¾ÍÊǽ«¼üÇó¹þÏ£ÔÙ¶ÔRȡģ¡£

3.a compare function¡£Õâ¸öº¯ÊýÓÃÓÚReduce×÷ÒµÅÅÐò£¬Õâ¸öº¯Êý¶¨ÒåÁ˼üµÄ´óС¹ØÏµ¡£

4.an output writer¡£¸ºÔ𽫽á¹ûдÈëµ×²ã·Ö²¼Ê½Îļþϵͳ¡£

5.a combiner function¡£Êµ¼Ê¾ÍÊÇreduceº¯Êý£¬ÕâÊÇÓÃÓÚÇ°ÃæÌáµ½µÄÓÅ»¯µÄ£¬±ÈÈçͳ¼Æ´ÊƵʱ£¬Èç¹ûÿ¸ö<w, "1">Òª¶ÁÒ»´Î£¬ÒòΪreduceºÍmapͨ³£²»ÔÚһ̨»úÆ÷£¬·Ç³£ÀË·Ñʱ¼ä£¬ËùÒÔ¿ÉÒÔÔÚmapÖ´Ðеĵط½ÏÈÔËÐÐÒ»´Îcombiner£¬ÕâÑùreduceÖ»ÐèÒª¶ÁÒ»´Î<w, "n">ÁË¡£

6.mapºÍreduceº¯Êý¾Í²»¶à˵ÁË¡£

7. MapReduceµÄʵÏÖ

ĿǰMapReduceÒѾ­ÓжàÖÖʵÏÖ£¬³ýÁ˹ȸè×Ô¼ºµÄʵÏÖÍ⣬»¹ÓÐÖøÃûµÄhadoop£¬Çø±ðÊǹȸèÊÇc++£¬¶øhadoopÊÇÓÃjava¡£ÁíÍâ˹̹¸£´óѧʵÏÖÁËÒ»¸öÔÚ¶àºË/¶à´¦ÀíÆ÷¡¢¹²ÏíÄÚ´æ»·¾³ÄÚÔËÐеÄMapReduce£¬³ÆÎªPhoenix£¨½éÉÜ£©£¬Ïà¹ØµÄÂÛÎÄ·¢±íÔÚ07ÄêµÄHPCA£¬Êǵ±ÄêµÄ×î¼ÑÂÛÎÄŶ£¡

¹È¸è¼¼Êõ"Èý±¦"Ö®BigTable

1 ʲôÊÇBigTable

BigtableÊÇÒ»¸öΪ¹ÜÀí´ó¹æÄ£½á¹¹»¯Êý¾Ý¶øÉè¼ÆµÄ·Ö²¼Ê½´æ´¢ÏµÍ³£¬¿ÉÒÔÀ©Õ¹µ½PB¼¶Êý¾ÝºÍÉÏǧ̨·þÎñÆ÷¡£ºÜ¶àgoogleµÄÏîĿʹÓÃBigtable´æ´¢Êý¾Ý£¬ÕâЩӦÓöÔBigtableÌá³öÁ˲»Í¬µÄÌôÕ½£¬±ÈÈçÊý¾Ý¹æÄ£µÄÒªÇó¡¢ÑÓ³ÙµÄÒªÇó¡£BigtableÄÜÂú×ãÕâЩ¶à±äµÄÒªÇó£¬ÎªÕâЩ²úÆ·³É¹¦µØÌṩÁËÁé»î¡¢¸ßÐÔÄܵĴ洢½â¾ö·½°¸¡£

Bigtable¿´ÆðÀ´ÏñÒ»¸öÊý¾Ý¿â£¬²ÉÓÃÁ˺ܶàÊý¾Ý¿âµÄʵÏÖ²ßÂÔ¡£µ«ÊÇBigtable²¢²»Ö§³ÖÍêÕûµÄ¹ØÏµÐÍÊý¾ÝÄ£ÐÍ£»¶øÊÇΪ¿Í»§¶ËÌṩÁËÒ»ÖÖ¼òµ¥µÄÊý¾ÝÄ£ÐÍ£¬¿Í»§¶Ë¿ÉÒÔ¶¯Ì¬µØ¿ØÖÆÊý¾ÝµÄ²¼¾ÖºÍ¸ñʽ£¬²¢ÇÒÀûÓõײãÊý¾Ý´æ´¢µÄ¾Ö²¿ÐÔÌØÕ÷¡£Bigtable½«Êý¾Ýͳͳ¿´³ÉÎÞÒâÒåµÄ×Ö½Ú´®£¬¿Í»§¶ËÐèÒª½«½á¹¹»¯ºÍ·Ç½á¹¹»¯Êý¾Ý´®Ðл¯ÔÙ´æÈëBigtable¡£

ÏÂÎĶÔBigTableµÄÊý¾ÝÄ£Ðͺͻù±¾¹¤×÷Ô­Àí½øÐнéÉÜ£¬¶ø¸÷ÖÖÓÅ»¯¼¼Êõ£¨ÈçѹËõ¡¢Bloom FilterµÈ£©²»ÔÚÌÖÂÛ·¶Î§¡£

2 BigTableµÄÊý¾ÝÄ£ÐÍ

Bigtable²»ÊǹØÏµÐÍÊý¾Ý¿â£¬µ«ÊÇÈ´ÑØÓÃÁËºÜ¶à¹ØÏµÐÍÊý¾Ý¿âµÄÊõÓÏñtable£¨±í£©¡¢row£¨ÐУ©¡¢column£¨ÁУ©µÈ¡£ÕâÈÝÒ×ÈöÁÕßÎóÈëÆç;£¬½«ÆäÓë¹ØÏµÐÍÊý¾Ý¿âµÄ¸ÅÄî¶ÔÓ¦ÆðÀ´£¬´Ó¶øÄÑÒÔÀí½âÂÛÎÄ¡£Understanding HBase and BigTableÊÇÆªºÜÓÅÐãµÄÎÄÕ£¬¿ÉÒÔ°ïÖú¶ÁÕß´Ó¹ØÏµÐÍÊý¾ÝÄ£Ð͵Ä˼ά¶¨ÊÆÖÐ×ß³öÀ´¡£

±¾ÖÊÉÏ˵£¬BigtableÊÇÒ»¸ö¼üÖµ£¨key-value£©Ó³Éä¡£°´×÷ÕßµÄ˵·¨£¬BigtableÊÇÒ»¸öÏ¡ÊèµÄ£¬·Ö²¼Ê½µÄ£¬³Ö¾Ã»¯µÄ£¬¶àάµÄÅÅÐòÓ³Éä¡£

ÏÈÀ´¿´¿´¶àά¡¢ÅÅÐò¡¢Ó³Éä¡£BigtableµÄ¼üÓÐÈýά£¬·Ö±ðÊÇÐмü£¨row key£©¡¢Áмü£¨column key£©ºÍʱ¼ä´Á£¨timestamp£©£¬ÐмüºÍÁмü¶¼ÊÇ×Ö½Ú´®£¬Ê±¼ä´ÁÊÇ64λÕûÐÍ£»¶øÖµÊÇÒ»¸ö×Ö½Ú´®¡£¿ÉÒÔÓà (row:string, column:string, time:int64)¡ústring À´±íʾһÌõ¼üÖµ¶Ô¼Ç¼¡£

Ðмü¿ÉÒÔÊÇÈÎÒâ×Ö½Ú´®£¬Í¨³£ÓÐ10-100×Ö½Ú¡£ÐеĶÁд¶¼ÊÇÔ­×ÓÐԵġ£Bigtable°´ÕÕÐмüµÄ×ÖµäÐò´æ´¢Êý¾Ý¡£BigtableµÄ±í»á¸ù¾ÝÐмü×Ô¶¯»®·ÖΪƬ£¨tablet£©£¬Æ¬ÊǸºÔؾùºâµÄµ¥Ôª¡£×î³õ±í¶¼Ö»ÓÐÒ»¸öƬ£¬µ«Ëæ×Å±í²»¶ÏÔö´ó£¬Æ¬»á×Ô¶¯·ÖÁÑ£¬Æ¬µÄ´óС¿ØÖÆÔÚ100-200MB¡£ÐÐÊDZíµÄµÚÒ»¼¶Ë÷Òý£¬ÎÒÃÇ¿ÉÒ԰ѸÃÐеÄÁС¢Ê±¼äºÍÖµ¿´³ÉÒ»¸öÕûÌ壬¼ò»¯ÎªÒ»Î¬¼üÖµÓ³É䣬ÀàËÆÓÚ£º

table{  
02. "1" : {sth.},//Ò»ÐÐ
03. "aaaaa" : {sth.},
04. "aaaab" : {sth.},
05. "xyz" : {sth.},
06. "zzzzz" : {sth.}
07.}

ÁÐÊǵڶþ¼¶Ë÷Òý£¬Ã¿ÐÐÓµÓеÄÁÐÊDz»ÊÜÏÞÖÆµÄ£¬¿ÉÒÔËæÊ±Ôö¼Ó¼õÉÙ¡£ÎªÁË·½±ã¹ÜÀí£¬Áб»·ÖΪ¶à¸öÁÐ×壨column family£¬ÊÇ·ÃÎÊ¿ØÖƵĵ¥Ôª£©£¬Ò»¸öÁÐ×åÀïµÄÁÐÒ»°ã´æ´¢ÏàͬÀàÐ͵ÄÊý¾Ý¡£Ò»ÐеÄÁÐ×åºÜÉٱ仯£¬µ«ÊÇÁÐ×åÀïµÄÁпÉÒÔËæÒâÌí¼Óɾ³ý¡£Áмü°´ÕÕfamily:qualifier¸ñʽÃüÃûµÄ¡£Õâ´ÎÎÒÃǽ«ÁÐÄóöÀ´£¬½«Ê±¼äºÍÖµ¿´³ÉÒ»¸öÕûÌ壬¼ò»¯Îª¶þά¼üÖµÓ³É䣬ÀàËÆÓÚ£º

table{  
02. // ...
03. "aaaaa" : { //Ò»ÐÐ
04. "A:foo" : {sth.},//Ò»ÁÐ
05. "A:bar" : {sth.},//Ò»ÁÐ
06. "B:" : {sth.} //Ò»ÁУ¬ÁÐ×åÃûΪB£¬µ«ÊÇÁÐÃûÊÇ¿Õ×Ö´®
07. },
08. "aaaab" : { //Ò»ÐÐ
09. "A:foo" : {sth.},
10. "B:" : {sth.}
11. },
12. // ...
13.}

»òÕß¿ÉÒÔ½«ÁÐ×åµ±×÷Ò»²ãеÄË÷Òý£¬ÀàËÆÓÚ£º

table{  
02. // ...
03. "aaaaa" : { //Ò»ÐÐ
04. "A" : { //ÁÐ×åA
05. "foo" : {sth.}, //Ò»ÁÐ
06. "bar" : {sth.}
07. },
08. "B" : { //ÁÐ×åB
09. "" : {sth.}
10. }
11. },
12. "aaaab" : { //Ò»ÐÐ
13. "A" : {
14. "foo" : {sth.},
15. },
16. "B" : {
17. "" : "ocean"
18. }
19. },
20. // ...
21.}

ʱ¼ä´ÁÊǵÚÈý¼¶Ë÷Òý¡£BigtableÔÊÐí±£´æÊý¾ÝµÄ¶à¸ö°æ±¾£¬°æ±¾Çø·ÖµÄÒÀ¾Ý¾ÍÊÇʱ¼ä´Á¡£Ê±¼ä´Á¿ÉÒÔÓÉBigtable¸³Öµ£¬´ú±íÊý¾Ý½øÈëBigtableµÄ׼ȷʱ¼ä£¬Ò²¿ÉÒÔÓɿͻ§¶Ë¸³Öµ¡£Êý¾ÝµÄ²»Í¬°æ±¾°´ÕÕʱ¼ä´Á½µÐò´æ´¢£¬Òò´ËÏȶÁµ½µÄÊÇ×îа汾µÄÊý¾Ý¡£ÎÒÃǼÓÈëʱ¼ä´Áºó£¬¾ÍµÃµ½ÁËBigtableµÄÍêÕûÊý¾ÝÄ£ÐÍ£¬ÀàËÆÓÚ£º

table{  
02. // ...
03. "aaaaa" : { //Ò»ÐÐ
04. "A:foo" : { //Ò»ÁÐ
05. 15 : "y", //Ò»¸ö°æ±¾
06. 4 : "m"
07. },
08. "A:bar" : { //Ò»ÁÐ
09. 15 : "d",
10. },
11. "B:" : { //Ò»ÁÐ
12. 6 : "w"
13. 3 : "o"
14. 1 : "w"
15. }
16. },
17. // ...
18.}

²éѯʱ£¬Èç¹ûÖ»¸ø³öÐÐÁУ¬ÄÇô·µ»ØµÄÊÇ×îа汾µÄÊý¾Ý£»Èç¹û¸ø³öÁËÐÐÁÐʱ¼ä´Á£¬ÄÇô·µ»ØµÄÊÇʱ¼äСÓÚ»òµÈÓÚʱ¼ä´ÁµÄÊý¾Ý¡£±ÈÈ磬ÎÒÃDzéѯ"aaaaa"/"A:foo"£¬·µ»ØµÄÖµÊÇ"y"£»²éѯ"aaaaa"/"A:foo"/10£¬·µ»ØµÄ½á¹û¾ÍÊÇ"m"£»²éѯ"aaaaa"/"A:foo"/2£¬·µ»ØµÄ½á¹ûÊǿա£

ͼ1ÊÇBigtableÂÛÎÄÀï¸ø³öµÄÀý×Ó£¬Webtable±í´æ´¢ÁË´óÁ¿µÄÍøÒ³ºÍÏà¹ØÐÅÏ¢¡£ÔÚWebtable£¬Ã¿Ò»Ðд洢һ¸öÍøÒ³£¬Æä·´×ªµÄurl×÷ΪÐмü£¬±ÈÈçmaps.google.com/index.htmlµÄÊý¾Ý´æ´¢ÔÚ¼üΪcom.google.maps/index.htmlµÄÐÐÀ·´×ªµÄÔ­ÒòÊÇΪÁËÈÃͬһ¸öÓòÃûϵÄ×ÓÓòÃûÍøÒ³Äܾۼ¯ÔÚÒ»Æð¡£Í¼1ÖеÄÁÐ×å"anchor"±£´æÁ˸ÃÍøÒ³µÄÒýÓÃÕ¾µã£¨±ÈÈçÒýÓÃÁËCNNÖ÷Ò³µÄÕ¾µã£©£¬qualifierÊÇÒýÓÃÕ¾µãµÄÃû³Æ£¬¶øÊý¾ÝÊÇÁ´½ÓÎı¾£»ÁÐ×å"contents"±£´æµÄÊÇÍøÒ³µÄÄÚÈÝ£¬Õâ¸öÁÐ×åÖ»ÓÐÒ»¸ö¿ÕÁÐ"contents:"¡£Í¼1ÖÐ"contents:"ÁÐϱ£´æÁËÍøÒ³µÄÈý¸ö°æ±¾£¬ÎÒÃÇ¿ÉÒÔÓÃ("com.cnn.www", "contents:", t5)À´ÕÒµ½CNNÖ÷Ò³ÔÚt5ʱ¿ÌµÄÄÚÈÝ¡£

ÔÙÀ´¿´¿´×÷Õß˵µÄÆäËüÌØÕ÷£ºÏ¡Ê裬·Ö²¼Ê½£¬³Ö¾Ã»¯¡£³Ö¾Ã»¯µÄÒâ˼ºÜ¼òµ¥£¬BigtableµÄÊý¾Ý×îÖÕ»áÒÔÎļþµÄÐÎʽ·Åµ½GFSÈ¥¡£Bigtable½¨Á¢ÔÚGFSÖ®Éϱ¾Éí¾ÍÒâζ×Å·Ö²¼Ê½£¬µ±È»·Ö²¼Ê½µÄÒâÒ廹²»½öÏÞÓÚ´Ë¡£Ï¡ÊèµÄÒâ˼ÊÇ£¬Ò»¸ö±íÀﲻͬµÄÐУ¬ÁпÉÄÜÍêÍêȫȫ²»Ò»Ñù¡£

3 Ö§³Å¼¼Êõ

BigtableÒÀÀµÓÚgoogleµÄ¼¸Ïî¼¼Êõ¡£ÓÃGFSÀ´´æ´¢ÈÕÖ¾ºÍÊý¾ÝÎļþ£»°´SSTableÎļþ¸ñʽ´æ´¢Êý¾Ý£»ÓÃChubby¹ÜÀíÔªÊý¾Ý¡£

GFS²Î¼û¹È¸è¼¼Êõ"Èý±¦"Ö®¹È¸èÎļþϵͳ¡£BigTableµÄÊý¾ÝºÍÈÕÖ¾¶¼ÊÇдÈëGFSµÄ¡£

SSTableµÄÈ«³ÆÊÇSorted Strings Table£¬ÊÇÒ»ÖÖ²»¿ÉÐ޸ĵÄÓÐÐòµÄ¼üÖµÓ³É䣬ÌṩÁ˲éѯ¡¢±éÀúµÈ¹¦ÄÜ¡£Ã¿¸öSSTableÓÉһϵÁеĿ飨block£©×é³É£¬Bigtable½«¿éĬÈÏÉèΪ64KB¡£ÔÚSSTableµÄβ²¿´æ´¢×Å¿éË÷Òý£¬ÔÚ·ÃÎÊSSTableʱ£¬Õû¸öË÷Òý»á±»¶ÁÈëÄÚ´æ¡£BigTableÂÛÎÄûÓÐÌáµ½SSTableµÄ¾ßÌå½á¹¹£¬LevelDbÈÕ֪¼֮ËÄ£º SSTableÎļþÕâÆªÎÄÕ¶ÔLevelDbµÄSSTable¸ñʽ½øÐÐÁ˽éÉÜ£¬ÒòΪLevelDBµÄ×÷ÕßJeffreyDeanÕýÊÇBigTableµÄÉè¼ÆÊ¦£¬ËùÒÔ¼«¾ß²Î¿¼¼ÛÖµ¡£Ã¿Ò»¸öƬ£¨tablet£©ÔÚGFSÀï¶¼Êǰ´ÕÕSSTableµÄ¸ñʽ´æ´¢µÄ£¬Ã¿¸öƬ¿ÉÄܶÔÓ¦¶à¸öSSTable¡£

ChubbyÊÇÒ»Öָ߿ÉÓõķֲ¼Ê½Ëø·þÎñ£¬ChubbyÓÐÎå¸ö»îÔ¾¸±±¾£¬Í¬Ê±Ö»ÓÐÒ»¸öÖ÷¸±±¾Ìṩ·þÎñ£¬¸±±¾Ö®¼äÓÃPaxosË㷨ά³ÖÒ»ÖÂÐÔ£¬ChubbyÌṩÁËÒ»¸öÃüÃû¿Õ¼ä£¨°üÀ¨Ò»Ð©Ä¿Â¼ºÍÎļþ£©£¬Ã¿¸öĿ¼ºÍÎļþ¾ÍÊÇÒ»¸öËø£¬ChubbyµÄ¿Í»§¶Ë±ØÐëºÍChubby±£³Ö»á»°£¬¿Í»§¶ËµÄ»á»°Èô¹ýÆÚÔò»á¶ªÊ§ËùÓеÄËø¡£¹ØÓÚChubbyµÄÏêϸÐÅÏ¢¿ÉÒÔ¿´googleµÄÁíһƪÂÛÎÄ£ºThe Chubby lock service for loosely-coupled distributed systems¡£ChubbyÓÃÓÚÆ¬¶¨Î»£¬Æ¬·þÎñÆ÷µÄ״̬¼à¿Ø£¬·ÃÎÊ¿ØÖÆÁÐ±í´æ´¢µÈÈÎÎñ¡£

4 Bigtable¼¯Èº

Bigtable¼¯Èº°üÀ¨Èý¸öÖ÷Òª²¿·Ö£ºÒ»¸ö¹©¿Í»§¶ËʹÓõĿ⣬һ¸öÖ÷·þÎñÆ÷£¨master server£©£¬Ðí¶àƬ·þÎñÆ÷£¨tablet server£©¡£

ÕýÈçÊý¾ÝÄ£ÐÍС½ÚËù˵£¬Bigtable»á½«±í£¨table£©½øÐÐ·ÖÆ¬£¬Æ¬£¨tablet£©µÄ´óСά³ÖÔÚ100-200MB·¶Î§£¬Ò»µ©³¬³ö·¶Î§¾Í½«·ÖÁѳɸüСµÄƬ£¬»òÕߺϲ¢³É¸ü´óµÄƬ¡£Ã¿¸öƬ·þÎñÆ÷¸ºÔðÒ»¶¨Á¿µÄƬ£¬´¦Àí¶ÔÆäƬµÄ¶ÁдÇëÇó£¬ÒÔ¼°Æ¬µÄ·ÖÁÑ»òºÏ²¢¡£Æ¬·þÎñÆ÷¿ÉÒÔ¸ù¾Ý¸ºÔØËæÊ±Ìí¼ÓºÍɾ³ý¡£ÕâÀïÆ¬·þÎñÆ÷²¢²»ÕæÊµ´æ´¢Êý¾Ý£¬¶øÏ൱ÓÚÒ»¸öÁ¬½ÓBigtableºÍGFSµÄ´úÀí£¬¿Í»§¶ËµÄһЩÊý¾Ý²Ù×÷¶¼Í¨¹ýƬ·þÎñÆ÷´úÀí¼ä½Ó·ÃÎÊGFS¡£

Ö÷·þÎñÆ÷¸ºÔð½«Æ¬·ÖÅä¸øÆ¬·þÎñÆ÷£¬¼à¿ØÆ¬·þÎñÆ÷µÄÌí¼ÓºÍɾ³ý£¬Æ½ºâƬ·þÎñÆ÷µÄ¸ºÔØ£¬´¦Àí±íºÍÁÐ×åµÄ´´½¨µÈ¡£×¢Ò⣬Ö÷·þÎñÆ÷²»´æ´¢ÈÎºÎÆ¬£¬²»ÌṩÈκÎÊý¾Ý·þÎñ£¬Ò²²»ÌṩƬµÄ¶¨Î»ÐÅÏ¢¡£

¿Í»§¶ËÐèÒª¶ÁдÊý¾Ýʱ£¬Ö±½ÓÓëÆ¬·þÎñÆ÷ÁªÏµ¡£ÒòΪ¿Í»§¶Ë²¢²»ÐèÒª´ÓÖ÷·þÎñÆ÷»ñȡƬµÄλÖÃÐÅÏ¢£¬ËùÒÔ´ó¶àÊý¿Í»§¶Ë´ÓÀ´²»ÐèÒª·ÃÎÊÖ÷·þÎñÆ÷£¬Ö÷·þÎñÆ÷µÄ¸ºÔØÒ»°ãºÜÇá¡£

5 ƬµÄ¶¨Î»

Ç°ÃæÌáµ½Ö÷·þÎñÆ÷²»ÌṩƬµÄλÖÃÐÅÏ¢£¬ÄÇô¿Í»§¶ËÊÇÈçºÎ·ÃÎÊÆ¬µÄÄØ£¿À´¿´¿´ÂÛÎĸøµÄʾÒâͼ£¬BigtableʹÓÃÒ»¸öÀàËÆB+Ê÷µÄÊý¾Ý½á¹¹´æ´¢Æ¬µÄλÖÃÐÅÏ¢¡£

Ê×ÏÈÊǵÚÒ»²ã£¬Chubby file¡£ÕâÒ»²ãÊÇÒ»¸öChubbyÎļþ£¬Ëü±£´æ×Åroot tabletµÄλÖá£Õâ¸öChubbyÎļþÊôÓÚChubby·þÎñµÄÒ»²¿·Ö£¬Ò»µ©Chubby²»¿ÉÓ㬾ÍÒâζ×ŶªÊ§ÁËroot tabletµÄλÖã¬Õû¸öBigtableÒ²¾Í²»¿ÉÓÃÁË¡£

µÚ¶þ²ãÊÇroot tablet¡£root tabletÆäʵÊÇÔªÊý¾Ý±í£¨METADATA table£©µÄµÚÒ»¸ö·ÖƬ£¬Ëü±£´æ×ÅÔªÊý¾Ý±íÆäËüƬµÄλÖá£root tabletºÜÌØ±ð£¬ÎªÁ˱£Ö¤Ê÷µÄÉî¶È²»±ä£¬root tablet´Ó²»·ÖÁÑ¡£

µÚÈý²ãÊÇÆäËüµÄÔªÊý¾ÝƬ£¬ËüÃǺÍroot tabletÒ»Æð×é³ÉÍêÕûµÄÔªÊý¾Ý±í¡£Ã¿¸öÔªÊý¾ÝƬ¶¼°üº¬ÁËÐí¶àÓû§Æ¬µÄλÖÃÐÅÏ¢¡£

¿ÉÒÔ¿´³öÕû¸ö¶¨Î»ÏµÍ³ÆäʵֻÊÇÁ½²¿·Ö£¬Ò»¸öChubbyÎļþ£¬Ò»¸öÔªÊý¾Ý±í¡£×¢ÒâÔªÊý¾Ý±íËäÈ»ÌØÊ⣬µ«Ò²ÈÔÈ»·þ´ÓǰÎĵÄÊý¾ÝÄ£ÐÍ£¬Ã¿¸ö·ÖƬҲ¶¼ÊÇÓÉרÃŵᬷþÎñÆ÷¸ºÔð£¬Õâ¾ÍÊDz»ÐèÒªÖ÷·þÎñÆ÷ÌṩλÖÃÐÅÏ¢µÄÔ­Òò¡£¿Í»§¶Ë»á»º´æÆ¬µÄλÖÃÐÅÏ¢£¬Èç¹ûÔÚ»º´æÀïÕÒ²»µ½Ò»¸öƬµÄλÖÃÐÅÏ¢£¬¾ÍÐèÒª²éÕÒÕâ¸öÈý²ã½á¹¹ÁË£¬°üÀ¨·ÃÎÊÒ»´ÎChubby·þÎñ£¬·ÃÎÊÁ½´ÎƬ·þÎñÆ÷¡£

6 ƬµÄ´æ´¢ºÍ·ÃÎÊ

ƬµÄÊý¾Ý×îÖÕ»¹ÊÇдµ½GFSÀïµÄ£¬Æ¬ÔÚGFSÀïµÄÎïÀíÐÎ̬¾ÍÊÇÈô¸É¸öSSTableÎļþ¡£Í¼5չʾÁ˶Áд²Ù×÷»ù±¾Çé¿ö¡£

µ±Æ¬·þÎñÆ÷ÊÕµ½Ò»¸öдÇëÇ󣬯¬·þÎñÆ÷Ê×Ïȼì²éÇëÇóÊÇ·ñºÏ·¨¡£Èç¹ûºÏ·¨£¬ÏȽ«Ð´ÇëÇóÌá½»µ½ÈÕ־ȥ£¬È»ºó½«Êý¾ÝдÈëÄÚ´æÖеÄmemtable¡£memtableÏ൱ÓÚSSTableµÄ»º´æ£¬µ±memtable³É³¤µ½Ò»¶¨¹æÄ£»á±»¶³½á£¬BigtableËæÖ®´´½¨Ò»¸öеÄmemtable£¬²¢ÇÒ½«¶³½áµÄmemtableת»»ÎªSSTable¸ñʽдÈëGFS£¬Õâ¸ö²Ù×÷³ÆÎªminor compaction¡£

µ±Æ¬·þÎñÆ÷ÊÕµ½Ò»¸ö¶ÁÇëÇó£¬Í¬ÑùÒª¼ì²éÇëÇóÊÇ·ñºÏ·¨¡£Èç¹ûºÏ·¨£¬Õâ¸ö¶Á²Ù×÷»á²é¿´ËùÓÐSSTableÎļþºÍmemtableµÄºÏ²¢ÊÓͼ£¬ÒòΪSSTableºÍmemtable±¾Éí¶¼ÊÇÒÑÅÅÐòµÄ£¬ËùÒԺϲ¢Ï൱¿ì¡£

ÿһ´Îminor compaction¶¼»á²úÉúÒ»¸öеÄSSTableÎļþ£¬SSTableÎļþÌ«¶à¶Á²Ù×÷µÄЧÂʾͽµµÍÁË£¬ËùÒÔBigtable¶¨ÆÚÖ´ÐÐmerging compaction²Ù×÷£¬½«¼¸¸öSSTableºÍmemtableºÏ²¢ÎªÒ»¸öеÄSSTable¡£BigTable»¹Óиö¸üÀ÷º¦µÄ½Ðmajor compaction£¬Ëü½«ËùÓÐSSTableºÏ²¢ÎªÒ»¸öеÄSSTable¡£

Òź¶µÄÊÇ£¬BigTable×÷ÕßûÓнéÉÜmemtableºÍSSTableµÄÏêϸÊý¾Ý½á¹¹¡£

7 BigTableºÍGFSµÄ¹ØÏµ

¼¯Èº°üÀ¨Ö÷·þÎñÆ÷ºÍƬ·þÎñÆ÷£¬Ö÷·þÎñÆ÷¸ºÔð½«Æ¬·ÖÅä¸øÆ¬·þÎñÆ÷£¬¶ø¾ßÌåµÄÊý¾Ý·þÎñÔòȫȨÓÉÆ¬·þÎñÆ÷¸ºÔð¡£µ«ÊDz»ÒªÎóÒÔΪƬ·þÎñÆ÷ÕæµÄ´æ´¢ÁËÊý¾Ý£¨³ýÁËÄÚ´æÖÐmemtableµÄÊý¾Ý£©£¬Êý¾ÝµÄÕæÊµÎ»ÖÃÖ»ÓÐGFS²ÅÖªµÀ£¬Ö÷·þÎñÆ÷½«Æ¬·ÖÅä¸øÆ¬·þÎñÆ÷µÄÒâ˼Ӧ¸ÃÊÇ£¬Æ¬·þÎñÆ÷»ñÈ¡ÁËÆ¬µÄËùÓÐSSTableÎļþÃû£¬Æ¬·þÎñÆ÷ͨ¹ýһЩË÷Òý»úÖÆ¿ÉÒÔÖªµÀËùÐèÒªµÄÊý¾ÝÔÚÄĸöSSTableÎļþ£¬È»ºó´ÓGFSÖжÁÈ¡SSTableÎļþµÄÊý¾Ý£¬Õâ¸öSSTableÎļþ¿ÉÄÜ·Ö²¼Ôںü¸Ì¨chunkserverÉÏ¡£

8 ÔªÊý¾Ý±íµÄ½á¹¹

ÔªÊý¾Ý±í£¨METADATA table£©ÊÇÒ»ÕÅÌØÊâµÄ±í£¬Ëü±»ÓÃÓÚÊý¾ÝµÄ¶¨Î»ÒÔ¼°Ò»Ð©ÔªÊý¾Ý·þÎñ£¬²»¿Éν²»ÖØÒª¡£µ«ÊÇBigtableÂÛÎÄÀïÖ»¸ø³öÁËÉÙÁ¿ÏßË÷£¬¶ø¶Ô±íµÄ¾ßÌå½á¹¹Ã»ÓÐ˵Ã÷¡£ÕâÀïÎÒÊÔͼ¸ù¾ÝÂÛÎĵÄһЩÏßË÷£¬²Â²âһϱíµÄ½á¹¹¡£Ê×ÏÈÁгöÂÛÎÄÖеÄÏßË÷£º

1.The METADATA table stores the location of a tablet under a row key that is an encoding of the tablet's table identifier and its end row.

2.Each METADATA row stores approximately 1KB of data in memory£¨ÒòΪ·ÃÎÊÁ¿±È½Ï´ó£¬ÔªÊý¾Ý±íÊÇ·ÅÔÚÄÚ´æÀïµÄ£¬Õâ¸öÓÅ»¯ÔÚÂÛÎĵÄlocality groupsÖÐÌáµ½£©.This feature£¨½«locality group·Åµ½ÄÚ´æÖеÄÌØÐÔ£© is useful for small pieces of data that are accessed frequently: we use it internally for the location column family in the METADATA table.

3.We also store secondary information in the METADATA table, including a log of all events pertaining to each tablet(such as when a server begins
serving it).

µÚÒ»ÌõÏßË÷£¬ÔªÊý¾Ý±íµÄÐмüÊÇÓÉÆ¬ËùÊô±íÃûµÄidºÍƬ×îºóÒ»ÐбàÂë¶ø³É£¬ËùÒÔÿ¸öƬÔÚÔªÊý¾Ý±íÖÐÕ¼¾ÝÒ»Ìõ¼Ç¼£¨Ò»ÐУ©£¬¶øÇÒÐмü¼È°üº¬ÁËÆäËùÊô±íµÄÐÅÏ¢Ò²°üº¬ÁËÆäËùÓµÓеÄÐеķ¶Î§¡£Æ©Èç²ÉÈ¡×î¼òµ¥µÄ±àÂ뷽ʽ£¬ÔªÊý¾Ý±íµÄÐмüµÈÓÚstrcat(±íÃû£¬Æ¬×îºóÒ»ÐеÄÐмü)¡£

µÚ¶þµãÏßË÷£¬³ýÁËÖªµÀÔªÊý¾Ý±íµÄµØÖ·²¿·ÖÊdz£×¤ÄÚ´æÒÔÍ⣬»¹¿ÉÒÔ·¢ÏÖÔªÊý¾Ý±íÓÐÒ»¸öÁÐ×å³ÆÎªlocation£¬ÎÒÃÇÒѾ­ÖªµÀÔªÊý¾Ý±íÿһÐдú±íÒ»¸öƬ£¬ÄÇôΪʲôÐèÒªÒ»¸öÁÐ×åÀ´´æ´¢µØÖ·ÄØ£¿ÒòΪÿ¸öƬ¶¼¿ÉÄÜÓɶà¸öSSTableÎļþ×é³É£¬ÁÐ×å¿ÉÒÔÓÃÀ´´æ´¢ÈÎÒâ¶à¸öSSTableÎļþµÄλÖá£Ò»¸öºÏÀíµÄ¼ÙÉè¾ÍÊÇÿ¸öSSTableÎļþµÄλÖÃÐÅÏ¢Õ¼¾ÝÒ»ÁУ¬ÁÐÃûΪlocation:filename¡£µ±È»²»Ò»¶¨·ÇµÃÓÃÁмü´æ´¢ÍêÕûÎļþÃû£¬¸ü´óµÄ¿ÉÄÜÐÔÊǰÑSSTableÎļþÃû´æÔÚÖµÀï¡£»ñÈ¡ÁËÎļþÃû¾Í¿ÉÒÔÏòGFSË÷ÒªÊý¾ÝÁË¡£

µÚÈý¸öÏßË÷¸æËßÎÒÃÇÔªÊý¾Ý±í²»Ö¹´æ´¢Î»ÖÃÐÅÏ¢£¬Ò²¾ÍÊÇ˵ÁÐ×å²»Ö¹location£¬ÕâЩÊý¾ÝÔÝʱ²»ÊÇÔÛÃǹØÐĵġ£

ͨ¹ýÒÔÉÏÐÅÏ¢£¬ÎÒ»­ÁËÒ»¸ö¼ò»¯µÄBigtable½á¹¹Í¼£º

½á¹¹Í¼ÒÔWebtable±íΪÀý£¬±íÖд洢ÁËÍøÒס¢°Ù¶ÈºÍ¶¹°êµÄ¼¸¸öÍøÒ³¡£µ±ÎÒÃÇÏë²éÕÒ°Ù¶ÈÌù°É×òÌìµÄÍøÒ³ÄÚÈÝ£¬¿ÉÒÔÏòBigtable·¢³ö²éѯWebtable±íµÄ(com.baidu.tieba, contents:, yesterday)¡£

¼ÙÉè¿Í»§¶ËûÓиûº´æ£¬ÄÇôBigtable·ÃÎÊroot tabletµÄƬ·þÎñÆ÷£¬Ï£ÍûµÃµ½¸ÃÍøÒ³ËùÊôµÄƬµÄλÖÃÐÅÏ¢ÔÚÄĸöÔªÊý¾ÝƬÖС£Ê¹ÓÃMETADATA.Webtable.com.baidu.tiebaΪÐмüÔÚroot tabletÖвéÕÒ£¬¶¨Î»µ½×îºóÒ»¸ö±ÈËü´óµÄÊÇMETADATA.Webtable.com.baidu.www£¬ÓÚÊÇÈ·¶¨ÐèÒªµÄ¾ÍÊÇÔªÊý¾Ý±íµÄƬA¡£·ÃÎÊÆ¬AµÄƬ·þÎñÆ÷£¬¼ÌÐø²éÕÒWebtable.com.baidu.tieba£¬¶¨Î»µ½Webtable.com.baidu.wwwÊDZÈËü´óµÄ£¬È·¶¨ÐèÒªµÄÊÇWebtable±íµÄƬB¡£·ÃÎÊÆ¬BµÄƬ·þÎñÆ÷£¬»ñµÃÊý¾Ý¡£

ÕâÀïÐèҪעÒâµÄÊÇ£¬Ã¿¸öƬʵ¼Ê¶¼ÓÉÈô¸ÉSSTableÎļþºÍmemtable×é³É£¬¶øÇÒÕâЩSSTableºÍmemtable¶¼ÊÇÒÑÅÅÐòµÄ¡£Õâ¾Íµ¼Ö²éÕÒÆ¬Bʱ£¬¿ÉÄÜÐèÒª½«ËùÓÐSSTableºÍmemtable¶¼²éÕÒÒ»±é£»ÁíÍâ¿Í»§¶ËÓ¦¸Ã²»»áÖ±½Ó´ÓÔªÊý¾Ý±í»ñµÃSSTableµÄÎļþÃû£¬¶øÖ»ÊÇ»ñµÃƬÊôÓÚÆ¬·þÎñÆ÷µÄÐÅÏ¢£¬Í¨¹ýƬ·þÎñÆ÷Ϊ´úÀí·ÃÎÊSSTable¡£

   
5146 ´Îä¯ÀÀ       27
Ïà¹ØÎÄÕÂ

»ùÓÚEAµÄÊý¾Ý¿â½¨Ä£
Êý¾ÝÁ÷½¨Ä££¨EAÖ¸ÄÏ£©
¡°Êý¾Ýºþ¡±£º¸ÅÄî¡¢ÌØÕ÷¡¢¼Ü¹¹Óë°¸Àý
ÔÚÏßÉ̳ÇÊý¾Ý¿âϵͳÉè¼Æ ˼·+Ч¹û
 
Ïà¹ØÎĵµ

GreenplumÊý¾Ý¿â»ù´¡Åàѵ
MySQL5.1ÐÔÄÜÓÅ»¯·½°¸
ijµçÉÌÊý¾ÝÖÐ̨¼Ü¹¹Êµ¼ù
MySQL¸ßÀ©Õ¹¼Ü¹¹Éè¼Æ
Ïà¹Ø¿Î³Ì

Êý¾ÝÖÎÀí¡¢Êý¾Ý¼Ü¹¹¼°Êý¾Ý±ê×¼
MongoDBʵս¿Î³Ì
²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
PostgreSQLÊý¾Ý¿âʵսÅàѵ
×îл¼Æ»®
DeepSeekÔÚÈí¼þ²âÊÔÓ¦ÓÃʵ¼ù 4-12[ÔÚÏß]
DeepSeek´óÄ£ÐÍÓ¦Óÿª·¢Êµ¼ù 4-19[ÔÚÏß]
UAF¼Ü¹¹ÌåϵÓëʵ¼ù 4-11[±±¾©]
AIÖÇÄÜ»¯Èí¼þ²âÊÔ·½·¨Óëʵ¼ù 5-23[ÉϺ£]
»ùÓÚ UML ºÍEA½øÐзÖÎöÉè¼Æ 4-26[±±¾©]
ÒµÎñ¼Ü¹¹Éè¼ÆÓ뽨ģ 4-18[±±¾©]

MySQLË÷Òý±³ºóµÄÊý¾Ý½á¹¹
MySQLÐÔÄܵ÷ÓÅÓë¼Ü¹¹Éè¼Æ
SQL ServerÊý¾Ý¿â±¸·ÝÓë»Ö¸´
ÈÃÊý¾Ý¿â·ÉÆðÀ´ 10´óDB2ÓÅ»¯
oracleµÄÁÙʱ±í¿Õ¼äдÂú´ÅÅÌ
Êý¾Ý¿âµÄ¿çƽ̨Éè¼Æ


²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿â
¸ß¼¶Êý¾Ý¿â¼Ü¹¹Éè¼ÆÊ¦
HadoopÔ­ÀíÓëʵ¼ù
Oracle Êý¾Ý²Ö¿â
Êý¾Ý²Ö¿âºÍÊý¾ÝÍÚ¾ò
OracleÊý¾Ý¿â¿ª·¢Óë¹ÜÀí


GE Çø¿éÁ´¼¼ÊõÓëʵÏÖÅàѵ
º½Ìì¿Æ¹¤Ä³×Ó¹«Ë¾ Nodejs¸ß¼¶Ó¦Óÿª·¢
ÖÐÊ¢Òæ»ª ׿Խ¹ÜÀíÕß±ØÐë¾ß±¸µÄÎåÏîÄÜÁ¦
ijÐÅÏ¢¼¼Êõ¹«Ë¾ PythonÅàѵ
ij²©²ÊITϵͳ³§ÉÌ Ò×ÓÃÐÔ²âÊÔÓëÆÀ¹À
ÖйúÓÊ´¢ÒøÐÐ ²âÊÔ³ÉÊì¶ÈÄ£Ðͼ¯³É(TMMI)
ÖÐÎïÔº ²úÆ·¾­ÀíÓë²úÆ·¹ÜÀí