¹È¸è¼¼Êõ"Èý±¦"Ö®MapReduce
1. MapReduceÊǸÉɶµÄ
ÒòΪûÕÒµ½¹È¸èµÄʾÒâͼ£¬ËùÒÔÎÒÏë½èÓÃÒ»ÕÅHadoopÏîÄ¿µÄ½á¹¹Í¼À´ËµÃ÷ÏÂMapReduceËù´¦µÄλÖã¬ÈçÏÂͼ¡£

Hadoopʵ¼ÊÉϾÍÊǹȸèÈý±¦µÄ¿ªÔ´ÊµÏÖ£¬Hadoop MapReduce¶ÔÓ¦Google
MapReduce£¬HBase¶ÔÓ¦BigTable£¬HDFS¶ÔÓ¦GFS¡£HDFS£¨»òGFS£©ÎªÉϲãÌṩ¸ßЧµÄ·Ç½á¹¹»¯´æ´¢·þÎñ£¬HBase£¨»òBigTable£©ÊÇÌṩ½á¹¹»¯Êý¾Ý·þÎñµÄ·Ö²¼Ê½Êý¾Ý¿â£¬Hadoop
MapReduce£¨»òGoogle MapReduce£©ÊÇÒ»ÖÖ²¢ÐмÆËãµÄ±à³ÌÄ£ÐÍ£¬ÓÃÓÚ×÷Òµµ÷¶È¡£
GFSºÍBigTableÒѾΪÎÒÃÇÌṩÁ˸ßÐÔÄÜ¡¢¸ß²¢·¢µÄ·þÎñ£¬µ«ÊDz¢Ðбà³Ì¿É²»ÊÇËùÓгÌÐòÔ±¶¼ÍæµÃתµÄ»î¶ù£¬Èç¹ûÎÒÃǵÄÓ¦Óñ¾Éí²»Äܲ¢·¢£¬ÄÇGFS¡¢BigTableÒ²¶¼ÊÇûÓÐÒâÒåµÄ¡£MapReduceµÄΰ´óÖ®´¦¾ÍÔÚÓÚÈò»ÊìϤ²¢Ðбà³ÌµÄ³ÌÐòÔ±Ò²Äܳä·Ö·¢»Ó·Ö²¼Ê½ÏµÍ³µÄÍþÁ¦¡£
¼òµ¥¸ÅÀ¨µÄ˵£¬MapReduceÊǽ«Ò»¸ö´ó×÷Òµ²ð·ÖΪ¶à¸öС×÷ÒµµÄ¿ò¼Ü£¨´ó×÷ÒµºÍС×÷ÒµÓ¦¸Ã±¾ÖÊÊÇÒ»ÑùµÄ£¬Ö»ÊǹæÄ£²»Í¬£©£¬Óû§ÐèÒª×öµÄ¾ÍÊǾö¶¨²ð³É¶àÉÙ·Ý£¬ÒÔ¼°¶¨Òå×÷Òµ±¾Éí¡£
ÏÂÃæÓÃÒ»¸ö¹á´©È«ÎĵÄÀý×ÓÀ´½âÊÍMapReduceÊÇÈçºÎ¹¤×÷µÄ¡£
2. Àý×Ó£ºÍ³¼Æ´ÊƵ
Èç¹ûÎÒÏëͳ¼ÆÏ¹ýÈ¥10Äê¼ÆËã»úÂÛÎijöÏÖ×î¶àµÄ¼¸¸öµ¥´Ê£¬¿´¿´´ó¼Ò¶¼ÔÚÑо¿Ð©Ê²Ã´£¬ÄÇÎÒÊÕ¼¯ºÃÂÛÎĺ󣬸ÃÔõô°ìÄØ£¿
·½·¨Ò»£ºÎÒ¿ÉÒÔдһ¸öС³ÌÐò£¬°ÑËùÓÐÂÛÎݴ˳Ðò±éÀúÒ»±é£¬Í³¼ÆÃ¿Ò»¸öÓöµ½µÄµ¥´ÊµÄ³öÏÖ´ÎÊý£¬×îºó¾Í¿ÉÒÔÖªµÀÄöµ¥´Ê×îÈÈÃÅÁË¡£
ÕâÖÖ·½·¨ÔÚÊý¾Ý¼¯±È½ÏСʱ£¬ÊǷdz£ÓÐЧµÄ£¬¶øÇÒʵÏÖ×î¼òµ¥£¬ÓÃÀ´½â¾öÕâ¸öÎÊÌâºÜºÏÊÊ¡£
·½·¨¶þ£ºÐ´Ò»¸ö¶àÏ̳߳ÌÐò£¬²¢·¢±éÀúÂÛÎÄ¡£
Õâ¸öÎÊÌâÀíÂÛÉÏÊÇ¿ÉÒԸ߶Ȳ¢·¢µÄ£¬ÒòΪͳ¼ÆÒ»¸öÎļþʱ²»»áÓ°Ïìͳ¼ÆÁíÒ»¸öÎļþ¡£µ±ÎÒÃǵĻúÆ÷ÊǶàºË»òÕß¶à´¦ÀíÆ÷£¬·½·¨¶þ¿Ï¶¨±È·½·¨Ò»¸ßЧ¡£µ«ÊÇдһ¸ö¶àÏ̳߳ÌÐòÒª±È·½·¨Ò»À§ÄѶàÁË£¬ÎÒÃDZØÐë×Ô¼ºÍ¬²½¹²ÏíÊý¾Ý£¬±ÈÈçÒª·ÀÖ¹Á½¸öÏß³ÌÖØ¸´Í³¼ÆÎļþ¡£
·½·¨Èý£º°Ñ×÷Òµ½»¸ø¶à¸ö¼ÆËã»úÈ¥Íê³É¡£
ÎÒÃÇ¿ÉÒÔʹÓ÷½·¨Ò»µÄ³ÌÐò£¬²¿Êðµ½N̨»úÆ÷ÉÏÈ¥£¬È»ºó°ÑÂÛÎֳÉN·Ý£¬Ò»Ì¨»úÆ÷ÅÜÒ»¸ö×÷Òµ¡£Õâ¸ö·½·¨ÅܵÃ×ã¹»¿ì£¬µ«ÊDz¿ÊðÆðÀ´ºÜÂé·³£¬ÎÒÃÇÒªÈ˹¤°Ñ³ÌÐòcopyµ½±ðµÄ»úÆ÷£¬ÒªÈ˹¤°ÑÂÛÎļ¯·Ö¿ª£¬×îÍ´¿àµÄÊÇ»¹Òª°ÑN¸öÔËÐнá¹û½øÐÐÕûºÏ£¨µ±È»ÎÒÃÇÒ²¿ÉÒÔÔÙдһ¸ö³ÌÐò£©¡£
·½·¨ËÄ£ºÈÃMapReduceÀ´°ï°ïÎÒÃǰɣ¡
MapReduce±¾ÖÊÉϾÍÊÇ·½·¨Èý£¬µ«ÊÇÈçºÎ²ð·ÖÎļþ¼¯£¬ÈçºÎcopy³ÌÐò£¬ÈçºÎÕûºÏ½á¹ûÕâЩ¶¼ÊÇ¿ò¼Ü¶¨ÒåºÃµÄ¡£ÎÒÃÇÖ»Òª¶¨ÒåºÃÕâ¸öÈÎÎñ£¨Óû§³ÌÐò£©£¬ÆäËü¶¼½»¸øMapReduce¡£
ÔÚ½éÉÜMapReduceÈçºÎ¹¤×÷֮ǰ£¬ÏȽ²½²Á½¸öºËÐĺ¯ÊýmapºÍreduceÒÔ¼°MapReduceµÄα´úÂë¡£
3. mapº¯ÊýºÍreduceº¯Êý
mapº¯ÊýºÍreduceº¯ÊýÊǽ»¸øÓû§ÊµÏֵģ¬ÕâÁ½¸öº¯Êý¶¨ÒåÁËÈÎÎñ±¾Éí¡£
1.mapº¯Êý£º½ÓÊÜÒ»¸ö¼üÖµ¶Ô£¨key-value pair£©£¬²úÉúÒ»×éÖмä¼üÖµ¶Ô¡£MapReduce¿ò¼Ü»á½«mapº¯Êý²úÉúµÄÖмä¼üÖµ¶ÔÀï¼üÏàͬµÄÖµ´«µÝ¸øÒ»¸öreduceº¯Êý¡£
2.reduceº¯Êý£º½ÓÊÜÒ»¸ö¼ü£¬ÒÔ¼°Ïà¹ØµÄÒ»×éÖµ£¬½«Õâ×éÖµ½øÐкϲ¢²úÉúÒ»×鹿ģ¸üСµÄÖµ£¨Í¨³£Ö»ÓÐÒ»¸ö»òÁã¸öÖµ£©¡£
ͳ¼Æ´ÊƵµÄMapReduceº¯ÊýµÄºËÐÄ´úÂë·Ç³£¼ò¶Ì£¬Ö÷Òª¾ÍÊÇʵÏÖÕâÁ½¸öº¯Êý¡£
map(String key, String value): 02. // key: document name 03. // value: document contents 04. for each word w in value: 05. EmitIntermediate(w, "1"); 06. 07.reduce(String key, Iterator values): 08. // key: a word 09. // values: a list of counts 10. int result = 0; 11. for each v in values: 12. result += ParseInt(v); 13. Emit(AsString(result));
|
ÔÚͳ¼Æ´ÊƵµÄÀý×ÓÀmapº¯Êý½ÓÊܵļüÊÇÎļþÃû£¬ÖµÊÇÎļþµÄÄÚÈÝ£¬mapÖð¸ö±éÀúµ¥´Ê£¬Ã¿Óöµ½Ò»¸öµ¥´Êw£¬¾Í²úÉúÒ»¸öÖмä¼üÖµ¶Ô<w,
"1">£¬Õâ±íʾµ¥´ÊwÔÛÓÖÕÒµ½ÁËÒ»¸ö£»MapReduce½«¼üÏàͬ£¨¶¼Êǵ¥´Êw£©µÄ¼üÖµ¶Ô´«¸øreduceº¯Êý£¬ÕâÑùreduceº¯Êý½ÓÊܵļü¾ÍÊǵ¥´Êw£¬ÖµÊÇÒ»´®"1"£¨×î»ù±¾µÄʵÏÖÊÇÕâÑù£¬µ«¿ÉÒÔÓÅ»¯£©£¬¸öÊýµÈÓÚ¼üΪwµÄ¼üÖµ¶ÔµÄ¸öÊý£¬È»ºó½«ÕâЩ¡°1¡±ÀۼӾ͵õ½µ¥´ÊwµÄ³öÏÖ´ÎÊý¡£×îºóÕâЩµ¥´ÊµÄ³öÏÖ´ÎÊý»á±»Ð´µ½Óû§¶¨ÒåµÄλÖ㬴洢ÔڵײãµÄ·Ö²¼Ê½´æ´¢ÏµÍ³£¨GFS»òHDFS£©¡£
4. MapReduceÊÇÈçºÎ¹¤×÷µÄ

ÉÏͼÊÇÂÛÎÄÀï¸ø³öµÄÁ÷³Ìͼ¡£Ò»Çж¼ÊÇ´Ó×îÉÏ·½µÄuser program¿ªÊ¼µÄ£¬user
programÁ´½ÓÁËMapReduce¿â£¬ÊµÏÖÁË×î»ù±¾µÄMapº¯ÊýºÍReduceº¯Êý¡£Í¼ÖÐÖ´ÐеÄ˳Ðò¶¼ÓÃÊý×Ö±ê¼ÇÁË¡£
1.MapReduce¿âÏȰÑuser programµÄÊäÈëÎļþ»®·ÖΪM·Ý£¨MΪÓû§¶¨Ò壩£¬Ã¿Ò»·Ýͨ³£ÓÐ16MBµ½64MB£¬Èçͼ×ó·½Ëùʾ·Ö³ÉÁËsplit0~4£»È»ºóʹÓÃfork½«Óû§½ø³Ì¿½±´µ½¼¯ÈºÄÚÆäËü»úÆ÷ÉÏ¡£
2.user programµÄ¸±±¾ÖÐÓÐÒ»¸ö³ÆÎªmaster£¬ÆäÓà³ÆÎªworker£¬masterÊǸºÔðµ÷¶ÈµÄ£¬Îª¿ÕÏÐworker·ÖÅä×÷Òµ£¨Map×÷Òµ»òÕßReduce×÷Òµ£©£¬workerµÄÊýÁ¿Ò²ÊÇ¿ÉÒÔÓÉÓû§Ö¸¶¨µÄ¡£
3.±»·ÖÅäÁËMap×÷ÒµµÄworker£¬¿ªÊ¼¶ÁÈ¡¶ÔÓ¦·ÖƬµÄÊäÈëÊý¾Ý£¬Map×÷ÒµÊýÁ¿ÊÇÓÉM¾ö¶¨µÄ£¬ºÍsplitÒ»Ò»¶ÔÓ¦£»Map×÷Òµ´ÓÊäÈëÊý¾ÝÖгéÈ¡³ö¼üÖµ¶Ô£¬Ã¿Ò»¸ö¼üÖµ¶Ô¶¼×÷Ϊ²ÎÊý´«µÝ¸ømapº¯Êý£¬mapº¯Êý²úÉúµÄÖмä¼üÖµ¶Ô±»»º´æÔÚÄÚ´æÖС£
4.»º´æµÄÖмä¼üÖµ¶Ô»á±»¶¨ÆÚдÈë±¾µØ´ÅÅÌ£¬¶øÇÒ±»·ÖΪR¸öÇø£¬RµÄ´óСÊÇÓÉÓû§¶¨ÒåµÄ£¬½«À´Ã¿¸öÇø»á¶ÔÓ¦Ò»¸öReduce×÷Òµ£»ÕâЩÖмä¼üÖµ¶ÔµÄλÖûᱻͨ±¨¸ømaster£¬master¸ºÔð½«ÐÅϢת·¢¸øReduce
worker¡£
5.master֪ͨ·ÖÅäÁËReduce×÷ÒµµÄworkerËü¸ºÔðµÄ·ÖÇøÔÚʲôλÖ㨿϶¨²»Ö¹Ò»¸öµØ·½£¬Ã¿¸öMap×÷Òµ²úÉúµÄÖмä¼üÖµ¶Ô¶¼¿ÉÄÜÓ³Éäµ½ËùÓÐR¸ö²»Í¬·ÖÇø£©£¬µ±Reduce
worker°ÑËùÓÐËü¸ºÔðµÄÖмä¼üÖµ¶Ô¶¼¶Á¹ýÀ´ºó£¬ÏȶÔËüÃǽøÐÐÅÅÐò£¬Ê¹µÃÏàͬ¼üµÄ¼üÖµ¶Ô¾Û¼¯ÔÚÒ»Æð¡£ÒòΪ²»Í¬µÄ¼ü¿ÉÄÜ»áÓ³É䵽ͬһ¸ö·ÖÇøÒ²¾ÍÊÇͬһ¸öReduce×÷Òµ£¨ËÈ÷ÖÇøÉÙÄØ£©£¬ËùÒÔÅÅÐòÊDZØÐëµÄ¡£
6.reduce worker±éÀúÅÅÐòºóµÄÖмä¼üÖµ¶Ô£¬¶ÔÓÚÿ¸öΨһµÄ¼ü£¬¶¼½«¼üÓë¹ØÁªµÄÖµ´«µÝ¸øreduceº¯Êý£¬reduceº¯Êý²úÉúµÄÊä³ö»áÌí¼Óµ½Õâ¸ö·ÖÇøµÄÊä³öÎļþÖС£
7.µ±ËùÓеÄMapºÍReduce×÷Òµ¶¼Íê³ÉÁË£¬master»½ÐÑÕý°æµÄuser
program£¬MapReduceº¯Êýµ÷Ó÷µ»Øuser programµÄ´úÂë¡£
ËùÓÐÖ´ÐÐÍê±Ïºó£¬MapReduceÊä³ö·ÅÔÚÁËR¸ö·ÖÇøµÄÊä³öÎļþÖУ¨·Ö±ð¶ÔÓ¦Ò»¸öReduce×÷Òµ£©¡£Óû§Í¨³£²¢²»ÐèÒªºÏ²¢ÕâR¸öÎļþ£¬¶øÊǽ«Æä×÷ΪÊäÈë½»¸øÁíÒ»¸öMapReduce³ÌÐò´¦Àí¡£Õû¸ö¹ý³ÌÖУ¬ÊäÈëÊý¾ÝÊÇÀ´×Եײã·Ö²¼Ê½Îļþϵͳ£¨GFS£©µÄ£¬ÖмäÊý¾ÝÊÇ·ÅÔÚ±¾µØÎļþϵͳµÄ£¬×îÖÕÊä³öÊý¾ÝÊÇдÈëµ×²ã·Ö²¼Ê½Îļþϵͳ£¨GFS£©µÄ¡£¶øÇÒÎÒÃÇҪעÒâMap/Reduce×÷ÒµºÍmap/reduceº¯ÊýµÄÇø±ð£ºMap×÷Òµ´¦ÀíÒ»¸öÊäÈëÊý¾ÝµÄ·ÖƬ£¬¿ÉÄÜÐèÒªµ÷Óöà´Îmapº¯ÊýÀ´´¦Àíÿ¸öÊäÈë¼üÖµ¶Ô£»Reduce×÷Òµ´¦ÀíÒ»¸ö·ÖÇøµÄÖмä¼üÖµ¶Ô£¬ÆÚ¼äÒª¶Ôÿ¸ö²»Í¬µÄ¼üµ÷ÓÃÒ»´Îreduceº¯Êý£¬Reduce×÷Òµ×îÖÕÒ²¶ÔÓ¦Ò»¸öÊä³öÎļþ¡£
ÎÒ¸üϲ»¶°ÑÁ÷³Ì·ÖΪÈý¸ö½×¶Î¡£µÚÒ»½×¶ÎÊÇ×¼±¸½×¶Î£¬°üÀ¨1¡¢2£¬Ö÷½ÇÊÇMapReduce¿â£¬Íê³É²ð·Ö×÷ÒµºÍ¿½±´Óû§³ÌÐòµÈÈÎÎñ£»µÚ¶þ½×¶ÎÊÇÔËÐн׶Σ¬°üÀ¨3¡¢4¡¢5¡¢6£¬Ö÷½ÇÊÇÓû§¶¨ÒåµÄmapºÍreduceº¯Êý£¬Ã¿¸öС×÷Òµ¶¼¶ÀÁ¢ÔËÐÐ×Å£»µÚÈý½×¶ÎÊÇɨβ½×¶Î£¬Õâʱ×÷ÒµÒѾÍê³É£¬×÷Òµ½á¹û±»·ÅÔÚÊä³öÎļþÀ¾Í¿´Óû§ÏëÔõô´¦ÀíÕâЩÊä³öÁË¡£
5. ´ÊƵÊÇÔõôͳ¼Æ³öÀ´µÄ
½áºÏµÚËĽڣ¬ÎÒÃǾͿÉÒÔÖªµÀµÚÈý½ÚµÄ´úÂëÊÇÈçºÎ¹¤×÷µÄÁË¡£¼ÙÉèÔÛÃǶ¨ÒåM=5£¬R=3£¬²¢ÇÒÓÐ6̨»úÆ÷£¬Ò»Ì¨master¡£

Õâ·ùͼÃèÊöÁËMapReduceÈçºÎ´¦Àí´ÊƵͳ¼Æ¡£ÓÉÓÚmap workerÊýÁ¿²»¹»£¬Ê×ÏÈ´¦ÀíÁË·ÖÆ¬1¡¢3¡¢4£¬²¢²úÉúÖмä¼üÖµ¶Ô£»µ±ËùÓÐÖмäÖµ¶¼×¼±¸ºÃÁË£¬Reduce×÷Òµ¾Í¿ªÊ¼¶ÁÈ¡¶ÔÓ¦·ÖÇø£¬²¢Êä³öͳ¼Æ½á¹û¡£
6. Óû§µÄȨÀû
Óû§×îÖ÷ÒªµÄÈÎÎñÊÇʵÏÖmapºÍreduce½Ó¿Ú£¬µ«»¹ÓÐһЩÓÐÓõĽӿÚÊÇÏòÓû§¿ª·ÅµÄ¡£
1.an input reader¡£Õâ¸öº¯Êý»á½«ÊäÈë·ÖΪM¸ö²¿·Ö£¬²¢ÇÒ¶¨ÒåÁËÈçºÎ´ÓÊý¾ÝÖгéÈ¡×î³õµÄ¼üÖµ¶Ô£¬±ÈÈç´ÊƵµÄÀý×ÓÖж¨ÒåÎļþÃûºÍÎļþÄÚÈÝÊǼüÖµ¶Ô¡£
2.a partition function¡£Õâ¸öº¯ÊýÓÃÓÚ½«mapº¯Êý²úÉúµÄÖмä¼üÖµ¶ÔÓ³Éäµ½Ò»¸ö·ÖÇøÀïÈ¥£¬×î¼òµ¥µÄʵÏÖ¾ÍÊǽ«¼üÇó¹þÏ£ÔÙ¶ÔRȡģ¡£
3.a compare function¡£Õâ¸öº¯ÊýÓÃÓÚReduce×÷ÒµÅÅÐò£¬Õâ¸öº¯Êý¶¨ÒåÁ˼üµÄ´óС¹ØÏµ¡£
4.an output writer¡£¸ºÔ𽫽á¹ûдÈëµ×²ã·Ö²¼Ê½Îļþϵͳ¡£
5.a combiner function¡£Êµ¼Ê¾ÍÊÇreduceº¯Êý£¬ÕâÊÇÓÃÓÚÇ°ÃæÌáµ½µÄÓÅ»¯µÄ£¬±ÈÈçͳ¼Æ´ÊƵʱ£¬Èç¹ûÿ¸ö<w,
"1">Òª¶ÁÒ»´Î£¬ÒòΪreduceºÍmapͨ³£²»ÔÚһ̨»úÆ÷£¬·Ç³£ÀË·Ñʱ¼ä£¬ËùÒÔ¿ÉÒÔÔÚmapÖ´Ðеĵط½ÏÈÔËÐÐÒ»´Îcombiner£¬ÕâÑùreduceÖ»ÐèÒª¶ÁÒ»´Î<w,
"n">ÁË¡£
6.mapºÍreduceº¯Êý¾Í²»¶à˵ÁË¡£
7. MapReduceµÄʵÏÖ
ĿǰMapReduceÒѾÓжàÖÖʵÏÖ£¬³ýÁ˹ȸè×Ô¼ºµÄʵÏÖÍ⣬»¹ÓÐÖøÃûµÄhadoop£¬Çø±ðÊǹȸèÊÇc++£¬¶øhadoopÊÇÓÃjava¡£ÁíÍâ˹̹¸£´óѧʵÏÖÁËÒ»¸öÔÚ¶àºË/¶à´¦ÀíÆ÷¡¢¹²ÏíÄÚ´æ»·¾³ÄÚÔËÐеÄMapReduce£¬³ÆÎªPhoenix£¨½éÉÜ£©£¬Ïà¹ØµÄÂÛÎÄ·¢±íÔÚ07ÄêµÄHPCA£¬Êǵ±ÄêµÄ×î¼ÑÂÛÎÄŶ£¡
¹È¸è¼¼Êõ"Èý±¦"Ö®BigTable
1 ʲôÊÇBigTable
BigtableÊÇÒ»¸öΪ¹ÜÀí´ó¹æÄ£½á¹¹»¯Êý¾Ý¶øÉè¼ÆµÄ·Ö²¼Ê½´æ´¢ÏµÍ³£¬¿ÉÒÔÀ©Õ¹µ½PB¼¶Êý¾ÝºÍÉÏǧ̨·þÎñÆ÷¡£ºÜ¶àgoogleµÄÏîĿʹÓÃBigtable´æ´¢Êý¾Ý£¬ÕâЩӦÓöÔBigtableÌá³öÁ˲»Í¬µÄÌôÕ½£¬±ÈÈçÊý¾Ý¹æÄ£µÄÒªÇó¡¢ÑÓ³ÙµÄÒªÇó¡£BigtableÄÜÂú×ãÕâЩ¶à±äµÄÒªÇó£¬ÎªÕâЩ²úÆ·³É¹¦µØÌṩÁËÁé»î¡¢¸ßÐÔÄܵĴ洢½â¾ö·½°¸¡£
Bigtable¿´ÆðÀ´ÏñÒ»¸öÊý¾Ý¿â£¬²ÉÓÃÁ˺ܶàÊý¾Ý¿âµÄʵÏÖ²ßÂÔ¡£µ«ÊÇBigtable²¢²»Ö§³ÖÍêÕûµÄ¹ØÏµÐÍÊý¾ÝÄ£ÐÍ£»¶øÊÇΪ¿Í»§¶ËÌṩÁËÒ»ÖÖ¼òµ¥µÄÊý¾ÝÄ£ÐÍ£¬¿Í»§¶Ë¿ÉÒÔ¶¯Ì¬µØ¿ØÖÆÊý¾ÝµÄ²¼¾ÖºÍ¸ñʽ£¬²¢ÇÒÀûÓõײãÊý¾Ý´æ´¢µÄ¾Ö²¿ÐÔÌØÕ÷¡£Bigtable½«Êý¾Ýͳͳ¿´³ÉÎÞÒâÒåµÄ×Ö½Ú´®£¬¿Í»§¶ËÐèÒª½«½á¹¹»¯ºÍ·Ç½á¹¹»¯Êý¾Ý´®Ðл¯ÔÙ´æÈëBigtable¡£
ÏÂÎĶÔBigTableµÄÊý¾ÝÄ£Ðͺͻù±¾¹¤×÷ÔÀí½øÐнéÉÜ£¬¶ø¸÷ÖÖÓÅ»¯¼¼Êõ£¨ÈçѹËõ¡¢Bloom
FilterµÈ£©²»ÔÚÌÖÂÛ·¶Î§¡£
2 BigTableµÄÊý¾ÝÄ£ÐÍ
Bigtable²»ÊǹØÏµÐÍÊý¾Ý¿â£¬µ«ÊÇÈ´ÑØÓÃÁËºÜ¶à¹ØÏµÐÍÊý¾Ý¿âµÄÊõÓÏñtable£¨±í£©¡¢row£¨ÐУ©¡¢column£¨ÁУ©µÈ¡£ÕâÈÝÒ×ÈöÁÕßÎóÈëÆç;£¬½«ÆäÓë¹ØÏµÐÍÊý¾Ý¿âµÄ¸ÅÄî¶ÔÓ¦ÆðÀ´£¬´Ó¶øÄÑÒÔÀí½âÂÛÎÄ¡£Understanding
HBase and BigTableÊÇÆªºÜÓÅÐãµÄÎÄÕ£¬¿ÉÒÔ°ïÖú¶ÁÕß´Ó¹ØÏµÐÍÊý¾ÝÄ£Ð͵Ä˼ά¶¨ÊÆÖÐ×ß³öÀ´¡£
±¾ÖÊÉÏ˵£¬BigtableÊÇÒ»¸ö¼üÖµ£¨key-value£©Ó³Éä¡£°´×÷ÕßµÄ˵·¨£¬BigtableÊÇÒ»¸öÏ¡ÊèµÄ£¬·Ö²¼Ê½µÄ£¬³Ö¾Ã»¯µÄ£¬¶àάµÄÅÅÐòÓ³Éä¡£
ÏÈÀ´¿´¿´¶àά¡¢ÅÅÐò¡¢Ó³Éä¡£BigtableµÄ¼üÓÐÈýά£¬·Ö±ðÊÇÐмü£¨row key£©¡¢Áмü£¨column
key£©ºÍʱ¼ä´Á£¨timestamp£©£¬ÐмüºÍÁмü¶¼ÊÇ×Ö½Ú´®£¬Ê±¼ä´ÁÊÇ64λÕûÐÍ£»¶øÖµÊÇÒ»¸ö×Ö½Ú´®¡£¿ÉÒÔÓÃ
(row:string, column:string, time:int64)¡ústring À´±íʾһÌõ¼üÖµ¶Ô¼Ç¼¡£
Ðмü¿ÉÒÔÊÇÈÎÒâ×Ö½Ú´®£¬Í¨³£ÓÐ10-100×Ö½Ú¡£ÐеĶÁд¶¼ÊÇÔ×ÓÐԵġ£Bigtable°´ÕÕÐмüµÄ×ÖµäÐò´æ´¢Êý¾Ý¡£BigtableµÄ±í»á¸ù¾ÝÐмü×Ô¶¯»®·ÖΪƬ£¨tablet£©£¬Æ¬ÊǸºÔؾùºâµÄµ¥Ôª¡£×î³õ±í¶¼Ö»ÓÐÒ»¸öƬ£¬µ«Ëæ×Å±í²»¶ÏÔö´ó£¬Æ¬»á×Ô¶¯·ÖÁÑ£¬Æ¬µÄ´óС¿ØÖÆÔÚ100-200MB¡£ÐÐÊDZíµÄµÚÒ»¼¶Ë÷Òý£¬ÎÒÃÇ¿ÉÒ԰ѸÃÐеÄÁС¢Ê±¼äºÍÖµ¿´³ÉÒ»¸öÕûÌ壬¼ò»¯ÎªÒ»Î¬¼üÖµÓ³É䣬ÀàËÆÓÚ£º
table{ 02. "1" : {sth.},//Ò»ÐÐ 03. "aaaaa" : {sth.}, 04. "aaaab" : {sth.}, 05. "xyz" : {sth.}, 06. "zzzzz" : {sth.} 07.}
|
ÁÐÊǵڶþ¼¶Ë÷Òý£¬Ã¿ÐÐÓµÓеÄÁÐÊDz»ÊÜÏÞÖÆµÄ£¬¿ÉÒÔËæÊ±Ôö¼Ó¼õÉÙ¡£ÎªÁË·½±ã¹ÜÀí£¬Áб»·ÖΪ¶à¸öÁÐ×壨column
family£¬ÊÇ·ÃÎÊ¿ØÖƵĵ¥Ôª£©£¬Ò»¸öÁÐ×åÀïµÄÁÐÒ»°ã´æ´¢ÏàͬÀàÐ͵ÄÊý¾Ý¡£Ò»ÐеÄÁÐ×åºÜÉٱ仯£¬µ«ÊÇÁÐ×åÀïµÄÁпÉÒÔËæÒâÌí¼Óɾ³ý¡£Áмü°´ÕÕfamily:qualifier¸ñʽÃüÃûµÄ¡£Õâ´ÎÎÒÃǽ«ÁÐÄóöÀ´£¬½«Ê±¼äºÍÖµ¿´³ÉÒ»¸öÕûÌ壬¼ò»¯Îª¶þά¼üÖµÓ³É䣬ÀàËÆÓÚ£º
table{ 02. // ... 03. "aaaaa" : { //Ò»ÐÐ 04. "A:foo" : {sth.},//Ò»ÁÐ 05. "A:bar" : {sth.},//Ò»ÁÐ 06. "B:" : {sth.} //Ò»ÁУ¬ÁÐ×åÃûΪB£¬µ«ÊÇÁÐÃûÊÇ¿Õ×Ö´® 07. }, 08. "aaaab" : { //Ò»ÐÐ 09. "A:foo" : {sth.}, 10. "B:" : {sth.} 11. }, 12. // ... 13.}
|
»òÕß¿ÉÒÔ½«ÁÐ×åµ±×÷Ò»²ãеÄË÷Òý£¬ÀàËÆÓÚ£º
table{ 02. // ... 03. "aaaaa" : { //Ò»ÐÐ 04. "A" : { //ÁÐ×åA 05. "foo" : {sth.}, //Ò»ÁÐ 06. "bar" : {sth.} 07. }, 08. "B" : { //ÁÐ×åB 09. "" : {sth.} 10. } 11. }, 12. "aaaab" : { //Ò»ÐÐ 13. "A" : { 14. "foo" : {sth.}, 15. }, 16. "B" : { 17. "" : "ocean" 18. } 19. }, 20. // ... 21.}
|
ʱ¼ä´ÁÊǵÚÈý¼¶Ë÷Òý¡£BigtableÔÊÐí±£´æÊý¾ÝµÄ¶à¸ö°æ±¾£¬°æ±¾Çø·ÖµÄÒÀ¾Ý¾ÍÊÇʱ¼ä´Á¡£Ê±¼ä´Á¿ÉÒÔÓÉBigtable¸³Öµ£¬´ú±íÊý¾Ý½øÈëBigtableµÄ׼ȷʱ¼ä£¬Ò²¿ÉÒÔÓɿͻ§¶Ë¸³Öµ¡£Êý¾ÝµÄ²»Í¬°æ±¾°´ÕÕʱ¼ä´Á½µÐò´æ´¢£¬Òò´ËÏȶÁµ½µÄÊÇ×îа汾µÄÊý¾Ý¡£ÎÒÃǼÓÈëʱ¼ä´Áºó£¬¾ÍµÃµ½ÁËBigtableµÄÍêÕûÊý¾ÝÄ£ÐÍ£¬ÀàËÆÓÚ£º
table{ 02. // ... 03. "aaaaa" : { //Ò»ÐÐ 04. "A:foo" : { //Ò»ÁÐ 05. 15 : "y", //Ò»¸ö°æ±¾ 06. 4 : "m" 07. }, 08. "A:bar" : { //Ò»ÁÐ 09. 15 : "d", 10. }, 11. "B:" : { //Ò»ÁÐ 12. 6 : "w" 13. 3 : "o" 14. 1 : "w" 15. } 16. }, 17. // ... 18.}
|
²éѯʱ£¬Èç¹ûÖ»¸ø³öÐÐÁУ¬ÄÇô·µ»ØµÄÊÇ×îа汾µÄÊý¾Ý£»Èç¹û¸ø³öÁËÐÐÁÐʱ¼ä´Á£¬ÄÇô·µ»ØµÄÊÇʱ¼äСÓÚ»òµÈÓÚʱ¼ä´ÁµÄÊý¾Ý¡£±ÈÈ磬ÎÒÃDzéѯ"aaaaa"/"A:foo"£¬·µ»ØµÄÖµÊÇ"y"£»²éѯ"aaaaa"/"A:foo"/10£¬·µ»ØµÄ½á¹û¾ÍÊÇ"m"£»²éѯ"aaaaa"/"A:foo"/2£¬·µ»ØµÄ½á¹ûÊǿա£

ͼ1ÊÇBigtableÂÛÎÄÀï¸ø³öµÄÀý×Ó£¬Webtable±í´æ´¢ÁË´óÁ¿µÄÍøÒ³ºÍÏà¹ØÐÅÏ¢¡£ÔÚWebtable£¬Ã¿Ò»Ðд洢һ¸öÍøÒ³£¬Æä·´×ªµÄurl×÷ΪÐмü£¬±ÈÈçmaps.google.com/index.htmlµÄÊý¾Ý´æ´¢ÔÚ¼üΪcom.google.maps/index.htmlµÄÐÐÀ·´×ªµÄÔÒòÊÇΪÁËÈÃͬһ¸öÓòÃûϵÄ×ÓÓòÃûÍøÒ³Äܾۼ¯ÔÚÒ»Æð¡£Í¼1ÖеÄÁÐ×å"anchor"±£´æÁ˸ÃÍøÒ³µÄÒýÓÃÕ¾µã£¨±ÈÈçÒýÓÃÁËCNNÖ÷Ò³µÄÕ¾µã£©£¬qualifierÊÇÒýÓÃÕ¾µãµÄÃû³Æ£¬¶øÊý¾ÝÊÇÁ´½ÓÎı¾£»ÁÐ×å"contents"±£´æµÄÊÇÍøÒ³µÄÄÚÈÝ£¬Õâ¸öÁÐ×åÖ»ÓÐÒ»¸ö¿ÕÁÐ"contents:"¡£Í¼1ÖÐ"contents:"ÁÐϱ£´æÁËÍøÒ³µÄÈý¸ö°æ±¾£¬ÎÒÃÇ¿ÉÒÔÓÃ("com.cnn.www",
"contents:", t5)À´ÕÒµ½CNNÖ÷Ò³ÔÚt5ʱ¿ÌµÄÄÚÈÝ¡£
ÔÙÀ´¿´¿´×÷Õß˵µÄÆäËüÌØÕ÷£ºÏ¡Ê裬·Ö²¼Ê½£¬³Ö¾Ã»¯¡£³Ö¾Ã»¯µÄÒâ˼ºÜ¼òµ¥£¬BigtableµÄÊý¾Ý×îÖÕ»áÒÔÎļþµÄÐÎʽ·Åµ½GFSÈ¥¡£Bigtable½¨Á¢ÔÚGFSÖ®Éϱ¾Éí¾ÍÒâζ×Å·Ö²¼Ê½£¬µ±È»·Ö²¼Ê½µÄÒâÒ廹²»½öÏÞÓÚ´Ë¡£Ï¡ÊèµÄÒâ˼ÊÇ£¬Ò»¸ö±íÀﲻͬµÄÐУ¬ÁпÉÄÜÍêÍêȫȫ²»Ò»Ñù¡£
3 Ö§³Å¼¼Êõ
BigtableÒÀÀµÓÚgoogleµÄ¼¸Ïî¼¼Êõ¡£ÓÃGFSÀ´´æ´¢ÈÕÖ¾ºÍÊý¾ÝÎļþ£»°´SSTableÎļþ¸ñʽ´æ´¢Êý¾Ý£»ÓÃChubby¹ÜÀíÔªÊý¾Ý¡£
GFS²Î¼û¹È¸è¼¼Êõ"Èý±¦"Ö®¹È¸èÎļþϵͳ¡£BigTableµÄÊý¾ÝºÍÈÕÖ¾¶¼ÊÇдÈëGFSµÄ¡£
SSTableµÄÈ«³ÆÊÇSorted Strings Table£¬ÊÇÒ»ÖÖ²»¿ÉÐ޸ĵÄÓÐÐòµÄ¼üÖµÓ³É䣬ÌṩÁ˲éѯ¡¢±éÀúµÈ¹¦ÄÜ¡£Ã¿¸öSSTableÓÉһϵÁеĿ飨block£©×é³É£¬Bigtable½«¿éĬÈÏÉèΪ64KB¡£ÔÚSSTableµÄβ²¿´æ´¢×Å¿éË÷Òý£¬ÔÚ·ÃÎÊSSTableʱ£¬Õû¸öË÷Òý»á±»¶ÁÈëÄÚ´æ¡£BigTableÂÛÎÄûÓÐÌáµ½SSTableµÄ¾ßÌå½á¹¹£¬LevelDbÈÕ֪¼֮ËÄ£º
SSTableÎļþÕâÆªÎÄÕ¶ÔLevelDbµÄSSTable¸ñʽ½øÐÐÁ˽éÉÜ£¬ÒòΪLevelDBµÄ×÷ÕßJeffreyDeanÕýÊÇBigTableµÄÉè¼ÆÊ¦£¬ËùÒÔ¼«¾ß²Î¿¼¼ÛÖµ¡£Ã¿Ò»¸öƬ£¨tablet£©ÔÚGFSÀï¶¼Êǰ´ÕÕSSTableµÄ¸ñʽ´æ´¢µÄ£¬Ã¿¸öƬ¿ÉÄܶÔÓ¦¶à¸öSSTable¡£
ChubbyÊÇÒ»Öָ߿ÉÓõķֲ¼Ê½Ëø·þÎñ£¬ChubbyÓÐÎå¸ö»îÔ¾¸±±¾£¬Í¬Ê±Ö»ÓÐÒ»¸öÖ÷¸±±¾Ìṩ·þÎñ£¬¸±±¾Ö®¼äÓÃPaxosË㷨ά³ÖÒ»ÖÂÐÔ£¬ChubbyÌṩÁËÒ»¸öÃüÃû¿Õ¼ä£¨°üÀ¨Ò»Ð©Ä¿Â¼ºÍÎļþ£©£¬Ã¿¸öĿ¼ºÍÎļþ¾ÍÊÇÒ»¸öËø£¬ChubbyµÄ¿Í»§¶Ë±ØÐëºÍChubby±£³Ö»á»°£¬¿Í»§¶ËµÄ»á»°Èô¹ýÆÚÔò»á¶ªÊ§ËùÓеÄËø¡£¹ØÓÚChubbyµÄÏêϸÐÅÏ¢¿ÉÒÔ¿´googleµÄÁíһƪÂÛÎÄ£ºThe
Chubby lock service for loosely-coupled distributed
systems¡£ChubbyÓÃÓÚÆ¬¶¨Î»£¬Æ¬·þÎñÆ÷µÄ״̬¼à¿Ø£¬·ÃÎÊ¿ØÖÆÁÐ±í´æ´¢µÈÈÎÎñ¡£
4 Bigtable¼¯Èº
Bigtable¼¯Èº°üÀ¨Èý¸öÖ÷Òª²¿·Ö£ºÒ»¸ö¹©¿Í»§¶ËʹÓõĿ⣬һ¸öÖ÷·þÎñÆ÷£¨master server£©£¬Ðí¶àƬ·þÎñÆ÷£¨tablet
server£©¡£
ÕýÈçÊý¾ÝÄ£ÐÍС½ÚËù˵£¬Bigtable»á½«±í£¨table£©½øÐÐ·ÖÆ¬£¬Æ¬£¨tablet£©µÄ´óСά³ÖÔÚ100-200MB·¶Î§£¬Ò»µ©³¬³ö·¶Î§¾Í½«·ÖÁѳɸüСµÄƬ£¬»òÕߺϲ¢³É¸ü´óµÄƬ¡£Ã¿¸öƬ·þÎñÆ÷¸ºÔðÒ»¶¨Á¿µÄƬ£¬´¦Àí¶ÔÆäƬµÄ¶ÁдÇëÇó£¬ÒÔ¼°Æ¬µÄ·ÖÁÑ»òºÏ²¢¡£Æ¬·þÎñÆ÷¿ÉÒÔ¸ù¾Ý¸ºÔØËæÊ±Ìí¼ÓºÍɾ³ý¡£ÕâÀïÆ¬·þÎñÆ÷²¢²»ÕæÊµ´æ´¢Êý¾Ý£¬¶øÏ൱ÓÚÒ»¸öÁ¬½ÓBigtableºÍGFSµÄ´úÀí£¬¿Í»§¶ËµÄһЩÊý¾Ý²Ù×÷¶¼Í¨¹ýƬ·þÎñÆ÷´úÀí¼ä½Ó·ÃÎÊGFS¡£
Ö÷·þÎñÆ÷¸ºÔð½«Æ¬·ÖÅä¸øÆ¬·þÎñÆ÷£¬¼à¿ØÆ¬·þÎñÆ÷µÄÌí¼ÓºÍɾ³ý£¬Æ½ºâƬ·þÎñÆ÷µÄ¸ºÔØ£¬´¦Àí±íºÍÁÐ×åµÄ´´½¨µÈ¡£×¢Ò⣬Ö÷·þÎñÆ÷²»´æ´¢ÈÎºÎÆ¬£¬²»ÌṩÈκÎÊý¾Ý·þÎñ£¬Ò²²»ÌṩƬµÄ¶¨Î»ÐÅÏ¢¡£
¿Í»§¶ËÐèÒª¶ÁдÊý¾Ýʱ£¬Ö±½ÓÓëÆ¬·þÎñÆ÷ÁªÏµ¡£ÒòΪ¿Í»§¶Ë²¢²»ÐèÒª´ÓÖ÷·þÎñÆ÷»ñȡƬµÄλÖÃÐÅÏ¢£¬ËùÒÔ´ó¶àÊý¿Í»§¶Ë´ÓÀ´²»ÐèÒª·ÃÎÊÖ÷·þÎñÆ÷£¬Ö÷·þÎñÆ÷µÄ¸ºÔØÒ»°ãºÜÇá¡£
5 ƬµÄ¶¨Î»
Ç°ÃæÌáµ½Ö÷·þÎñÆ÷²»ÌṩƬµÄλÖÃÐÅÏ¢£¬ÄÇô¿Í»§¶ËÊÇÈçºÎ·ÃÎÊÆ¬µÄÄØ£¿À´¿´¿´ÂÛÎĸøµÄʾÒâͼ£¬BigtableʹÓÃÒ»¸öÀàËÆB+Ê÷µÄÊý¾Ý½á¹¹´æ´¢Æ¬µÄλÖÃÐÅÏ¢¡£

Ê×ÏÈÊǵÚÒ»²ã£¬Chubby file¡£ÕâÒ»²ãÊÇÒ»¸öChubbyÎļþ£¬Ëü±£´æ×Åroot
tabletµÄλÖá£Õâ¸öChubbyÎļþÊôÓÚChubby·þÎñµÄÒ»²¿·Ö£¬Ò»µ©Chubby²»¿ÉÓ㬾ÍÒâζ×ŶªÊ§ÁËroot
tabletµÄλÖã¬Õû¸öBigtableÒ²¾Í²»¿ÉÓÃÁË¡£
µÚ¶þ²ãÊÇroot tablet¡£root tabletÆäʵÊÇÔªÊý¾Ý±í£¨METADATA table£©µÄµÚÒ»¸ö·ÖƬ£¬Ëü±£´æ×ÅÔªÊý¾Ý±íÆäËüƬµÄλÖá£root
tabletºÜÌØ±ð£¬ÎªÁ˱£Ö¤Ê÷µÄÉî¶È²»±ä£¬root tablet´Ó²»·ÖÁÑ¡£
µÚÈý²ãÊÇÆäËüµÄÔªÊý¾ÝƬ£¬ËüÃǺÍroot tabletÒ»Æð×é³ÉÍêÕûµÄÔªÊý¾Ý±í¡£Ã¿¸öÔªÊý¾ÝƬ¶¼°üº¬ÁËÐí¶àÓû§Æ¬µÄλÖÃÐÅÏ¢¡£
¿ÉÒÔ¿´³öÕû¸ö¶¨Î»ÏµÍ³ÆäʵֻÊÇÁ½²¿·Ö£¬Ò»¸öChubbyÎļþ£¬Ò»¸öÔªÊý¾Ý±í¡£×¢ÒâÔªÊý¾Ý±íËäÈ»ÌØÊ⣬µ«Ò²ÈÔÈ»·þ´ÓǰÎĵÄÊý¾ÝÄ£ÐÍ£¬Ã¿¸ö·ÖƬҲ¶¼ÊÇÓÉרÃŵᬷþÎñÆ÷¸ºÔð£¬Õâ¾ÍÊDz»ÐèÒªÖ÷·þÎñÆ÷ÌṩλÖÃÐÅÏ¢µÄÔÒò¡£¿Í»§¶Ë»á»º´æÆ¬µÄλÖÃÐÅÏ¢£¬Èç¹ûÔÚ»º´æÀïÕÒ²»µ½Ò»¸öƬµÄλÖÃÐÅÏ¢£¬¾ÍÐèÒª²éÕÒÕâ¸öÈý²ã½á¹¹ÁË£¬°üÀ¨·ÃÎÊÒ»´ÎChubby·þÎñ£¬·ÃÎÊÁ½´ÎƬ·þÎñÆ÷¡£
6 ƬµÄ´æ´¢ºÍ·ÃÎÊ
ƬµÄÊý¾Ý×îÖÕ»¹ÊÇдµ½GFSÀïµÄ£¬Æ¬ÔÚGFSÀïµÄÎïÀíÐÎ̬¾ÍÊÇÈô¸É¸öSSTableÎļþ¡£Í¼5չʾÁ˶Áд²Ù×÷»ù±¾Çé¿ö¡£

µ±Æ¬·þÎñÆ÷ÊÕµ½Ò»¸öдÇëÇ󣬯¬·þÎñÆ÷Ê×Ïȼì²éÇëÇóÊÇ·ñºÏ·¨¡£Èç¹ûºÏ·¨£¬ÏȽ«Ð´ÇëÇóÌá½»µ½ÈÕ־ȥ£¬È»ºó½«Êý¾ÝдÈëÄÚ´æÖеÄmemtable¡£memtableÏ൱ÓÚSSTableµÄ»º´æ£¬µ±memtable³É³¤µ½Ò»¶¨¹æÄ£»á±»¶³½á£¬BigtableËæÖ®´´½¨Ò»¸öеÄmemtable£¬²¢ÇÒ½«¶³½áµÄmemtableת»»ÎªSSTable¸ñʽдÈëGFS£¬Õâ¸ö²Ù×÷³ÆÎªminor
compaction¡£
µ±Æ¬·þÎñÆ÷ÊÕµ½Ò»¸ö¶ÁÇëÇó£¬Í¬ÑùÒª¼ì²éÇëÇóÊÇ·ñºÏ·¨¡£Èç¹ûºÏ·¨£¬Õâ¸ö¶Á²Ù×÷»á²é¿´ËùÓÐSSTableÎļþºÍmemtableµÄºÏ²¢ÊÓͼ£¬ÒòΪSSTableºÍmemtable±¾Éí¶¼ÊÇÒÑÅÅÐòµÄ£¬ËùÒԺϲ¢Ï൱¿ì¡£
ÿһ´Îminor compaction¶¼»á²úÉúÒ»¸öеÄSSTableÎļþ£¬SSTableÎļþÌ«¶à¶Á²Ù×÷µÄЧÂʾͽµµÍÁË£¬ËùÒÔBigtable¶¨ÆÚÖ´ÐÐmerging
compaction²Ù×÷£¬½«¼¸¸öSSTableºÍmemtableºÏ²¢ÎªÒ»¸öеÄSSTable¡£BigTable»¹Óиö¸üÀ÷º¦µÄ½Ðmajor
compaction£¬Ëü½«ËùÓÐSSTableºÏ²¢ÎªÒ»¸öеÄSSTable¡£
Òź¶µÄÊÇ£¬BigTable×÷ÕßûÓнéÉÜmemtableºÍSSTableµÄÏêϸÊý¾Ý½á¹¹¡£
7 BigTableºÍGFSµÄ¹ØÏµ
¼¯Èº°üÀ¨Ö÷·þÎñÆ÷ºÍƬ·þÎñÆ÷£¬Ö÷·þÎñÆ÷¸ºÔð½«Æ¬·ÖÅä¸øÆ¬·þÎñÆ÷£¬¶ø¾ßÌåµÄÊý¾Ý·þÎñÔòȫȨÓÉÆ¬·þÎñÆ÷¸ºÔð¡£µ«ÊDz»ÒªÎóÒÔΪƬ·þÎñÆ÷ÕæµÄ´æ´¢ÁËÊý¾Ý£¨³ýÁËÄÚ´æÖÐmemtableµÄÊý¾Ý£©£¬Êý¾ÝµÄÕæÊµÎ»ÖÃÖ»ÓÐGFS²ÅÖªµÀ£¬Ö÷·þÎñÆ÷½«Æ¬·ÖÅä¸øÆ¬·þÎñÆ÷µÄÒâ˼Ӧ¸ÃÊÇ£¬Æ¬·þÎñÆ÷»ñÈ¡ÁËÆ¬µÄËùÓÐSSTableÎļþÃû£¬Æ¬·þÎñÆ÷ͨ¹ýһЩË÷Òý»úÖÆ¿ÉÒÔÖªµÀËùÐèÒªµÄÊý¾ÝÔÚÄĸöSSTableÎļþ£¬È»ºó´ÓGFSÖжÁÈ¡SSTableÎļþµÄÊý¾Ý£¬Õâ¸öSSTableÎļþ¿ÉÄÜ·Ö²¼Ôںü¸Ì¨chunkserverÉÏ¡£
8 ÔªÊý¾Ý±íµÄ½á¹¹
ÔªÊý¾Ý±í£¨METADATA table£©ÊÇÒ»ÕÅÌØÊâµÄ±í£¬Ëü±»ÓÃÓÚÊý¾ÝµÄ¶¨Î»ÒÔ¼°Ò»Ð©ÔªÊý¾Ý·þÎñ£¬²»¿Éν²»ÖØÒª¡£µ«ÊÇBigtableÂÛÎÄÀïÖ»¸ø³öÁËÉÙÁ¿ÏßË÷£¬¶ø¶Ô±íµÄ¾ßÌå½á¹¹Ã»ÓÐ˵Ã÷¡£ÕâÀïÎÒÊÔͼ¸ù¾ÝÂÛÎĵÄһЩÏßË÷£¬²Â²âһϱíµÄ½á¹¹¡£Ê×ÏÈÁгöÂÛÎÄÖеÄÏßË÷£º
1.The METADATA table stores the location of a tablet
under a row key that is an encoding of the tablet's
table identifier and its end row.
2.Each METADATA row stores approximately 1KB of data
in memory£¨ÒòΪ·ÃÎÊÁ¿±È½Ï´ó£¬ÔªÊý¾Ý±íÊÇ·ÅÔÚÄÚ´æÀïµÄ£¬Õâ¸öÓÅ»¯ÔÚÂÛÎĵÄlocality groupsÖÐÌáµ½£©.This
feature£¨½«locality group·Åµ½ÄÚ´æÖеÄÌØÐÔ£© is useful for small
pieces of data that are accessed frequently: we use
it internally for the location column family in the
METADATA table.
3.We also store secondary information in the METADATA
table, including a log of all events pertaining to each
tablet(such as when a server begins
serving it).
µÚÒ»ÌõÏßË÷£¬ÔªÊý¾Ý±íµÄÐмüÊÇÓÉÆ¬ËùÊô±íÃûµÄidºÍƬ×îºóÒ»ÐбàÂë¶ø³É£¬ËùÒÔÿ¸öƬÔÚÔªÊý¾Ý±íÖÐÕ¼¾ÝÒ»Ìõ¼Ç¼£¨Ò»ÐУ©£¬¶øÇÒÐмü¼È°üº¬ÁËÆäËùÊô±íµÄÐÅÏ¢Ò²°üº¬ÁËÆäËùÓµÓеÄÐеķ¶Î§¡£Æ©Èç²ÉÈ¡×î¼òµ¥µÄ±àÂ뷽ʽ£¬ÔªÊý¾Ý±íµÄÐмüµÈÓÚstrcat(±íÃû£¬Æ¬×îºóÒ»ÐеÄÐмü)¡£
µÚ¶þµãÏßË÷£¬³ýÁËÖªµÀÔªÊý¾Ý±íµÄµØÖ·²¿·ÖÊdz£×¤ÄÚ´æÒÔÍ⣬»¹¿ÉÒÔ·¢ÏÖÔªÊý¾Ý±íÓÐÒ»¸öÁÐ×å³ÆÎªlocation£¬ÎÒÃÇÒѾ֪µÀÔªÊý¾Ý±íÿһÐдú±íÒ»¸öƬ£¬ÄÇôΪʲôÐèÒªÒ»¸öÁÐ×åÀ´´æ´¢µØÖ·ÄØ£¿ÒòΪÿ¸öƬ¶¼¿ÉÄÜÓɶà¸öSSTableÎļþ×é³É£¬ÁÐ×å¿ÉÒÔÓÃÀ´´æ´¢ÈÎÒâ¶à¸öSSTableÎļþµÄλÖá£Ò»¸öºÏÀíµÄ¼ÙÉè¾ÍÊÇÿ¸öSSTableÎļþµÄλÖÃÐÅÏ¢Õ¼¾ÝÒ»ÁУ¬ÁÐÃûΪlocation:filename¡£µ±È»²»Ò»¶¨·ÇµÃÓÃÁмü´æ´¢ÍêÕûÎļþÃû£¬¸ü´óµÄ¿ÉÄÜÐÔÊǰÑSSTableÎļþÃû´æÔÚÖµÀï¡£»ñÈ¡ÁËÎļþÃû¾Í¿ÉÒÔÏòGFSË÷ÒªÊý¾ÝÁË¡£
µÚÈý¸öÏßË÷¸æËßÎÒÃÇÔªÊý¾Ý±í²»Ö¹´æ´¢Î»ÖÃÐÅÏ¢£¬Ò²¾ÍÊÇ˵ÁÐ×å²»Ö¹location£¬ÕâЩÊý¾ÝÔÝʱ²»ÊÇÔÛÃǹØÐĵġ£
ͨ¹ýÒÔÉÏÐÅÏ¢£¬ÎÒ»ÁËÒ»¸ö¼ò»¯µÄBigtable½á¹¹Í¼£º

½á¹¹Í¼ÒÔWebtable±íΪÀý£¬±íÖд洢ÁËÍøÒס¢°Ù¶ÈºÍ¶¹°êµÄ¼¸¸öÍøÒ³¡£µ±ÎÒÃÇÏë²éÕÒ°Ù¶ÈÌù°É×òÌìµÄÍøÒ³ÄÚÈÝ£¬¿ÉÒÔÏòBigtable·¢³ö²éѯWebtable±íµÄ(com.baidu.tieba,
contents:, yesterday)¡£
¼ÙÉè¿Í»§¶ËûÓиûº´æ£¬ÄÇôBigtable·ÃÎÊroot tabletµÄƬ·þÎñÆ÷£¬Ï£ÍûµÃµ½¸ÃÍøÒ³ËùÊôµÄƬµÄλÖÃÐÅÏ¢ÔÚÄĸöÔªÊý¾ÝƬÖС£Ê¹ÓÃMETADATA.Webtable.com.baidu.tiebaΪÐмüÔÚroot
tabletÖвéÕÒ£¬¶¨Î»µ½×îºóÒ»¸ö±ÈËü´óµÄÊÇMETADATA.Webtable.com.baidu.www£¬ÓÚÊÇÈ·¶¨ÐèÒªµÄ¾ÍÊÇÔªÊý¾Ý±íµÄƬA¡£·ÃÎÊÆ¬AµÄƬ·þÎñÆ÷£¬¼ÌÐø²éÕÒWebtable.com.baidu.tieba£¬¶¨Î»µ½Webtable.com.baidu.wwwÊDZÈËü´óµÄ£¬È·¶¨ÐèÒªµÄÊÇWebtable±íµÄƬB¡£·ÃÎÊÆ¬BµÄƬ·þÎñÆ÷£¬»ñµÃÊý¾Ý¡£
ÕâÀïÐèҪעÒâµÄÊÇ£¬Ã¿¸öƬʵ¼Ê¶¼ÓÉÈô¸ÉSSTableÎļþºÍmemtable×é³É£¬¶øÇÒÕâЩSSTableºÍmemtable¶¼ÊÇÒÑÅÅÐòµÄ¡£Õâ¾Íµ¼Ö²éÕÒÆ¬Bʱ£¬¿ÉÄÜÐèÒª½«ËùÓÐSSTableºÍmemtable¶¼²éÕÒÒ»±é£»ÁíÍâ¿Í»§¶ËÓ¦¸Ã²»»áÖ±½Ó´ÓÔªÊý¾Ý±í»ñµÃSSTableµÄÎļþÃû£¬¶øÖ»ÊÇ»ñµÃƬÊôÓÚÆ¬·þÎñÆ÷µÄÐÅÏ¢£¬Í¨¹ýƬ·þÎñÆ÷Ϊ´úÀí·ÃÎÊSSTable¡£
|