
1
ÒýÑÔ
ʲôÊǵ÷¶È?ͨ³£Ëù˵µÄµ÷¶È(schedule)ÊǺÍʱ¼äÓйص쬱ÈÈçÎÒ½ñÌìµÄscheduleºÜ½ô(Èçͼ1Ëùʾ)¡£Ê±¼ä×÷ΪΨһµÄ²»¿ÉÄæ×ªµÄ×ÊÔ´£¬Ò»°ãÊÇ»®·ÖΪ¶à¸öʱ¼äƬÀ´Ê¹Ó᣾ͼÆËã»ú¶øÑÔ£¬ÓÉÓÚCPUµÄËÙ¶È¿ìµÄ¶à£¬ËùÒÔ¾ÍÓÐÁËÕë¶ÔCPUʱ¼äƬµÄµ÷¶È£¬Èöà¸öÈÎÎñÔÚͬһ¸öCPUÉÏÔËÐÐÆðÀ´¡£È»¶øÕâÊÇÒ»¸ö¼ÙÏó£¬Ä³Ò»Ê±¿ÌCPU»¹Êǵ¥ÈÎÎñÔËÐеġ£

ͼ1 ʱ¼äƬµÄ»®·Ö
ΪÁËÔÚͬһʱ¼äÔËÐиü¶àµÄÈÎÎñ£¬»òÕß¶à¸ö´¦ÀíÆ÷Ò»Æð¹¤×÷Íê³ÉÒ»¸öÈÎÎñÄ¿±ê£¬¾ÍÐèÒªÒ»¸öе÷Õß¡ª¡ªÕâ¾Í³ÉΪһ¸ö·Ö²¼Ê½ÏµÍ³£¬¾Íµ¥¸öÊý¾ÝÖÐÐÄ»òÕßС·¶Î§À´Ëµ£¬Õâ¾ÍÊǼ¯Èº¡£Èç¹ûÈÃÒ»¸ö·Ö²¼Ê½ÏµÍ³ÔËÐжà¸öÈÎÎñ£¬Ã¿¸öÈÎÎñ¶Ô·Ö²¼Ê½ÏµÍ³ÖеÄ×ÊÔ´±ØÈ»²úÉú¾ºÕù£¬Ê±¼äµ÷¶È¾Í·¢Õ¹µ½×ÊÔ´µ÷¶È¡£
ºê¹ÛÉÏÀ´Ëµµ÷¶ÈÖ÷Ìâ°üÀ¨Á˵¥»ú²Ù×÷ϵͳ¡¢C/Sϵͳ¡¢B/Sϵͳ¡¢P2Pϵͳ¡¢¼¯ÈºÏµÍ³¡¢·Ö²¼Ê½ÏµÍ³µÈµÈ£¬ÒÔ¼°ÍøÂçÐÒéÕ»¡¢´æ´¢ÐÒéÕ»µÄ¸÷ÖÖµ÷¶È»úÖÆ¡£±¾ÎÄÖ÷Òª×ܽáÁ˼¯Èºµ÷¶È·¢Õ¹µÄÈý¸ö½×¶Î£ººêµ÷¶È¡¢Á½²ãµ÷¶ÈºÍ¹²Ïí״̬µ÷¶È£¬²¢±È½ÏÁËÈýÕßÖ®¼äµÄÓÅȱµã¡£
2 ¼¯Èºµ÷¶È
2.1 ºêµ÷¶È(Monolithic schedulers)
ºêµ÷¶È£ºÔÚͬһ¸ö´úÂëÄ£¿éÖÐʵÏÖµ÷¶È²ßÂÔ£¬µ¥¸öʵÀý£¬Ã»Óв¢ÐС£³£¼ûÓÚHPC(high-performance
computing)ÊÀ½çÖС£

ͼ2 Hadoop1ºÍMapReduce
µÄºêµ÷¶È¼Ü¹¹
Èçͼ2Ëùʾ£¬ÒÔΪMapReduceΪÀý£¬Ò»¸ö³ÆÖ®ÎªJobTrackerµÄMaster½ø³ÌÊÇËùÓÐMapReduceÈÎÎñµÄÖÐÐĵ÷¶ÈÆ÷¡£Ã¿Ò»¸ö½ÚµãÉÏÃæ¶¼ÔËÐÐÒ»¸öTaskTracker½ø³ÌÀ´¹ÜÀí¸÷¸ö½ÚµãÉϵÄÈÎÎñ¡£¸÷¸öTaskTrackerÒªºÍMaster½ÚµãÉϵÄJobTrackerͨÐŲ¢½ÓÊÜJobTrackerµÄ¿ØÖÆ¡£ºÍ´ó¶àÊý×ÊÔ´¹ÜÀíÆ÷ÀàËÆ£¬MapReduceµÄJobTrackerÖ§³ÖÁ½ÖÖµ÷¶È²ßÂÔ£¬Capacityµ÷¶È²ßÂÔºÍFairµ÷¶È²ßÂÔ¡£
ÔÚJobTrackerÖУ¬×ÊÔ´µÄµ÷¶ÈºÍ×÷ÒµµÄ¹ÜÀí¹¦ÄÜÈ«²¿·Åµ½Ò»¸ö½ø³ÌÖÐÍê³É¡£ÕâÖÖÉè¼Æ·½Ê½µÄȱµãÊÇÀ©Õ¹ÐԲÊ×ÏÈ£¬¼¯Èº¹æÄ£ÊÜÏÞ;Æä´Î£¬Ðµĵ÷¶È²ßÂÔÄÑÒÔÈÚÈëÏÖÓдúÂëÖУ¬±ÈÈç֮ǰ½öÖ§³ÖÅú´¦Àí×÷Òµ£¬ÏÖÔÚÒªÖ§³ÖÁ÷ʽ×÷Òµ£¬¶ø½«Á÷ʽ×÷ÒµµÄµ÷¶È²ßÂÔǶÈëµ½ÖÐÑëʽµ÷¶ÈÆ÷ÖÐÊÇÒ»ÏîºÜÄѵŤ×÷¡£
2.2 ¾²Ì¬·ÖÇø(Statically partitioned schedulers)
»ùÓÚ¾²Ì¬·ÖÇøµÄ×ÊÔ´»®·ÖºÍµ÷¶ÈÒ²±»³ÆÎªÔƼÆËãÖеĵ÷¶È£¬Í¨¹ýÔÚÔÆÆ½Ì¨ÖзÖÅäºÍ¶¨ÒåÐéÄâ»ú½ÇÉ«£¬ÊµÏÖ×ÊÔ´¼¯ºÏµÄÈ«Ãæ¿ØÖÆ¡£ÒµÎñϵͳÍùÍù²¿ÊðÔÚרÃŵġ¢¾²Ì¬»®·ÖµÄ¼¯ÈºµÄÒ»¸ö×Ó¼¯ÉÏ¡ª¡ª°Ñ¼¯Èº»®·ÖΪ²»Í¬µÄ²¿·Ö£¬·Ö±ðÖ§³Ö²»Í¬µÄÒµÎñ¡£ÏÖÔÚ´ó¶àÊýÆóÒµ¼¶µÄÔÆ¼ÆËã¶¼ÊDzÉÓÃÕâÑù¼Æ»®¾¼ÃʽµÄ×ÊÔ´·ÖÅ䷽ʽ¡ª¡ªÔÚϵͳ²¿Êð֮ǰ×öºÃÈÝÁ¿¹æ»®ºÍ×ÊÔ´·ÖÅä
2.3 Á½²ãµ÷¶È(Two-level scheduling)
ΪÁË´¦Àíºêµ÷¶ÈºÍ¼¯Èº¾²Ì¬·ÖÇøµÄÖÖÖÖÏÞÖÆ£¬Ò»¸öÖ±½ÓµÄ½â¾ö·½°¸¾ÍÊÇÁ½²ãµ÷¶È¡£Í¨¹ýÒýÈëÒ»¸öÖÐÑëµÄе÷×é¼þÀ´¾ö¶¨Ã¿Ò»¸ö×Ó¼¯ÈºÐèÒª·ÖÅäµÄ×ÊÔ´ÊýÁ¿£¬´Ó¶ø¶¯Ì¬µÄµ÷Õû·ÖÅä¸øÃ¿Ò»¸öµ÷¶ÈÆ÷(¿ò¼Üµ÷¶ÈÆ÷)µÄ×ÊÔ´¡£Á½²ãµ÷¶È±¾ÖÊÉÏÊÇÔÚµ÷¶ÈÖзÖÀë×ÊÔ´·ÖÅäºÍÈÎÎñ·ÖÅ䣬ÈöÉÒ»²¿·Ö¾ö²ßȨÁ¦¸øÓ¦Óÿò¼Ü£¬½â¾ö²»Í¬Ó¦Óÿò¼ÜµÄÐèÇóÒì¹¹ÎÊÌâ¡£Èçͼ3Ëùʾ£¬YARNΪÉϲ㲻ͬµÄÓ¦Óÿò¼ÜÌṩÁËͳһµÄ×ÊÔ´µ÷¶È²ã¡£ºóÎĶÔMesosµÄ½éÉÜÒ»½Ú»áÏêϸ½éÉÜÁ½²ãµ÷¶ÈµÄÐèÇó±³¾°¡£

ͼ3 Hadoop1ºÍMapReduce µÄºêµ÷¶È¼Ü¹¹
¸÷¸ö¿ò¼Üµ÷¶ÈÆ÷²¢²»ÖªµÀÕû¸ö¼¯Èº×ÊԴʹÓÃÇé¿ö£¬Ö»ÊDZ»¶¯µÄ½ÓÊÕ×ÊÔ´¡£ÖÐÑëе÷×é¼þ½ö½«¿ÉÓõÄ×ÊÔ´ÍÆË͸ø¸÷¸ö¿ò¼Ü£¬¶ø¿ò¼Ü×Ô¼ºÑ¡ÔñʹÓû¹ÊǾܾøÕâЩ×ÊÔ´¡£Ò»µ©¿ò¼Ü(±ÈÈçJobTracker)½ÓÊÕµ½ÐÂ×ÊÔ´ºó£¬ÔÙ½øÒ»²½½«×ÊÔ´·ÖÅ䏸ÆäÄÚ²¿µÄ¸÷¸öÓ¦ÓóÌÐò(¸÷¸öMapReduce×÷Òµ)£¬½ø¶øÊµÏÖË«²ãµ÷¶È¡£
Ë«²ãµ÷¶ÈÆ÷ÓÐÁ½¸öȱµã£¬ÆäÒ»£¬¸÷¸ö¿ò¼ÜÎÞ·¨ÖªµÀÕû¸ö¼¯ÈºµÄʵʱ×ÊԴʹÓÃÇé¿ö;Æä¶þ£¬²ÉÓñ¯¹ÛËø£¬²¢·¢Á£¶ÈС¡£
2.3.1 YARN
YARN±»³ÆÖ®ÎªApache Hadoop Next Generation Compute Platform£¬ÊÇhadoop1ºÍhadoop2Ö®¼ä×î´óµÄÇø±ðÈçͼ4Ëùʾ¡£

ͼ4 ÔÚHadoop 2.0ÖÐÒýÈëYARN
Hadoop2(MRv2)µÄ»ù´¡Ë¼Ïë¾ÍÊǰÑJobTrackerµÄ¹¦ÄÜ»®·Ö³ÉÁ½¸ö¶ÀÁ¢µÄ½ø³Ì£ºÈ«¾ÖµÄ×ÊÔ´¹ÜÀíResourceManagerºÍÿ¸ö½ø³ÌµÄ¼à¿ØºÍµ÷¶ÈApplicationMaster¡£Õâ¸ö½ø³Ì¿ÉÒÔÊÇMap-Reduce
ÖÐÒ»¸öÈÎÎñ»òÕßÊÇDAGÖÐÒ»¸öÈÎÎñ¡£

ͼ5 ÔÚHadoop 2.0ÖÐÒýÈëYARN
ÔÚYARNµÄÉè¼ÆÖУ¬¼¯ÈºÖпÉÒÔÓжà¸öApplicationMasters£¬Ã¿Ò»¸öApplicationMasters¿ÉÒÔÓжà¸öContainers
(ÀýÈ磬ͼ5ÖÐÓÐÁ½¸öApplicationMasters£¬ºìÉ«ºÍÀ¶É«¡£ºìÉ«µÄÓÐÈý¸öContainers£¬À¶É«µÄÓÐÒ»¸öContainer)¡£¹Ø¼üµÄÒ»µãÊÇApplicationMasters
²»ÊÇResourceManagerµÄ²¿·Ö£¬Õâ¾Í¼õÇáÁËÖÐÐĵ÷¶ÈÆ÷µÄѹÁ¦£¬²¢ÇÒ£¬Ã¿Ò»¸öApplicationMasters¶¼¿ÉÒÔ¶¯Ì¬µÄµ÷Õû×Ô¼º¿ØÖƵÄcontainer¡£
¶ø ResourceManager ÊÇÒ»¸ö´¿´âµÄµ÷¶ÈÆ÷(²»¼à¿ØºÍ×·×Ù½ø³ÌµÄÖ´ÐÐ״̬£¬Ò²²»¸ºÔðÖØÆô¹ÊÕϵĽø³Ì)£¬ËüΨһµÄÄ¿µÄ¾ÍÊÇÔÚ¶à¸öÓ¦ÓÃÖ®¼ä¹ÜÀí¿ÉÓõÄ×ÊÔ´(ÒÔContainersµÄÁ£¶È)¡£ResourceManagerÊÇ×ÊÔ´·ÖÅäµÄÖÕ¼«È¨Íþ¡£Èç¹û˵ResourceManagerÊÇMaster£¬NodeManager¾ÍÊÇÆäslave¡£ResourceManager²¢ÇÒÖ§³Öµ÷¶È²ßÂԵIJå¼þ»¯£¬CapacityScheduler
ºÍ FairScheduler¾ÍÊÇÕâÑùµÄ²å¼þ¡£
ApplicationMaster ¸ºÔðÈÎÎñÌá½»£¬Í¨¹ýÐÉ̺Í̸ÅдÓResourceManagerÄÇÀïÒÔContainersµÄÐÎʽ»ñµÃ×ÊÔ´(¸ºÔð̸ÅлñµÃÊÊºÏÆäÓ¦ÓÃÐèÒªµÄContainers)¡£È»ºó¾Ítrack½ø³ÌµÄÔËÐÐ״̬¡£ApplicationMastersÊÇÌØ¶¨ÓÚ¾ßÌåµÄÓ¦Ó㬿ÉÒÔ¸ù¾Ý²»Í¬µÄÓ¦ÓÃÀ´±àд²»Í¬µÄApplicationMasters¡£ÀýÈ磬¡°YARN
includes a distributed Shell framework that runs a
shell script on multiple nodes on the cluster. ¡± ÁíÍ⣬ApplicationMaster
Ìṩ×Ô¶¯ÖØÆôµÄ·þÎñ¡£ApplicationMaster¿ÉÒÔÀí½âΪӦÓóÌÐò¿ÉÒÔ×Ô¼ºÊµÏֵĽӿڿ⡣
ApplicationMastersÇëÇóºÍ¹ÜÀíContainers¡£ContainersÖ¸¶¨ÁËÒ»¸öÓ¦ÓÃÔÚijһ̨Ö÷»úÉÏ¿ÉÒÔʹÓöàÉÙ×ÊÔ´
(°üÀ¨memory, CPUµÈ) £¬ÕâÀàËÆÓÚHPCµ÷¶ÈÖеÄ×ÊÔ´³Ø¡£ApplicationMasterÒ»µ©´Ó
ResourceManagerÄÇÀï»ñµÃ×ÊÔ´£¬Ëü¾Í»áÁªÏµNodeManager À´Æô¶¯Ä³¸öÌØ¶¨µÄÈÎÎñ¡£ÀýÈçÈç¹ûʹÓÃ
MapReduce ¿ò¼Ü£¬ÕâЩÈÎÎñ¿ÉÄܾÍÊÇMapper ºÍ Reducer ½ø³Ì¡£²»Í¬µÄ¿ò¼Ü»áÓв»Í¬µÄ½ø³Ì¡£
NodeManagerÊÇÿһ¸ö»úÆ÷ÉÏ¿ò¼Ü´úÀí£¬¸ºÔð¸Ã»úÉϵÄContainers£¬²¢ÇÒ¼à¿Ø¿ÉÓõÄ×ÊÔ´(CPU,
memory, disk, network)¡£²¢ÇÒ×ÊԴ״̬±¨¸æ¸øResourceManager¡£
±íÃæÉÏ¿´À´£¬YARNÒ²ÊÇÒ»¸öÁ½²ãµ÷¶È¡£ÔÚYARNÖУ¬×ÊÔ´ÇëÇó´ÓApplication Masters·¢³öµ½Ò»¸öÖÐÐĵÄÈ«¾Öµ÷¶ÈÆ÷ÉÏ£¬ÖÐÐĵ÷¶ÈÆ÷¸ù¾ÝÓ¦ÓõÄÐèÒªÔÚ¼¯ÈºÖеĶà¸ö½ÚµãÉÏ·ÖÅä×ÊÔ´¡£µ«ÊÇYARNÖеÄApplication
MastersÌṩµÄ½ö½öÊÇÒ»¸öÈÎÎñ¹ÜÀí·þÎñ£¬²¢²»ÊÇÒ»¸öÕæÕýµÄ¶þ²ãµ÷¶ÈÆ÷¡£Òò´Ë±¾ÖÊÉÏYARNÈÔ¾ÉÊÇÒ»¸öºêµ÷¶È¼Ü¹¹¡£½ØÖ¹Ä¿Ç°£¬YARNÖ»Ö§³Ö×ÊÔ´ÀàÐÍ(ÄÚ´æ)µÄµ÷¶È¡£
Hadoop2(MRv2)µÄAPIÊǺóÏò¼æÈݵģ¬Ö§³ÖMap-ReduceµÄÈÎÎñÖ»ÐèÒªÖØÐ±àÒëһϾͿÉÒÔÔËÐÐÔÚHadoop2(MRv2)ÉÏ¡£
2.3.2 Mesos
´óÁ¿·Ö²¼Ê½¼ÆËã¿ò¼Ü(Hadoop, Giraph, MPI, etc)µÄ³öÏÖ£¬Ã¿Ò»¸ö¼ÆËã¿ò¼ÜÐèÒª¹ÜÀí×Ô¼ºµÄ¼ÆË㼯Ⱥ¡£ÕâЩ¼ÆËã¿ò¼ÜÍùÍù°ÑÈÎÎñ·Ö¸î³ÉºÜ¶àСÈÎÎñ£¬ÈüÆËã¿¿½üÊý¾Ý£¬´Ó¶ø¿ÉÒÔÌá¸ß¼¯ÈºµÄÀûÓÃÂÊ¡£µ«ÊÇÕâЩ¿ò¼Ü¶¼ÊǶÀÁ¢¿ª·¢µÄ£¬²»¿ÉÄÜÔÚÓ¦Óÿò¼ÜÖ®¼ä¹²Ïí×ÊÔ´¡£ÐÎÏóµÄ±íʾÈçͼ6Ëùʾ¡£

ͼ6 ¼¯ÈºµÄ¾²Ì¬·ÖÇøºÍ¶¯Ì¬¹²Ïí
ÎÒÃÇÏ£ÍûÔÚͬһ¸ö¼¯ÈºÉÏ¿ÉÒÔÔËÐжà¸öÓ¦Óÿò¼Ü¡£ MesosÊÇͨ¹ýÌṩһ¸öͨÓÃ×ÊÔ´¹²Ïí²ã£¬¶à¸ö²»Í¬µÄÓ¦Óÿò¼Ü¿ÉÒÔÔËÐÐÔÚÕâ¸ö×ÊÔ´¹²Ïí²ãÖ®ÉÏ¡£ÐÎÏóµÄ±íʾÈçͼ7Ëùʾ¡£

ͼ7 MesosΪ²»Í¬Ó¦Óÿò¼ÜÌṩͳһµ÷¶È½Ó¿Ú
µ«ÎÒÃDz»Ï£ÍûʹÓüòµ¥µÄ¾²Ì¬·ÖÇøµÄ·½·¨£¬Èçͼ8Ëùʾ¡£

ͼ8 ¼¯ÈºÄÚ¾²Ì¬·ÖÇø
MesosµÄÓ¢Î͍ÒåΪ£º¡° Mesos, which is an open source platform
for fine-grained resource sharing between multiple
diverse cluster computing frameworks.¡±
Mesos×î´óµÄºÃ´¦¾ÍÊÇÌá¸ß¼¯ÈºµÄÀûÓÃÂÊ¡£¿ÉÒԺܺõĸôÀë²úÆ·»·¾³ºÍʵÑé»·¾³£¬¿ÉÒÔͬʱ²¢·¢ÔËÐжà¸ö¿ò¼Ü¡£Æä´Î£¬¿ÉÒÔÔÚ¶à¸ö¼¯ÈºÖ®¼ä¹²ÏíÊý¾Ý¡£µÚÈý£¬¿ÉÒÔ½µµÍά»¤³É±¾¡£
Mesos×î´óÌôÕ½ÊÇÈçºÎÖ§³Ö´óÁ¿µÄÓ¦Óÿò¼Ü¡£ÒòΪÿһ¸ö¿ò¼Ü¶¼Óв»Í¬µÄµ÷¶ÈÐèÇ󣺱à³ÌÄ£ÐÍ¡¢Í¨ÐÅ·¶ÐÍ¡¢ÈÎÎñÒÀÀµºÍÊý¾Ý·ÅÖá£ÁíÍâ
MesosµÄµ÷¶ÈϵͳÐèÒªÄܹ»À©Õ¹µ½Êýǧ¸ö½Úµã£¬ÔËÐÐÊý°ÙÍò¸öÈÎÎñ¡£ÓÉÓÚ¼¯ÈºÖеÄËùÓÐÈÎÎñ¶¼ÒÀÀµÓÚ Mesos£¬µ÷¶Èϵͳ±ØÐëÊÇÈÝ´íºÍ¸ß¿ÉÓõġ£
MesosµÄÉè¼Æ¾ö²ß(Éè¼ÆÕÜѧ)£º²»²ÉÓÃÖÐÐÄ»¯µÄ£¬Éè¼ÆÖÜÈ«µÄ(Ó¦ÓÃÐèÇ󣬿ÉÓÃ×ÊÔ´£¬×éÖ¯²ßÂÔ)£¬ÊÊÓÃÓÚËùÓÐÈÎÎñµÄÈ«¾Öµ÷¶È²ßÂÔ¡£¶ø²ÉÓÃίÅɵ÷¶ÈÈÎÎñ¸øÓ¦Óÿò¼Ü(°Ñµ÷¶ÈºÍÖ´ÐеŦÄܽ»¸øÓ¦Óÿò¼Ü)¡£
MesosÉù³Æ£ºÕâÑùµÄÉè¼Æ²ßÂÔ¿ÉÄܲ»»á´ïµ½È«¾Ö×îÓŵĵ÷¶È£¬µ«ÔÚʵ¼ÊÔËÐÐÖгöÆæµÄºÃ£¬¿ÉÒÔʹµÃÓ¦Óÿò¼Ü½üºõÍêÃÀµÄ´ïµ½Ä¿±ê¡£ÆäÉù³ÆÖеÄÓŵãÖ÷ÒªÓÐÁ½¸ö£ºÓ¦Óÿò¼ÜµÄÑÝ»¯¶ÀÁ¢ºÍ±£³Ö
MesosµÄ¼ò½à¡£
MesosµÄÖ÷Òª×é¼þ°üÀ¨Master daemon£¬Slave daemonsºÍÔÚslavesÖ®ÉÏÔËÐеÄ
Mesos applications (Ò²±»³ÆÎª frameworks)(Èçͼ9Ëùʾ)¡£Master¸ù¾ÝÏàÓ¦µÄ²ßÂÔ(¹«Æ½µ÷¶È£¬ÓÅÏȼ¶µ÷¶ÈµÈ)¾ö¶¨¸øÃ¿Ò»¸öÓ¦Ó÷ÖÅä¶àÉÙ×ÊÔ´¡£Ä£¿é»¯¼Ü¹¹Ö§³Ö¶àÖÖ²ßÂÔ¡£Resource
offerÊÇ×ÊÔ´µÄ³éÏó±íʾ£¬»ùÓÚ¸Ã×ÊÔ´£¬Ó¦Óÿò¼Ü¿ÉÒÔÔÚ¼¯ÈºÖеÄij¸önodeÉÏʵÀý»¯·ÖÅäoffer£¬²¢ÔËÐÐÈÎÎñ¡£Ã¿Ò»¸öResource
offer ¾ÍÊÇÒ»¸ö·Ö²¼ÔÚ¶à¸önodeÉϵĿÕÏÐ×ÊÔ´ÁÐ±í¡£Mesos»ùÓÚÒ»¶¨µÄËã·¨²ßÂÔ(È繫ƽµ÷¶È)¾ö¶¨ÓжàÉÙ×ÊÔ´¿ÉÒÔ·ÖÅ䏸ӦÓÿò¼Ü£¬¶øÓ¦Óÿò¼Ü¾ö¶¨Ê¹ÓÃ(½ÓÊÜ)ÄÄЩ×ÊÔ´£¬ÔËÐÐÄÄЩÈÎÎñ¡£MesosÉÏÔËÐеÄÓ¦Óÿò¼ÜÓÉÁ½²¿·Ö×é³É£ºÓ¦Óõ÷¶ÈÆ÷ºÍslaveÉÏÔËÐдúÀí¡£Ó¦Óõ÷¶ÈÆ÷Ïò
Mesos×¢²á¡£Master¾ö¶¨Ïò×¢²áµÄ¿ò¼ÜÌṩ¶àÉÙ×ÊÔ´£¬Ó¦Óõ÷¶ÈÆ÷¾ö¶¨Master·ÖÅäµÄ×ÊÔ´ÖÐÄÄЩÀ´Ê¹Óᣵ÷¶ÈÍê³ÉÖ®ºó£¬Ó¦Óõ÷¶ÈÆ÷°Ñ½ÓÊܵÄ×ÊÔ´·¢Ë͸ø
Mesos£¬´Ó¶ø¾ö¶¨ÁËʹÓÃÄÄЩslave¡£È»ºóÓ¦Óÿò¼ÜÖеÄÈÎÎñµÄÖ´ÐпÉÒÔÔÚslaveÉÏÔËÐС£µ±ÈÎÎñºÜС²¢ÇÒÊÇ¶ÌÆÚÈÎÎñ(ÿ¸öÈÎÎñ¶¼Æµ·±µÄÈöÉ×Ô¼ºÎÕ×ŵÄ×ÊÔ´µÄʱ)£¬Mesos¹¤×÷µÄºÜºÃ¡£

ͼ9 Mesosµ÷¶È¿ò¼Ü
ÔÚ MesosÖУ¬Ò»¸öÖÐÑë×ÊÔ´·ÖÅäÆ÷¶¯Ì¬µÄ»®·Ö¼¯Èº£¬·ÖÅä×ÊÔ´¸ø²»Í¬µÄµ÷¶È¿ò¼Ü(scheduler frameworks)¡£×ÊÔ´¿ÉÒÔÔÚ²»Í¬µÄµ÷¶È¿ò¼ÜÖ®¼äÒÔ¡°offers¡±µÄÐÎʽÈÎÒâ·ÖÅ䣬offers±íʾÁ˵±Ç°¿ÉÓõÄ×ÊÔ´¡£×ÊÔ´·ÖÅäÆ÷ΪÁ˱ÜÃⲻͬµ÷¶È¿ò¼Ü¶Ôͬһ×ÊÔ´³åÍ»ÉêÇ룬ֻÔÊÐíÒ»´ÎÖ»ÄÜ·ÖÅä¸øÒ»¸öµ÷¶È¿ò¼Ü¡£ÔÚµ÷¶È¾ö²ßµÄ¹ý³ÌÖУ¬×ÊÔ´·ÖÅäÆ÷ʵÖÊÉÏÆðµ½ÁËËøµÄ×÷Óá£Òò´Ë
MesosÖеIJ¢·¢µ÷¶ÈÊDZ¯¹Û²ßÂԵġ£
MasterʹÓÃresource offer»úÖÆÔÚ¶à¸ö¿ò¼ÜÖ®¼äϸÁ£¶ÈµÄ¹²Ïí×ÊÔ´¡£Ã¿Ò»¸öresource
offer¿ÕÏÐ×ÊÔ´ÁÐ±í£¬·Ö²¼ÔÚ¶à¸öslaveÉÏ¡£Master¾ö¶¨¸øÃ¿Ò»¸öÓ¦Óÿò¼ÜÌṩ¶àÉÙ×ÊÔ´£¬ÒÀ¾ÝÊǹ«Æ½·½·¨»òÕßÓÅÏȼ¶·½·¨¡£µÚÈý·¢¿ÉÒÔÒԿɲå°ÎÄ£¿éµÄ·½Ê½¶¨ÖƲßÂÔ¡£
Reject»úÖÆ£º²µ»Ø MesosÌṩµÄ×ÊÔ´·½°¸¡£ÎªÁ˱£³Ö½Ó¿ÚµÄ¼òµ¥ÐÔ£¬ Mesos²»ÔÊÐíÓ¦Óÿò¼ÜÖ¸¶¨×ÊÔ´ÐèÇóµÄÏÞÖÆÐÅÏ¢£¬¶øÊÇÔÊÐíÓ¦Óÿò¼Ü¾Ü¾ø
MesosÌṩµÄ×ÊÔ´·½°¸¡£Ó¦Óÿò¼ÜÈç¹ûÓöµ½Ã»ÓÐÂú×ãÆäÐèÇóµÄ×ÊÔ´Ìṩ·½°¸£¬Ôò»á¾Ü¾øµÈ´ý¡£ MesosÉù³Æ¾Ü¾ø»úÖÆ¿ÉÒÔÖ§³ÖÈÎÒ⸴ÔÓµÄ×ÊÔ´ÏÞÖÆ£¬Í¬Ê±±£³ÖÀ©Õ¹ÐԺͼòµ¥¡£
Reject»úÖÆ´øÀ´µÄÒ»¸öÎÊÌâÊÇÔÚÓ¦Óÿò¼ÜÊÕµ½Ò»¸öÂú×ãÆäÐèÇóµÄ·½°¸Ö®Ç°¿ÉÄÜÐèÒªµÈ´ýºÜ³¤Ê±¼ä¡£ÓÉÓÚ²»ÖªµÀÓ¦Óÿò¼ÜµÄÐèÇó£¬
Mesos¿ÉÄÜ»á°Ñͬһ¸ö×ÊÔ´·½°¸·¢¸ø¶à¸öÓ¦Óÿò¼Ü¡£Òò´Ë£¬ÒýÈëfliter»úÖÆ£º MesosÖеÄÒ»¸öµ÷¶È¿ò¼ÜʹÓÃfilterÀ´ÃèÊöËüÆÚÍû±»·þÎñµÄ×ÊÔ´ÀàÐÍ(ÔÊÐíÓ¦Óÿò¼ÜÉèÖÃÒ»¸öfilter±íʾ¸ÃÓ¦Óÿò¼Ü»áÓÀÔ¶µÄ¾Ü¾øÄ³Àà×ÊÔ´)¡£Òò´Ë£¬Ëü²»ÐèÒª·ÃÎÊÕû¸ö¼¯Èº£¬ËüÖ»ÐèÒª·ÃÎÊËü±»offerµÄ½Úµã¼´¿É¡£ÕâÖÖ²ßÂÔ´øÀ´µÄȱµãÊDz»ÄÜÖ§³Ö»ùÓÚÕû¸ö¼¯Èº×´Ì¬µÄÇÀÕ¼ºÍ²ßÂÔ£ºÒ»¸öµ÷¶È¿ò¼Ü²»ÖªµÀ·ÖÅ䏸ÆäËûµ÷¶È¿ò¼ÜµÄ×ÊÔ´¡£
MesosÌṩÁËÒ»ÖÖ×ÊÔ´´¢´æµÄ²ßÂÔÀ´Ö§³ÖGangµ÷¶È¡£ÀýÈ磬ӦÓÿò¼Ü¿ÉÒÔÖ¸¶¨Ò»¸öÆä¿ÉÒÔÔËÐÐnode°×Ãûµ¥ÁÐ±í¡£Õâ²»ÊǶ¯Ì¬µÄ¼¯Èº·ÖÇøÂð?
Mesos½øÒ»²½½âÊÍfilter»úÖÆ£ºfilterÖ»ÊÇÒ»¸ö×ÊÔ´·ÖÅäÄ£Ð͵ÄÐÔÄÜÓÅ»¯·½°¸£¬Ó¦Óÿò¼ÜÓÐÄÄЩÈÎÎñÔËÐÐÔÚÄÄЩnodeÉÏ×îÖÕ¾ö¶¨È¨¡£

ͼ10 Mesosµ÷¶ÈÁ÷³Ì
MesosÈÎÎñµ÷¶È¹ý³ÌÈçͼ10Ëùʾ£¬¾ßÌåÁ÷³ÌÈçÏ£º
Slave 1 ÏòMaster»ã±¨ËüÓÐ4¸öCPUs ºÍ 4 GBµÄ¿ÕÏÐÄڴ棬Master µÄallocationÄ£¿é»á¸ù¾ÝÏàÓ¦µÄ·ÖÅä²ßÂÔ֪ͨframework
1 ¿ÉÒÔʹÓÃËùÓпÉÓÃ×ÊÔ´¡£
Master°Ñslave 1ÉϵĿÉÓÃ×ÊÔ´·¢Ë͸øframework 1(ÒÔresource offerµÄ·½Ê½)¡£
frameworkµÄµ÷¶ÈÆ÷ÏìÓ¦Masterµ÷¶ÈÆ÷£º×¼±¸ÔÚslaveÉÏÔËÐÐÁ½¸öÈÎÎñ£¬Ê¹ÓõÄ×ÊÔ´·Ö±ðÊÇ£ºµÚÒ»¸öÈÎÎñ<2
CPUs, 1 GB RAM> £¬µÚ¶þ¸öÈÎÎñ <1 CPUs, 2 GB RAM>
¡£
×îºó£¬Master°ÑÈÎÎñ·¢Ë͸øslave£¬È»ºó°ÑÏàÓ¦µÄ×ÊÔ´·ÖÅ䏸frameworkµÄÖ´ÐÐÆ÷¡£È»ºóÖ´ÐÐÆ÷Æô¶¯Á½¸öÈÎÎñ¡£ÓÉÓÚslave1ÉÏ»¹ÓÐ1
CPU ºÍ1 GBµÄÄÚ´æÃ»ÓзÖÅ䣬·ÖÅäÄ£¿é¿ÉÒÔ°Ñ×ÊÔ´·ÖÅ䏸framework 2¡£
ÁíÍ⣬Mesos MasterµÄAllocation moduleÊÇpluggable¡£Ê¹ÓÃZooKeeper
À´ÊµÏÖ Mesos MasterµÄFailover¡£
2.4 ״̬¹²Ïíµ÷¶È(Shared-state scheduling)
ÔÚ״̬¹²Ïíµ÷¶ÈÖУ¬Ã¿Ò»¸öµ÷¶ÈÆ÷¶¼¿ÉÒÔ·ÃÎÊÕû¸ö¼¯Èº×´Ì¬¡£µ±¶à¸öµ÷¶ÈÆ÷ͬʱ¸üм¯Èº×´Ì¬Ê±Ê¹ÓÃÀÖ¹Û²¢·¢¿ØÖÆ¡£Shared-stateµ÷¶È¿ÉÒÔ½â¾öÁ½²ãµ÷¶ÈµÄÁ½¸öÎÊÌ⣺±¯¹Û²¢·¢¿ØÖÆËù´øÀ´µÄ²¢ÐÐÏÞÖÆºÍµ÷¶È¿ò¼Ü¶ÔÕû¸ö¼¯Èº×ÊÔ´µÄ¿É¼ûÐÔ¡£ÀÖ¹Û²¢·¢¿ØÖÆËù´øÀ´µÄÎÊÌâÊǵ±ÀÖ¹Û¼ÙÉè²»³ÉÁ¢Ê±£¬ÐèÒªÖØÐµ÷¶È¡£
ΪÁ˿˷þË«²ãµ÷¶ÈÆ÷µÄÒÔÉÏÁ½¸öȱµã(Omega paperÖ÷Òª¹Ø×¢ÁËÕâ¸öÎÊÌâ)£¬Google¿ª·¢ÁËÏÂÒ»´ú×ÊÔ´¹ÜÀíϵͳOmega£¬OmegaÊÇÒ»ÖÖ»ùÓÚ¹²Ïí״̬µÄµ÷¶ÈÆ÷£¬¸Ãµ÷¶ÈÆ÷½«Ë«²ãµ÷¶ÈÆ÷Öеļ¯ÖÐʽ×ÊÔ´µ÷¶ÈÄ£¿é¼ò»¯³ÉÁËһЩ³Ö¾Ã»¯µÄ¹²ÏíÊý¾Ý(״̬)ºÍÕë¶ÔÕâЩÊý¾ÝµÄÑéÖ¤´úÂ룬¶øÕâÀïµÄ¡°¹²ÏíÊý¾Ý¡±Êµ¼ÊÉϾÍÊÇÕû¸ö¼¯ÈºµÄʵʱ×ÊԴʹÓÃÐÅÏ¢¡£Ò»µ©ÒýÈë¹²ÏíÊý¾Ýºó£¬¹²ÏíÊý¾ÝµÄ²¢·¢·ÃÎÊ·½Ê½¾Í³ÉΪ¸ÃϵͳÉè¼ÆµÄºËÐÄ£¬¶øOmegaÔò²ÉÓÃÁË´«Í³Êý¾Ý¿âÖлùÓÚ¶à°æ±¾µÄ²¢·¢·ÃÎÊ¿ØÖÆ·½Ê½(Ò²³ÆÎª¡°ÀÖ¹ÛËø¡±,
MVCC, Multi-Version Concurrency Control)£¬Õâ´ó´óÌáÉýÁËOmegaµÄ²¢·¢ÐÔ¡£ÔÚOmegaÖÐûÓÐÖÐÐĵÄ×ÊÔ´·ÖÅäÆ÷£¬µ÷¶ÈÆ÷×Ô¼º×ö³ö×ÊÔ´·ÖÅäµÄ¾ö²ß¡£
2.4.1 Omega
ºêµ÷¶ÈµÄȱµãÊÇÄÑÒÔÔö¼Óµ÷¶È²ßÂÔºÍרÃŵÄʵÏÖ£¬²¢ÇÒ²»ÄÜËæ×ż¯ÈºµÄÀ©Õ¹¶øÀ©Õ¹¡£Á½²ãµ÷¶Èȷʵ¿ÉÒÔÌṩÁé»îÐԺͲ¢ÐÐÐÔ£¬µ«ÊÇÔÚʵ¼ùÖÐËûÃǵÄ×ÊÔ´¿É¼ûÐÔÈ´ÊDZ£Êصģ¬ÄÑÒÔÊÊӦһЩÌôÌÞÐ͵ÄÈÎÎñºÍһЩÐèÒª·ÃÎÊÕû¸ö¼¯Èº×ÊÔ´µÄÈÎÎñ¡£OmegaµÄ½â¾ö·½°¸ÊÇÌá³öÁËÒ»¸öеIJ¢Ðе÷¶È¿ò¼Ü£º»ùÓÚ¹²Ïí״̬µÄ¡¢ÎÞËøµÄ¡¢ÀÖ¹Û²¢·¢¿ØÖÆ£¬¿ÉÀ©Õ¹µÄ¡£
Èçͼ11Ëùʾ£¬OmegaÖÐûÓÐÖÐÐĵÄ×ÊÔ´·ÖÅäÆ÷£¬ËùÓеÄ×ÊÔ´·ÖÅä¾ö²ß¶¼ÊÇÓÉÓ¦Óõĵ÷¶ÈÆ÷×Ô¼ºÍê³ÉµÄ¡£Omegaά»¤ÁËÒ»¸ö³ÉΪcell
stateµÄ×ÊÔ´·ÖÅä״̬ÐÅÏ¢Ö÷¿½±´¡£Ã¿Ò»¸öÓ¦Óõĵ÷¶ÈÆ÷¶¼Î¬»¤ÁËÒ»¸ö±¾µØË½Óе쬯µ·±¸üеÄcell state¿½±´£¬ÓÃÀ´×öµ÷¶È¾ö²ß¡£µ÷¶ÈÆ÷¿ÉÒÔ¿´µ½È«¾ÖµÄËùÓÐ×ÊÔ´£¬²¢¸ù¾ÝȨÏÞºÍÓÅÏȼ¶À´×ÔÒÔΪÊǵÄÒªÇóÐèÒªµÄ×ÊÔ´¡£µ±µ÷¶ÈÆ÷¾ö¶¨×ÊÔ´·½°¸Ê±£¬ÒÔÔ×ӵķ½Ê½¸üй²ÏíµÄcell
state£º´ó¶àÊýʱºòÕâÑùcommit½«»á³É¹¦(Õâ¾ÍÊÇÀÖ¹Û·½·¨)¡£µ±³åÍ»·¢Éúʱ£¬µ÷¶È¾ö²ß½«»áÒÔÊÂÎñµÄ·½Ê½Ê§°Ü¡£ÎÞÂÛµ÷¶È³É¹¦»¹ÊÇʧ°Ü£¬µ÷¶ÈÆ÷¶¼»áÖØÐÂͬ²½±¾µØµÄcell
stateºÍ¹²ÏíµÄcell state¡£È»ºó£¬Èç¹ûÐèÒª£¬ÖØÆôµ÷¶È¹ý³Ì¡£
OmegaµÄµ÷¶ÈÆ÷ÍêÈ«ÊDz¢Ðе쬲»ÐèÒªµÈ´ýÆäËûµ÷¶ÈÆ÷¡£ÎªÁ˱ÜÃâ³åÍ»Ôì³ÉµÄ¼¢¶ö£¬Omegaµ÷¶ÈÆ÷ʹÓÃÔöÁ¿µ÷¶È¡ª¡ªAccept
all but the conflict things£¬ÕâÑù¿ÉÒÔ±ÜÃâ×ÊÔ´¶Ú»ý¡£Èç¹ûʹÓÃall or
nothing µÄ²ßÂÔ¿ÉÒÔʹÓÃGangµ÷¶È(Either all tasks of a job are
scheduled together, or none are, and the scheduler
must try to schedule the entire job again.)¡£Gangµ÷¶ÈÒªµÈ´ýËùÓÐ×ÊÔ´¾ÍÐ÷£¬²ÅcommitÕû¸öÈÎÎñ£¬¾ÍÔì³ÉÁË×ÊÔ´¶Ú»ý¡£
ÿһ¸öÓ¦Óõ÷¶ÈÆ÷¶¼¿ÉÒÔʵÏÖ×Ô¼ºµÄµ÷¶È²ßÂÔ¡£µ«ÊÇËüÃDZØÐë¾Í×ÊÔ´·ÖÅäºÍÈÎÎñµÄÓÅÏȼ¶´ï³ÉÒ»Ö¡£Á½²ãµ÷¶ÈµÄÖÐÐÄ×ÊÔ´¹ÜÀíÆ÷¿ÉÒÔÇáËÉʵÏÖÕâÒ»µã¡£ÕâÊÇÒ»¸ö¿ª·ÅµÄ»°Ì⣬¿ÉÒÔ½øÒ»²½ÌÖÂÛ£ºGoogleÈÏΪ¹«Æ½ÐÔ²»ÊÇÒ»¸ö¹Ø¼üÐèÇ󣬸÷¸öµ÷¶ÈÆ÷Ö»ÊÇÂú×ã×Ô¼ºµÄÒµÎñÐèÇó¡£Òò´Ë£¬ÏÞÖÆÃ¿Ò»¸öÓ¦Óõ÷¶ÈÆ÷µÄ×ÊÔ´ÉÏÏÞºÍÈÎÎñÌá½»ÉÏÏÞ¡£

ͼ11 Omegaµ÷¶È¿ò¼Ü
2.4 ¶Ô±È·ÖÎö
¼¯Èºµ÷¶ÈµÄÖ÷ҪĿ±êÊÇÌá¸ß¼¯ÈºµÄÀûÓÃÂʺÍʹÓÃЧÂÊ¡£Èçͼ12ËùʾΪÈýÖÖµ÷¶È·½Ê½¡£Èçͼ13ËùʾΪÈýÖÖµ÷¶È·½Ê½¡£

ͼ12 ÈýÖÖµ÷¶È·½Ê½µÄ¶Ô±È
ºêµ÷¶È(Monolithic schedulers)ΪËùÓÐÈÎÎñ¶¼Ê¹ÓÃÒ»¸öÖÐÐĵ÷¶ÈËã·¨¡£ÆäȱµãÊDz»Ò×Ôö¼Óеĵ÷¶È²ßÂÔ£¬Ò²²»ÄÜËæ×ż¯ÈºµÄÀ©Õ¹¶øÀ©Õ¹¡£
Á½²ãµ÷¶È(Two-level schedulers)±¾ÖÊÉÏÊÇÔÚµ÷¶ÈÖзÖÀë×ÊÔ´·ÖÅäºÍÈÎÎñ·ÖÅ䡣ʹÓÃÒ»¸ö¶¯Ì¬×ÊÔ´¹ÜÀíÆ÷Ìṩ¼ÆËã×ÊÔ´»òÕß´æ´¢×ÊÔ´¸ø¶à¸ö²¢Ðеĵ÷¶È¿ò¼Ü¡£Ã¿Ò»¸öµ÷¶È¿ò¼ÜËùÓµÓеͼÊÇÒ»¸öÕû¸ö×ÊÔ´µÄÒ»¸ö×Ó¼¯¡£ÎªÊ²Ã´ÊÇÒ»¸ö¶¯Ì¬×ÊÔ´¹ÜÀíÆ÷ÄØ?ÊÇÏà¶ÔÓÚ¾²Ì¬µÄ¼¯Èº·ÖÇøÀ´ËµµÄ¡£ÎÒÃÇ¿ÉÒÔ¾²Ì¬µÄ°Ñ¼¯Èº·ÖΪ¼¸¸öÇø£¬·Ö±ð·þÎñÓÚ²»Í¬µÄÓ¦Óá£ÉÏÃæµÄ¶¯Ì¬×ÊÔ´¹ÜÀíÆ÷Íê³É¹¤×÷¾ÍÊǰѾ²Ì¬µÄ·ÖÇø¹¤×÷¶¯Ì¬»¯¡£ÓÉÓÚÁ½²ãµ÷¶ÈÎÞ·¨´¦ÀíÄÑÒÔµ÷¶ÈµÄÌôÌÞÈÎÎñ£¬ÇÒ²»Äܸù¾ÝÕû¸ö¼¯ÈºµÄ״̬×ö³ö¾ö²ß£¬GoogleÒýÈë¹²Ïí״̬µ÷¶È¼Ü¹¹¡£
¹²Ïí״̬µ÷¶È(Shared state schedulers)ʹÓÃÎÞËøµÄÀÖ¹Û²¢·¢¿ØÖÆËã·¨¡£Omega,GoogleµÄÏÂÒ»´úµ÷¶ÈϵͳÖÐʹÓÃÁ˸üܹ¹¡£ÄÇô¶Ô±ÈÆðÀ´£¬Á½²ãµ÷¶È(Two-level
schedulers)±¾ÖÊÉÏÊDZ¯¹Ûµ÷¶ÈËã·¨¡£
ÔÚOmega¿´À´£¬ MesosµÄoffer µÄ»úÖÆ±¾ÖÊÉÏÊÇÒ»¸ö¶¯Ì¬µÄ¹ýÂË»úÖÆ£¬ÕâÑù Mesos MasterÏòÓ¦Óÿò¼ÜÌṩµÄÖ»ÊÇÒ»¸ö×ÊÔ´³ØµÄ×Ó¼¯¡£µ±È»¿ÉÒÔ°ÑÕâ¸ö×Ó¼¯À©´óΪһ¸öÈ«¼¯£¬Ò²¾ÍÊÇShare
stateµÄ£¬µ«Æä½Ó¿ÚÒÀÈ»ÊDZ¯¹Û²ßÂԵġ£
ÔÚOmega¿´À´£¬YARNÖеÄApplication MastersÌṩµÄ½ö½öÊÇÒ»¸öÈÎÎñ¹ÜÀí·þÎñ£¬²¢²»ÊÇÒ»¸öÕæÕýµÄ¶þ²ãµ÷¶ÈÆ÷¡£Æä´Î£¬µ½Ä¿Ç°ÎªÖ¹£¬YARNÖ»Ö§³ÖÒ»ÖÖ×ÊÔ´ÀàÐÍ¡£ÁíÍ⣬¾¡¹ÜYARNÖеÄApplication
Masters¿ÉÒÔÇëÇóÒ»¸öÌØ¶¨½ÚµãµÄ×ÊÔ´£¬µ«ÊÇÆä¾ßÌå²ßÂÔÊDz»ÇåÎúµÄ¡£
ͼ13Ëùʾ¼¸ÖÖµ÷¶È²ßÂԵĶԱȣº

ͼ13 µ÷¶È²ßÂԵĶԱÈ(°üº¬¾²Ì¬·ÖÇø)
3ÏÂÒ»²½¹¤×÷
¼¯Èºµ÷¶È¼¼ÊõÈÔÔÚ·¢Õ¹Ö®ÖУ¬OSDI 16½«»á·¢²¼Ò»Ð©×îÐµĹØÓÚµ÷¶ÈµÄÎÄÕ£¬°üÀ¨GoogleµÄRapid:Fast,
Centralized Cluster Scheduling at Scale£¬ºóÃæ»áÖ§³Ö¹Ø×¢¡£×ÊÔ´¸ÐÖªµ÷¶È¡£ÀûÓûúÆ÷ѧϰ´ÓÀúÊ·¸ºÔر仯ÖÐÔ¤²â×ÊÔ´ÐèÇóÄ£ÐÍ£¬Îªµ÷¶È¾ö²ßÌṩÒÀ¾Ý¡£ |