Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Modeler   Code  
»áÔ±   
 
   
 
 
     
   
 ¶©ÔÄ
  ¾èÖú
Tachyon£ºSparkÉú̬ϵͳÖеķֲ¼Ê½ÄÚ´æÎļþϵͳ
 
×÷Õß ÁõÉÙɽ À´Ô´£º¡¶³ÌÐòÔ±¡·µç×Ó¿¯ »ðÁú¹ûÈí¼þ  ·¢²¼ÓÚ 2015-6-29
  2866  次浏览      29
 

ÕªÒª£ºTachyon°ÑÄÚ´æ´æ´¢µÄ¹¦ÄÜ´ÓSparkÖзÖÀë³öÀ´£¬ ʹSpark¿ÉÒÔ¸üרע¼ÆËãµÄ±¾Éí£¬ ÒÔÇóͨ¹ý¸üϸµÄ·Ö¹¤´ïµ½¸ü¸ßµÄÖ´ÐÐЧÂÊ¡£

TachyonÊÇSparkÉú̬ϵͳÄÚ¿ìËÙáÈÆðµÄÒ»¸öÐÂÏîÄ¿¡£ ±¾ÖÊÉÏ£¬ TachyonÊǸö·Ö²¼Ê½µÄÄÚ´æÎļþϵͳ£¬ ËüÔÚ¼õÇáSparkÄÚ´æÑ¹Á¦µÄͬʱ£¬Ò²¸³ÓèÁËSparkÄÚ´æ¿ìËÙ´óÁ¿Êý¾Ý¶ÁдµÄÄÜÁ¦¡£Tachyon°ÑÄÚ´æ´æ´¢µÄ¹¦ÄÜ´ÓSparkÖзÖÀë³öÀ´£¬ ʹSpark¿ÉÒÔ¸üרע¼ÆËãµÄ±¾Éí£¬ ÒÔÇóͨ¹ý¸üϸµÄ·Ö¹¤´ïµ½¸ü¸ßµÄÖ´ÐÐЧÂÊ¡£ ±¾ÎĽ«ÏÈÏò¶ÁÕß½éÉÜTachyonÔÚSparkÉú̬ϵͳÖеÄʹÓ㬠Ҳ½«·ÖÏí°Ù¶ÈÔÚ´óÊý¾Ýƽ̨ÉÏÀûÓÃTachyonÈ¡µÃµÄÐÔÄܸÄÉÆµÄÓÃÀý£¬ÒÔ¼°ÔÚʵ¼ÊʹÓÃTachyon¹ý³ÌÖÐÓöµ½µÄһЩÎÊÌâºÍ½â¾ö·½°¸¡£×îºóÎÒÃǽ«½éÉÜÒ»ÏÂTachyonµÄһЩй¦ÄÜ¡£

Tachyon¼ò½é

Sparkƽ̨ÒÔ·Ö²¼Ê½ÄÚ´æ¼ÆËãµÄģʽ´ïµ½¸ü¸ßµÄ¼ÆËãÐÔÄÜ£¬ÔÚ×î½üÒýÆðÁËÒµ½çµÄ¹ã·º¹Ø×¢£¬Æä¿ªÔ´ÉçÇøÒ²Ê®·Ö»îÔ¾¡£ÒÔ°Ù¶ÈΪÀý£¬ÔÚ°Ù¶ÈÄÚ²¿¼ÆËãÆ½Ì¨ÒѾ­´î½¨²¢ÔËÐÐÁËǧ̨¹æÄ£µÄSpark¼ÆË㼯Ⱥ£¬°Ù¶ÈҲͨ¹ýÆäBMRµÄ¿ª·ÅÔÆÆ½Ì¨¶ÔÍâÌṩSpark¼ÆËãÆ½Ì¨·þÎñ¡£È»¶ø£¬·Ö²¼Ê½ÄÚ´æ¼ÆËãµÄģʽҲÊÇÒ»±úË«Èн££¬ÔÚÌá¸ßÐÔÄܵÄͬʱ²»µÃ²»Ãæ¶Ô·Ö²¼Ê½Êý¾Ý´æ´¢Ëù²úÉúµÄÎÊÌ⣬¾ßÌåÎÊÌâÖ÷ÒªÓÐÒÔϼ¸¸ö£º

  1. µ±Á½¸öSpark×÷ÒµÐèÒª¹²ÏíÊý¾Ýʱ£¬±ØÐëͨ¹ýд´ÅÅ̲Ù×÷¡£±ÈÈ磺×÷Òµ1ÒªÏȰÑÉú³ÉµÄÊý¾ÝдÈëHDFS£¬È»ºó×÷Òµ2ÔÙ´ÓHDFS°ÑÊý¾Ý¶Á³öÀ´¡£ÔÚ´Ë£¬´ÅÅ̵ĶÁд¿ÉÄÜÔì³ÉÐÔÄÜÆ¿¾±¡£
  2. ÓÉÓÚSpark»áÀûÓÃ×ÔÉíµÄJVM¶ÔÊý¾Ý½øÐлº´æ£¬µ±Spark³ÌÐò±ÀÀ£Ê±£¬JVM½ø³ÌÍ˳ö£¬Ëù»º´æÊý¾ÝÒ²ËæÖ®¶ªÊ§£¬Òò´ËÔÚ¹¤×÷ÖØÆôʱÓÖÐèÒª´ÓHDFS°ÑÊý¾ÝÔٴζÁ³ö¡£
  3. µ±Á½¸öSpark×÷ÒµÐè²Ù×÷ÏàͬµÄÊý¾Ýʱ£¬Ã¿¸ö×÷ÒµµÄJVM¶¼ÐèÒª»º´æÒ»·ÝÊý¾Ý£¬²»µ«Ôì³É×ÊÔ´ÀË·Ñ£¬Ò²¼«Ò×Òý·¢Æµ·±µÄÀ¬»øÊÕ¼¯£¬Ôì³ÉÐÔÄܵĽµµÍ¡£

×Ðϸ·ÖÎöÕâЩÎÊÌâºó£¬¿ÉÒÔÈ·ÈÏÎÊÌâµÄ¸ùÔ´À´×ÔÓÚÊý¾Ý´æ´¢£¬ÓÉÓÚ¼ÆËãÆ½Ì¨³¢ÊÔ×ÔÐнøÐд洢¹ÜÀí£¬ÒÔÖÁÓÚSpark²»ÄÜרעÓÚ¼ÆËã±¾Éí£¬Ôì³ÉÕûÌåÖ´ÐÐЧÂʵĽµµÍ¡£

TachyonµÄÌá³ö¾ÍÊÇΪÁ˽â¾öÕâЩÎÊÌ⣺±¾ÖÊÉÏ£¬TachyonÊǸö·Ö²¼Ê½µÄÄÚ´æÎļþϵͳ£¬ËüÔÚ¼õÇáSparkÄÚ´æÑ¹Á¦µÄͬʱ¸³ÓèÁËSparkÄÚ´æ¿ìËÙ´óÁ¿Êý¾Ý¶ÁдµÄÄÜÁ¦¡£Tachyon°Ñ´æ´¢ÓëÊý¾Ý¶ÁдµÄ¹¦ÄÜ´ÓSparkÖзÖÀ룬ʹµÃSpark¸üרעÔÚ¼ÆËãµÄ±¾Éí£¬ÒÔÇóͨ¹ý¸üϸµÄ·Ö¹¤´ïµ½¸ü¸ßµÄÖ´ÐÐЧÂÊ¡£


ͼ1: TachyonµÄ²¿Êð

ͼ1ÏÔʾÁËTachyonµÄ²¿Êð½á¹¹¡£Tachyon±»²¿ÊðÔÚ¼ÆËãÆ½Ì¨£¨Spark£¬MR£©Ö®ÏÂÒÔ¼°´æ´¢Æ½Ì¨£¨HDFS£¬ S3£©Ö®ÉÏ£¬Í¨¹ýÈ«¾ÖµØ¸ôÀë¼ÆËãÆ½Ì¨Óë´æ´¢Æ½Ì¨£¬ Tachyon¿ÉÒÔÓÐЧµØ½â¾öÉÏÎÄÁоٵöÎÊÌ⣬£º

  1. µ±Á½¸öSpark×÷ÒµÐèÒª¹²ÏíÊý¾Ýʱ£¬ÎÞÐèÔÙͨ¹ýд´ÅÅÌ£¬¶øÊǽèÖúTachyon½øÐÐÄÚ´æ¶Áд£¬´Ó¶øÌá¸ß¼ÆËãЧÂÊ¡£
  2. ÔÚʹÓÃTachyon¶ÔÊý¾Ý½øÐлº´æºó£¬¼´±ãÔÚSpark³ÌÐò±ÀÀ£JVM½ø³ÌÍ˳öºó£¬Ëù»º´æÊý¾ÝÒ²²»»á¶ªÊ§¡£ÕâÑù£¬Spark¹¤×÷ÖØÆôʱ¿ÉÒÔÖ±½Ó´ÓTachyonÄÚ´æ¶ÁÈ¡Êý¾ÝÁË¡£
  3. µ±Á½¸öSpark×÷ÒµÐèÒª²Ù×÷ÏàͬµÄÊý¾Ýʱ£¬ËüÃÇ¿ÉÒÔÖ±½Ó´ÓTachyon»ñÈ¡£¬²¢²»ÐèÒª¸÷×Ô»º´æÒ»·ÝÊý¾Ý£¬´Ó¶ø½µµÍJVMÄÚ´æÑ¹Á¦£¬¼õÉÙÀ¬»øÊÕ¼¯·¢ÉúµÄƵÂÊ¡£

Tachyonϵͳ¼Ü¹¹

ÔÚÉÏÒ»ÕÂÎÒÃǽéÉÜÁËTachyonµÄÉè¼Æ£¬±¾ÕÂÎÒÃÇÀ´¼òµ¥¿´¿´TachyonµÄϵͳ¼Ü¹¹ÒÔ¼°ÊµÏÖ¡£ ͼ2ÏÔʾÁËTachyonÔÚSparkƽ̨µÄ²¿Êð£º×ܵÄÀ´Ëµ£¬TachyonÓÐÈý¸öÖ÷ÒªµÄ²¿¼þ£ºMaster£¬ Client£¬ÓëWorker¡£ÔÚÿ¸öSpark Worker½ÚµãÉÏ£¬¶¼²¿ÊðÁËÒ»¸öTachyon Worker£¬Spark Workerͨ¹ýTachyon Client·ÃÎÊTachyon½øÐÐÊý¾Ý¶Áд¡£ËùÓеÄTachyon Worker¶¼±»Tachyon MasterËù¹ÜÀí£¬Tachyon Masterͨ¹ýTachyon Worker¶¨Ê±·¢³öµÄÐÄÌøÀ´ÅжÏWorkerÊÇ·ñÒѾ­±ÀÀ£ÒÔ¼°Ã¿¸öWorkerÊ£ÓàµÄÄÚ´æ¿Õ¼äÁ¿¡£


ͼ2: TachyonÔÚSparkƽ̨µÄ²¿Êð

ͼ3ÏÔʾÁËTachyon MasterµÄ½á¹¹£¬ÆäÖ÷Òª¹¦ÄÜÈçÏ£ºÊ×ÏÈ£¬Tachyon MasterÊǸöÖ÷¹ÜÀíÆ÷£¬´¦Àí´Ó¸÷¸öClient·¢³öµÄÇëÇó£¬ÕâһϵÁеŤ×÷ÓÉService HandlerÀ´Íê³É¡£ÕâЩÇëÇó°üÀ¨£º»ñÈ¡WorkerµÄÐÅÏ¢£¬¶ÁÈ¡FileµÄBlockÐÅÏ¢£¬ ´´½¨FileµÈµÈ£»Æä´Î£¬Tachyon MasterÊǸöName Node£¬´æ·Å×ÅËùÓÐÎļþµÄÐÅÏ¢£¬Ã¿¸öÎļþµÄÐÅÏ¢¶¼±»·â×°³ÉÒ»¸öInode£¬Ã¿¸öInode¶¼¼Ç¼×ÅÊôÓÚÕâ¸öÎļþµÄËùÓÐBlockÐÅÏ¢¡£ÔÚTachyonÖУ¬BlockÊÇÎļþϵͳ´æ´¢µÄ×îСµ¥Î»£¬¼ÙÉèÿ¸öBlockÊÇ256MB£¬Èç¹ûÓÐÒ»¸öÎļþµÄ´óСÊÇ1GB£¬ÄÇôÕâ¸öÎļþ»á±»ÇÐΪ4¸öBlock¡£Ã¿¸öBlock¿ÉÄÜ´æÔÚ¶à¸ö¸±±¾£¬±»´æ´¢ÔÚ¶à¸öTachyon WorkerÖУ¬Òò´ËMasterÀïÃæÒ²±ØÐë¼Ç¼ÿ¸öBlock±»´æ´¢µÄWorkerµØÖ·£»µÚÈý£¬Tachyon Masterͬʱ¹ÜÀí×ÅËùÓеÄWorker£¬Worker»á¶¨Ê±ÏòMaster·¢ËÍÐÄÌøÍ¨Öª±¾´Î»îԾ״̬ÒÔ¼°Ê£Óà´æ´¢¿Õ¼ä¡£MasterÊÇͨ¹ýMaster Worker InfoÈ¥¼Ç¼ÿ¸öWorkerµÄÉÏ´ÎÐÄÌøÊ±¼ä£¬ÒÑʹÓõÄÄÚ´æ¿Õ¼ä£¬ÒÔ¼°×Ü´æ´¢¿Õ¼äµÈÐÅÏ¢¡£


ͼ3: TachyonµÄMasterÉè¼Æ

ͼ4ÏÔʾÁËTachyon WorkerµÄ½á¹¹£¬ËüÖ÷Òª¸ºÔð´æ´¢¹ÜÀí£ºÊ×ÏÈ£¬Tachyon WorkerµÄService Handler´¦ÀíÀ´×ÔClient·¢À´µÄÇëÇó£¬ÕâЩÇëÇó°üÀ¨£º¶Áȡij¸öBlockµÄÐÅÏ¢£¬»º´æÄ³¸öBlock£¬Ëø×¡Ä³¸öBlock£¬Ïò±¾µØÄÚ´æ´æ´¢ÒªÇó¿Õ¼äµÈµÈ¡£µÚ¶þ£¬Tachyon WorkerµÄÖ÷Òª²¿¼þÊÇWorker Storage£¬Æä×÷ÓÃÊǹÜÀíLocal Data£¨±¾µØµÄÄÚ´æÎļþϵͳ£©ÒÔ¼°Under File System£¨TachyonÒÔϵĴÅÅÌÎļþϵͳ£¬±ÈÈçHDFS£©¡£µÚÈý£¬Tachyon Worker»¹ÓиöData ServerÒԱ㴦ÀíÆäËûµÄClient¶ÔÆä·¢ÆðµÄÊý¾Ý¶ÁдÇëÇó¡£µ±ÓÉÇëÇó´ïµ½Ê±£¬Tachyon»áÏÈÔÚ±¾µØµÄÄÚ´æ´æ´¢ÕÒÊý¾Ý£¬Èç¹ûûÓÐÕÒµ½Ôò»á³¢ÊÔÈ¥ÆäËûµÄTachyon WorkerµÄÄÚ´æ´æ´¢ÖнøÐвéÕÒ¡£Èç¹ûÊý¾ÝÍêÈ«²»ÔÚTachyonÀÔòÐèҪͨ¹ýUnder File SystemµÄ½Ó¿ÚÈ¥´ÅÅÌÎļþϵͳ£¨HDFS£©ÖжÁÈ¡¡£


ͼ4: TachyonµÄWorkerÉè¼Æ

ͼ5ÏÔʾÁËTachyon ClientµÄ½á¹¹£¬ËüÖ÷Òª¹¦ÄÜÊÇÏòÓû§³éÏóÒ»¸öÎļþϵͳ½Ó¿ÚÒÔÆÁ±Îµôµ×²ãʵÏÖϸ½Ú¡£Ê×ÏÈ£¬Tachyon Client»áͨ¹ýMaster Client²¿¼þ¸úTachyon Master½»»¥£¬±ÈÈç¿ÉÒÔÏòTachyon Master²éѯij¸öÎļþµÄij¸öBlockÔÚÄÄÀï¡£Tachyon ClientÒ²»áͨ¹ýWorker Client²¿¼þ¸úTachyon Worker½»»¥£¬ ±ÈÈçÏòij¸öTachyon WorkerÇëÇó´æ´¢¿Õ¼ä¡£ÔÚTachyon ClientʵÏÖÖÐ×îÖ÷ÒªµÄÊÇTachyon FileÕâ¸ö²¿¼þ¡£ÔÚTachyon FileÏÂʵÏÖÁËBlock Out Stream£¬ÆäÖ÷ÒªÓÃÓÚд±¾µØÄÚ´æÎļþ£»ÊµÏÖÁËBlock In StreamÖ÷Òª¸ºÔð¶ÁÄÚ´æÎļþ¡£ÔÚBlock In StreamÄÚ°üº¬ÁËÁ½¸ö²»Í¬µÄʵÏÖ£ºLocal Block In StreamÖ÷ÒªÊÇÓÃÀ´¶Á±¾µØµÄÄÚ´æÎļþ£¬¶øRemote Block In StreamÖ÷ÒªÊǶÁ·Ç±¾µØµÄÄÚ´æÎļþ¡£Çë×¢Ò⣬·Ç±¾µØ¿ÉÒÔÊÇÔÚÆäËüµÄTachyon WorkerµÄÄÚ´æÎļþÀҲ¿ÉÒÔÊÇÔÚUnder File SystemµÄÎļþÀï¡£


ͼ5: TachyonµÄClientÉè¼Æ

ÏÖÔÚÎÒÃÇͨ¹ýÒ»¸ö¼òµ¥µÄ³¡¾°°Ñ¸÷¸ö²¿¼þ¶¼´®ÆðÀ´£º¼ÙÉèÒ»¸öSpark×÷Òµ·¢ÆðÁËÒ»¸ö¶ÁÇëÇó£¬ËüÊ×ÏÈ»áͨ¹ýTachyon ClientÈ¥Tachyon Master²éѯËùÐèÒªµÄBlockËùÔÚµÄλÖá£Èç¹ûËùÔÚµÄBlock²»ÔÚ±¾µØµÄTachyon WorkerÀ´ËClientÔò»áͨ¹ýRemote Block In StreamÏò±ðµÄTachyon Worker·¢³ö¶ÁÇëÇó£¬Í¬Ê±ÔÚBlock¶ÁÈëµÄ¹ý³ÌÖУ¬ClientÒ²»áͨ¹ýBlock Out Stream°ÑBlockдÈëµ½±¾µØµÄÄÚ´æ´æ´¢ÀÕâÑù¾Í¿ÉÒÔ±£Ö¤Ï´ÎͬÑùµÄÇëÇó¿ÉÒÔÓɱ¾»úÍê³É¡£

TachyonÔÚ°Ù¶ÈÄÚ²¿µÄʹÓÃ

ÔÚ°Ù¶ÈÄÚ²¿£¬ÎÒÃÇʹÓÃSpark SQL½øÐдóÊý¾Ý·ÖÎö¹¤×÷, ÓÉÓÚSparkÊǸö»ùÓÚÄÚ´æµÄ¼ÆËãÆ½Ì¨£¬ÎÒÃÇÔ¤¼Æ¾ø´ó²¿·ÖµÄÊý¾Ý²éѯӦ¸ÃÔÚ¼¸Ãë»òÕßÊ®¼¸ÃëÍê³ÉÒÔ´ïµ½»¥¶¯²éѯµÄÄ¿µÄ¡£¿ÉÊÇÔÚSpark¼ÆËãÆ½Ì¨µÄÔËÐÐÖУ¬ÎÒÃÇÈ´·¢ÏÖ²éѯ¶¼ÐèÒªÉϰÙÃë²ÅÄÜÍê³É£¬ÆäÔ­ÒòÈçͼ6Ëùʾ£ºÎÒÃǵļÆËã×ÊÔ´(Data Center 1)ÓëÊý¾Ý²Ö¿â(Data Center 2)¿ÉÄܲ¢²»ÔÚͬһ¸öÊý¾ÝÖÐÐÄÀïÃæ£¬ÔÚÕâÖÖÇé¿öÏ£¬ÎÒÃÇÿһ´ÎÊý¾Ý²éѯ¶¼¿ÉÄÜÐèÒª´ÓÔ¶¶ËµÄÊý¾ÝÖÐÐĶÁÈ¡Êý¾Ý£¬ÓÉÓÚÊý¾ÝÖÐÐļäµÄÍøÂç´ø¿íÒÔ¼°ÑÓʱµÄÎÊÌ⣬µ¼ÖÂÿ´Î²éѯ¶¼ÐèÒª½Ï³¤µÄʱ¼ä£¨>100Ã룩²ÅÄÜÍê³É¡£¸üÔã¸âµÄÊÇ£¬ºÜ¶à²éѯµÄÖØ¸´ÐԺܸߣ¬Í¬ÑùµÄÊý¾ÝºÜ¿ÉÄܻᱻ²éѯ¶à´Î£¬Èç¹ûÿ´Î¶¼´ÓÔ¶¶ËµÄÊý¾ÝÖÐÐĶÁÈ¡£¬±ØÈ»Ôì³É×ÊÔ´ÀË·Ñ¡£

ΪÁ˽â¾öÕâ¸öÎÊÌ⣬ÎÒÃǽèÖúTachyon°ÑÊý¾Ý»º´æÔÚ±¾µØ£¬¾¡Á¿±ÜÃâ¿çÊý¾ÝÖÐÐĵ÷Êý¾Ý¡£µ±Tachyon±»²¿Êðµ½SparkËùÔÚµÄÊý¾ÝÖÐÐĺó£¬Ã¿´ÎÊý¾ÝÀä²éѯʱ£¬ÎÒÃÇ»¹ÊÇ´ÓÔ¶¶ËÊý¾Ý²Ö¿âÀ­Êý¾Ý£¬µ«Êǵ±Êý¾ÝÔٴα»²éѯʱ£¬Spark½«´ÓͬһÊý¾ÝÖÐÐĵÄTachyonÖжÁÈ¡Êý¾Ý£¬´Ó¶øÌá¸ß²éѯÐÔÄÜ¡£ÊµÑé±íÃ÷£ºÈç¹û´Ó·Ç±¾»úµÄTachyon¶ÁÈ¡Êý¾Ý£¬ºÄʱ½µµ½10µ½15Ã룬±ÈÔ­À´µÄÐÔÄÜÌá¸ßÁË10±¶£»×îºÃµÄÇé¿öÏ£¬Èç¹û´Ó±¾»úµÄTachyon¶ÁÊý¾Ý£¬²éѯ½öÐè5Ã룬±ÈÔ­À´µÄÐÔÄÜÌá¸ßÁË30±¶£¬Ð§¹ûÏ൱Ã÷ÏÔ¡£

ÔÚʹÓÃÁËÕâ¸öÓÅ»¯ºó£¬ÈȲéѯÐÔÄÜ´ïµ½ÁË»¥¶¯²éѯµÄÒªÇ󣬿ÉÊÇÀä²éѯµÄÓû§ÌåÑ黹ÊǺܲ·ÖÎöÁËÓû§ÐÐΪºó£¬ÎÒÃÇ·¢ÏÖÓû§²éѯµÄģʽ±È½Ï¹Ì¶¨£º±ÈÈçºÜ¶àÓû§Ã¿Ìì¶¼»áÅÜͬһ¸ö²éѯ£¬Ö»ÊÇËùʹÓùýÂËÊý¾ÝµÄÈÕÆÚ»á·¢Éú¸Ä±ä¡£½èÖúÕâ´ÎÌØÐÔ£¬ÎÒÃÇ¿ÉÒÔ¸ù¾ÝÓû§µÄÐèÇó½øÐÐÏßÏÂÔ¤²éѯ£¬Ìáǰ°ÑËùÐèÒªµÄÊý¾Ýµ¼ÈëTachyon£¬´Ó¶ø±ÜÃâÓû§Àä²éѯ¡£


ͼ6: TachyonÔÚ°Ù¶È´óÊý¾Ýƽ̨µÄ²¿Êð

ÔÚʹÓÃTachyon¹ý³ÌÖУ¬ÎÒÃÇÒ²Óöµ½ÁËһЩÎÊÌ⣺ÔÚ¸Õ¿ªÊ¼²¿ÊðTachyonµÄʱºò£¬ ÎÒÃÇ·¢ÏÖÊý¾ÝÍêÈ«²»Äܱ»»º´æ£¬µÚÒ»´ÎÓëºóÐøµÄ²éѯºÄʱÊÇÒ»ÑùµÄ¡£Èçͼ7µÄÔ´´úÂëËùʾ£ºÖ»ÓÐÕû¸öÊý¾ÝBlock±»¶ÁÈ¡ºó£¬Õâ¸öBlock²Å»á±»»º´æ×¡£»·ñÔò»º´æµÄ²Ù×÷»á±»È¡Ïû¡£±ÈÈçÒ»¸öBlockÊÇ256MB£¬Èç¹ûÄã¶ÁÁËÆäÖеÄ255MB£¬Õâ¸öBlock»¹ÊDz»»á±»»º´æ£¬ÒòΪËüÖ»Ðè¶ÁÈ¡Õû¸öblockÖеIJ¿·ÖÊý¾Ý¡£ÔÚ°Ù¶ÈÄÚ²¿£¬ÎÒÃǺܶàÊý¾ÝÊÇÓÃÐÐÁÐʽ´æ´¢µÄ£¬±ÈÈçORCÓëParquetÎļþ£¬Ã¿´Î²éѯֻ»á¶ÁÆäÖеÄij¼¸ÁУ¬ Òò´Ë²»»á¶ÁÈ¡ÍêÕûµÄBlock, ÒÔÖÂblock»º´æÊ§°Ü¡£ÎªÁ˽â¾öÕâ¸öÎÊÌ⣬ÎÒÃǶÔTachyon½øÐÐÁËÐ޸ģ¬Èç¹ûÊý¾ÝBlock²»ÊÇÌ«´óµÄ»°£¬Àä²éѯʱ¼´Ê¹Óû§ÇëÇóµÄÖ»ÊÇÆäÖм¸ÁУ¬ÎÒÃÇÒ²»á°ÑÕû¸öBlock¶¼¶Á½øÀ´£¬±£Ö¤Õû¸öBlockÄܱ»»º´æ×¡£¬È»ºóÔٴβéѯµÄ»°¾Í¿ÉÒÔÖ±½Ó´ÓTachyon¶ÁÈ¡ÁË¡£ÔÚʹÓÃÁËÐ޸ĵİ汾ºó£¬Tachyon´ïµ½ÁËÎÒÃÇÆÚ´ýµÄЧ¹û£¬´ó²¿·Ö²éѯ¿ÉÒÔÔÚ10ÃëÄÚÍê³É¡£


ͼ7: Tachyon»º´æÊý¾ÝÂß¼­

TachyonµÄһЩй¦ÄÜ

ÎÒÃǰÑTachyonµ±×÷»º´æÀ´Ê¹Ó㬵«ÊÇÿ̨»úÆ÷µÄÄÚ´æÓÐÏÞ£¬ÄÚ´æºÜ¿ì»á±»ÓÃÍê¡£ Èç¹ûÎÒÃÇÓÐ50̨»úÆ÷£¬Ã¿Ì¨·ÖÅä20GBµÄÄÚ´æ¸øTachyon£¬ÄÇô×ܹ²Ò²Ö»ÓÐ1TBµÄ»º´æ¿Õ¼ä£¬Ô¶Ô¶²»ÄÜÂú×ãÎÒÃǵÄÐèÒª¡£ÔÚTachyon×îа汾ÓÐÒ»¸öÐµĹ¦ÄÜ£º Hierarchical Storage,¼´Ê¹Óò»Í¬µÄ´æ´¢Ã½½é¶ÔÊý¾Ý·Ö²ã´Î»º´æ¡£Èçͼ8Ëùʾ£¬ËüÀàÓÚCPUµÄ»º´æÉè¼Æ£ºÄÚ´æµÄ¶ÁдËÙ¶È×î¿ìËùÒÔ¿ÉÒÔÓÃÓÚµÚ0¼¶»º´æ£¬È»ºóSSD¿ÉÒÔÓÃÓÚµÚ1¼¶»º´æ£¬×îºó±¾µØ´ÅÅÌ¿ÉÒÔ×÷Ϊµ×²ã»º´æ¡£ÕâÑùµÄÉè¼Æ¿ÉÒÔΪÎÒÃÇÌṩ¸ü´óµÄ»º´æ¿Õ¼ä£¬Í¬Ñù50̨»úÆ÷£¬ÏÖÔÚÎÒÃÇÿ̨¿É¹±Ï׳ö20TBµÄ»º´æ¿Õ¼ä£¬Ê¹×Ü»º´æ¿Õ¼ä´ïµ½1PB£¬»ù±¾¿ÉÒÔÂú×ãÎÒÃǵĴ¢´æÐèÇó¡£ÓëCPU»º´æÀàËÆ£¬Èç¹ûTachyonµÄblock Replacement PolicyÉè¼ÆµÃµ±£¬99%µÄÇëÇó¿ÉÒÔ±»µÚ0¼¶»º´æ£¨Äڴ棩ËùÂú×㣬´Ó¶øÔÚ¾ø´ó²¿·Öʱ¼ä¿ÉÒÔ×öµ½Ãë¼¶ÏìÓ¦¡£


ͼ8: Tachyon Hierarchical Storage

µ±TachyonÊÕµ½¶ÁÇëÇóʱ£¬ËüÊ×Ïȼì²éÊý¾ÝÊÇ·ñÔÚµÚ0²ã£¬Èç¹ûÃüÖУ¬Ö±½Ó·µ»ØÊý¾Ý£¬·ñÔòËü»á²éѯÏÂÒ»²ã»º´æ£¬Ö±µ½ÕÒµ½±»ÇëÇóµÄÊý¾ÝΪֹ¡£Êý¾ÝÕÒµ½ºó»áÖ±½Ó·µ»Ø¸øÓû§£¬Í¬Ê±Ò²»á±»Promoteµ½µÚ0²ã»º´æ£¬È»ºóµÚ0²ã±»Ìæ»»µÄÊý¾ÝBlock»á±»LRUËã·¨Öû»µ½ÏÂÒ»²ã»º´æ¡£Èç´ËÒ»À´£¬Èç¹ûÓû§ÔÙ´ÎÇëÇóÏàͬµÄÊý¾Ý¾Í»áÖ±½Ó´ÓµÚ0²ã¿ìËٵõ½£¬´Ó¶ø³ä·Ö·¢»Ó»º´æµÄLocalityÌØÐÔ¡£

µ±TachyonÊÕµ½Ð´ÇëÇóʱ£¬ËüÊ×Ïȼì²éµÚ0²ãÊÇ·ñÓÐ×ã¹»¿Õ¼ä£¬Èç¹ûÓУ¬ÔòÖ±½ÓдÈëÊý¾Ýºó·µ»Ø¡£·ñÔòËü»á²éѯÏÂÒ»²ã»º´æ£¬Ö±µ½ÕÒµ½Ò»²ã»º´æÓÐ×ã¹»¿Õ¼ä£¬È»ºó°ÑÉÏÒ»²ãµÄÒ»¸öBlockÓÃLRUËã·¨ÍÆµ½ÏÂÒ»²ã£¬Èç´ËÀàÍÆ£¬Ö±µ½°ÑµÚ0²ãÓÐ×ã¹»¿Õ¼äÒÔдÈëеÄÊý¾Ý£¬È»ºóÔÙ·µ»Ø¡£Õâô×öµÄÄ¿µÄÊDZ£Ö¤Êý¾Ý±»Ð´ÈëµÚ0²ã£¬Èç¹û¶ÁÇëÇóÂíÉÏ·¢ÉúÔÚдÇëÇóºó£¬Êý¾Ý¿ÉÒÔ¿ìËÙ±»¶ÁÈ¡¡£¿ÉÊÇ£¬ÕâÑù×öµÄ»°Ð´µÄÐÔÄÜÓпÉÄܱäµÄºÜ²î£º±ÈÈçÍ·Á½²ã»º´æ¶¼ÂúµÄ»°£¬ËüÐèÒª°ÑÒ»¸öBlock´ÓµÚ1²ã¶ªµ½µÚ2²ã£¬ÔÙ°ÑÒ»¸öBlock´ÓµÚ0²ã¶ªµ½µÚ1²ã£¬È»ºó²ÅÄÜдÊý¾Ýµ½µÚ0²ã£¬ÔÙ·µ»Ø¸øÓû§¡£

¶Ô´ËÎÒÃÇ×öÁ˸öÓÅ»¯£¬ ÓëÆä²ã²ãÀàÍÆÌÚ³ö¿Õ¼ä£¬ÎÒÃǵÄËã·¨Ö±½Ó°ÑÊý¾ÝдÈëÓÐ×ã¹»¿Õ¼äµÄ»º´æ²ã£¬È»ºó¿ìËÙ·µ»Ø¸øÓû§¡£Èç¹û»º´æÈ«Âú£¬Ôò°Ñµ×²ãµÄÒ»¸öBlockÖû»µô£¬È»ºó°ÑÊý¾ÝдÈëµ×²ã»º´æºó·µ»Ø¡£¾­¹ýʵÑ飬ÎÒÃÇ·¢ÏÖÓÅ»¯ºóµÄ×ö·¨»á°ÑдÑÓʱ½µµÍÔ¼50%£¬´ó´óµÄÌá¸ßÁËдµÄЧÂÊ¡£µ«ÊǶÁµÄЧÂÊÓÖÈçºÎÄØ£¬ÓÉÓÚÔÚTACHYONÀдÊÇͨ¹ýMemory-Mapped File½øÐеģ¬ËùÒÔÊÇÏÈдÈëÄڴ棬ÔÙFlushµ½´ÅÅÌ£¬Èç¹û¶ÁÊÇÂíÉÏ·¢ÉúÔÚд֮ºóµÄ»°£¬Æäʵ»á´Ó²Ù×÷ϵͳµÄBuffer£¬Ò²¾ÍÊÇÄÚ´æÀï¶ÁÊý¾Ý£¬Òò´Ë¶ÁµÄÐÔÄÜÒ²²»»áϽµ¡£

Hierarchical StorageºÜºÃµØ½â¾öÁËÎÒÃÇ»º´æ²»¹»ÓõÄÎÊÌ⣬ÏÂÒ»²½ÎÒÃǽ«¼ÌÐø¶ÔÆä½øÐÐÓÅ»¯¡£±ÈÈ磬ÏÖÔÚËüÖ»ÓÐLRUÒ»ÖÖÖû»Ëã·¨£¬²¢²»ÄÜÂú×ãËùÓеÄÓ¦Óó¡¾°£¬ ÎÒÃǽ«Õë¶Ô²»Í¬µÄ³¡¾°Éè¼Æ¸ü¸ßЧµÄÖû»Ëã·¨£¬¾¡Á¿Ìá¸ß»º´æÃüÖÐÂÊ¡£

½áÓï

ÎÒ¸öÈËÏàПüϸµÄ·Ö¹¤»á´ïµ½¸ü¸ßµÄЧÂÊ£¬Spark×÷Ϊһ¸öÄÚ´æ¼ÆËãÆ½Ì¨£¬Èç¹ûʹÓùý¶àµÄ×ÊÔ´È¥»º´æÊý¾Ý£¬»áÒý·¢Æµ·±µÄÀ¬»øÊÕ¼¯£¬Ôì³ÉϵͳµÄ²»Îȶ¨£¬»òÕßÓ°ÏìÐÔÄÜ¡£ÔÚÎÒÃÇʹÓÃSparkµÄ³õÆÚ£¬ÏµÍ³²»Îȶ¨ÊÇÎÒÃÇÃæÁÙµÄ×î´óÌôÕ½£¬¶øÆµ·±µÄÀ¬»øÊÕ¼¯ÕýÊÇÒýÆðϵͳ²»Îȶ¨×î´óµÄÔ­Òò¡£±ÈÈçµ±Ò»´ÎÀ¬»øÊÕ¼¯ºÄʱ¹ý³¤Ê±£¬Spark Worker±äµÄÏìÓ¦·Ç³£²»¼°Ê±£¬ºÜÈÝÒ×±»ÎóÈÏΪÒѾ­±ÀÀ££¬µ¼ÖÂÈÎÎñÖØÐÂÖ´ÐС£Tachyonͨ¹ý°ÑÄÚ´æ´æ´¢µÄ¹¦ÄÜ´ÓSparkÖзÖÀë³öÀ´£¬ÈÃSpark¸üרעÔÚ¼ÆËã±¾Éí£¬´Ó¶øºÜºÃµÄ½â¾öÁËÕâ¸öÎÊÌâ¡£Ëæ×ÅÄÚ´æ±äµÄÔ½À´Ô½±ãÒË£¬ÎÒÃÇ¿ÉÒÔÔ¤ÆÚδÀ´Ò»¶Îʱ¼äÀÎÒÃǵķþÎñÆ÷Àï¿ÉʹÓõÄÄÚ´æ»á²»¶ÏÔö³¤£¬Tachyon»áÔÚ´óÊý¾Ýƽ̨Öз¢»ÓÔ½À´Ô½ÖØÒªµÄ×÷Óá£ÏÖÔÚ»¹ÊÇTachyon·¢Õ¹µÄ³õÆÚ£¬ÔÚ±¾ÎÄÍê³ÉʱTachyon²Å×¼±¸·¢²¼0.6°æ£¬»¹Óкܶ๦ÄÜØ½ÐèÍêÉÆ£¬ÕâÒ²ÊÇÒ»¸öºÃ»úÓö£¬ÓÐÐËȤµÄͬѧÃÇ¿ÉÒÔ¶à¹Ø×¢Tachyon£¬µ½ÉçÇøÀï½øÐм¼ÊõÌÖÂÛÒÔ¼°¹¦ÄÜ¿ª·¢¡£

   
2866 ´Îä¯ÀÀ       29
 
Ïà¹ØÎÄÕÂ

ÔÆ¼ÆËãµÄ¼Ü¹¹
¶ÔÔÆ¼ÆËã·þÎñÄ£ÐÍ
ÔÆ¼ÆËãºËÐļ¼ÊõÆÊÎö
Á˽âÔÆ¼ÆËãµÄ©¶´
 
Ïà¹ØÎĵµ

ÔÆ¼ÆËã¼ò½é
ÔÆ¼ÆËã¼ò½éÓëÔÆ°²È«
ÏÂÒ»´úÍøÂç¼ÆËã--ÔÆ¼ÆËã
ÈídzÎöÔÆ¼ÆËã
 
Ïà¹Ø¿Î³Ì

ÔÆ¼ÆËãÔ­ÀíÓëÓ¦ÓÃ
ÔÆ¼ÆËãÓ¦ÓÃÓ뿪·¢
CMMIÌåϵÓëʵ¼ù
»ùÓÚCMMI±ê×¼µÄÈí¼þÖÊÁ¿±£Ö¤
×îл¼Æ»®
DeepSeek´óÄ£ÐÍÓ¦Óÿª·¢ 6-12[ÏÃÃÅ]
È˹¤ÖÇÄÜ.»úÆ÷ѧϰTensorFlow 6-22[Ö±²¥]
»ùÓÚ UML ºÍEA½øÐзÖÎöÉè¼Æ 6-30[±±¾©]
ǶÈëʽÈí¼þ¼Ü¹¹-¸ß¼¶Êµ¼ù 7-9[±±¾©]
Óû§ÌåÑé¡¢Ò×ÓÃÐÔ²âÊÔÓëÆÀ¹À 7-25[Î÷°²]
ͼÊý¾Ý¿âÓë֪ʶͼÆ× 8-23[±±¾©]

ר¼ÒÊӽǿ´ITÓë¼Ü¹¹
Èí¼þ¼Ü¹¹Éè¼Æ
ÃæÏò·þÎñÌåϵ¼Ü¹¹ºÍÒµÎñ×é¼þµÄ˼¿¼
ÈËÈËÍøÒÆ¶¯¿ª·¢¼Ü¹¹
¼Ü¹¹¸¯»¯Ö®ÃÕ
̸ƽ̨¼´·þÎñPaaS
Ïà¹ØÅàѵ¿Î³Ì

ÔÆ¼ÆËãÔ­ÀíÓëÓ¦ÓÃ
Windows Azure ÔÆ¼ÆËãÓ¦ÓÃ

ĦÍÐÂÞÀ­ ÔÆÆ½Ì¨µÄ¹¹½¨ÓëÓ¦ÓÃ
ͨÓù«Ë¾GE DockerÔ­ÀíÓëʵ¼ù
ijÑз¢ÖÐÐÄ Openstackʵ¼ù
ÖªÃûµç×Ó¹«Ë¾ ÔÆÆ½Ì¨¼Ü¹¹ÓëÓ¦ÓÃ
ijµçÁ¦ÐÐÒµ »ùÓÚÔÆÆ½Ì¨¹¹½¨ÔÆ·þÎñ
ÔÆ¼ÆËãÓëWindows AzureÅàѵ
±±¾© ÔÆ¼ÆËãÔ­ÀíÓëÓ¦ÓÃ