Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Modeler   Code  
»áÔ±   
 
   
 
 
     
   
 ¶©ÔÄ
  ¾èÖú
΢µêµÄ´óÊý¾Ýƽ̨½¨Éèʵ¼ùÓë̽ÌÖ
 
×÷ÕߣºÍõ·æ À´Ô´: ³ÌÐòÔ±ÔÓÖ¾   »ðÁú¹ûÈí¼þ  ·¢²¼ÓÚ 2015-9-23
  2775  次浏览      27
 

ÕªÒª£ºÎ¢µêÊÇÈ«ÇòÁìÏȵÄÒÆ¶¯µçÉÌÍøÂ磬´´ÔìÁËÒ»¸ö±ãÀûµÄÊÖ»ú¹ºÎï»·¾³£¬Ä¿Ç°Óг¬¹ý3000ÍòµÄµêÖ÷ʹÓÃ΢µêÏúÊÛÉÌÆ·¡£Î¢µê´óÊý¾Ý¼Ü¹¹Ê¦Íõ·æ£¬½«ÖصãÃèÊö´óÊý¾Ý´¦ÀíÆ½Ì¨ÖÐÊý¾Ý²É¼¯¡¢´«Êä¡¢´æ´¢¡¢·ÖÎö¹ý³ÌÖеĹ«¹²»ù´¡¼¼Êõ²¿·Ö¡£

¡°ÈËÀàÕý´ÓITʱ´ú×ßÏòDTʱ´ú¡±£¬2014ÄêÈýÔÂÔÚ±±¾©¾ÙÐеÄÒ»³¡´óÊý¾Ý²úÒµÍÆ½é»áÉÏ£¬°¢Àï°Í°Í¼¯ÍÅ´´Ê¼ÈËÂíÔÆÔÚÖ÷ÌâÑݽ²Öз¢±íÁËËûµÄÕâÒ»¹Ûµã¡£Õâ¸ö¹ÛÄîÌá·¨ºÜ¿ì¾Í±»¹ã·º´«²¥¿ªÀ´£¬²¢±»ÈËÃÇËù½ÓÊÜ¡£ÕâÀï±ÊÕß²»×¼±¸´ó̸DTʱ´ú£¬µ«ÊÇÏàÐÅDTʱ´úÒ»¶¨ÊÇÒÔÊý¾Ý´¦ÀíΪºËÐĵģ¬Òò´Ë´óÊý¾Ý¼¼ÊõÔÚÕâÀïÓÐÖÁ¹ØÖØÒªµÄµØÎ»£¬ºÜÓÐÐÒ±ÊÕß¼°¸÷λ¿´¹ÙÕýÔÚÕâ¸öÁìÓòŬÁ¦¡£

Ôø¿´µ½Ò»ÆªÎÄÕ£¬ÀïÃæÓиö¹Ûµã£¬¡°DTʱ´úµÄ¹Ç÷À¡ª¡ª´óÊý¾Ý´¦ÀíÆ½Ì¨¡±£¬·´Ó³ÁË´óÊý¾Ý´¦ÀíÆ½Ì¨ÔÚ»¥ÁªÍø»òÕßÒÆ¶¯»¥ÁªÍø¹«Ë¾µÄÖØÒªÐÔ¡£´óÊý¾Ý´¦ÀíÆ½Ì¨Æäʵ°üº¬ÁËÕû¸ö´óÊý¾Ý´¦Àí¹ý³Ì£¬Ëü³ÐÔØÁË´ÓÊý¾Ý²É¼¯¡¢´«Êä¡¢´æ´¢¡¢·ÖÎöÍÚ¾ò£¨ÀëÏß OR¡¢ÊµÊ± OR¡¢¼´Ï¯²éѯ£©¡¢¿ÉÊÓ»¯¡¢¼ÛÖµÌåÏÖµÄÕûÌåÁ÷³Ì¡£ÕâЩÔÚ´óµÄ»¥ÁªÍø¹«Ë¾£¬ÓÈÆäÒÔBATΪÊ×£¬ÒѾ­Öð²½³ÉÊ죬¶øÇÒ¼ÛÖµÌåÏÖ²»¶Ï·Å´ó¡£¶øÔÚ³õ´´¹«Ë¾»òÕß¾ßÓÐÒ»¶¨¹æÄ£µÄ´´Òµ¹«Ë¾£¬´óÊý¾Ý´¦ÀíÆ½Ì¨µÄ»ù´¡ÉèÊ©»ò¿ªÊ¼´î½¨£¬»ò´¦ÓڽϳõʼµÄ״̬£¬»òÕßÔÚÖ𲽹淶ÖС£¿ÉÄÜÓÐÈË»áÓÐÁíÍâµÄÏë·¨£ºÎÒÃǹ«Ë¾¹æÄ£Ã»ÓÐÄÇô´ó£¬ÓбØÒªÕûÕâôһÌ×ô£¿Êǵģ¬Èç¹ûÊý¾ÝÁ¿ºÜС£¬Ã¿ÌìÐÂÔöÊý¾Ý£¨±ÈÈçÓ¦ÓÃÈÕÖ¾£©¶¼ÊÇMB¼¶±ð£¬»òÕßGB¼¶±ð£¬¶øÒÔºóÒ²²»»áÓб¬·¢Ê½Ôö³¤£¬Ò²Ã»±ØÒªÌ«ÕÛÌÚ¡£ÎÞÂÛÈçºÎ£¬ÓÐÒ»¸öÇ÷ÊÆ·Ç³£Ã÷È·£¬Ëæ×Ź«Ë¾ÒµÎñ·¢Õ¹£¬Êý¾ÝÁ¿µÄ±¬·¢Ê½Ôö³¤£¬´óÊý¾Ý´¦ÀíÆ½Ì¨µÄ½¨ÉèÊÆÔÚ±ØÐС£

´óÊý¾Ý´¦ÀíÆ½Ì¨½¨ÉèÊǶÔÊý¾Ý²É¼¯¡¢Êý¾Ý´«Êä¡¢´æ´¢¡¢·ÖÎöÍÚ¾ò£¨ÀëÏß OR ʵʱ OR ¼´Ï¯²éѯ£©¡¢Êý¾ÝÕ¹ÏÖ¡¢¼ÛÖµÌåÏÖµÄÕûÌåÁ÷³ÌÊáÀí¡£Î¢µêÊÇĿǰȫÇòÁìÏȵÄÒÆ¶¯µçÉÌÍøÂ磨ÔÚ΢µêÉú̬Ìåϵ£¬¹«Ë¾ÆìÏ»¹Óпڴü¹ºÎ΢µêÈ«Çò¹º¡¢Î¢µêÂò¼Ò°æ¡¢½ñÈÕ°ë¼Û¡¢YouShopµÈ5´óÓÅÊÆÆ½Ì¨£©£¬´´ÔìÁËÒ»¸ö±ãÀûµÄÊÖ»ú¹ºÎï»·¾³£¬ÊÇÈ«ÇòÄêÇáÈËϲ°®µÄÒÆ¶¯¹ºÎïÍøÂ硣ĿǰÓг¬¹ý3000ÍòµÄµêÖ÷ʹÓÃ΢µêÏúÊÛÉÌÆ·£¬ÔÚÕâÑùµÄ±³¾°Ï£¬¼¼Êõ²¿ÃÅ¿ª·¢²¿ÊðµÄ¸÷ÖÖÓ¦ÓÃÿÌìÐèÒª·þÎñ¾ÞÁ¿ÈÕÖ¾Êý¾Ý£¬ÕâЩÊý¾Ý¼È°üº¬Óû§µÄÐÐÎªÌØÕ÷¡¢ÐËȤ°®ºÃ£¬Ò²°üº¬ÁËÓ¦ÓõķþÎñÖÊÁ¿Çé¿ö£¬ÕâЩ¶¼ÊÇÒª½øÐÐÉî¶È·ÖÎö·¢¾òµÄÊý¾Ý£¬ÖØÒªÐÔ²»ÑÔ¶øÓ÷¡£»ùÓÚ´Ë£¬¸ºÔð´óÊý¾Ý»ù´¡ÉèÊ©½¨ÉèµÄÎÒÃdzе£ÆðÁË´óÊý¾Ý´¦ÀíÆ½Ì¨µÄ½¨ÉèÈÎÎñ£¬ÎªÒµÎñ·ÖÎö²¿ÃÅÌṩ¹«¹²»ù´¡Ö§³Å¡£½ÓÏÂÀ´£¬±¾ÎĽ«ÖصãÃèÊö´óÊý¾Ý´¦ÀíÆ½Ì¨ÖÐÊý¾Ý²É¼¯¡¢´«Êä¡¢´æ´¢¡¢·ÖÎö¹ý³ÌÖеĹ«¹²»ù´¡¼¼Êõ²¿·Ö¡£

ʲôÊÇÊý¾Ý¼¯

Ëæ×ÅÒµÎñµÄ±¬·¢Ê½Ôö³¤£¬¹«Ë¾²¿ÊðÁ˸÷ÖÖ¸÷ÑùµÄÓ¦Ó÷þÎñ£¬ÐµķþÎñÒ²²»¶Ï±»¿ª·¢³öÀ´¡£ÈÕÖ¾Êý¾ÝÓÉÓ¦Ó÷þÎñ²úÉú£¬Ó¦Ó÷þÎñÓÉÒµÎñ¿ª·¢ÈËÔ±¿ª·¢£¬ÓÉÒµÎñÔËάÈËÔ±²¿Êðά»¤£»·ÖÎöÍÚ¾òÕâЩÊý¾ÝµÄÊÇÊý¾Ý·ÖÎöÈËÔ±¡¢ÍƼöËã·¨¿ª·¢ÈËÔ±µÈµÈ£¬ÔÚʵ¼Ê¹¤×÷¹ý³ÌÖУ¬ÓÉÓÚ¸÷·½¹Ø×¢½Ç¶È²»Í¬£¬´øÀ´ºÜ¶à²»±ØÒªµÄ¹µÍ¨½»Á÷³É±¾¡£Êý¾Ý¼¯£¨DATASET£©ÕýÊÇΪÁËÔÚÊý¾Ý²É¼¯¡¢´«Êä¡¢´æ´¢¡¢·ÖÎö¹ý³ÌÖУ¬Êý¾Ý¹ØÁª¸÷·½¶ÔÄ¿±êÊý¾ÝÓÐͳһµÄ³ÆÎ½¡¢Í¬Ê±¹æ·¶Êý¾ÝµÄʹÓá£

ͼ1ÏÔʾÁËÊý¾Ý¼¯µÄÒ»Ð©ÖØÒªÊôÐÔ£¬Ô­ÔòÉÏÓÉÒµÎñ¿ª·¢²¿ÃÅÉêÇë´´½¨ÐµÄÊý¾Ý¼¯£¬ÉêÇëÕß×÷ΪÊý¾ÝµÄowner£¬Í¬Ê±±êʶ³öÆäËùÊô²úÆ·Ïß¡¢ÏîÄ¿¡¢Êý¾ÝÀàÐÍ£¬Äâ²ÉÓõÄÊý¾ÝÊÕ¼¯·½Ê½¡¢´æ´¢·½Ê½£¬Êý¾Ý¹æÄ£Çé¿öÔ¤¹ÀÒÔ¼°Òª´æ´¢µÄʱ¼ä¡£ÆäÖÐÊý¾ÝÀàÐͰüº¬wwwÈÕÖ¾£¨access log£©¡¢Ó¦ÓÃÈÕÖ¾¡¢´íÎóÈÕÖ¾¡¢MySQLÈÕÖ¾µÈµÈ£»Êý¾ÝÊÕ¼¯°üÀ¨£ºAgentʵʱÊÕ¼¯¡¢Rsync´«Êä¡¢HdfsClientÉÏ´«¡¢APIÍÆËÍ£»´æ´¢·½Ê½·ÖΪ£ºHDFS¡¢·Ö²¼Ê½ÏûÏ¢¶ÓÁÐKafka¡¢ÊµÊ±Êý¾ÝËÑË÷Elasticsearch¡¢µÚÈý·½´æ´¢£»Êý¾Ý¹æÄ£Ô¤¹À¿ÉÒÔ¶ÔÒªÊÕ¼¯µÄÊý¾Ý¹æÄ£½øÐÐÆÀ¹À£¬´«Êä²ã¼°´æ´¢²ãÊÇ·ñ¿ÉÒÔ³ÐÔØµÄÒ»¸ö³õ²½Åжϡ£´æ´¢Ê±¼äÈ·¶¨¸ÃÊý¾Ý¼¯±£´æÊ±¼ä£¬µ½ÆÚºóÓÉÆ½Ì¨·½¶ÔÊý¾Ý¼¯Í³Ò»ÇåÀí¡£

ÔÚÊý¾Ý¼¯´´½¨ºó£¬ÓÉÊý¾Ý²É¼¯¶Ë²É¼¯£¬¾­ÓÉÊý¾Ý´«Êä²ã½øÈëÊý¾Ý´æ´¢²ã¡£ÔÚÕâ¸ö¹ý³ÌÖУ¬categoryÊÇÊý¾Ý¼¯µÄÒ»¸ö´úÃû´Ê¡£category×î³õÊÇFacebook¿ªÔ´µÄscribeÅäÖÃÖÐÒ»¸öºÜÖØÒªµÄÊôÐÔ£¬±êʶÊý¾Ý´«Êä¶ÔÏó£¬ÕâÀïÎÒÃÇÑØÓÃÁËÕâ¸öµ¥´Ê£¬²¢´Ó¿ªÊ¼µ½´æ´¢ÂäµØÈ«³Ì±»Ð¯´ø¡£

Êý¾Ý¼¯µÄ»®·ÖÊǺÜÖØÒªµÄÒ»¸ö¹ý³Ì£¬¾ö¶¨ÁËÊý¾ÝÈçºÎ´«Êä¡¢´æ´¢£¬²¢±»ÈçºÎ·ÖÎö´¦Àí¡£Ò»°ãÓÉÒµÎñ²¿Ãż°·ÖÎö²¿ÃÅÈ·¶¨¡£Êý¾Ý¼¯ÄÚÊý¾Ý¸ñʽӦһÖ£¬·½±ã½øÐд¦Àí¡£µ«ÔÚʵ¼Ê³¡¾°Ï£¬ÓÈÆä´´Òµ¹«Ë¾£¬µ¥¸öÒµÎñ²¿ÃÅÄÚÊý¾Ý¸ñʽҲδ±ØÍ³Ò»£¬Êý¾ÝÉ¢ÂäÔÚ¶à¸öÈÕÖ¾ÎļþÖУ¬µ¥¸öÌå»ýÏà¶Ô½ÏС£¬¶ø·ÖÎöÈËÔ±Ò²»á¹Ø×¢ÕâЩÊý¾Ý£¬ÕâÖÖÇé¿öÏÂΪÁË·½±ã´¦Àí£¬¿ÉÒÔ½«ÕâЩ»®·Öµ½Ò»¸öÊý¾Ý¼¯Ï£¬Í¬Ê±Ôڲɼ¯¶Ë¶ÔÊý¾Ý½øÐбê×¢¡£µäÐÍ·½·¨£¬ÈçÔÚʵʱ²É¼¯Ê±ÈÕÖ¾ÐÐÖмÓÈëheader£¬ÓÉÎļþÃû»òÕ߯äËûÌØÕ÷Çø·ÖÊý¾Ý¡£¾ÍÏñÍòÊÂÍòÎïÓÐÆäÉúÃü¹æÂÉÒ»Ñù£¬Êý¾Ý¼¯Ò²²»ÀýÍ⡣ͼ2ÃèÊöÁËÊý¾Ý¼¯µÄÉúÃüÖÜÆÚ¡£

Êý¾Ý²É¼¯²ã

ijһÌ죬һ¸ö·ÖÎöÈËÔ±Ð˳å³å¹ýÀ´£¬¡°Ä³Ä³Ä³£¬ÎÒÒª·ÖÎöxxx·þÎñ´ò³öµÄÈÕÖ¾£¬xxx·þÎñ×òÌìÉÏÏßÁË£¬Õâ¸öÐèÇó·Ç³£ÖØÒª£¬balabalabala......¡±¡£È»ºóÎÒÃǸæËßËû£¬ÈÃÒµÎñ¿ª·¢²¿ÃÅÉêÇë¸öÊý¾Ý¼¯°É£¬Êý¾Ý¼¯´«Êä¹ýÀ´Äã¾Í¿ÉÒÔ·ÖÎöÁË£º£©¡£

Êý¾Ý¼¯ÔÚ´´½¨ºó£¬ËùÊô²úÆ·Ïß¡¢ÏîÄ¿¡¢Êý¾ÝÀàÐÍ£¬Äâ²ÉÓõÄÊý¾ÝÊÕ¼¯·½Ê½¡¢´æ´¢·½Ê½£¬Êý¾Ý¹æÄ£Çé¿öÔ¤¹ÀÒÔ¼°Òª´æ´¢µÄʱ¼äһһȷ¶¨¡£ÒÔAgentʵʱ²É¼¯ÎªÀý£¬Êý¾Ý²É¼¯Á÷³ÌÈçͼ3Ëùʾ¡£

  1. ÓÉÒµÎñ¿ª·¢²¿ÃÅÉêÇëÊý¾Ý¼¯
  2. ´óÊý¾Ý×é·¢²¼DataAgent
  3. ÒµÎñÔËάÈËÔ±ÔÚÒµÎñ»úÆ÷²¿ÊðDataAgent
  4. DataAgent²É¼¯Êý¾Ý²¢´«Êä

Ŀǰ´ó²¿·ÖÒµÎñµÄÈÕÖ¾Êý¾Ý²ÉÓÃÕâÖÖ·½Ê½²É¼¯¡£DataAgent»ùÓÚFlumeʵÏÖ£¬×Ô¿ª·¢Flume²å¼þTailsourceÖ§³Ö¶àÊý¾Ý¼¯¡¢¶àÎļþʵʱtail£¬DataAgent¾ßÓÐÒÔÏÂÌØÐÔ£º

  1. Ö§³ÖÊý¾Ý¼¯(category)ÅäÖã¬Ö§³Öͬʱtail¶à¸öÊý¾ÝÎļþ
  2. Ö§³Öcheckpoint£¬¶¨ÆÚ£¨Ä¬ÈÏ10s£©½«¶Á³öµÄÎļþoffsetдÈë±¾µØ´ÅÅÌ
  3. ¿ª·¢ÏÞËÙÄ£¿é£¬¿ÉÅäÖã¬Ö§³ÖÔÚÌØÊⳡ¾°ÏµÄÏÞËÙ´«Êä
  4. Ö§³Ö°´ÕÕÎļþÃûtailÎļþ£¬Í¬Ê±Ö§³Ö¸ù¾ÝinodeÎļþ²éÕÒ
  5. Ö§³ÖÎļþÈíÁ¬½Ó£¬ÔÚÈíÁ¬½Ó¸Ä±äºó¶ÁȡԴÈÕÖ¾ÎļþÊ£ÓàÄÚÈÝ
  6. ÐÞ¸ÄFlumeÔ´ÂëÖ§³Ö½«Event HeaderдÈëԭʼÊý¾ÝÖÐ
  7. ½è¼øÃÀÍÅDualChannel£¬¿ª·¢ÁËÎÒÃÇ×Ô¼ºµÄDualChannel£¬Ö§³ÖMemChannel+FileChannel¡£
  8. Ö§³ÖKafkachannel£¬²¢ÐÞ¸ÄkafkachannelÔ´Â룬֧³Ö½«Ô­Ê¼Êý¾ÝдÈëKafka£¬¶ÔÒµÎñ·ÖÎö³ÌÐò͸Ã÷
  9. Agent×Ôά»¤¼°ÖÇÄÜÉý¼¶
  10. Agent¶Ë½«¼à¿ØÖ¸±ê·¢µ½Ö¸¶¨ganglia¼à¿Ø¶Ë¿Ú£¬Í³Ò»ÓÉ¼à¿Ø²ãÊÕ¼¯£¬Ö§³ÖÊý¾Ý±È¶Ô£¬²¢Ö§³Ö¸ù¾ÝÓ¦ÓòÎÊýÉèÖñ¨¾¯¡£

DataAgent²É¼¯·½Ê½¾ßÌåʹÓÃFlume£¬ºÎÖÖchannelÓÉÊý¾ÝÀàÐÍ¡¢´æ´¢·½Ê½¡¢Êý¾ÝÁ¿¼°ÒµÎñ³¡¾°×ÛºÏÈ·¶¨¡£¸ù¾ÝÎÒÃǵIJâÊÔ£¬µ¥¸öAgent£¬MemoryChannelÔںܶೡ¾°Ï£¬¶¼¿ÉÒÔ´ïµ½6w+/s£»KafkaChannel¿ÉÒÔµ½µ½2.5w-3w+ÿÃ룬¶øFileChannel×î¸ßÔÚ1w/s£¬ÓÐЩ³¡¾°ÏÂÉõÖÁÔÚ5000/sÒÔÏ¡£¶ÔÓ¦ÓÃÈÕÖ¾£¬ÎÒÃÇÐèÒª±£Ö¤Êý¾ÝµÄ¸ß¿É¿¿ÐÔ´«Ê䣬ͬʱÐèÒª±£Ö¤Ð§ÂÊ£¬ËùÒÔĿǰ´óÁ¿²ÉÓÃtailsource+Kafkachannel·½Ê½£»¶ø·ÃÎÊÈÕÖ¾Ö÷Òª²ÉÓÃtailsource+DualChannel+AVROSink·½Ê½¡£

һЩҵÎñÊý¾ÝÒ²»á²ÉÓÃRsync·½Ê½£¨´æ´¢·½Ê½½öÏÞÓÚHDFS´æ´¢£©£ºÔÚÊý¾Ý¼¯È·¶¨ºó£¬´óÊý¾Ý×é·ÖÅärsyncȨÏÞ£¬ÓÉÒµÎñÔËάÈËԱʹÓÃRsync¾­¹ýÖмäLVS²ã£¬½«Êý¾ÝÍÆË͵½databusÖ¸¶¨µÄRsync model£¨ÓÉcategoryÈ·¶¨£©£¬×îºóÓÉ×Ô¿ª·¢µÄHADOOPLoader×é¼þuploadµ½HDFS¡£

²É¼¯²ãÖ§³ÖAPIÍÆËÍ£¬Ò»Ð©ÉÙÁ¿Êý¾Ý³¡¾°Ï£¬ÒµÎñ¶Ë¿ÉÒÔÖ±½Óµ÷ÓÃÎÒÃÇÌṩµÄÊý¾ÝAPI£¬½«Êý¾ÝÖ±½ÓдÈëKAFKA¡£

ÁíÍâÖ§³ÖÒµÎñ¶ËÖ±½ÓʹÓÃHDFSClientдÈëHDFS£¬ÕâÖÖ·½Ê½Ä¿Ç°Ö÷Òª´æÔÚÓÚÒÔǰÒÅÁôµÄһЩÊý¾ÝÊÕ¼¯ÉÏ¡£ÒòΪHadoop¼¯ÈºÊ¹Óð×Ãûµ¥·½Ê½¶ÔдÈë¶ËIP½øÐÐÊÚȨ£¬Èç¹û´æÔÚ´óÁ¿µÄÕâÀà¿Í»§¶Ë£¬»áÑÏÖØ½µµÍÊý¾ÝµÄ´«ÊäЧÂÊ£¬Í¬Ê±Ìá¸ßÁ˿ͻ§¶ËµÄά»¤³É±¾¡£

Êý¾Ý´«Êä²ã

ÒµÎñÔËάÈËÔ±²¿ÊðDataAgent£¬»òÕ߯äËûÊÕ¼¯·½Ê½ºó£¬Êý¾Ý¼¯½øÈëÊý¾Ý´«Êä²ã¡£Í¼4ÊÇÊý¾Ý´«Êä²ãµÄÕûÌå¼Ü¹¹¡£

DataBusͳһ¸ºÔð¶ÔÊý¾Ý¼¯µÄÖмä²ã´«Êä¡¢Êý¾ÝÁ÷ת¼°Êý¾ÝÂ䵨£¬Êý¾Ý´ÓÒµÎñ¶Ë»úÆ÷·¢³öºóÖм侭¹ýLVS¸ºÔؾùºâ²ã£¬½øÈëDatabus¡£DatabusÓɼ¸²¿·Ö×é³É£¬°üÀ¨£º

  • »ùÓÚFlumeµÄAvroÊý¾Ý½ÓÊղ㣬½ÓÊÕAgent¶ËAvroSink·¢³öµÄÊý¾Ý£»
  • ʹÓÃKafkaChannelʵʱÏû·ÑKafkaÊý¾Ý£»
  • ½ÓÊÕsyslogÊÕ¼¯·½Ê½´«ÈëµÄÊý¾Ý£¬Èç½»»»»úÈÕÖ¾£»
  • HadoopLoader½ÓÊÕRsync´«ÈëµÄÊý¾ÝдÈëHDFS£»
  • ½ÓÊÕAPI postµÄÊý¾Ý

Ö§³ÖµÄ´æ´¢·½Ê½°üÀ¨£º

  • HDFS´æ´¢¼¯Èº
  • Kafka·Ö²¼Ê½ÏûÏ¢¶ÓÁÐ
  • Elasticsearch¼¯Èº
  • µÚÈý·½´æ´¢

ÆäÖУ¬Êý¾ÝдÈëKafkaµÄtopicÓÉÊý¾Ý¼¯£¨»òÕßcategory£©Î¨Ò»È·¶¨£¬·ÖÎö¿ª·¢ÈËÔ±ÔÚ×Ô¼ºµÄkafka consumer¶ËÅäÖÃtopicΪcategory¼´¿ÉÏû·ÑÊý¾Ý¡£

¶ÔÓÚÏòElasticsearchµÄдÈë¸ñʽ»¯Êý¾ÝÐèÇó£¬ÔÚDatabus¶Ë£¬ÎÒÃÇÌṩÁ˾ßÓнÏǿͨÓÃÐÔµÄÖ§³Ö¡£»ùÓÚFlume ElasticsearchSink£¬ÐÞ¸ÄÔ´Â룬֧³ÖÕýÔò¼°·Ö¸ô·ûµÄ×Ö¶ÎÇи²¢¿ÉÅäÖ㬽«Databus´«Êä¹ýÀ´µÄÊý¾Ý¼¯Ô­Ê¼Êý¾Ý£¬¸ù¾ÝÅäÖõĽâÎö·½Ê½¼°×ֶΣ¬¸ñʽ»¯Êý¾ÝΪ½á¹¹»¯Êý¾ÝÊÊÅäElasticsearch£¬Ð´ÈëES¼¯Èº¡£

³ý·ÃÎÊÈÕÖ¾¼°Ó¦ÓÃÈÕÖ¾ÒÔÍ⣬DatabusÖ§³ÖÒÔsyslog·½Ê½ÊÕ¼¯ÍøÂçÉ豸Êý¾Ý¡£½»»»»úÉ豸µÄÎȶ¨¶ÔÒµÎñ·þÎñÖÁ¹ØÖØÒª¡£ÒÔǰÎÒÃÇȱ·¦¶Ô½»»»»úµÄ¼à¿Ø£¬ÔÚ6Ôµף¬ÎÒÃÇרÃŶԹ«Ë¾ÄÚ¸÷»ú·¿¼¸ºõËùÓн»»»»úÒÔsyslog·½Ê½ÊÕ¼¯É豸ÈÕÖ¾µ½Kafka£¬²¢¶ÔÈÕÖ¾½øÐÐʵʱ·ÖÎö£¬·¢ÏÖÒì³£¼°Ê±±¨¾¯¡£

¾ø´ó²¿·ÖÊý¾ÝÐèҪдÈëHDFSÊý¾Ý³¤Ê±¼ä´æ´¢¡£ÎÒÃÇʹÓøÄÔìºóFlume HdfsSinkдÈëHDFS¡£Ô­ÉúµÄHdfsSinkÓÐһЩȱµã£¬ÎÒÃǶԲ¿·ÖÔ´Âë½øÐиÄÔ죺

  1. ÔÚÎÒÃǵij¡¾°ÖУ¬µ¥¸ö»úÆ÷É϶à¸öHdfsSink½ø³ÌÓгöÏÖÎļþͬÃûµÄ·çÏÕ£¬ÐÞ¸ÄÆäÔ´Â룬ÔÚĿǰfilepath+fileprefix+ʱ¼ä´Á+filesuffix»ù´¡ÉÏ£¬ÔÚʱ¼ä´Á¼°filesuffixÖ®¼äÔö¼Ó4Î»Ëæ»úÊý£¬Ê¹Óùý³ÌÖÐûÓÐÔÙ³öÏÖÎļþͬÃûÇé¿ö¡£
  2. HdfsSinkÔÚ½âÎöfilepath¼°fileprefix¹ý³ÌÖÐʹÓÃÕýÔòmatcherȥƥÅ䣬²¢ÇÒÔÚÿ¸öEvent´¦Àí¹ý³ÌÖж¼»á×ßÕâ¸ö¹ý³Ì£¬Ð§Âʺܵͣ¨¶ÔÕýÔò½âÎö´úÂë¶Îµ¥¶À²âÊÔ500w event£¬ÕýÔò½âÎö´úÂë¶ÎºÄʱ53s£©£¬ÒòΪÎÒÃÇдÈëHDFSʱ°´ÕÕÊý¾Ý¼¯Í³Ò»´æ´¢¹æ·¶Ð´È룬ËùÒÔ½«Â·¾¶½âÎöÖØÐ´ÓÅ»¯£¬²¢Ôö¼Ó×Ô¼ºµÄÅäÖÃÊôÐÔ£¬ÓÅ»¯ºó£¬Ð´ÈëHDFSЧÂÊÌáÉý40%ÒÔÉÏ£¨lzoѹËõ£©¡£
  3. дÈëHDFSͳһʹÓÃlzo·½Ê½Ð´È룬´ïµ½Ò»¶¨´óС»òÕß³¬¹ýÅäÖÃʱ¼ä½øÐлعö¡£

ĿǰDatabusдÈëHDFS»òÕßKafkaÅäÖñȽϷ±Ëö£¬ºóÃæÐèÒªÕë¶Ô´Ë½øÐÐÓÅ»¯¡£

HadoopLoaderÊÇÎÒÃÇ×ÔÐпª·¢µÄ×é¼þ£¬ÓÃÒÔ¶¨ÆÚɨÃèRsyncÍÆË͹ýÀ´µÄ±¾µØ´ÅÅÌÊý¾Ý¼¯´æ´¢Ä¿Â¼£¬¸ù¾Ýͳһ´æ´¢¹æ·¶ÉÏ´«ÖÁHDFS¡£¼òµ¥Á÷³ÌÈçÏ£º

  • ¶Ôÿ¸öÊý¾Ý¼¯ÔÚÄÚ´æÖÐά»¤Ò»¸öuploadingQueue¡£É¨ÃèÏ̷߳¢ÏÖ´ýÉÏ´«Îļþºó£¬ÑéÖ¤ÎļþÊÇ·ñÍêÕû£¨¸ù¾Ý¶ÔÓ¦md5ÑéÖ¤ÂëÈ·¶¨£©£¬È»ºó½«´ËÎļþ¼ÓÈë´ËQueue¡£
  • ÉÏ´«Ï̴߳ÓQueueÖÐÄÃÒªÉÏ´«µÄÎļþ£¬´Ó±¾µØ´ÅÅÌmvµ½uploadingĿ¼Ï£¬²¢ÉÏ´«¡£
  • ÉÏ´«½áÊø£¬½«ÒÑÉÏ´«Îļþmvµ½±¾µØ´ÅÅÌdoneĿ¼Ï¡£Í¬Ê±½«±¾´ÎÉÏ´«Îļþ·¾¶£¬ËùÊôÊý¾Ý¼¯¡¢´óС¡¢md5ÑéÖ¤Âë¡¢ÉÏ´«Ê±¼ä¡¢HDFS·¾¶µÈÐÅÏ¢Èë¿â¡£

¿Í»§¶ËʹÓÃAPI postÊý¾ÝĿǰ»¹ÔÚ¿ª·¢ÑéÖ¤½×¶Î£¬ÔÝʱ²»±ã͸©¸ü¶à¡£DatabusÖ§³ÖÏòµÚÈý·½×ª·¢£¬»ùÓÚFlume replica²ßÂÔÅäÖÃʵÏÖ¡£

Êý¾Ý´æ´¢¼°·ÖÎö²ã

ÉÏÎÄÒѾ­Ìáµ½£¬Êý¾Ý¼¯ÔÚDatabusÖÐÖ§³ÖÏòHDFS¡¢Kafka¡¢ElasticsearchдÈëÊý¾Ý¡£ÕâÀïÖ÷Òª¶ÔHDFS´æ´¢¼°¹«¹²·ÖÎöƽ̨´î½¨Öصã½éÉÜ¡£

¶ÔÓÚº£Á¿Êý¾ÝµÄ·Ö²¼Ê½´æ´¢£¬Hadoop/HDFSÒѾ­³ÉΪÊÂʵ±ê×¼£¬Ä¿Ç°²»½öÔÚ¸÷´ó»¥ÁªÍø¹«Ë¾£¬ÉõÖÁÔÚµçÐÅÁìÓòÒÔ¼°ÒøÐÐÒ²¶¼¿ªÊ¼Â½ÐøÂ䵨¡£Hadoop2¶Ô±ÈHadoop1£¬ÎÞÂÛÔÚHA¡¢namenodeÀ©Õ¹ÐÔ¡¢È¨ÏÞ¿ØÖÆ¡¢×ÊÔ´µ÷¶È¼°·ÖÅä¡¢×ÊÔ´¸ôÀëµÈ¶¼Óм«´óÌáÉý¡£Ä¿Ç°ÎÒÃÇʹÓÃHadoop 2.6.0×÷Ϊ¹«Ë¾×îм¯ÈºÊ¹Óð汾£¬²¢¶ÔÒÑÖªµÄÖØÒªbug´òÁËpatch¡£

ÏàÐÅÔںܶ๫˾£¬ÓÈÆäÊÇ´´ÒµÐ͹«Ë¾£¬³õÆÚÒµÎñ¿ìËÙÀ©ÕÅ£¬ÎªÁË·½±ã£¬ÄÚ²¿´æÔÚ¶à¸ö¼¯Èº£¬ÇÒ¼¯Èº¹æÄ£¿ÉÄܶ¼²»ÊǺܴ󣬸÷ÒµÎñʹÓõļ¯Èº°æ±¾¿ÉÄÜÒ²²»Ò»Ñù£¬Ï໥ÒÀÀµÒ²ºÜÉÙ¡£³õÆÚµÄÉ¢Áв¿Êð½á¹¹£¬¿ÉÒÔÇáËÉÓ¦¶ÔÒµÎñµÄѸËÙ·¢Õ¹¡£Ëæ×ÅÒµÎñµÄÖð²½·¢Õ¹£¬¸÷¸öÒµÎñ²¿ÃÅÊý¾Ý¹²ÏíÐèÇóÔ½À´Ô½Ç¿ÁÒ£¬Í¬Ê±Êý¾ÝÒÀÀµ¹ØÏµÒ²Ô½À´Ô½¸´ÔÓ£¬·ÖÎöÊý¾ÝÖм¯Èº¼äÊý¾ÝÀ´»Ø°á¶¯Ô½À´Ô½¶à£¬Í¬Ê±Ëæ×ÅÊý¾ÝÁ¿µÄѸËÙÃÍÔö£¬¸÷¼¯Èº´æ´¢¿Õ¼äѹÁ¦¼Ó´ó£¬Õâʱ¼¯Èº¼ä×ÊÔ´ÕûºÏ¾ÍÔ½À´Ô½±ØÒª£¬É¢Áеļ¯Èº²¿Êð½á¹¹×è°­ÁËÊý¾ÝµÄ¹²Ïí£¬Ôö¼ÓÁËÊý¾Ý´¦Àí¹ý³ÌÍâµÄÐí¶àÊý¾ÝÇ¨ÒÆ»·½Ú£¬½µµÍÁËÊý¾Ý´¦ÀíµÄÐÔÄÜ£¬²¢ÇÒ²»ÀûÓÚ¼¯Èº×ÊÔ´µÄ×î´ó»¯ÀûÓ㬼¯Èº¹ÜÀí³É±¾Ì«¸ß¡£Ôø¼ûµ½ÓиöÒµÎñÿÌ콫½ü20¸öTBµÄÊý¾ÝÔÚ¶à¸ö¼¯Èº¼äÀ´»ØÕÛÌڵݸÀý£¨²¢·Ç¶à»ú·¿ÔÖ±¸£©£¬Ê®·ÖµäÐÍ¡£

ÔÚ΢µêͬÑùÈç´Ë£¬µ¥¸ö»ú·¿ÄÚ´æÔÚ×ÅÈô¸É¸ö´ó´óССµÄ¼¯Èº£¬¼¯Èº¹æÄ£ÔÚ¼¸¸ö½Úµãµ½½ü°Ù¸ö½Úµã²»µÈ£¬×îС¹æÄ£²Å4¸ö½Úµã£¬°æ±¾Ò²²»½üÏàͬ¡£×ÊÔ´ÕûºÏÓÈÎªÖØÒª£¬Í¬Ê±¼æ¹Ë¸÷ÒµÎñ²¿ÃŵÄЧÂÊ¡£Îª´ó¼Òı¸£Àû£¬²ÅÄܸüºÃµÄÍÆ½ø×ÊÔ´ÕûºÏ¹¤×÷¡£ÔÚʵ¼ÊÕûºÏ¹ý³ÌÖУ¬¼¯Èº²»Í¬µÄÒµÎñ´¦ÀíÀàÐÍ£¬¼ÆËãÒýÇæ£¬¾ö¶¨ÈçºÎÈ¥×ÊÔ´ÕûºÏ¡£ÎÒÃÇÕûºÏµÄÔ­ÔòÊÇ´æ´¢¹²ÏíÓÅÏÈ£¬¼ÆËãÀàÐÍ·ÖÀ࣬¼æ¹ËÌØÊâÒµÎñÐèÇó¡£ÔÚ´ËÔ­ÔòÏ£¬ÎÒÃǶà¸ö¼¯Èº½«¹²ÏíͳһµÄHDFS´æ´¢×ÊÔ´£¬½â¾öÊý¾ÝÀ´»Ø°áÔ˵ÄÎÊÌ⣬ͬʱ¸÷¸ö¼¯ÈºÍ³Ò»°æ±¾£¬·½±ã¼¯Èº¹ÜÀí£»°´ÕÕ¼ÆËãÀàÐͽøÐÐÕûºÏ£¬ÕûºÏºó½«»áÓУº

  • ¹«¹²¼ÆË㼯Ⱥ£¬¸ºÔðMR¡¢Hive¡¢Pig¡¢Streaming×÷ÒµµÄ´¦Àí£»
  • Spark¼¯Èº£¬¶ÔÄÚ´æ×ÊÔ´ÐèÇó´ó£¬×¨ÃÅÅÜSpark×÷Òµ£»
  • GPU¼¯Èº£¬¸ºÔð¸ßÐÔÄܼÆË㣻
  • UDC¼¯Èº£¬×¨ÃÅ´¦ÀíÁìµ¼¹ØÐĵÄʱ¼äÒªÇó¸ßµÄÒµÎñÖ¸±êÊý¾Ý±¨±í¡£

ÕûºÏºó£¬¼¯ÈºÊ¹ÓÃͳһµÄHDFS¼¯Èº£¨¹æÄ£300¸ö½Úµã£©£¬¸÷¼ÆË㼯ȺÎïÀí¸ôÀ룬·þÎñÆ÷ÀàÐ͵¥¶ÀÅäÖã¬ÓÐÀûÓڳɱ¾½ÚÔ¼¡£

´æ´¢¹²Ïíºó£¬Êý¾ÝµÄ´æ´¢¹æ·¶¡¢Êý¾Ý°²È«·ÃÎÊ¡¢¶ÁдȨÏ޹淶µÈؽ´ý½¨Á¢¡£Í¬Ê±ÐèÒªÓÐͳһµÄ¹©Êý¾Ý·ÖÎö¿ª·¢ÈËԱʹÓõĴóÊý¾Ý´¦ÀíÆ½Ì¨Portal£¬×÷ΪΨһµÄÓû§ÊÚȨ¡¢ÔªÊý¾Ý·ÃÎÊ¡¢Ìá½»²¢¹ÜÀí×÷Òµ¡¢È¨ÏÞÉêÇë¡¢¼¯Èº×ÊԴʹÓÃÇé¿ö²éѯ¡¢×ÊÔ´ÏÞ¶îµÈµÈ¹¦ÄܵÄÈë¿Ú¡£Í¼5ÊǶÔ×ÊÔ´ÕûºÏºóµÄÊý¾Ý´æ´¢¼°·ÖÎö´¦ÀíÁ÷³Ì¼òͼ¡£

·ÖÎö¿ª·¢ÈËÔ±ÓÉͳһPortal·ÃÎÊ´óÊý¾Ý»ù´¡×ÊÔ´£¬Ö§³ÖÓû§¶ÔÓÐȨÏÞµÄÊý¾Ý¼¯²éѯÊý¾Ý¼¯ÊôÐÔÐÅÏ¢¡¢Êý¾Ý¼¯Êý¾Ý£»°´Ìõ¼þ²éÕÒÊý¾Ý¼¯¡¢È¨ÏÞÉêÇ룻֧³ÖȨÏ޵ľ«Ï¸»¯¹ÜÀí£¨ÈçÒµÎñ×éÄÚȨÏÞ·ÖÅ䣩£»×÷Òµ¹ÜÀí£¨Ìá½»¡¢ÔËÐС¢Í£Ö¹ÀëÏßORʵʱ·ÖÎöÈÎÎñ¡¢Spark×÷ÒµµÈµÈ£©¡¢Êý¾ÝÁ÷ת¹ØÏµ£»²é¿´×ÊԴʹÓÃÇé¿ö±¨±íµÈµÈ¡£Ìá½»µÄ×÷ÒµÓÉ×÷Òµµ÷¶ÈÖÐÐĽøÐе÷¶È£»Ö§³Ö¹«¹²UDFÀà¿â¡£ÔªÊý¾Ý¹ÜÀíÌṩ¶ÔÒµÎñÊý¾Ý²Ö¿âÔªÊý¾ÝµÄ¹²ÏíÖ§³Ö¡£

µ±Ç°Çé¿öÏ£¬´æÔÚןܶà¿Í»§»ú£¨ÈÎÎñÌá½»»ú£©£¬ÓÃÀ´Ìá½»×÷Òµ¡£¿Í»§»ú±ØÐë¾­¹ýƽ̨¹ÜÀí·½ÊÚȨ²Å¿É·ÃÎʼ¯Èº¡£

·ÖÎö¿ª·¢ÈËÔ±¶ÔÊý¾Ý¼¯½øÐзÖÎö´¦Àí£¬ÐèÒª¾­¹ýÊý¾Ý¼¯»òHive¿â±íµÄÊÚȨ£¬²¢Ìá½»µ½Ö¸¶¨µÄ¶ÓÁУ¨Óɼ¯Èº¹ÜÀí·¿Ìáǰ½¨Á¢£¬¶Ô·ÖÎöÈËԱ͸Ã÷£©¡£Ö÷Òª°üÀ¨£º

  • ¿Í»§»úÊÚȨ¡£·ÃÎÊHadoop¼¯ÈºµÄ·þÎñÆ÷³ÆÎª¿Í»§»ú£¬ÊÚȨ²ÅÄÜ·ÃÎÊ¡£
  • Óû§¼°Óû§×é¡£µ±Ç°Õ˺ÅÑØÓÃLinuxµÄuser¼°group£»½«À´»áʹÓÃLDAP£»Óû§×é°´ÕÕÒµÎñ²¿ÃÅ»ò²úÆ·Ïß»®·Ö£¬Áé»îÖ§³ÖÒµÎñ·½µÄȨÏÞÐèÇó¡£
  • Êý¾Ý¼¯ÊÚȨ¡£¶ÔÊý¾Ý¼¯ÓжÁ/дȨÏ޲ſɽøÐÐÏàÓ¦²Ù×÷£¨µÃÒæÓÚhadoop2.4ÐÂÔöµÄaclÌØÐÔ£©¡£

3-1. ԭʼÊý¾Ý£ºOwnerΪ³¬¼¶¹ÜÀíÔ±£¬ÒµÎñ²¿ÃÅÖ»ÔÊÐíÓжÁȨÏÞ£»ÉúÃüÖÜÆÚÓɳ¬¼¶¹ÜÀíԱͳһ¹ÜÀí¡£

3-2. ¹éµµÊý¾Ý£ºÎªÀÏÊý¾Ý(>6month)£¬Í³Ò»Ê¹ÓÃLZMAѹËõ£¬Ìá¸ßѹËõ±È¡£

3-3. ½á¹ûÊý¾Ý£ºOwnerΪҵÎñ·½£¬½¨ÒéʹÓÃͳһ´æ´¢½á¹¹Í³Ò»¹ÜÀí¡£

3-4. Óû§Ä¿Â¼£ºOwnerΪҵÎñ·½£¬²ÉÓÃÈÝÁ¿Åä¶î¹ÜÀí¡£

3-5. tmpĿ¼£º¶¼¿É¶Áд£¬´æ·ÅÁÙʱÊý¾Ý£¬ÓɹÜÀí·½¶¨Ê±ÇåÀí¡£

4. Hive·þÎñÊÚȨ¡£Í³Ò»µÄHive MetaStore·þÎñ£¬°´ÕÕÒµÎñ²¿ÃÅ»ò²úÆ·Ïß¶ÔDB¼°±í»®·ÖȨÏÞ£¬²¢ÅäºÏʹÓÃHDFSÊÚȨ¡£

5. ¶ÓÁÐÊÚȨ¡£°´ÕÕÒµÎñ×é»®·Ö¶ÓÁУ¬²¢·ÖÅä×ÊÔ´;Ö§³Ö¶ÓÁÐǶÌס£¡¾×¢£ºHiveÔ­Éú´úÂëÎÞ·¨×öµ½³¬¼¶¹ÜÀíÔ±½ÇÉ«£¬ÐèÒª×ÔÐÐÐ޸ĴúÂëʵÏÖ¡£¡¿

¼à¿Ø²ã

´óÊý¾Ý´¦ÀíÆ½Ì¨µÄ×îºóÒ»»·ÎÞÒÉÊÇ¼à¿Ø¡£¼à¿ØÏñÊÇÎÒÃǵÄÑÛ¾¦£¬ÎÞʱÎ޿̶¢×Å´óÊý¾Ýƽ̨µÄÕû¸ö´¦ÀíÁ÷³Ì£¬µ±½«Òª³öÏÖÎÊÌâʱ´¥·¢±¨¾¯£¬Æ½Ì¨¹ÜÀíÈËÔ±¼°Ê±ÇÐÈë±ÜÃâ¹ÊÕÏ·¢Éú¡£ÎÒÃÇͳһʹÓÃGanglia´Ó²É¼¯¶Ë¡¢´«Êä²ãµ½´æ´¢²ã¡¢·ÖÎö²ãµÄ»ù´¡×ÊÔ´Ö¸±ê¡¢Ó¦ÓÃÖ¸±êдÈëGanglia£¬²¢Ê¹ÓÃNagios½øÐб¨¾¯¡£Í¼6¡¢Í¼7·Ö±ðÊÇÆ½Ì¨Ï¸÷»ù´¡×é¼þµÄ¼à¿Ø²¼¾Ö¼°DataAgent¶Ë°´ÒµÎñ·ÖÀà¼à¿Ø¡£

ÓÉÓÚʱ¼ä²Ö´Ù£¬Î´ÄÜÓиü¶àµÄʱ¼äУ¶Ô£¬ÎÄÕÂÖÐÄÑÃâÓÐç¢Â©£¬»¶Ó­¿´¹ÙÖ¸Õý¡£ÁíÍâ΢µêÕýÔÚÃæÁÙÊý¾Ý±¬·¢Ê½Ôö³¤£¬´óÊý¾Ý¼¼Êõ¡¢HadoopÏà¹Ø¿ª·¢ÈËÔ±¼±È±£¬ÓÐÖ¾ÓÚ´óÊý¾Ý·½Ïò£¬²¢ÇÒÀÖÓÚÉî¸ûµÄ¼¼ÊõÈË£¬»¶Ó­½«¼òÀúÔÒÀ´£¬ÓÊÏ䵨ַ£ºwangfeng@weidian.com¡£

×÷Õß¼ò½é£ºÍõ·æ¡£ÔøÈÎÖ°²¢¸ºÔðÐÂÀËÑз¢dip·ÖÎöƽ̨¼Ü¹¹Éè¼Æ¡¢¿ª·¢¹¤×÷£¬³ÐÔØÁËÐÂÀ˼°Î¢²©¸÷²úÆ·ÏßµÄÀëÏß¡¢ÊµÊ±µÈ¸÷ÀàÒµÎñ·ÖÎöÐèÇó¡£Ä¿Ç°ÈÎְ΢µê´óÊý¾Ý¼Ü¹¹Ê¦£¬¸ºÔð΢µê´óÊý¾Ý£¨hadoop£©»ù´¡¼¼Êõ¼Ü¹¹¼°·þÎñÔËÓª£¬²¢¸ºÔðÍê³ÉÒµÎñÀ༰ÔËάÀàÖ¸±ê·ÖÎöÐèÇó£¬Öð²½¹¹½¨Î¢µêµÄ¼à¿Ø·ÖÎöƽ̨¡£

 

   
2775 ´Îä¯ÀÀ       27
Ïà¹ØÎÄÕÂ

»ùÓÚEAµÄÊý¾Ý¿â½¨Ä£
Êý¾ÝÁ÷½¨Ä££¨EAÖ¸ÄÏ£©
¡°Êý¾Ýºþ¡±£º¸ÅÄî¡¢ÌØÕ÷¡¢¼Ü¹¹Óë°¸Àý
ÔÚÏßÉ̳ÇÊý¾Ý¿âϵͳÉè¼Æ ˼·+Ч¹û
 
Ïà¹ØÎĵµ

GreenplumÊý¾Ý¿â»ù´¡Åàѵ
MySQL5.1ÐÔÄÜÓÅ»¯·½°¸
ijµçÉÌÊý¾ÝÖÐ̨¼Ü¹¹Êµ¼ù
MySQL¸ßÀ©Õ¹¼Ü¹¹Éè¼Æ
Ïà¹Ø¿Î³Ì

Êý¾ÝÖÎÀí¡¢Êý¾Ý¼Ü¹¹¼°Êý¾Ý±ê×¼
MongoDBʵս¿Î³Ì
²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
PostgreSQLÊý¾Ý¿âʵսÅàѵ
×îл¼Æ»®
DeepSeekÔÚÈí¼þ²âÊÔÓ¦ÓÃʵ¼ù 4-12[ÔÚÏß]
DeepSeek´óÄ£ÐÍÓ¦Óÿª·¢Êµ¼ù 4-19[ÔÚÏß]
UAF¼Ü¹¹ÌåϵÓëʵ¼ù 4-11[±±¾©]
AIÖÇÄÜ»¯Èí¼þ²âÊÔ·½·¨Óëʵ¼ù 5-23[ÉϺ£]
»ùÓÚ UML ºÍEA½øÐзÖÎöÉè¼Æ 4-26[±±¾©]
ÒµÎñ¼Ü¹¹Éè¼ÆÓ뽨ģ 4-18[±±¾©]

MySQLË÷Òý±³ºóµÄÊý¾Ý½á¹¹
MySQLÐÔÄܵ÷ÓÅÓë¼Ü¹¹Éè¼Æ
SQL ServerÊý¾Ý¿â±¸·ÝÓë»Ö¸´
ÈÃÊý¾Ý¿â·ÉÆðÀ´ 10´óDB2ÓÅ»¯
oracleµÄÁÙʱ±í¿Õ¼äдÂú´ÅÅÌ
Êý¾Ý¿âµÄ¿çƽ̨Éè¼Æ


²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿â
¸ß¼¶Êý¾Ý¿â¼Ü¹¹Éè¼ÆÊ¦
HadoopÔ­ÀíÓëʵ¼ù
Oracle Êý¾Ý²Ö¿â
Êý¾Ý²Ö¿âºÍÊý¾ÝÍÚ¾ò
OracleÊý¾Ý¿â¿ª·¢Óë¹ÜÀí


GE Çø¿éÁ´¼¼ÊõÓëʵÏÖÅàѵ
º½Ìì¿Æ¹¤Ä³×Ó¹«Ë¾ Nodejs¸ß¼¶Ó¦Óÿª·¢
ÖÐÊ¢Òæ»ª ׿Խ¹ÜÀíÕß±ØÐë¾ß±¸µÄÎåÏîÄÜÁ¦
ijÐÅÏ¢¼¼Êõ¹«Ë¾ PythonÅàѵ
ij²©²ÊITϵͳ³§ÉÌ Ò×ÓÃÐÔ²âÊÔÓëÆÀ¹À
ÖйúÓÊ´¢ÒøÐÐ ²âÊÔ³ÉÊì¶ÈÄ£Ðͼ¯³É(TMMI)
ÖÐÎïÔº ²úÆ·¾­ÀíÓë²úÆ·¹ÜÀí