Äú¿ÉÒÔ¾èÖú£¬Ö§³ÖÎÒÃǵĹ«ÒæÊÂÒµ¡£

1Ôª 10Ôª 50Ôª





ÈÏÖ¤Â룺  ÑéÖ¤Âë,¿´²»Çå³þ?Çëµã»÷Ë¢ÐÂÑéÖ¤Âë ±ØÌî



  ÇóÖª ÎÄÕ ÎÄ¿â Lib ÊÓÆµ iPerson ¿Î³Ì ÈÏÖ¤ ×Éѯ ¹¤¾ß ½²×ù Modeler   Code  
»áÔ±   
 
   
 
 
     
   
 ¶©ÔÄ
  ¾èÖú
Hadoop¼¯ÈºÅäÖã¨×îÈ«Ãæ×ܽᣩ
 
×÷ÕߣºÕæÊµµÄ¹éËÞ À´Ô´£ºCSDN ·¢²¼ÓÚ£º2014-12-19
  3399  次浏览      27
 

1 ÏȾöÌõ¼þ

È·±£ÔÚÄ㼯ȺÖеÄÿ¸ö½ÚµãÉ϶¼°²×°ÁËËùÓбØÐèÈí¼þ£ºsun-JDK £¬ssh£¬Hadoop

JavaTM1.5.x£¬±ØÐë°²×°£¬½¨ÒéÑ¡ÔñSun¹«Ë¾·¢ÐеÄJava°æ±¾¡£

ssh ±ØÐë°²×°²¢ÇÒ±£Ö¤ sshdÒ»Ö±ÔËÐУ¬ÒÔ±ãÓÃHadoop ½Å±¾¹ÜÀíÔ¶¶ËHadoopÊØ»¤½ø³Ì¡£

2 ʵÑé»·¾³´î½¨

2.1 ×¼±¸¹¤×÷

²Ù×÷ϵͳ£ºUbuntu

²¿Êð£ºVmvare

ÔÚvmvare°²×°ºÃһ̨UbuntuÐéÄâ»úºó£¬¿ÉÒÔµ¼³ö»òÕ߿ˡ³öÁíÍâÁ½Ì¨ÐéÄâ»ú¡£

˵Ã÷£º

±£Ö¤ÐéÄâ»úµÄipºÍÖ÷»úµÄipÔÚͬһ¸öip¶Î£¬ÕâÑù¼¸¸öÐéÄâ»úºÍÖ÷»úÖ®¼ä¿ÉÒÔÏ໥ͨÐÅ¡£

ΪÁ˱£Ö¤ÐéÄâ»úµÄipºÍÖ÷»úµÄipÔÚͬһ¸öip¶Î£¬ÐéÄâ»úÁ¬½ÓÉèÖÃΪÇÅÁ¬¡£

×¼±¸»úÆ÷£ºÒ»Ì¨master£¬Èô¸Ę́slave£¬ÅäÖÃÿ̨»úÆ÷µÄ/etc/hosts±£Ö¤¸÷̨»úÆ÷Ö®¼äͨ¹ý»úÆ÷Ãû¿ÉÒÔ»¥·Ã£¬ÀýÈ磺

10.64.56.76 node1£¨master£© 
10.64.56.77 node2 £¨slave1£©
10.64.56.78 node3 £¨slave2£©

Ö÷»úÐÅÏ¢:

2.2 °²×°JDK

#°²×°JDK
$ sudo apt-get install sun-java6-jdk1.2.3 

Õâ¸ö°²×°£¬javaÖ´ÐÐÎļþ×Ô¶¯Ìí¼Óµ½/usr/bin/Ŀ¼¡£

ÑéÖ¤ shellÃüÁî £ºjava -version ¿´ÊÇ·ñÓëÄãµÄ°æ±¾ºÅÒ»Ö¡£

2.3ÏÂÔØ¡¢´´½¨Óû§

$ useradd hadoop
$ cd /home/hadoop

ÔÚËùÓеĻúÆ÷É϶¼½¨Á¢ÏàͬµÄĿ¼£¬Ò²¿ÉÒԾͽ¨Á¢ÏàͬµÄÓû§£¬×îºÃÊÇÒÔ¸ÃÓû§µÄhome·¾¶À´×öhadoopµÄ°²×°Â·¾¶¡£

ÀýÈçÔÚËùÓеĻúÆ÷Éϵݲװ·¾¶¶¼ÊÇ£º/home/hadoop/hadoop-0.20.203£¬Õâ¸ö²»ÐèÒªmkdir£¬ÔÚ/home/hadoop/Ͻâѹhadoop°üµÄʱºò£¬»á×Ô¶¯Éú³É£©

£¨µ±È»¿ÉÒÔ°²×°/usr/local/Ŀ¼Ï£¬ÀýÈç/usr/local/hadoop-0.20.203/
chown -R hadoop /usr/local/hadoop-0.20.203/
chgrp -R hadoop /usr/local/hadoop-0.20.203/£©
(×îºÃ²»ÒªÊ¹ÓÃroot°²×°,ÒòΪ²»ÍƼö¸÷¸ö»úÆ÷Ö®¼äʹÓÃroot·ÃÎÊ )

2.4 °²×°sshºÍÅäÖÃ

1£© °²×°£ºsudo apt-get install ssh

Õâ¸ö°²×°Íêºó£¬¿ÉÒÔÖ±½ÓʹÓÃsshÃüÁî ÁË¡£

Ö´ÐÐ$ netstat -nat ²é¿´22¶Ë¿ÚÊÇ·ñ¿ªÆôÁË¡£

²âÊÔ£ºssh localhost¡£

ÊäÈ뵱ǰÓû§µÄÃÜÂ룬»Ø³µ¾ÍokÁË¡£ËµÃ÷°²×°³É¹¦£¬Í¬Ê±sshµÇ¼ÐèÒªÃÜÂë¡£

£¨ÕâÖÖĬÈϰ²×°·½Ê½Íêºó£¬Ä¬ÈÏÅäÖÃÎļþÊÇÔÚ/etc/ssh/Ŀ¼Ï¡£sshdÅäÖÃÎļþÊÇ£º/etc/ssh/sshd_config£©£º

×¢Ò⣺ÔÚËùÓлú×Ó¶¼ÐèÒª°²×°ssh¡£

2£© ÅäÖãº

ÔÚHadoopÆô¶¯ÒÔºó£¬NamenodeÊÇͨ¹ýSSH£¨Secure Shell£©À´Æô¶¯ºÍÍ£Ö¹¸÷¸ödatanodeÉϵĸ÷ÖÖÊØ»¤½ø³ÌµÄ£¬Õâ¾ÍÐëÒªÔÚ½ÚµãÖ®¼äÖ´ÐÐÖ¸ÁîµÄʱºòÊDz»ÐëÒªÊäÈëÃÜÂëµÄÐÎʽ£¬¹ÊÎÒÃÇÐëÒªÅäÖÃSSHÔËÓÃÎÞÃÜÂ빫ԿÈÏÖ¤µÄÐÎʽ¡£

ÒÔ±¾ÎÄÖеÄÈý̨»úÆ÷ΪÀý£¬ÏÖÔÚnode1ÊÇÖ÷½Úµã£¬ËûÐëÒªÁ¬½Ónode2ºÍnode3¡£ÐëҪȷ¶¨Ã¿Ì¨»úÆ÷É϶¼°²×°ÁËssh£¬²¢ÇÒdatanode»úÆ÷ÉÏsshd·þÎñÒѾ­Æô¶¯¡£

( ˵Ã÷£ºhadoop@hadoop~]$ssh-keygen -t rsa Õâ¸öÃüÁΪhadoopÉϵÄÓû§hadoopÉú³ÉÆäÃÜÔ¿¶Ô£¬Ñ¯ÎÊÆä±£´æÂ·¾¶Ê±Ö±½Ó»Ø³µ²ÉÓÃĬÈÏ·¾¶£¬µ±ÌáʾҪΪÉú³ÉµÄÃÜÔ¿ÊäÈëpassphraseµÄʱºò£¬Ö±½Ó»Ø³µ£¬Ò²¾ÍÊǽ«ÆäÉ趨Ϊ¿ÕÃÜÂë¡£Éú³ÉµÄÃÜÔ¿¶Ôid_rsa£¬id_rsa.pub£¬Ä¬ÈÏ´æ´¢ÔÚ/home/hadoop/.sshĿ¼ÏÂÈ»ºó½«id_rsa.pubµÄÄÚÈݸ´ÖƵ½Ã¿¸ö»úÆ÷(Ò²°üÀ¨±¾»ú)µÄ/home/dbrg/.ssh/authorized_keysÎļþÖУ¬Èç¹û»úÆ÷ÉÏÒѾ­ÓÐauthorized_keysÕâ¸öÎļþÁË£¬¾ÍÔÚÎļþĩβ¼ÓÉÏid_rsa.pubÖеÄÄÚÈÝ£¬Èç¹ûûÓÐauthorized_keysÕâ¸öÎļþ£¬Ö±½Ó¸´ÖƹýÈ¥¾ÍÐÐ.)

3) Ê×ÏÈÉèÖÃnamenodeµÄsshΪÎÞÐèÃÜÂëµÄ¡¢×Ô¶¯µÇ¼¡£

Çл»µ½hadoopÓû§( ±£Ö¤Óû§hadoop¿ÉÒÔÎÞÐèÃÜÂëµÇ¼£¬ÒòΪÎÒÃǺóÃæ°²×°µÄhadoopÊôÖ÷ÊÇhadoopÓû§¡£)

$ su hadoop
cd /home/hadoop
$ ssh-keygen -t rsa

È»ºóÒ»Ö±°´»Ø³µ

Íê³Éºó£¬ÔÚhome¸úĿ¼Ï»á²úÉúÒþ²ØÎļþ¼Ð.ssh

$ cd .ssh

Ö®ºóls ²é¿´Îļþ

cp id_rsa.pub authorized_keys

²âÊÔ£º

$ssh localhost

»òÕߣº

$ ssh node1 

µÚÒ»´Îssh»áÓÐÌáʾÐÅÏ¢£º

The authenticity of host ¡®node1 (10.64.56.76)¡¯ can¡¯t be established.
RSA key fingerprint is 03:e0:30:cb:6e:13:a8:70:c9:7e:cf:ff:33:2a:67:30.
Are you sure you want to continue connecting (yes/no)?

ÊäÈë yes À´¼ÌÐø¡£Õâ»á°Ñ¸Ã·þÎñÆ÷Ìí¼Óµ½ÄãµÄÒÑÖªÖ÷»úµÄÁбíÖÐ

·¢ÏÖÁ´½Ó³É¹¦£¬²¢ÇÒÎÞÐèÃÜÂë¡£

4 ) ¸´ÖÆauthorized_keysµ½node2 ºÍnode3 ÉÏ

ΪÁ˱£Ö¤node1¿ÉÒÔÎÞÐèÃÜÂë×Ô¶¯µÇ¼µ½node2ºÍnode3£¬ÏÈÔÚnode2ºÍnode3ÉÏÖ´ÐÐ

$ su hadoop
cd /home/hadoop
$ ssh-keygen -t rsa

һ·°´»Ø³µ.

È»ºó»Øµ½node1£¬¸´ÖÆauthorized_keysµ½node2 ºÍnode3

[hadoop@hadoop .ssh]$ scp authorized_keys node2:/home/hadoop/.ssh/
[hadoop@hadoop .ssh]$ scp authorized_keys node3:/home/hadoop/.ssh/

ÕâÀï»áÌáʾÊäÈëÃÜÂë,ÊäÈëhadoopÕ˺ÅÃÜÂë¾Í¿ÉÒÔÁË¡£

¸Ä¶¯ÄãµÄ authorized_keys ÎļþµÄÐí¿ÉȨÏÞ

[hadoop@hadoop .ssh]$chmod 644 authorized_keys

²âÊÔ£ºssh node2»òÕßssh node3£¨µÚÒ»´ÎÐèÒªÊäÈëyes£©¡£

Èç¹û²»ÐëÒªÊäÈëÃÜÂëÔòÅäÖóɹ¦£¬Èç¹û»¹ÐëÒªÇë¼ì²éÉÏÃæµÄÅäÖÃÄܲ»ÄÜÕýÈ·¡£

2.5 °²×°Hadoop

#Çл»ÎªhadoopÓû§
su hadoop
wgethttp://apache.mirrors.tds.net//hadoop/common/hadoop-0.20.203.0/hadoop-0.20.203.0rc1.tar.gz

ÏÂÔØ°²×°°üºó£¬Ö±½Ó½âѹ°²×°¼´¿É£º

$ tar -zxvfhadoop-0.20.203.0rc1.tar.gz 

1 ) °²×°Hadoop¼¯ÈºÍ¨³£Òª½«°²×°Èí¼þ½âѹµ½¼¯ÈºÄÚµÄËùÓлúÆ÷ÉÏ¡£²¢ÇÒ°²×°Â·¾¶ÒªÒ»Ö£¬Èç¹ûÎÒÃÇÓÃHADOOP_HOMEÖ¸´ú°²×°µÄ¸ù·¾¶£¬Í¨³££¬¼¯ÈºÀïµÄËùÓлúÆ÷µÄHADOOP_HOME·¾¶Ïàͬ¡£

2 ) Èç¹û¼¯ÈºÄÚ»úÆ÷µÄ»·¾³ÍêȫһÑù£¬¿ÉÒÔÔÚһ̨»úÆ÷ÉÏÅäÖúã¬È»ºó°ÑÅäÖúõÄÈí¼þ¼´hadoop-0.20.203Õû¸öÎļþ¼Ð¿½±´µ½ÆäËû»úÆ÷µÄÏàͬλÖü´¿É¡£

3 ) ¿ÉÒÔ½«MasterÉϵÄHadoopͨ¹ýscp¿½±´µ½Ã¿Ò»¸öSlaveÏàͬµÄĿ¼Ï£¬Í¬Ê±¸ù¾Ýÿһ¸öSlaveµÄJava_HOME µÄ²»Í¬ÐÞ¸ÄÆähadoop-env.sh ¡£

4) ΪÁË·½±ã£¬Ê¹ÓÃhadoopÃüÁî»òÕßstart-all.shµÈÃüÁÐÞ¸ÄMasterÉÏ/etc/profile ÐÂÔöÒÔÏÂÄÚÈÝ£º

export HADOOP_HOME=/home/hadoop/hadoop-0.20.203
exportPATH=$PATH:$HADOOP_HOME/bin

ÐÞ¸ÄÍê±Ïºó£¬Ö´ÐÐsource /etc/profile À´Ê¹ÆäÉúЧ¡£

6£©ÅäÖÃconf/hadoop-env.shÎļþ

ÅäÖÃconf/hadoop-env.shÎļþ

#Ìí¼Ó
export JAVA_HOME=/usr/lib/jvm/java-6-sun/

ÕâÀïÐÞ¸ÄΪÄãµÄjdkµÄ°²×°Î»Öá£

²âÊÔhadoop°²×°£º

Bin/hadoop jar hadoop-0.20.2-examples.jarwordcount 
                          conf/ /tmp/out

3. ¼¯ÈºÅäÖã¨ËùÓнڵãÏàͬ£©

3.1ÅäÖÃÎļþ£ºconf/core-site.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl"href="configuration.xsl"?>
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://node1:49000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/hadoop_home/var</value>
</property>
</configuration>

1£©fs.default.nameÊÇNameNodeµÄURI¡£hdfs://Ö÷»úÃû:¶Ë¿Ú/

2£©hadoop.tmp.dir £ºHadoopµÄĬÈÏÁÙʱ·¾¶£¬Õâ¸ö×îºÃÅäÖã¬Èç¹ûÔÚÐÂÔö½Úµã»òÕ߯äËûÇé¿öÏÂĪÃûÆäÃîµÄDataNodeÆô¶¯²»ÁË£¬¾Íɾ³ý´ËÎļþÖеÄtmpĿ¼¼´¿É¡£²»¹ýÈç¹ûɾ³ýÁËNameNode»úÆ÷µÄ´ËĿ¼£¬ÄÇô¾ÍÐèÒªÖØÐÂÖ´ÐÐNameNode¸ñʽ»¯µÄÃüÁî¡£

3.2ÅäÖÃÎļþ£ºconf/mapred-site.xml

<?xmlversion="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>node1:49001</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/hadoop/hadoop_home/var</value>
</property>
</configuration>

1£©mapred.job.trackerÊÇJobTrackerµÄÖ÷»ú£¨»òÕßIP£©ºÍ¶Ë¿Ú¡£Ö÷»ú:¶Ë¿Ú¡£

3.3ÅäÖÃÎļþ£ºconf/hdfs-site.xml

<?xmlversion="1.0"?>
<?xml-stylesheettype="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/name1, /home/hadoop/name2</value> #hadoopµÄnameĿ¼·¾¶
<description> </description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/data1, /home/hadoop/data2</value>
<description> </description>
</property>
<property>
<name>dfs.replication</name>
<!-- ÎÒÃǵļ¯ÈºÓÖÁ½¸ö½áµã£¬ËùÒÔrepÁ½·Ý -->
<value>2</vaue>
</property>
</configuration>

1£© dfs.name.dirÊÇNameNode³Ö¾Ã´æ´¢Ãû×ֿռ估ÊÂÎñÈÕÖ¾µÄ±¾µØÎļþϵͳ·¾¶¡£ µ±Õâ¸öÖµÊÇÒ»¸ö¶ººÅ·Ö¸îµÄĿ¼Áбíʱ£¬nametableÊý¾Ý½«»á±»¸´ÖƵ½ËùÓÐĿ¼ÖÐ×öÈßÓ౸·Ý¡£

2£© dfs.data.dirÊÇDataNode´æ·Å¿éÊý¾ÝµÄ±¾µØÎļþϵͳ·¾¶£¬¶ººÅ·Ö¸îµÄÁÐ±í¡£ µ±Õâ¸öÖµÊǶººÅ·Ö¸îµÄĿ¼Áбíʱ£¬Êý¾Ý½«±»´æ´¢ÔÚËùÓÐĿ¼Ï£¬Í¨³£·Ö²¼ÔÚ²»Í¬É豸ÉÏ¡£

3£©dfs.replicationÊÇÊý¾ÝÐèÒª±¸·ÝµÄÊýÁ¿£¬Ä¬ÈÏÊÇ3£¬Èç¹û´ËÊý´óÓÚ¼¯ÈºµÄ»úÆ÷Êý»á³ö´í¡£

×¢Ò⣺´Ë´¦µÄname1¡¢name2¡¢data1¡¢data2Ŀ¼²»ÄÜÔ¤ÏÈ´´½¨£¬hadoop¸ñʽ»¯Ê±»á×Ô¶¯´´½¨£¬Èç¹ûÔ¤ÏÈ´´½¨·´¶ø»áÓÐÎÊÌâ¡£

3.4ÅäÖÃmastersºÍslavesÖ÷´Ó½áµã

ÅäÖÃconf/mastersºÍconf/slavesÀ´ÉèÖÃÖ÷´Ó½áµã£¬×¢Òâ×îºÃʹÓÃÖ÷»úÃû£¬²¢ÇÒ±£Ö¤»úÆ÷Ö®¼äͨ¹ýÖ÷»úÃû¿ÉÒÔ»¥Ïà·ÃÎÊ£¬Ã¿¸öÖ÷»úÃûÒ»ÐС£

vi masters£º

ÊäÈ룺

node1

vi slaves£º

ÊäÈ룺

node2
node3

ÅäÖýáÊø£¬°ÑÅäÖúõÄhadoopÎļþ¼Ð¿½±´µ½ÆäËû¼¯ÈºµÄ»úÆ÷ÖУ¬²¢ÇÒ±£Ö¤ÉÏÃæµÄÅäÖöÔÓÚÆäËû»úÆ÷¶øÑÔÕýÈ·£¬ÀýÈ磺Èç¹ûÆäËû»úÆ÷µÄJava°²×°Â·¾¶²»Ò»Ñù£¬ÒªÐÞ¸Äconf/hadoop-env.sh

$ scp -r /home/hadoop/hadoop-0.20.203 root@node2: /home/hadoop/

4 hadoopÆô¶¯

4.1 ¸ñʽ»¯Ò»¸öеķֲ¼Ê½Îļþϵͳ

Ïȸñʽ»¯Ò»¸öеķֲ¼Ê½Îļþϵͳ

$ cd hadoop-0.20.203
$ bin/hadoop namenode -format

³É¹¦Çé¿öÏÂϵͳÊä³ö£º

12/02/06 00:46:50 INFO namenode.NameNode:STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = ubuntu/127.0.1.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 0.20.203.0
STARTUP_MSG:   build =http://svn.apache.org/repos/asf/hadoop/common
/branches/branch-0.20-security-203-r 1099333; compiled by 'oom' on Wed May 4 07:57:50 PDT 2011
************************************************************/

12/02/0600:46:50 INFO namenode.FSNamesystem: fsOwner=root,root
12/02/06 00:46:50 INFO namenode.FSNamesystem:supergroup=supergroup
12/02/06 00:46:50 INFO namenode.FSNamesystem:isPermissionEnabled=true
12/02/06 00:46:50 INFO common.Storage: Imagefile of size 94 saved in 0 seconds.
12/02/06 00:46:50 INFO common.Storage: Storagedirectory /opt/hadoop/hadoopfs/name1 has been successfully formatted.
12/02/06 00:46:50 INFO common.Storage: Imagefile of size 94 saved in 0 seconds.
12/02/06 00:46:50 INFO common.Storage: Storagedirectory /opt/hadoop/hadoopfs/name2 has been successfully formatted.
12/02/06 00:46:50 INFO namenode.NameNode:SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode atv-jiwan-ubuntu-0/127.0.0.1
************************************************************/

²é¿´Êä³ö±£Ö¤·Ö²¼Ê½Îļþϵͳ¸ñʽ»¯³É¹¦

Ö´ÐÐÍêºó¿ÉÒÔµ½master»úÆ÷ÉÏ¿´µ½/home/hadoop//name1ºÍ/home/hadoop//name2Á½¸öĿ¼¡£ÔÚÖ÷½ÚµãmasterÉÏÃæÆô¶¯hadoop£¬Ö÷½Úµã»áÆô¶¯ËùÓдӽڵãµÄhadoop¡£

4.2 Æô¶¯ËùÓнڵã

Æô¶¯·½Ê½1£º

$ bin/start-all.sh £¨Í¬Ê±Æô¶¯HDFSºÍMap/Reduce£©

ϵͳÊä³ö£º

starting namenode, logging to /usr/local/hadoop/logs/hadoop-hadoop-namenode-ubuntu.out
node2: starting datanode, loggingto /usr/local/hadoop/logs/hadoop-hadoop-datanode-ubuntu.out
node3: starting datanode, loggingto /usr/local/hadoop/logs/hadoop-hadoop-datanode-ubuntu.out
node1: starting secondarynamenode,logging to /usr/local/hadoop/logs/hadoop-hadoop-secondarynamenode-ubuntu.out
starting jobtracker, logging to/usr/local/hadoop/logs/hadoop-hadoop-jobtracker-ubuntu.out
node2: starting tasktracker,logging to /usr/local/hadoop/logs/hadoop-hadoop-tasktracker-ubuntu.out
node3: starting tasktracker,logging to /usr/local/hadoop/logs/hadoop-hadoop-tasktracker-ubuntu.out
As you can see in slave's output above, it will automatically format it's storage directory
(specified by dfs.data.dir) if it is not formattedalready. It will also create the directory if it does not exist yet.

Ö´ÐÐÍêºó¿ÉÒÔµ½master£¨node1£©ºÍslave£¨node1£¬node2£©»úÆ÷ÉÏ¿´µ½/home/hadoop/hadoopfs/data1ºÍ/home/hadoop/data2Á½¸öĿ¼¡£

Æô¶¯·½Ê½2£º

Æô¶¯Hadoop¼¯ÈºÐèÒªÆô¶¯HDFS¼¯ÈººÍMap/Reduce¼¯Èº¡£

ÔÚ·ÖÅäµÄNameNodeÉÏ£¬ÔËÐÐÏÂÃæµÄÃüÁîÆô¶¯HDFS£º

                          $ bin/start-dfs.sh£¨µ¥¶ÀÆô¶¯HDFS¼¯Èº£©

bin/start-dfs.sh½Å±¾»á²ÎÕÕNameNodeÉÏ${HADOOP_CONF_DIR}/slavesÎļþµÄÄÚÈÝ£¬ÔÚËùÓÐÁгöµÄslaveÉÏÆô¶¯DataNodeÊØ»¤½ø³Ì¡£

ÔÚ·ÖÅäµÄJobTrackerÉÏ£¬ÔËÐÐÏÂÃæµÄÃüÁîÆô¶¯Map/Reduce£º

$bin/start-mapred.sh £¨µ¥¶ÀÆô¶¯Map/Reduce£©

bin/start-mapred.sh½Å±¾»á²ÎÕÕJobTrackerÉÏ${HADOOP_CONF_DIR}/slavesÎļþµÄÄÚÈÝ£¬ÔÚËùÓÐÁгöµÄslaveÉÏÆô¶¯TaskTrackerÊØ»¤½ø³Ì¡£

4.3 ¹Ø±ÕËùÓнڵã

´ÓÖ÷½Úµãmaster¹Ø±Õhadoop£¬Ö÷½Úµã»á¹Ø±ÕËùÓдӽڵãµÄhadoop¡£

$ bin/stop-all.sh

HadoopÊØ»¤½ø³ÌµÄÈÕ־дÈëµ½ ${HADOOP_LOG_DIR} Ŀ¼ (ĬÈÏÊÇ ${HADOOP_HOME}/logs).

${HADOOP_HOME}¾ÍÊǰ²×°Â·¾¶.

5 .²âÊÔ

1£©ä¯ÀÀNameNodeºÍJobTrackerµÄÍøÂç½Ó¿Ú£¬ËüÃǵĵØÖ·Ä¬ÈÏΪ£º

NameNode - http://node1:50070/
JobTracker - http://node2:50030/

 


3£© ʹÓÃnetstat ¨Cnat²é¿´¶Ë¿Ú49000ºÍ49001ÊÇ·ñÕýÔÚʹÓá£

4£© ʹÓÃjps²é¿´½ø³Ì

ÒªÏë¼ì²éÊØ»¤½ø³ÌÊÇ·ñÕýÔÚÔËÐУ¬¿ÉÒÔʹÓà jps ÃüÁÕâÊÇÓÃÓÚ JVM ½ø³ÌµÄps ʵÓóÌÐò£©¡£Õâ¸öÃüÁîÁгö 5 ¸öÊØ»¤½ø³Ì¼°Æä½ø³Ì±êʶ·û¡£

5£©½«ÊäÈëÎļþ¿½±´µ½·Ö²¼Ê½Îļþϵͳ£º

$ bin/hadoop fs -mkdir input
$ bin/hadoop fs -put conf/core-site.xml input

ÔËÐз¢ÐаæÌṩµÄʾÀý³ÌÐò£º

$ bin/hadoop jar hadoop-0.20.2-examples.jar grep input  output 'dfs[a-z.]+'

6.²¹³ä

Q: bin/hadoop jar hadoop-0.20.2-examples.jar grep input output 'dfs[a-z.]+' ʲôÒâ˼°¡£¿

A: bin/hadoop jar£¨Ê¹ÓÃhadoopÔËÐÐjar°ü£© hadoop-0.20.2_examples.jar£¨jar°üµÄÃû×Ö£© grep £¨ÒªÊ¹ÓõÄÀ࣬ºó±ßµÄÊDzÎÊý£©input output 'dfs[a-z.]+'

Õû¸ö¾ÍÊÇÔËÐÐhadoopʾÀý³ÌÐòÖеÄgrep£¬¶ÔÓ¦µÄhdfsÉϵÄÊäÈëĿ¼Ϊinput¡¢Êä³öĿ¼Ϊoutput¡£

Q: ʲôÊÇgrep?

A: A map/reduce program that counts the matches of a regex in the input.

²é¿´Êä³öÎļþ£º

½«Êä³öÎļþ´Ó·Ö²¼Ê½Îļþϵͳ¿½±´µ½±¾µØÎļþϵͳ²é¿´£º

$ bin/hadoop fs -get output output 
$ cat output/*

»òÕß

ÔÚ·Ö²¼Ê½ÎļþϵͳÉϲ鿴Êä³öÎļþ£º

$ bin/hadoop fs -cat output/*

ͳ¼Æ½á¹û£º

root@v-jiwan-ubuntu-0:~/hadoop/hadoop-0.20.2-bak/hadoop-0.20.2#bin/hadoop fs -cat output/part-00000
3 dfs.class
2 dfs.period
1 dfs.file
1 dfs.replication
1 dfs.servers
1 dfsadmin

7. HDFS³£ÓòÙ×÷

hadoopdfs -ls ÁгöHDFSϵÄÎļþ
hadoop dfs -ls in ÁгöHDFSÏÂij¸öÎĵµÖеÄÎļþ
hadoop dfs -put test1.txt test ÉÏ´«Îļþµ½Ö¸¶¨Ä¿Â¼²¢ÇÒÖØÐÂÃüÃû£¬Ö»ÓÐËùÓеÄDataNode¶¼½ÓÊÕÍêÊý¾Ý²ÅËã³É¹¦
hadoop dfs -get in getin ´ÓHDFS»ñÈ¡Îļþ²¢ÇÒÖØÐÂÃüÃûΪgetin£¬Í¬putÒ»Ñù¿É²Ù×÷ÎļþÒ²¿É²Ù×÷Ŀ¼
hadoop dfs -rmr out ɾ³ýÖ¸¶¨Îļþ´ÓHDFSÉÏ
hadoop dfs -cat in/* ²é¿´HDFSÉÏinĿ¼µÄÄÚÈÝ
hadoop dfsadmin -report ²é¿´HDFSµÄ»ù±¾Í³¼ÆÐÅÏ¢£¬½á¹ûÈçÏÂ
hadoop dfsadmin -safemode leave Í˳ö°²È«Ä£Ê½
hadoop dfsadmin -safemode enter ½øÈ밲ȫģʽ

8.Ìí¼Ó½Úµã

¿ÉÀ©Õ¹ÐÔÊÇHDFSµÄÒ»¸öÖØÒªÌØÐÔ£¬Ê×ÏÈÔÚмӵĽڵãÉϰ²×°hadoop£¬È»ºóÐÞ¸Ä$HADOOP_HOME/conf/masterÎļþ£¬¼ÓÈë NameNodeÖ÷»úÃû£¬È»ºóÔÚNameNode½ÚµãÉÏÐÞ¸Ä$HADOOP_HOME/conf/slavesÎļþ£¬¼ÓÈëмӽڵãÖ÷»úÃû£¬ÔÙ½¨Á¢µ½Ð¼ӽڵãÎÞÃÜÂëµÄSSHÁ¬½Ó

ÔËÐÐÆô¶¯ÃüÁ

start-all.sh

È»ºó¿ÉÒÔͨ¹ýhttp://(MasternodeµÄÖ÷»úÃû):50070²é¿´ÐÂÌí¼ÓµÄDataNode

9¸ºÔؾùºâ

start-balancer.sh£¬¿ÉÒÔʹDataNode½ÚµãÉÏÑ¡Ôñ²ßÂÔÖØÐÂÆ½ºâDataNodeÉϵÄÊý¾Ý¿éµÄ·Ö²¼

½áÊøÓï:Óöµ½ÎÊÌâʱ£¬ÏȲ鿴logs£¬ºÜÓаïÖú¡£

10 SHell×Ô¶¯°²×°½Å±¾

#!/bin/bash  
  
#validate user or group  
validate() {  
 if [ 'id -u' == 0 ];then  
   echo "must not be root!"  
   exit 0  
 else  
   echo "---------welcome to hadoop---------"  
 fi  
}  
  
#hadoop install  
hd-dir() {  
 if [ ! -d /home/hadoop/ ];then  
   mkdir /home/hadoop/  
 else  
   echo "download hadoop will begin"  
 fi  
}  
  
download-hd() {  
 wget -c http://archive.apache.org/dist/hadoop/core/stable/hadoop-1.0.4.tar.gz -O /home/hadoop/hadoop-1.0.4.tar.gz  
 tar -xzvf /home/hadoop/hadoop-1.0.4.tar.gz -C /home/hadoop  
 rm /home/hadoop/hadoop-1.0.4.tar.gz  
 Ln -s /home/hadoop/hadoop-1.0.4 /home/hadoop/hadoop1.0.4  
}  
  
#hadoop conf  
hd-conf() {  
 echo "export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386" >> /home/hadoop/hadoop1.0.4/conf/hadoop-env.sh  
 echo "#set path jdk" >> /home/hadoop/.profile  
 echo "export JAVA_HOME=/usr/lib/jvm/java-6-openjdk-i386" >> /home/hadoop/.profile  
 echo "#hadoop path" >> /home/hadoop/.profile  
 echo "export HADOOP_HOME=/home/hadoop/hadoop1.0.4" >> /home/hadoop/.profile  
 echo "PATH=$PATH:$HADOOP_HOME/bin:$JAVA_HOME/bin" >> /home/hadoop/.profile  
 echo "HADOOP_HOME_WARN_SUPPRESS=1" >> /home/hadoop/.profile  
#hadoop core-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
 echo "fs.default.name" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
 echo "hdfs://hadoop-master:9000" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
 echo "hadoop.tmp.dir" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
 echo "/home/hadoop/tmp" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/core-site.xml  
#hadoop hdfs-site.xml  
  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "dfs.name.dir" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "/home/hadoop/name" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "dfs.data.dir" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "/home/hadoop/data" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "dfs.replication" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "1" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/hdfs-site.xml  
# hadoop mapred-site.xml  
  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/mapred-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/mapred-site.xml  
 echo "mapred.job.tracker" >> /home/hadoop/hadoop1.0.4/conf/mapred-site.xml  
 echo "hadoop-master:9001" >> /home/hadoop/hadoop1.0.4/conf/mapred-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/mapred-site.xml  
 echo "" >> /home/hadoop/hadoop1.0.4/conf/mapred-site.xml  
#hadoop master  
 echo "hadoop-master" >> /home/hadoop/hadoop1.0.4/conf/masters  
  
#hadoop slaves  
 echo "hadoop-master" >> /home/hadoop/hadoop1.0.4/conf/slaves  
source /home/hadoop/.profile  
}  
  
hd-start() {  
hadoop namenode -format  
}  
  
  
yes-or-no() {  
  echo "Is your name $* ?"  
  while true  
  do  
     echo -n "Enter yes or no: "  
     read x  
     case "$x" in  
     y | yes ) return 0;;  
     n | no ) return 1;;  
     * ) echo "Answer yes or no";;  
   esac  
  done  
}  
  
echo "Original params are $*"  
  
if yes-or-no "$1"  
then  
  echo "HI $1,nice name!"  
  validate  
  hd-dir  
  download-hd  
  hd-conf  
else  
  echo "Never mind!"  
fi  
   
3399 ´Îä¯ÀÀ       27
Ïà¹ØÎÄÕÂ

»ùÓÚEAµÄÊý¾Ý¿â½¨Ä£
Êý¾ÝÁ÷½¨Ä££¨EAÖ¸ÄÏ£©
¡°Êý¾Ýºþ¡±£º¸ÅÄî¡¢ÌØÕ÷¡¢¼Ü¹¹Óë°¸Àý
ÔÚÏßÉ̳ÇÊý¾Ý¿âϵͳÉè¼Æ ˼·+Ч¹û
 
Ïà¹ØÎĵµ

GreenplumÊý¾Ý¿â»ù´¡Åàѵ
MySQL5.1ÐÔÄÜÓÅ»¯·½°¸
ijµçÉÌÊý¾ÝÖÐ̨¼Ü¹¹Êµ¼ù
MySQL¸ßÀ©Õ¹¼Ü¹¹Éè¼Æ
Ïà¹Ø¿Î³Ì

Êý¾ÝÖÎÀí¡¢Êý¾Ý¼Ü¹¹¼°Êý¾Ý±ê×¼
MongoDBʵս¿Î³Ì
²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿âÉè¼ÆÓëÓÅ»¯
PostgreSQLÊý¾Ý¿âʵսÅàѵ
×îл¼Æ»®
DeepSeek´óÄ£ÐÍÓ¦Óÿª·¢ 6-12[ÏÃÃÅ]
È˹¤ÖÇÄÜ.»úÆ÷ѧϰTensorFlow 6-22[Ö±²¥]
»ùÓÚ UML ºÍEA½øÐзÖÎöÉè¼Æ 6-30[±±¾©]
ǶÈëʽÈí¼þ¼Ü¹¹-¸ß¼¶Êµ¼ù 7-9[±±¾©]
Óû§ÌåÑé¡¢Ò×ÓÃÐÔ²âÊÔÓëÆÀ¹À 7-25[Î÷°²]
ͼÊý¾Ý¿âÓë֪ʶͼÆ× 8-23[±±¾©]

MySQLË÷Òý±³ºóµÄÊý¾Ý½á¹¹
MySQLÐÔÄܵ÷ÓÅÓë¼Ü¹¹Éè¼Æ
SQL ServerÊý¾Ý¿â±¸·ÝÓë»Ö¸´
ÈÃÊý¾Ý¿â·ÉÆðÀ´ 10´óDB2ÓÅ»¯
oracleµÄÁÙʱ±í¿Õ¼äдÂú´ÅÅÌ
Êý¾Ý¿âµÄ¿çƽ̨Éè¼Æ


²¢·¢¡¢´óÈÝÁ¿¡¢¸ßÐÔÄÜÊý¾Ý¿â
¸ß¼¶Êý¾Ý¿â¼Ü¹¹Éè¼ÆÊ¦
HadoopÔ­ÀíÓëʵ¼ù
Oracle Êý¾Ý²Ö¿â
Êý¾Ý²Ö¿âºÍÊý¾ÝÍÚ¾ò
OracleÊý¾Ý¿â¿ª·¢Óë¹ÜÀí


GE Çø¿éÁ´¼¼ÊõÓëʵÏÖÅàѵ
º½Ìì¿Æ¹¤Ä³×Ó¹«Ë¾ Nodejs¸ß¼¶Ó¦Óÿª·¢
ÖÐÊ¢Òæ»ª ׿Խ¹ÜÀíÕß±ØÐë¾ß±¸µÄÎåÏîÄÜÁ¦
ijÐÅÏ¢¼¼Êõ¹«Ë¾ PythonÅàѵ
ij²©²ÊITϵͳ³§ÉÌ Ò×ÓÃÐÔ²âÊÔÓëÆÀ¹À
ÖйúÓÊ´¢ÒøÐÐ ²âÊÔ³ÉÊì¶ÈÄ£Ðͼ¯³É(TMMI)
ÖÐÎïÔº ²úÆ·¾­ÀíÓë²úÆ·¹ÜÀí