如何在Linux下搭建hadoop集群环境 小残's Blog 怎样在linux系统上搭建Hadoop集群？
\u5982\u4f55\u5728linux\u4e0b\u642d\u5efahadoop\u96c6\u7fa4\u73af\u5883\u5c0f\u6b8b'sblog\u3000\u3000\u524d\u671f\u51c6\u5907
\u3000\u3000l \u4e24\u53f0linux\u865a\u62df\u673a\uff08\u672c\u6587\u4f7f\u7528redhat5\uff0cIP\u5206\u522b\u4e3a IP1\u3001IP2\uff09
\u3000\u3000l JDK\u73af\u5883\uff08\u672c\u6587\u4f7f\u7528jdk1.6\uff0c\u7f51\u4e0a\u5f88\u591a\u914d\u7f6e\u65b9\u6cd5\uff0c\u672c\u6587\u7701\u7565\uff09
\u3000\u3000l Hadoop\u5b89\u88c5\u5305\uff08\u672c\u6587\u4f7f\u7528Hadoop1.0.4\uff09
\u3000\u3000\u642d\u5efa\u76ee\u6807
\u3000\u3000210\u4f5c\u4e3a\u4e3b\u673a\u548c\u8282\u70b9\u673a\uff0c211\u4f5c\u4e3a\u8282\u70b9\u673a\u3002
\u3000\u3000\u642d\u5efa\u6b65\u9aa4
\u3000\u30001\u4fee\u6539hosts\u6587\u4ef6
\u3000\u3000\u5728/etc/hosts\u4e2d\u589e\u52a0\uff1a
\u3000\u3000IP1 hadoop1
\u3000\u3000IP2 hadoop2
\u3000\u3000
\u3000\u30002 \u5b9e\u73b0ssh\u65e0\u5bc6\u7801\u767b\u9646
\u3000\u30002.1 \u4e3b\u673a\uff08master\uff09\u65e0\u5bc6\u7801\u672c\u673a\u767b\u9646
\u3000\u3000ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
\u3000\u3000\u76f4\u63a5\u56de\u8f66\uff0c\u5b8c\u6210\u540e\u4f1a\u5728 ~/.ssh/ \u751f\u6210\u4e24\u4e2a\u6587\u4ef6\uff1a id_dsa \u548c id_dsa.pub \u3002
\u3000\u3000\u8fd9\u4e24\u4e2a\u662f\u6210\u5bf9\u51fa\u73b0\uff0c\u7c7b\u4f3c\u94a5\u5319\u548c\u9501\u3002
\u3000\u3000\u518d\u628a id_dsa.pub \u8ffd\u52a0\u5230\u6388\u6743 key \u91cc\u9762 ( \u5f53\u524d\u5e76\u6ca1\u6709 authorized_key s\u6587\u4ef6 ) \uff1a
\u3000\u3000cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
\u3000\u3000
\u3000\u3000ssh localhost hostname
\u3000\u3000\u8fd8\u662f\u8981\u8f93\u5165\u5bc6\u7801\uff0c\u4e00\u822c\u8fd9\u79cd\u60c5\u51b5\u90fd\u662f\u56e0\u4e3a\u76ee\u5f55\u6216\u6587\u4ef6\u7684\u6743\u9650\u95ee\u9898\uff0c\u770b\u770b\u7cfb\u7edf\u65e5\u5fd7\uff0c\u786e\u5b9e\u662f\u6743\u9650\u95ee\u9898
\u3000\u3000.ssh\u4e0b\u7684authorized_keys\u6743\u9650\u4e3a600\uff0c\u5176\u7236\u76ee\u5f55\u548c\u7956\u7236\u76ee\u5f55\u5e94\u4e3a755
\u3000\u30002.2 \u65e0\u5bc6\u7801\u767b\u9646\u8282\u70b9\u673a\uff08slave\uff09
\u3000\u3000slave\u4e0a\u6267\u884c\uff1a
\u3000\u3000ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
\u3000\u3000
\u3000\u3000\u751f\u6210.ssh\u76ee\u5f55\u3002
\u3000\u3000\u5c06master\u4e0a\u7684authorized_keys\u590d\u5236\u5230slave\u4e0a\uff1a
\u3000\u3000scp authorized_keys hadoop2:~/.ssh/
\u3000\u3000
\u3000\u3000\u5b9e\u9a8c\uff1a\u5728master\u4e0a\u6267\u884c
\u3000\u3000ssh hadoop2
\u3000\u3000
\u3000\u3000\u5b9e\u73b0\u65e0\u5bc6\u7801\u767b\u9646\u3002
\u3000\u30003 \u914d\u7f6eHadoop
\u3000\u30003.1\u62f7\u8d1dhadoop
\u3000\u3000\u5c06hadoop-1.0.4.tar.gz ,\u62f7\u8d1d\u5230usr/local \u6587\u4ef6\u5939\u4e0b\uff0c\u7136\u540e\u89e3\u538b\u3002
\u3000\u3000\u89e3\u538b\u547d\u4ee4\uff1a
\u3000\u3000tar \u2013zxvf hadoop-1.0.4.tar.gz
\u3000\u3000
\u3000\u30003.2\u67e5\u770b cat /etc/hosts
\u3000\u3000IP1 hadoop1
\u3000\u3000IP2 hadoop2
\u3000\u3000
\u3000\u30003.3 \u914d\u7f6e conf/masters \u548c conf/slaves
\u3000\u3000conf/masters\uff1a
\u3000\u30001
\u3000\u3000
\u3000\u3000IP1
\u3000\u3000
\u3000\u3000conf/slaves\uff1a
\u3000\u30001
\u3000\u30002
\u3000\u3000
\u3000\u3000IP2
\u3000\u3000IP2
\u3000\u3000
\u3000\u30003.4 \u914d\u7f6e conf/hadoop-env.sh
\u3000\u3000\u52a0\u5165
\u3000\u30001
\u3000\u3000
\u3000\u3000export JAVA_HOME=/home/elvis/soft/jdk1.7.0_17
\u3000\u3000
\u3000\u30003.5 \u914d\u7f6e conf/core-site.xml
\u3000\u30001
\u3000\u30002
\u3000\u30003
\u3000\u30004
\u3000\u3000
\u3000\u3000
\u3000\u3000fs.default.name
\u3000\u3000hdfs://IP1:9000
\u3000\u3000
\u3000\u3000
\u3000\u30003.6 \u914d\u7f6e conf/hdfs-site.xml
\u3000\u3000\u52a0\u5165
\u3000\u3000
\u3000\u3000dfs.http.address
\u3000\u3000IP1:50070
\u3000\u3000
\u3000\u3000
\u3000\u3000dfs.name.dir
\u3000\u3000/usr/local/hadoop/namenode
\u3000\u3000
\u3000\u3000
\u3000\u3000dfs.data.dir
\u3000\u3000/usr/local/hadoop/data
\u3000\u3000
\u3000\u3000
\u3000\u3000dfs.replication
\u3000\u30002
\u3000\u3000
\u3000\u3000
\u3000\u30003.7 \u914d\u7f6econf/mapred-site.xml
\u3000\u3000\u52a0\u5165
\u3000\u3000
\u3000\u3000mapred.job.tracker
\u3000\u3000192.168.1.50:8012
\u3000\u3000
\u3000\u3000
\u3000\u30003.8 \u5efa\u7acb\u76f8\u5173\u7684\u76ee\u5f55
\u3000\u30001
\u3000\u3000
\u3000\u3000/usr/local/hadoop/ //hadoop\u6570\u636e\u548cnamenode\u76ee\u5f55
\u3000\u3000
\u3000\u3000\u3010\u6ce8\u610f\u3011\u53ea\u521b\u5efa\u5230hadoop\u76ee\u5f55\u5373\u53ef\uff0c\u4e0d\u8981\u624b\u52a8\u521b\u5efadata\u548cnamenode\u76ee\u5f55\u3002
\u3000\u3000\u5176\u4ed6\u8282\u70b9\u673a\u4e5f\u540c\u6837\u5efa\u7acb\u8be5\u76ee\u5f55\u3002
\u3000\u30003.9 \u62f7\u8d1dhadoop\u6587\u4ef6\u5230\u5176\u4ed6\u8282\u70b9\u673a
\u3000\u3000\u5c06hadoop\u6587\u4ef6\u8fdc\u7a0bcopy\u5230\u5176\u4ed6\u8282\u70b9\uff08\u8fd9\u6837\u524d\u9762\u7684\u914d\u7f6e\u5c31\u90fd\u6620\u5c04\u5230\u4e86\u5176\u4ed6\u8282\u70b9\u4e0a\uff09\uff0c
\u3000\u3000\u547d\u4ee4\uff1a
\u3000\u30001
\u3000\u3000
\u3000\u3000scp -r hadoop-1.0.4 IP2:/usr/local/
\u3000\u3000
\u3000\u30003.10 \u683c\u5f0f\u5316Active master
\u3000\u3000\u547d\u4ee4\uff1a
\u3000\u3000bin/hadoop namenode -format
\u3000\u3000
\u3000\u30003.11 \u542f\u52a8\u96c6\u7fa4 ./start-all.sh
\u3000\u3000\u73b0\u5728\u96c6\u7fa4\u542f\u52a8\u8d77\u6765\u4e86\uff0c\u770b\u4e00\u4e0b\uff0c\u547d\u4ee4\uff1a
\u3000\u30001
\u3000\u3000
\u3000\u3000bin/hadoop dfsadmin -report
\u3000\u3000
\u3000\u30002\u4e2adatanode\uff0c\u6253\u5f00web\u770b\u4e00\u4e0b
\u3000\u3000\u6d4f\u89c8\u5668\u8f93\u5165:IP1:50070
\u3000\u3000\u6253\u5b8c\u6536\u5de5\uff0c\u96c6\u7fa4\u5b89\u88c5\u5b8c\u6210\uff01
\uff081\uff09\u4e0b\u8f7djdk\uff0c\u5728\u5b98\u7f51\u4e0b\u8f7d\uff0c\u4e0b\u8f7drpm\u7684\u5305
\uff082\uff09hadoop\u5305\u7684\u4e0b\u8f7d\uff0c\u5b98\u7f51\u4e0a\u4e0b\u8f7d
\u3000\u3000\u3000download hadoop->release->mirror site(\u955c\u50cf\u7ad9)->\u968f\u4fbf\u9009\u62e9\u79bb\u81ea\u5df1\u8fd1\u7684\uff08HTTP\u4e0b\u7684\u7b2c\u4e00\u4e2a\uff09->\u9009\u62e92.7.2->\u4e0b\u8f7d.tar.gz
\uff083\uff09\u5c06\u4e24\u4e2a\u5305\u8fdc\u7a0b\u4f20\u8f93\u5230linux\u865a\u62df\u673a\u4e2d
\uff084\uff09\u5c06\u4e3b\u673a\u540d\u548cip\u5730\u5740\u8fdb\u884c\u9002\u914d\uff0c\u8ba9\u6211\u4eec\u7684ip\u5730\u5740\u548c\u4e3b\u673a\u540d\uff08\u5982bigdata\uff09\u76f8\u5339\u914d\uff1a\u5199\u5230/etc/hosts\u91cc\u9762
\u3000\u3000\u3000vi /etc/hosts
\u3000\u3000\u3000\u6309\u201ci\u201d\u8fdb\u5165\u63d2\u5165\u72b6\u6001 \u3000\u3000\u3000\u5c06\u539f\u6709\u7684\u5730\u5740\u6ce8\u91ca\u6389
\u3000\u3000\u3000\u5728\u65b0\u7684\u4e00\u884c\u8f93\u5165\uff1aip\u5730\u5740 \u4e3b\u673a\u540d\uff08\u5982172.17.171.42 bigdata\uff09\uff08\u6ce8\uff1a\u53ef\u4ee5\u53cc\u51fbxshell\u7684\u7a97\u53e3\u518d\u6253\u5f00\u4e00\u4e2a\u8fde\u63a5\u7a97\u53e3\uff0c\u53ef\u4ee5\u5728\u65b0\u7684\u7a97\u53e3\u67e5\u8be2ip\u5730\u5740\u5e76\u8fdb\u884c\u590d\u5236\uff09
\u3000\u3000\u3000\u6309\u201cEsc\u201d\u9000\u51fa\u63d2\u5165\u72b6\u6001
\u3000\u3000\u3000\u8f93\u5165\uff1awq\u4fdd\u5b58\u9000\u51fa
\u3000\u3000\u3000\u4fee\u6539\u5b8c\u4e4b\u540e\u53ef\u4ee5\u8f93\u5165hostname\u56de\u8f66\uff0c\u67e5\u770b\u662f\u5426\u6210\u529f
\u3000\u3000\u3000reboot\uff1a\u91cd\u542f\uff0c\u4f7f\u5f97\u521a\u521a\u7684\u4fee\u6539\u751f\u6548
\uff085\uff09\u5c06\u5305\u653e\u5230opt\u4e0b\uff1acp hadoop-2.7.2.tar.gz /opt/
\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000cp jdk-8u111-linux-x64.rpm /opt/
\u3000\u3000\u3000\u8fdb\u5165opt\uff1acd /opt/
\u3000\u3000\u3000\u67e5\u770bopt\u4e0b\u7684\u6587\u4ef6\uff1all
\uff086\uff09\u5b89\u88c5jdk\uff0c\u914d\u7f6ejdk\u7684\u73af\u5883\u53d8\u91cf
\u3000\u3000\u3000\u5b89\u88c5\u547d\u4ee4\uff1arpm -ivh jdk-Bu101-linux-x64.rpm
\u3000\u3000\u3000\u914d\u7f6e\u73af\u5883\u53d8\u91cf\uff1a\u8fdb\u5165profile\u8fdb\u884c\u7f16\u8f91\uff1avi /etc/profile
\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u5e76\u6309\u7167\u4e0a\u9762\u7684\u65b9\u5f0f\u5728\u6700\u540e\u5199\u5165\u5e76\u4fdd\u5b58\uff1aJAVA_HOME=/usr/java/default/\uff08/usr/java/default/\u662fjdk\u7684\u5b89\u88c5\u76ee\u5f55\uff09
\u3000\u3000\u3000\u6253\u5370JAVA_HOME\u68c0\u9a8c\u662f\u5426\u914d\u7f6e\u597d\uff1aecho $JAVA_HOME\u7ed3\u679c\u53d1\u73b0\u6253\u5370\u51fa\u6765\u7684\u6ca1\u6709\u5185\u5bb9\u56e0\u4e3a\u6211\u4eec\u5bf9/etc/profile\u7684\u4fee\u6539\u9700\u8981\u901a\u8fc7\u4ee5\u4e0b\u547d\u4ee4\u5bf9\u5b83\u751f\u6548source /etc/profile\u3002\u518d\u6b21\u8f93\u5165echo $JAVA_HOME\uff0c\u6253\u5370\u7ed3\u679c\u4e3a/usr/java/default/
\uff087\uff09\u9a8c\u8bc1jdk\u5b89\u88c5\u597d\uff1ajava -version
\uff088\uff09\u914d\u7f6eSSH\uff08\u514d\u5bc6\u7801\u767b\u5f55\uff09
\u3000\u3000\u3000\u56de\u5230\u6839\u76ee\u5f55\uff1acd \u3000\u3000\u3000\u5b89\u88c5SSH\u79d8\u94a5\uff1assh-keygen -t rsa\u4f1a\u81ea\u52a8\u5728/root/.shh/\u76ee\u5f55\u4e0b\u751f\u6210
\u3000\u3000\u3000\u67e5\u770b\u76ee\u5f55\uff1all .ssh/\u6709\u4e24\u4e2a\u65b0\u751f\u6210\u7684\u6587\u4ef6id_rsa(\u79c1\u94a5)\uff0cid_rsa.pub(\u516c\u94a5)
\u3000\u3000\u3000\u8fdb\u5165.ssh/:cd .ssh/
\u3000\u3000\u3000\u5c06\u516c\u94a5\u5199\u5165authorized_key\u4e2d\uff1acat id_rsa.pub >> authorized_keys
\u3000\u3000\u3000\u4fee\u6539authorized_keys\u6587\u4ef6\u7684\u6743\u9650\uff1achmod 644 authorized_keys
\u3000\u3000\u3000\u4fee\u6539\u5b8c\u540e\u9000\u51fa.ssh\u7684\u76ee\u5f55cd\u8fdb\u5165\u521d\u59cb\u76ee\u5f55\u8f93\u5165\uff1assh bigdata\uff08bigdata\u4e3a\u4f60\u8981\u8fdc\u7a0b\u767b\u5f55\u7684\u4e3b\u673a\u540d\u6216\u8005ip\u5730\u5740\uff09\u7b2c\u4e00\u6b21\u767b\u5f55\u9700\u8981\u786e\u8ba4\u662f\u5426\u9700\u8981\u7ee7\u7eed\u767b\u5f55\u8f93\u5165yes\u7ee7\u7eed\u767b\u5f55
\u3000\u3000\u3000\u9000\u51faexit
\uff089\uff09\u5b89\u88c5\u53ca\u914d\u7f6ehadoop
\u3000\u3000\u3000\u89e3\u538b:tar zxf hadoop-2.7.2.tar.gz
\u3000\u3000\u3000\u67e5\u770b/opt\u76ee\u5f55\u4e0b\u662f\u5426\u5df2\u7ecf\u5b58\u5728\u89e3\u538b\u7684\u6587\u4ef6\uff1all\uff08\u7ed3\u679c\u4e3a\u51fa\u73b0hadoop-2.7.2\uff09
\u3000\u3000\u3000\u7ee7\u7eed\u67e5\u770bhadoop-2.7.2\u91cc\u7684\u5185\u5bb9\uff1acd hadoop-2.7.2
\u3000\u3000\u3000\u914d\u7f6eHADOOP_HOME:\u4fee\u6539/etc/profile
\u3000\u3000\u3000\u8fdb\u5165hadoop\u7684\u914d\u7f6e\u6587\u4ef6\u76ee\u5f55cd /opt/hadoop-2.7.2/etc/hadoop/\uff0c\u4f1a\u7528\u7684\u7684\u914d\u7f6e\u6587\u4ef6\u5982\u4e0b\uff1a
\u3000\u3000\u3000core-site.xml
\u3000\u3000\u3000\u914d\u7f6ehadoop\u7684\u6587\u4ef6\u7cfb\u7edf\u5373HDFS\u7684\u7aef\u53e3\u662f\u4ec0\u4e48\u3002
\u3000\u3000\u3000\u914d\u7f6e\u98791\u4e3adefault.name\uff0c\u503c\u4e3ahdfs://bigdata:9000\uff08\u4e3b\u673a\u540d\uff1abigdata\u4e5f\u53ef\u4e5f\u5199\u6210ip\u5730\u5740\uff0c\u7aef\u53e39000\u4e60\u60ef\u7528\uff09
\u3000\u3000\u3000\u914d\u7f6e\u98792\u4e3ahadoop\u4e34\u65f6\u6587\u4ef6\uff0c\u5176\u5b9e\u5c31\u662f\u914d\u7f6e\u4e00\u4e2a\u76ee\u5f55\uff0c\u914d\u7f6e\u5b8c\u540e\u8981\u53bb\u521b\u5efa\u8fd9\u4e2a\u76ee\u5f55\uff0c\u5426\u5219\u4f1a\u5b58\u5728\u95ee\u9898\u3002
\u3000\u3000\u3000\u914d\u7f6e\u98793\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\u7684\u5783\u573e\u7bb1\uff0c\u503c\u4e3a4320\u8868\u793a3\u5206\u949f\u56de\u53bb\u6e05\u7406\u4e00\u6b21
fs.default.name
hdfs://bigdata:9000
hadoop.tmp.dir
/opt/hadoop-2.7.2/current/tmp
fs.trash.interval
4320
\u3000\u3000\u3000hdfs-site.xml
\u3000\u3000\u3000\u914d\u7f6e\u98791\uff0cnamenode\u7684\u7ec6\u8282\u5b9e\u9645\u4e0a\u5c31\u662f\u4e00\u4e2a\u76ee\u5f55
\u3000\u3000\u3000\u914d\u7f6e\u98792\uff0cdatanode\u7684\u7ec6\u8282\uff0c\u771f\u5b9e\u73af\u5883\u4e2ddatanode\u7684\u5185\u5bb9\u4e0d\u9700\u8981\u518dnamenode\u7684\u7cfb\u7edf\u4e0b\u914d\u7f6e\uff0c\u5728\u6b64\u914d\u7f6e\u7684\u539f\u56e0\u662f\u6211\u4eec\u7684\u7cfb\u7edf\u662f\u4f2a\u5206\u5e03\u5f0f\u7cfb\u7edf\uff0cnamenode\u548cdatanode\u5728\u4e00\u53f0\u673a\u5668\u4e0a
\u3000\u3000\u3000\u914d\u7f6e\u98793\uff0c\u526f\u672c\u7684\u6570\u91cf\uff0c\u5728hdfs\u4e2d\u6bcf\u4e2a\u5757\u6709\u51e0\u4e2a\u526f\u672c
\u3000\u3000\u3000\u914d\u7f6e\u98794\uff0cHDFS\u662f\u5426\u542f\u7528web
\u3000\u3000\u3000\u914d\u7f6e\u98795\uff0cHDFS\u7684\u7528\u6237\u7ec4
\u3000\u3000\u3000\u914d\u7f6e\u98796\uff0cHDFS\u7684\u6743\u9650\uff0c\u73b0\u5728\u914d\u7f6e\u4e3a\u4e0d\u5f00\u542f\u6743\u9650
dfs.namenode.name.dir
/opt/hadoop-2.7.2/current/dfs/name
dfs.datanode.data.dir
/opt/hadoop-2.7.2/current/data
dfs.replication
1
dfs.webhdfs.enabled
true
dfs.permissions.superusergroup
staff
dfs.permissions.enabled
false
\u3000\u3000\u3000\u521b\u5efa\u914d\u7f6e\u6587\u4ef6\u4e2d\u4e0d\u5b58\u5728\u7684\u76ee\u5f55\uff1amkdir -p /opt/hadoop-2.7.2/current/data
\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000mkdir -p /opt/hadoop-2.7.2/current/dfs/name
\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000\u3000mkdir -p /opt/hadoop-2.7.2/current/tmp
\u3000\u3000\u3000yarn-site.xml
\u3000\u3000\u3000\u914d\u7f6e\u98791\uff0cresourcemanager\u7684hostname\uff0c\u503c\u4e3a\u4f60\u8fd0\u884c\u7684\u90a3\u53f0\u673a\u5668\u7684\u4e3b\u673a\u540d\u6216IP\u5730\u5740
\u3000\u3000\u3000\u914d\u7f6e\u98792\uff0cnodemanager\u76f8\u5173\u7684\u4e1c\u897f
\u3000\u3000\u3000\u914d\u7f6e\u98793\uff0cnodemanager\u76f8\u5173\u7684\u4e1c\u897f
\u3000\u3000\u3000\u914d\u7f6e\u98794\uff0cresourcemanager\u7684\u7aef\u53e3\uff0c\u4e3b\u673a\u540d+\u7aef\u53e3\u53f7\uff08IP+\u7aef\u53e3\uff09
\u3000\u3000\u3000\u914d\u7f6e\u98795\uff0cresourcemanager\u8c03\u5ea6\u5668\u7684\u7aef\u53e3
\u3000\u3000\u3000\u914d\u7f6e\u98796\uff0cresourcemanager.resource-tracker,\u7aef\u53e3
\u3000\u3000\u3000\u914d\u7f6e\u98797\uff0c\u7aef\u53e3
\u3000\u3000\u3000\u914d\u7f6e\u98798\uff0c\u7aef\u53e3
\u3000\u3000\u3000\u914d\u7f6e\u98799\uff0c\u65e5\u5fd7\u662f\u5426\u542f\u52a8
\u3000\u3000\u3000\u914d\u7f6e\u987910\uff0c\u65e5\u5fd7\u4fdd\u7559\u7684\u65f6\u95f4\u957f\u77ed\uff08\u4ee5\u79d2\u4e3a\u5355\u4f4d\uff09
\u3000\u3000\u3000\u914d\u7f6e\u987911\uff0c\u65e5\u5fd7\u68c0\u67e5\u7684\u65f6\u95f4
\u3000\u3000\u3000\u914d\u7f6e\u987912\uff0c\u76ee\u5f55
\u3000\u3000\u3000\u914d\u7f6e\u987913\uff0c\u76ee\u5f55\u7684\u524d\u7f00
yarn.resourcemanager.hostname
bigdata
yarn.nodemanager.aux-services
mapreduce_shuffle
yarn.nodemanager.aux-services.mapreduce.shuffle.class
org.apache.hadoop.mapred.ShuffleHandler
yarn.resourcemanager.address
bigdata:18040
yarn.resourcemanager.scheduler.address
bigdata:18030
yarn.resourcemanager.resource-tracker.address
bigdata:18025
yarn.resourcemanager.admin.address
bigdata:18141
yarn.resourcemanager.webapp.address
bigdata:18088
yarn.log-aggregation-enable
true
yarn.log-aggregation.retain-seconds
86400
yarn.log-aggregation.retain-check-interval-seconds
86400
yarn.nodemanager.remote-app-log-dir
/tmp/logs
yarn.nodemanager.remote-app-log-dir-suffix
logs
\u3000\u3000\u3000mapred-site.xml
\u3000\u3000\u3000\u6ca1\u6709mapred-site.xml\uff0c\u8f93\u5165vi mapred-\u6309\u201cTAB\u201d\u53d1\u73b0\u6709mapred-site.xml.template\uff0c\u5bf9\u8be5\u6587\u4ef6\u8fdb\u884c\u590d\u5236
\u3000\u3000\u3000cp mapred-site.xml.template mapred-site.xml\u3000\u3000 \u3000\u3000\u3000\u914d\u7f6e\u98791\uff0cmapreduce\u7684\u6846\u67b6
\u3000\u3000\u3000\u914d\u7f6e\u98792\uff0cmapreduce\u7684\u901a\u4fe1\u7aef\u53e3
\u3000\u3000\u3000\u914d\u7f6e\u98793\uff0cmapreduce\u7684\u4f5c\u4e1a\u5386\u53f2\u8bb0\u5f55\u7aef\u53e3
\u3000\u3000\u3000\u914d\u7f6e\u98794\uff0cmapreduce\u7684\u4f5c\u4e1a\u5386\u53f2\u8bb0\u5f55\u7aef\u53e3
\u3000\u3000\u3000\u914d\u7f6e\u98795\uff0cmapreduce\u7684\u4f5c\u4e1a\u5386\u53f2\u8bb0\u5f55\u5df2\u5b8c\u6210\u7684\u65e5\u5fd7\u76ee\u5f55\uff0c\u5728hdfs\u4e0a
\u3000\u3000\u3000\u914d\u7f6e\u98796\uff0cmapreduce\u4e2d\u95f4\u5b8c\u6210\u60c5\u51b5\u65e5\u5fd7\u76ee\u5f55
\u3000\u3000\u3000\u914d\u7f6e\u98797\uff0cmapreduce\u7684ubertask\u662f\u5426\u5f00\u542f
mapreduce.framework.name
yarn
mapreduce.jobtracker.http.address
bigdata:50030
mapreduce.jobhisotry.address
bigdata:10020
mapreduce.jobhistory.webapp.address
bigdata:19888
mapreduce.jobhistory.done-dir
/jobhistory/done
mapreduce.intermediate-done-dir
/jobhisotry/done_intermediate
mapreduce.job.ubertask.enable
true
\u3000\u3000\u3000slaves
bigdata
\u3000\u3000\u3000hadoop-env.sh
JAVA_HOME\uff1d/usr/java/default/
\u3000\u3000\u3000\u683c\u5f0f\u5316\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\uff08hdfs\uff09\uff1ahdfs namenode -format
\u3000\u3000\u3000\u6210\u529f\u7684\u6807\u5fd7\uff1a INFO common.Storage: Storage directory /opt/hadoop-2.7.2/current/dfs/namehas been successfully formatted.
\u3000\u3000\u3000\u542f\u52a8Hadoop\u96c6\u7fa4\uff1a/opt/hadoop-2.7.2/sbin/start-all.sh
\u3000\u3000\u3000\u9a8c\u8bc1Hadoop\u96c6\u7fa4\u662f\u5426\u6b63\u5e38\u542f\u52a8\uff1a
\u3000\u3000\u3000jps\uff0c\u7cfb\u7edf\u4e2d\u8fd0\u884c\u7684java\u8fdb\u7a0b;
\u3000\u3000\u3000\u901a\u8fc7\u7aef\u53e3\u67e5\u770b\uff08\u5173\u95ed\u9632\u706b\u5899\u6216\u8005service iptables stop\u5728\u9632\u706b\u5899\u7684\u89c4\u5219\u4e2d\u5f00\u653e\u8fd9\u4e9b\u7aef\u53e3\uff09\uff1a
\u3000\u3000\u3000http://bigdata:50070(http://http://192.168.42.209/:50070)\uff0c\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edfhdfs\u7684\u60c5\u51b5
\u3000\u3000\u3000yarn http://bigdata:18088(http://http://192.168.42.209/:50070)
l 两台linux虚拟机（本文使用redhat5，IP分别为 IP1、IP2）
l JDK环境（本文使用jdk1.6，网上很多配置方法，本文省略）
l Hadoop安装包（本文使用Hadoop1.0.4）
搭建目标
210作为主机和节点机，211作为节点机。
搭建步骤
1修改hosts文件
在/etc/hosts中增加：
IP1 hadoop1
IP2 hadoop2
2 实现ssh无密码登陆
2.1 主机（master）无密码本机登陆
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
直接回车，完成后会在 ~/.ssh/ 生成两个文件： id_dsa 和 id_dsa.pub 。
这两个是成对出现，类似钥匙和锁。
再把 id_dsa.pub 追加到授权 key 里面 ( 当前并没有 authorized_key s文件 ) ：
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh localhost hostname
还是要输入密码，一般这种情况都是因为目录或文件的权限问题，看看系统日志，确实是权限问题
.ssh下的authorized_keys权限为600，其父目录和祖父目录应为755
2.2 无密码登陆节点机（slave）
slave上执行：
ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa
生成.ssh目录。
将master上的authorized_keys复制到slave上：
scp authorized_keys hadoop2:~/.ssh/
实验：在master上执行
ssh hadoop2
实现无密码登陆。
3 配置Hadoop
3.1拷贝hadoop
将hadoop-1.0.4.tar.gz ,拷贝到usr/local 文件夹下，然后解压。
解压命令：
tar –zxvf hadoop-1.0.4.tar.gz
3.2查看 cat /etc/hosts
IP1 hadoop1
IP2 hadoop2
3.3 配置 conf/masters 和 conf/slaves
conf/masters：
1
IP1
conf/slaves：
1
2
IP2
IP2
3.4 配置 conf/hadoop-env.sh
加入
1
export JAVA_HOME=/home/elvis/soft/jdk1.7.0_17
3.5 配置 conf/core-site.xml
1
2
3
4
<property>
<name>fs.default.name</name>
<value>hdfs://IP1:9000</value>
</property>
3.6 配置 conf/hdfs-site.xml
加入
<property>
<name>dfs.http.address</name>
<value>IP1:50070</value>
</property>
<property>
<name>dfs.name.dir</name>
<value>/usr/local/hadoop/namenode</value>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/local/hadoop/data</value>
</property>
<property>
<name>dfs.replication</name>
<value>2</value>
</property>
3.7 配置conf/mapred-site.xml
加入
<property>
<name>mapred.job.tracker</name>
<value>192.168.1.50:8012</value>
</property>
3.8 建立相关的目录
1
/usr/local/hadoop/ //hadoop数据和namenode目录
【注意】只创建到hadoop目录即可，不要手动创建data和namenode目录。
其他节点机也同样建立该目录。
3.9 拷贝hadoop文件到其他节点机
将hadoop文件远程copy到其他节点（这样前面的配置就都映射到了其他节点上），
命令：
1
scp -r hadoop-1.0.4 IP2:/usr/local/
3.10 格式化Active master
命令：
bin/hadoop namenode -format
3.11 启动集群 ./start-all.sh
现在集群启动起来了，看一下，命令：
1
bin/hadoop dfsadmin -report
2个datanode，打开web看一下
浏览器输入:IP1:50070
打完收工，集群安装完成！
大讲台 实战项目最多的Hadoop学习平台
扩展阅读：如何搭建自己的网站 ... linux网站入口链接 ... linux下搭建ftp服务器 ... 内网ftp服务器搭建win7 ... linux网站入口老票款 ... linux下搭建nas存储 ... 个人服务器搭建linux ... linux搭建hadoop步骤 ... 如何搭建vps的最详细教程 ...
