溫馨提示×
您好，登錄后才能下訂單哦！
點擊 登錄注冊 即表示同意《億速云用戶服務條款》
您好，登錄后才能下訂單哦！
硬件環境：
hddcluster1 10.0.0.197 redhat7
hddcluster2 10.0.0.228 centos7 這臺作為master
hddcluster3 10.0.0.202 redhat7
hddcluster4 10.0.0.181 centos7
軟件環境：
關閉所有防火墻firewall
openssh-clients
openssh-server
java-1.8.0-openjdk
java-1.8.0-openjdk-devel
hadoop-2.7.3.tar.gz
流程：
選定一臺機器作為 Master
在 Master 節點上配置 hadoop 用戶、安裝 SSH server、安裝 Java 環境
在 Master 節點上安裝 Hadoop，并完成配置
在其他 Slave 節點上配置 hadoop 用戶、安裝 SSH server、安裝 Java 環境
將 Master 節點上的 /usr/local/hadoop 目錄復制到其他 Slave 節點上
在 Master 節點上開啟 Hadoop
#節點的名稱與對應的 IP 關系 [hadoop@hddcluster2 ~]$ cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.0.0.228 hddcluster2 10.0.0.197 hddcluster1 10.0.0.202 hddcluster3 10.0.0.181 hddcluster4
創建hadoop用戶 su # 上述提到的以 root 用戶登錄 useradd -m hadoop -s /bin/bash # 創建新用戶hadoop passwd hadoop #設置hadoop密碼 visudo #root ALL=(ALL) ALL 這行下面添加hadoop ALL=(ALL) ALL
#登錄hadoop用戶，安裝SSH、配置SSH無密碼登陸 [hadoop@hddcluster2 ~]$ rpm -qa | grep ssh [hadoop@hddcluster2 ~]$ sudo yum install openssh-clients [hadoop@hddcluster2 ~]$ sudo yum install openssh-server [hadoop@hddcluster2 ~]$cd ~/.ssh/ # 若沒有該目錄，請先執行一次ssh localhost [hadoop@hddcluster2 ~]$ssh-keygen -t rsa # 會有提示，都按回車就可以 [hadoop@hddcluster2 ~]$ssh-copy-id -i ~/.ssh/id_rsa.pub localhost # 加入授權 [hadoop@hddcluster2 ~]$chmod 600 ./authorized_keys # 修改文件權限 [hadoop@hddcluster2 ~]$ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hddcluster1 [hadoop@hddcluster2 ~]$ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hddcluster3 [hadoop@hddcluster2 ~]$ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@hddcluster4
#解壓hadoop文件到/usr/local/hadoop [hadoop@hddcluster2 ~]$sudo tar -zxf hadoop-2.7.3.tar.gz -C /usr/local/ [hadoop@hddcluster2 ~]$sudo mv /usr/local/hadoop-2.7.3 /usr/local/hadoop [hadoop@hddcluster2 ~]$sudo chown -R hadoop:hadoop /usr/local/hadoop cd /usr/local/hadoop ./bin/hadoop version #安裝java環境 [hadoop@hddcluster2 ~]$sudo yum install java-1.8.0-openjdk java-1.8.0-openjdk-devel [hadoop@hddcluster2 ~]$ rpm -ql java-1.8.0-openjdk-devel | grep '/bin/javac' /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64/bin/javac [hadoop@hddcluster2 ~]$ vim ~/.bashrc export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-2.b15.el7_3.x86_64 export HADOOP_HOME=/usr/local/hadoop export HADOOP_INSTALL=$HADOOP_HOME export HADOOP_MAPRED_HOME=$HADOOP_HOME export HADOOP_COMMON_HOME=$HADOOP_HOME export HADOOP_HDFS_HOME=$HADOOP_HOME export YARN_HOME=$HADOOP_HOME export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin export HADOOP_PREFIX=$HADOOP_HOME export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib:$HADOOP_PREFIX/lib/native" #測試java環境 source ~/.bashrc java -version $JAVA_HOME/bin/java -version # 與直接執行 java -version 一樣
#修改hadoop文件配置 [hadoop@hddcluster2 hadoop]$ pwd /usr/local/hadoop/etc/hadoop [hadoop@hddcluster2 hadoop]$ cat core-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hddcluster2:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>file:/usr/local/hadoop/tmp</value> <description>Abase for other temporary directories.</description> </property> </configuration> [hadoop@hddcluster2 hadoop]$ cat hdfs-site.xml <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.namenode.secondary.http-address</name> <value>hddcluster2:50090</value> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>file:/usr/local/hadoop/tmp/dfs/data</value> </property> </configuration> [hadoop@hddcluster2 hadoop]$ [hadoop@hddcluster2 hadoop]$ cat mapred-site.xml <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobhistory.address</name> <value>hddcluster2:10020</value> </property> <property> <name>mapreduce.jobhistory.webapp.address</name> <value>hddcluster2:19888</value> </property> </configuration> [hadoop@hddcluster2 hadoop]$ [hadoop@hddcluster2 hadoop]$ cat yarn-site.xml <?xml version="1.0"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.resourcemanager.hostname</name> <value>hddcluster2</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration> [hadoop@hddcluster2 hadoop]$ [hadoop@hddcluster2 hadoop]$ cat slaves hddcluster1 hddcluster2 hddcluster3 hddcluster4
$cd /usr/local $sudo rm -r ./hadoop/tmp # 刪除 Hadoop 臨時文件 $sudo rm -r ./hadoop/logs/* # 刪除日志文件 $tar -zcf ~/hadoop.master.tar.gz ./hadoop # 先壓縮再復制 $cd ~ $scp ./hadoop.master.tar.gz hddcluster1:/home/hadoop $scp ./hadoop.master.tar.gz hddcluster3:/home/hadoop $scp ./hadoop.master.tar.gz hddcluster4:/home/hadoop
在salve節點上操作，安裝軟件環境并配置好.bashrc sudo tar -zxf ~/hadoop.master.tar.gz -C /usr/local sudo chown -R hadoop /usr/local/hadoop
[hadoop@hddcluster2 ~]$hdfs namenode -format # 首次運行需要執行初始化，之后不需要 接著可以啟動 hadoop 了，啟動需要在 Master 節點上進行啟動命令： $start-dfs.sh $start-yarn.sh $mr-jobhistory-daemon.sh start historyserver 通過命令 jps 可以查看各個節點所啟動的進程。正確的話， 在 Master 節點上可以看到 NameNode、ResourceManager、SecondrryNameNode、JobHistoryServer 進程， 另外還需要在 Master 節點上通過命令 hdfs dfsadmin -report 查看 DataNode 是否正常啟動，如果 Live datanodes 不為 0 ，則說明集群啟動成功。 [hadoop@hddcluster2 ~]$ hdfs dfsadmin -report Configured Capacity: 2125104381952 (1.93 TB) Present Capacity: 1975826509824 (1.80 TB) DFS Remaining: 1975824982016 (1.80 TB) DFS Used: 1527808 (1.46 MB) DFS Used%: 0.00% Under replicated blocks: 0 Blocks with corrupt replicas: 0 Missing blocks: 0 Missing blocks (with replication factor 1): 0 ------------------------------------------------- Live datanodes (4): 也可以通過 Web 頁面看到查看 DataNode 和 NameNode 的狀態：http://hddcluster2:50070/。如果不成功，可以通過啟動日志排查原因。
在 Slave 節點操作可以看到 DataNode 和 NodeManager 進程
測試hadoop分布式實例 首先創建 HDFS 上的用戶目錄： hdfs dfs -mkdir -p /user/hadoop 將 /usr/local/hadoop/etc/hadoop 中的配置文件作為輸入文件復制到分布式文件系統中： hdfs dfs -mkdir input hdfs dfs -put /usr/local/hadoop/etc/hadoop/*.xml input 通過查看 的DataNode 的狀態（占用大小有改變），輸入文件確實復制到了 DataNode 中。 接著就可以運行 MapReduce 作業了： hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep input output 'dfs[a-z.]+' 等待執行完畢后的輸出結果：
hadoop啟動命令： start-dfs.sh start-yarn.sh mr-jobhistory-daemon.sh start historyserver hadoop關閉命令： stop-dfs.sh stop-yarn.sh mr-jobhistory-daemon.sh stop historyserver
PS：如果集群有一兩臺無法啟動的話，先嘗試一下刪除hadoop臨時文件
cd /usr/local
sudo rm -r ./hadoop/tmp
sudo rm -r ./hadoop/logs/*
然后執行
hdfs namenode -format
再啟動
本文參考了一下網站并實驗成功：
http://www.powerxing.com/install-hadoop-cluster/
免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。
