溫馨提示×
您好，登錄后才能下訂單哦！
點擊 登錄注冊 即表示同意《億速云用戶服務條款》
您好，登錄后才能下訂單哦！
1. Java安裝與環境配置
Hadoop是基于Java的，所以首先需要安裝配置好java環境。從官網下載JDK，我用的是1.8版本。 在Mac下可以在終端下使用scp命令遠程拷貝到虛擬機linux中。
danieldu@daniels-MacBook-Pro-857 ~/Downloads scp jdk-8u121-linux-x64.tar.gz root@hadoop100:/opt/software root@hadoop100's password: danieldu@daniels-MacBook-Pro-857 ~/Downloads
其實我在Mac上裝了一個神器-Forklift。 可以通過SFTP的方式連接到遠程linux。然后在操作本地電腦一樣，直接把文件拖過去就行了。而且好像配置文件的編輯，也可以不用在linux下用vi，直接在Mac下用sublime遠程打開就可以編輯了 ：）
然后在linux虛擬機中（ssh 登錄上去）解壓縮到/opt/modules目錄下
[root@hadoop100 include]# tar -zxvf /opt/software/jdk-8u121-linux-x64.tar.gz -C /opt/modules/
然后需要設置一下環境變量， 打開 /etc/profile, 添加JAVA_HOME并設置PATH用vi打開也行，或者如果你也安裝了類似forklift這樣的可以遠程編輯文件的工具那更方便。
vi /etc/profile
按shift + G 跳到文件最后，按i切換到編輯模式，添加下面的內容，主要路徑要搞對。
#JAVA_HOME export JAVA_HOME=/opt/modules/jdk1.8.0_121 export PATH=$PATH:$JAVA_HOME/bin
按ESC ， 然后 ：wq存盤退出。
執行下面的語句使更改生效
[root@hadoop100 include]# source /etc/profile
檢查java是否安裝成功。如果能看到版本信息就說明安裝成功了。
[root@hadoop100 include]# java -version java version "1.8.0_121" Java(TM) SE Runtime Environment (build 1.8.0_121-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode) [root@hadoop100 include]#
2. Hadoop安裝與環境配置
Hadoop的安裝也是只需要把hadoop的tar包拷貝到linux，解壓，設置環境變量.然后用之前做好的xsync腳本，把更新同步到集群中的其他機器。如果你不知道xcall、xsync怎么寫的。可以翻一下之前的文章。這樣集群里的所有機器就都設置好了。
[root@hadoop100 include]# tar -zxvf /opt/software/hadoop-2.7.3.tar.gz -C /opt/modules/ [root@hadoop100 include]# vi /etc/profile 繼續添加HADOOP_HOME #JAVA_HOME export JAVA_HOME=/opt/modules/jdk1.8.0_121 export PATH=$PATH:$JAVA_HOME/bin #HADOOP_HOME export HADOOP_HOME=/opt/modules/hadoop-2.7.3 export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin [root@hadoop100 include]# source /etc/profile 把更改同步到集群中的其他機器 [root@hadoop100 include]# xsync /etc/profile [root@hadoop100 include]# xcall source /etc/profile [root@hadoop100 include]# xsync hadoop-2.7.3/
3. Hadoop分布式配置
然后需要對Hadoop集群環境進行配置。對于集群的資源配置是這樣安排的，當然hadoop100顯得任務重了一點 :)
編輯0/opt/modules/hadoop-2.7.3/etc/hadoop/mapred-env.sh、yarn-env.sh、hadoop-env.sh 這幾個shell文件中的JAVA_HOME，設置為真實的絕對路徑。
export JAVA_HOME=/opt/modules/jdk1.8.0_121
打開編輯 /opt/modules/hadoop-2.7.3/etc/hadoop/core-site.xml, 內容如下
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://hadoop100:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/opt/modules/hadoop-2.7.3/data/tmp</value> </property> </configuration
編輯/opt/modules/hadoop-2.7.3/etc/hadoop/hdfs-site.xml， 指定讓dfs復制5份，因為我這里有5臺虛擬機組成的集群。每臺機器都擔當DataNode的角色。暫時也把secondary name node也放在hadoop100上，其實這里不太好，最好能和主namenode分開在不同機器上。
<configuration> <property> <name>dfs.replication</name> <value>5</value> </property> <property> <name>dfs.namenode.secondary.http-address</name> <value>hadoop100:50090</value> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> </configuration>
YARN 是hadoop的集中資源管理服務，放在hadoop100上。 編輯/opt/modules/hadoop-2.7.3/etc/hadoop/yarn-site.xml
<configuration> <!-- Site specific YARN configuration properties --> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>hadoop100</value> </property> <property> <name>yarn.log-aggregation-enbale</name> <value>true</value> </property> <property> <name>yarn.log-aggregation.retain-seconds</name> <value>604800</value> </property> </configuration>
為了讓集群能一次啟動，編輯slaves文件（/opt/modules/hadoop-2.7.3/etc/hadoop/slaves），把集群中的幾臺機器都加入到slave文件中，一臺占一行。
hadoop100
hadoop101
hadoop102
hadoop103
hadoop104
最后，在hadoop100上全部做完相關配置更改后，把相關的修改同步到集群中的其他機器
xsync hadoop-2.7.3/
在啟動Hadoop之前需要format一下hadoop設置。
hdfs namenode -format
然后就可以啟動hadoop了。從下面的輸出過程可以看到整個集群從100到104的5臺機器都已經啟動起來了。通過jps可以查看當前進程。
[root@hadoop100 sbin]# ./start-dfs.sh Starting namenodes on [hadoop100] hadoop100: starting namenode, logging to /opt/modules/hadoop-2.7.3/logs/hadoop-root-namenode-hadoop100.out hadoop101: starting datanode, logging to /opt/modules/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop101.out hadoop102: starting datanode, logging to /opt/modules/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop102.out hadoop100: starting datanode, logging to /opt/modules/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop100.out hadoop103: starting datanode, logging to /opt/modules/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop103.out hadoop104: starting datanode, logging to /opt/modules/hadoop-2.7.3/logs/hadoop-root-datanode-hadoop104.out Starting secondary namenodes [hadoop100] hadoop100: starting secondarynamenode, logging to /opt/modules/hadoop-2.7.3/logs/hadoop-root-secondarynamenode-hadoop100.out [root@hadoop100 sbin]# jps 2945 NameNode 3187 SecondaryNameNode 3047 DataNode 3351 Jps [root@hadoop100 sbin]# ./start-yarn.sh starting yarn daemons starting resourcemanager, logging to /opt/modules/hadoop-2.7.3/logs/yarn-root-resourcemanager-hadoop100.out hadoop103: starting nodemanager, logging to /opt/modules/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop103.out hadoop102: starting nodemanager, logging to /opt/modules/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop102.out hadoop104: starting nodemanager, logging to /opt/modules/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop104.out hadoop101: starting nodemanager, logging to /opt/modules/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop101.out hadoop100: starting nodemanager, logging to /opt/modules/hadoop-2.7.3/logs/yarn-root-nodemanager-hadoop100.out [root@hadoop100 sbin]# jps 3408 ResourceManager 2945 NameNode 3187 SecondaryNameNode 3669 Jps 3047 DataNode 3519 NodeManager [root@hadoop100 sbin]#
4. Hadoop的使用
使用hadoop可以通過API調用，這里先看看使用命令調用，確保hadoop環境已經正常運行了。
這中間有個小插曲，我通過下面的命令查看hdfs上面的文件時，發現連接不上。
[root@hadoop100 ~]# hadoop fs -ls ls: Call From hadoop100/192.168.56.100 to hadoop100:9000 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
后來發現，是我中間更改過前面提到的xml配置文件，忘記format了。修改配置后記得要format。
hdfs namenode -format
hdfs 文件操作
[root@hadoop100 sbin]# hadoop fs -ls / [root@hadoop100 sbin]# hadoop fs -put ~/anaconda-ks.cfg / [root@hadoop100 sbin]# hadoop fs -ls / Found 1 items -rw-r--r-- 5 root supergroup 1233 2019-09-16 16:31 /anaconda-ks.cfg [root@hadoop100 sbin]# hadoop fs -cat /anaconda-ks.cfg
文件內容
[root@hadoop100 ~]# mkdir tmp [root@hadoop100 ~]# hadoop fs -get /anaconda-ks.cfg ./tmp/ [root@hadoop100 ~]# ll tmp/ total 4 -rw-r--r--. 1 root root 1233 Sep 16 16:34 anaconda-ks.cfg
執行MapReduce程序
hadoop中指向示例的MapReduce程序，wordcount，數數在一個文件中出現的詞的次數，我隨便找了個anaconda-ks.cfg試了一下：
[root@hadoop100 ~]# hadoop jar /opt/modules/hadoop-2.7.3/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /anaconda-ks.cfg ~/tmp 19/09/16 16:43:28 INFO client.RMProxy: Connecting to ResourceManager at hadoop100/192.168.56.100:8032 19/09/16 16:43:29 INFO input.FileInputFormat: Total input paths to process : 1 19/09/16 16:43:29 INFO mapreduce.JobSubmitter: number of splits:1 19/09/16 16:43:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1568622576365_0001 19/09/16 16:43:30 INFO impl.YarnClientImpl: Submitted application application_1568622576365_0001 19/09/16 16:43:31 INFO mapreduce.Job: The url to track the job: http://hadoop100:8088/proxy/application_1568622576365_0001/ 19/09/16 16:43:31 INFO mapreduce.Job: Running job: job_1568622576365_0001 19/09/16 16:43:49 INFO mapreduce.Job: Job job_1568622576365_0001 running in uber mode : false 19/09/16 16:43:49 INFO mapreduce.Job: map 0% reduce 0% 19/09/16 16:43:58 INFO mapreduce.Job: map 100% reduce 0% 19/09/16 16:44:10 INFO mapreduce.Job: map 100% reduce 100% 19/09/16 16:44:11 INFO mapreduce.Job: Job job_1568622576365_0001 completed successfully 19/09/16 16:44:12 INFO mapreduce.Job: Counters: 49 File System Counters FILE: Number of bytes read=1470 FILE: Number of bytes written=240535 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=1335 HDFS: Number of bytes written=1129 HDFS: Number of read operations=6 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Launched map tasks=1 Launched reduce tasks=1 Rack-local map tasks=1 Total time spent by all maps in occupied slots (ms)=6932 Total time spent by all reduces in occupied slots (ms)=7991 Total time spent by all map tasks (ms)=6932 Total time spent by all reduce tasks (ms)=7991 Total vcore-milliseconds taken by all map tasks=6932 Total vcore-milliseconds taken by all reduce tasks=7991 Total megabyte-milliseconds taken by all map tasks=7098368 Total megabyte-milliseconds taken by all reduce tasks=8182784 Map-Reduce Framework Map input records=46 Map output records=120 Map output bytes=1704 Map output materialized bytes=1470 Input split bytes=102 Combine input records=120 Combine output records=84 Reduce input groups=84 Reduce shuffle bytes=1470 Reduce input records=84 Reduce output records=84 Spilled Records=168 Shuffled Maps =1 Failed Shuffles=0 Merged Map outputs=1 GC time elapsed (ms)=169 CPU time spent (ms)=1440 Physical memory (bytes) snapshot=300003328 Virtual memory (bytes) snapshot=4159303680 Total committed heap usage (bytes)=141471744 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=1233 File Output Format Counters Bytes Written=1129 [root@hadoop100 ~]#
在web端管理界面中可以看到對應的application：
執行的結果，看到就是“#” 出現的最多，出現了12次，這也難怪，里面好多都是注釋嘛。
[root@hadoop100 tmp]# hadoop fs -ls /root/tmp Found 2 items -rw-r--r-- 5 root supergroup 0 2019-09-16 16:44 /root/tmp/_SUCCESS -rw-r--r-- 5 root supergroup 1129 2019-09-16 16:44 /root/tmp/part-r-00000 [root@hadoop100 tmp]# hadoop fs -cat /root/tmp/part-r-0000 cat: `/root/tmp/part-r-0000': No such file or directory [root@hadoop100 tmp]# hadoop fs -cat /root/tmp/part-r-00000 # 12 #version=DEVEL 1 $6$JBLRSbsT070BPmiq$Of51A9N3Zjn/gZ23mLMlVs8vSEFL6ybkfJ1K1uJLAwumtkt1PaLcko1SSszN87FLlCRZsk143gLSV22Rv0zDr/ 1 %addon 1 %anaconda 1 %end 3 %packages 1 --addsupport=zh_CN.UTF-8 1 --boot-drive=sda 1 --bootproto=dhcp 1 --device=enp0s3 1 --disable 1 --disabled="chronyd" 1 --emptyok 1 。。。
通過web 界面可以查看hdfs中的文件列表 http://192.168.56.100:50070/explorer.html#
hadoop還有好多好玩兒的東西，等待我去發現呢，過幾天再來更新。
總結
以上所述是小編給大家介紹的hadoop分布式環境搭建過程,希望對大家有所幫助，如果大家有任何疑問請給我留言，小編會及時回復大家的。在此也非常感謝大家對億速云網站的支持！
如果你覺得本文對你有幫助，歡迎轉載，煩請注明出處，謝謝！
免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。
