大數據之----部署安裝編譯打包hadoop終極篇

發布時間：2020-08-07 17:27:29 來源：網絡閱讀：973 作者：ycwyong 欄目：大數據

------------------------------軟件版本--------------------------------------
RHEL6.8 hadoop2.8.1 apache-maven-3.3.9

findbugs-1.3.9 protobuf-2.5.0.tar.gz jdk-8u45

------------------------------軟件版本---------------------------------------

大數據概述可以用“5V + 1C”來概括：

Variety 多樣化
Volume 海量
Velocity 快速
Vitality 靈活
Value 價值性
Complexity 復雜

1.Hadoop生態
宏觀: Hadoop為主的生態圈 hadoop flume........
狹義: apache hadoop hadoop.apache.org

2.Hadoop(存儲+計算+資源和作業調度)
hadoop1.x
    HDFS      存儲
    MapReduce 計算+資源和作業調度

hadoop2.x 企業正在用
    HDFS      存儲
    MapReduce 計算
    YARN      資源和作業調度平臺計算組件都會on yarn

hadoop3.x ???

EC技術：Erasure Encoding 簡稱EC，是Hadoop3給HDFS拓展的一種新特性，用來解決存儲空間文件。
YARN：提供YARN的時間軸服務V.2，以便用戶和開發人員可以對其進行測試，并提供反饋意見。
優化Hadoop Shell腳本
重構Hadoop Client Jar包
支持隨機Container
MapReduce任務級本地優化
支持多個NameNode
部分默認服務端口被改變
支持文件系統連接器
DataNode內部添加了負載均衡
重構后臺程序和任務對管理

2.Maven部署

blog
2.1解壓

[root@hadoop1 softwore]# pwd
/opt/softwore
[root@hadoop1 softwore]# ls
apache-maven-3.3.9-bin.zip hadoop-2.8.1-src.tar.gz jdk-8u45-linux-x64.gz
findbugs-1.3.9.zip hadoop-2.8.1.tar.gz protobuf-2.5.0.tar.gz
[root@hadoop1 softwore]# unzip apache-maven-3.3.9-bin.zip

--配置環境變量

[root@hadoop1 softwore]#

vi /etc/profile
export MAVEN_HOME=/opt/software/apache-maven-3.3.9
export MAVEN_OPTS="-Xms256m -Xmx512m"
export PATH=$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH

[root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 apache-maven-3.3.9]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: zh_CN, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-358.el6.x86_64", arch: "amd64", family: "unix"

Maven部署為二進制部署，解壓配置環境變量

2.2配置mavne目錄
2.3查看配置文件和解壓我們準備好的倉庫文件

3.Hadoop編譯

編譯hadoop 通過Requirements我看可以知道需要安裝jdk maven findbugs protocolBuffer cmake Zlib openssl-devel 等軟件
3.1解壓
3.2查看pom.xml
3.3查看BUILDING.txt
Requirements:編譯軟件環境要求

* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)

3.4 JDK部署

[root@hadoop1 softwore]# tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java[root@hadoop1 softwore]# ls -ld /usr/java/*
drwxr-xr-x 8 root root 4096 12月 15 2016 /usr/java/djdk1.7.0_79
drwxr-xr-x 8 uucp 143 4096 4月 11 2015 /usr/java/jdk1.8.0_45
drwxr-xr-x 8 uucp 143 4096 10月 7 2015 /usr/java/jdk1.8.0_65
[root@hadoop1 softwore]# vim /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_45
export JRE_HOME=/usr/java/jdk1.8.0_45/jre

[root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 softwore]# java -version
java version "1.8.0_45"
Java(TM) SE Runtime Environment (build 1.8.0_45-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.45-b02, mixed mode)

二進制版本解壓，配置環境變量即可

3.5 MAVEN

上面我們已經安裝maven和jdk檢測版本是否滿足我們編譯軟件要求
[root@hadoop1 softwore]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
[root@hadoop1 softwore]#

3.6 FINDBUGS

[root@hadoop1 softwore]# unzip findbugs-1.3.9.zip
[root@hadoop1 softwore]# vi /etc/profile
export FINDBUGS_HOME=/opt/software/findbugs-1.3.9
export PATH=$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH [root@hadoop1 softwore]# source /etc/profile
[root@hadoop1 softwore]# findbugs -version
1.3.9

二進制安裝解壓，配置環境變量即可

3.7 PROTOCBUF編譯安裝

[root@hadoop1 software]# tar xvf protobuf-2.5.0.tar.gz
[root@hadoop1 yum.repos.d]# yum install -y gcc gcc-c++ make cmake ---編譯環境
[root@hadoop1 yum.repos.d]#cd /opt/software/protobuf-2.5.0
[root@hadoop1 protobuf-2.5.0]# ./configure --prefix=/usr/local/protobuf
[root@hadoop1 protobuf-2.5.0]# make && make install
[root@hadoop1 protobuf-2.5.0]# vim /etc/profile
export PROTOC_HOME=/usr/local/protobuf
export PATH=$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MAVEN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop1 software]# source /etc/profile
[root@hadoop1 software]# protoc --version
libprotoc 2.5.0
[root@hadoop000 local]#

編譯安裝完成，配置環境變量

3.8 OTHER 依賴包

yum install -y openssl openssl-devel svn ncurses-devel zlib-devel libtool
yum install -y snappy snappy-devel bzip2 bzip2-devel \n
zo lzo-devel lzop autoconf automake

其他依賴包在系統安裝ios都包含，執行yum安裝即可

---------------------------------------------------------------------------------------------------------------------------------------

至此，編譯hadoop需要的編譯環境全部準備好，下面開啟編譯之路

---------------------------------------------------------------------------------------------------------------------------------------

3.9 編譯

[root@hadoop1 software]# tar xvf hadoop-2.8.1-src.tar.gz
[root@hadoop1 software]# cd hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]#
mvn clean package -Pdist,native -DskipTests –Dtar

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

----編譯好軟件包保存位置

直接編譯需要聯網找依賴包，一般網絡情況較好，需要幾個小時

提前準備好依賴包編譯方式

[root@hadoop1 software]# tar xvf hadoop-2.8.1-src.tar.gz
[root@hadoop1 software]# cd hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src

[root@hadoop1 hadoop-2.8.1-src]# pwd
/opt/software/hadoop-2.8.1-src
[root@hadoop1 hadoop-2.8.1-src]# ls pom.xml ---根據這個文件下載jar
pom.xml
[root@hado

-rw-r--r-- 1 root root 96721446 4月 25 09:57 .m2.tar.gz ---提前下好的倉庫軟件包

[root@hadoop1 software]#tar xvf .m2.tar.gz –C /root ---不想download，使用提前準備好的倉庫軟件--->解壓----->root家目錄

[root@hadoop1 hadoop-2.8.1-src]# mvn clean package -Pdist,native -DskipTests –Dtar

……..

[INFO] Apache Hadoop Azure support ........................ SUCCESS [ 5.562 s]
[INFO] Apache Hadoop Client ............................... SUCCESS [ 13.396 s]
[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [ 1.969 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [ 6.050 s]
[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 11.110 s]
[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [ 3.862 s]
[INFO] Apache Hadoop Tools ................................ SUCCESS [ 0.029 s]
[INFO] Apache Hadoop Distribution ......................... SUCCESS [01:01 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13:45 min
[INFO] Finished at: 2018-04-25T14:03:20+08:00
[INFO] Final Memory: 161M/479M
[INFO] ------------------------------------------------------------------------

[root@hadoop1 hadoop-2.8.1-src]# 編譯成功后的軟件保存路徑

[root@hadoop1 hadoop-2.8.1-src]# ls /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-*
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-javadoc.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-sources.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-dist-2.8.1-test-sources.jar
/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1:
bin etc include lib libexec LICENSE.txt NOTICE.txt README.txt sbin share
[root@hadoop1 hadoop-2.8.1-src]# du -sh /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz
186M /opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

/opt/software/hadoop-2.8.1-src/hadoop-dist/target/hadoop-2.8.1.tar.gz

總結：完成源碼的編譯好打包，得到安裝hadoop安裝包，難點在于軟件包的裝備和倉庫軟件的準備，接下來準備開始hadoop安裝之路。

向AI問一下細節

中文字幕av专区_日韩电影在线播放_精品国产精品久久一区免费式_av在线免费观看网站

大數據之----部署安裝編譯打包hadoop終極篇

猜你喜歡

RHEL6.8	hadoop2.8.1	apache-maven-3.3.9

findbugs-1.3.9	protobuf-2.5.0.tar.gz	jdk-8u45

Variety	多樣化
Volume	海量
Velocity	快速
Vitality	靈活
Value	價值性
Complexity	復雜

中文字幕av专区_日韩电影在线播放_精品国产精品久久一区免费式_av在线免费观看网站

大數據之----部署安裝編譯打包hadoop終極篇

猜你喜歡

最新資訊

相關推薦

相關標簽