centos 6.8下 CDH5.13.3安裝與配置（詳細部署）

centos 6.8下 CDH5.13.3安裝與配置（詳細部署）

一、安裝前準備

1.1、下載地址

wget http://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.13.3_x86_64.tar.gz

http://archive.cloudera.com/cdh6/parcels/5.13.3/

wget http://archive.cloudera.com/cdh6/parcels/5.13.3/manifest.json

wget http://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel

wget http://archive.cloudera.com/cdh6/parcels/5.13.3/CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1

1.2、防火墻關閉

/etc/init.d/iptables stop

chkconfig iptables off

1.3、設置selinux

vi /etc/selinux/config

1.4、設置時間同步ntp(每臺機器)

yum -y install ntpdate

crontab -e

*/5 * * * * ntpdate 0.rhel.pool.ntp.org

保存退出

vi /etc/hosts（hadoop01增加）

注意：Cloudera-Manager臺需要增加127.0.0.1   hadoop01

127.0.0.1   hadoop01

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

vi /etc/hosts（其他兩臺hadoo02、hadoop03）

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

二、依賴包安裝（每臺機器）

yum -y install chkconfig python bind-utils psmisc libxslt zlib sqlite fuse fuse-libs redhat-lsb cyrus-sasl-plain cyrus-sasl-gssapi  openssh-clients

三、jdk安裝（每臺機器）

tar zxvf jdk-8u111-linux-x64.gz

mv jdk1.8.0_111 /usr/local/jdk

配置環境變量

vi /etc/profile

##jdk

export JAVA_HOME=/usr/local/jdk

export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

export PATH=$JAVA_HOME/bin:$PATH

讓環境變量生產

source /etc/profile

四、ssh免密碼登錄配置

ssh-keygen  -t rsa

ssh-copy-id 192.168.1.201

ssh-copy-id 192.168.1.202

ssh-copy-id 192.168.1.202

五、安裝mysql(在haoop01上安裝)

5.1、安裝依賴包

yum -y install libaio*

yum -y install libnuma*

5.2、創建mysql用戶

useradd mysql

5.3、創建目錄

mkdir -p /data/app

mkdir /data/app/mysql3306

mkdir /data/app/undolog

5.4、解壓文件

tar zxvf mysql-5.6.41-linux-glibc2.12-x86_64.tar.gz

mv mysql-5.6.41-linux-glibc2.12-x86_64 /data/app/mysql5.6

mkdir /data/app/mysql5.6/logs

5.5、配置my.cnf

vi /data/app/mysql5.6/my.cnf

[client]

socket=/data/app/mysql5.6/mysqld.sock

port=3306

[mysql]

prompt=\\u@\\d \\r:\\m:\\s>

no-auto-rehash

default-character-set=utf8

[mysqld_safe]

log-error=/data/app/mysql3306/mysqld.error

[mysqld]

socket=/data/app/mysql5.6/mysqld.sock

pid-file=/data/app/mysql5.6/mysqld.pid

basedir=/data/app/mysql5.6

datadir=/data/app/mysql3306

port=3306

server-id=1201

character-set-server=utf8

skip-external-locking

#skip-name-resolve

max_connections=1000

max_connect_errors=1000

max_allowed_packet=128M

table_definition_cache=500

table_open_cache=500

sort_buffer_size = 1M

tmp_table_size = 64M

read_buffer_size = 1M

read_rnd_buffer_size = 1M

myisam_sort_buffer_size = 64M

thread_cache_size = 8

query_cache_type=0

query_cache_size=0

lower_case_table_names = 1

log_bin_trust_function_creators = 1

#################slow log####################

slow-query_log=1

slow-query_log_file=/data/app/mysql5.6/logs/mysql.slow

long_query_time=2

####################binlog######################

log-bin=mysql-bin

binlog-format=ROW

expire_logs_days=10

log-slave-updates=1

sync_binlog=0

auto_increment_increment=2

auto_increment_offset=1

###

###

###

innodb_undo_directory=/data/app/undolog

innodb_undo_logs = 128

innodb_undo_tablespaces = 3

################INNODB################

transaction-isolation=READ-COMMITTED

innodb_buffer_pool_size=2G

innodb_flush_log_at_trx_commit=2

innodb_strict_mode=1

innodb_flush_method=O_DIRECT

innodb_file_format=Barracuda

innodb_log_files_in_group=3

innodb_file_per_table=1

innodb_io_capacity=500

innodb_support_xa=1

[mysqldump]

quick

max_allowed_packet=128M

[mysqlhotcopy]

interactive_timeout     = 28400

5.6、增加權限

chown -R mysql:mysql /data/app

5.7、配置環境變量

vi /etc/profile

##mysql

export MYSQL_HOME=/data/app/mysql5.6

export MY_BASEDIR_VERSION=/data/app/mysql5.6

export PATH=/data/app/mysql5.6/bin:$PATH

export LD_LIBRARY_PATH=/data/app/mysql5.6/lib

環境生效

source /etc/profile

5.8、初始化數據庫

/data/app/mysql5.6/scripts/mysql_install_db --user=mysql --basedir=/data/app/mysql5.6   --datadir=/data/app/mysql3306

5.9、配置啟動文件

cp /data/app/mysql5.6/support-files/mysql.server /etc/init.d/mysqld

vi /etc/init.d/mysqld

basedir=/data/app/mysql5.6

datadir=/data/app/mysql3306

保存退出

chmod +x /etc/init.d/mysqld

chkconfig mysqld on

啟動

/etc/init.d/mysqld start

5.10、初始化密碼

/data/app/mysql5.6/bin/mysqladmin -u root password 'yb2018#08'

5.11、設置遠程權限

# mysql -uroot -pyb2018#08

Warning: Using a password on the command line interface can be insecure.

Welcome to the MySQL monitor.  Commands end with ; or \g.

Your MySQL connection id is 2

Server version: 5.6.41-log MySQL Community Server (GPL)

Copyright (c) 2000, 2018, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

root@(none) 10:28:44>grant all on *.* to 'scm'@'localhost' identified by 'temp' with grant option;

Query OK, 0 rows affected (0.00 sec)

root@(none) 10:28:58>grant all on *.* to 'cdh'@'%' identified by 'cdh#2018' with grant option;

Query OK, 0 rows affected (0.00 sec)

root@(none) 10:29:08>create database hive;

Query OK, 1 row affected (0.00 sec)

root@(none) 10:29:16>create database ooz;

Query OK, 1 row affected (0.00 sec)

root@(none) 10:29:27>create database hue;

Query OK, 1 row affected (0.00 sec)

root@(none) 10:29:36>flush privileges;

Query OK, 0 rows affected (0.00 sec)

六、安裝Cloudera-Manager

6.1、解壓包到指定目錄，所有服務器都要

mkdir -p /data/cloudera-manager

tar zxvf cloudera-manager-el6-cm5.13.3_x86_64.tar.gz -C /data/cloudera-manager

6.2創建cloudera-scm用戶(所有節點)

useradd -r -d /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-server -M -c "Cloudera SCM User" cloudera-scm

id cloudera-scm

6.3、配置從節點cloudera-manger-agent指向主節點服務器（hadoop02,hadoop03配置）

vi /data/cloudera-manager/cm-5.13.3/etc/cloudera-scm-agent/config.ini

將server_host改為CMS所在的主機名即hadoop01

6.4、主節點中創建parcel-repo倉庫目錄（這個只在主節點hadoop01上需要）

mkdir -p /data/cloudera/parcel-repo

chown cloudera-scm:cloudera-scm /data/cloudera/parcel-repo

cp CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 manifest.json /data/cloudera/parcel-repo/

cd /data/cloudera/parcel-repo/

mv CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha

注意：其中CDH-5.13.3-1.cdh6.13.3.p0.2-el6.parcel.sha1 后綴要把1去掉

6.5、所有節點創建parcels目錄（所有節點）

mkdir -p /data/cloudera/parcels

chown cloudera-scm:cloudera-scm /data/cloudera/parcels

解釋：Clouder-Manager將CDHs從主節點的/data/cloudera/parcel-repo目錄中抽取出來，分發解壓激活到各個節點的/data/cloudera/parcels目錄中

6.6、初始腳本配置數據庫scm_prepare_database.sh(在主節點上)

/data/cloudera-manager/cm-5.13.3/share/cmf/schema/scm_prepare_database.sh  mysql -uscm -ptemp scm scm temp

如果報錯，操作

mkdir /usr/share/java

cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

上圖說明成功

成功把記住把#127.0.0.1   localhost hadoop01 注釋掉，不然后面就會有問題

vi /etc/hosts

#127.0.0.1  hadoop01

192.168.1.201 hadoop01

192.168.1.202 hadoop02

192.168.1.203 hadoop03

6.7、啟動主節點cloudera-scm-server(hadoop01操作)

cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-server /etc/init.d/cloudera-scm-server

chmod +x /etc/init.d/cloudera-scm-server

啟動

/etc/init.d/cloudera-scm-server start

如果報File not found: /usr/sbin/cmf-server錯誤

vi /etc/init.d/cloudera-scm-server

找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}

修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default

啟動

/etc/init.d/cloudera-scm-server start

6.8、啟動cloudera-scm-agent所有節點(hadoop02,hadoop03操作)

mkdir /data/cloudera-manager/cm-5.13.3/run/cloudera-scm-agent

cp /data/cloudera-manager/cm-5.13.3/etc/init.d/cloudera-scm-agent /etc/init.d/cloudera-scm-agent

chmod +x /etc/init.d/cloudera-scm-agent

/etc/init.d/cloudera-scm-agent start

vi /etc/init.d/cloudera-scm-agent

找到CMF_DEFAULTS=${CMF_DEFAULTS:-/etc/default}

修改成CMF_DEFAULTS=/data/cloudera-manager/cm-5.13.3/etc/default

啟動

/etc/init.d/cloudera-scm-agent start

6.9、登錄用戶admin 密碼admin

http://192.168.1.201:7180

繼續

選擇免費，繼續

繼續

選擇當前管理主機選擇hadoop02,hadoop03,繼續

點擊更多選項配置

修改好后，保存修改

重啟后才后出下如下圖

/etc/init.d/cloudera-scm-server restart

重啟好后才能如上圖繼續，因為默認/opt下安裝的

繼續

每臺執行以下

echo 10 >/proc/sys/vm/swappiness

echo never >/sys/kernel/mm/transparent_hugepage/defrag

echo never >/sys/kernel/mm/transparent_hugepage/enabled

以圖根據自己需求選擇不一樣點繼續

以上Datanode修改兩臺

把連接主機名，數據庫名稱，用戶名，密碼輸入進去，點擊測試連接都Succesful成功后，點擊繼續

根據自己修改點擊繼續

繼續

安裝完成