使用cloudera manager安装cdh5.8.0
2016-12-02 14:35
351 查看
节点规划:
所有节点data-01 - data-06 6台阿里云服务器8核32G内存
管理节点data-01
基础环境准备工作:
1:hosts映射及系统设置
echo ‘10.0.0.1 data-01’ >> /etc/hosts
echo ‘10.0.0.2 data-02’ >> /etc/hosts
echo ‘10.0.0.3 data-03’ >> /etc/hosts
echo ‘10.0.0.4 data-04’ >> /etc/hosts
echo ‘10.0.0.5 data-05’ >> /etc/hosts
echo ‘10.0.0.6 data-06’ >> /etc/hosts
echo ‘01 * * * * root ntpdate -u data-06’ >> /etc/crontab
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/’ /etc/selinux/config
setenforce 0
service iptables stop
chkconfig iptables off
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo ‘echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag’ >> /etc/rc.local
echo ‘0’ > /proc/sys/vm/swappiness
echo ‘echo 0 > /proc/sys/vm/swappiness’ >> /etc/sysctl.conf
sysctl -p
2:ssh免密码登录
ssh-keygen
tar -xvf cdh.5.8.0.20161009.tar.gz -C /var/www/html/
yum install expat-devel-* cyrus-sasl-devel-* openldap-devel-* apr-* httpd-* deltarpm-* python-deltarpm-* createrepo-*
service httpd restart
chkconfig httpd on
4:配置ntp时间服务
sed -i ‘s/restrict default kod nomodify notrap nopeer noquery/restrict default kod nomodify /’ /etc/ntp.conf
echo ” >> /etc/ntp.conf
echo ‘server 127.127.1.0’ >> /etc/ntp.conf
echo ‘fudge 127.127.1.0 stratum 10’ >> /etc/ntp.conf
service ntpd restart
chkconfig ntpd on
service ntpd start
chkconfig ntpd on
5:Java环境
wget http://data-06/cdh/jdk-8u51-linux-x64.tar.gz -P ~
tar -xvf jdk-8u51-linux-x64.tar.gz -C /usr/java
vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_51
export CLASSPATH=.:JAVAHOME/jre/lib/rt.jar:JAVA_HOME/lib/dt.jar:JAVAHOME/lib/tools.jarexportPATH=PATH:$JAVA_HOME/bin
source /etc/profile
6:mysql安装
/usr/bin/mysql_install_db
service mysqld start
chkconfig mysqld on
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database reports_manager DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all on . to root@”%” Identified by “bigdata”;
use mysql;
UPDATE user SET password=PASSWORD(‘bigdata’) WHERE user=’root’;
flush privileges;
exit;
7:安装cloudera manager
tar -xvf cloudera-manager-el6-cm5.8.2_x86_64.tar.gz -C /opt/
rm -f cloudera-manager-el6-cm5.8.2_x86_64.tar.gz
wget http://data-06/cdh/CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel -P /opt/cloudera/parcel-repo/
wget http://data-06/cdh/CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha -P /opt/cloudera/parcel-repo/
wget http://data-06/cdh/manifest.json -P /opt/cloudera/parcel-repo/
wget http://data-06/cdh/mysql-connector-java.jar -P /opt/cm-5.8.2/share/cmf/lib/
wget http://data-06/cdh/mysql-connector-java.jar -P /usr/share/java/
sed -i ‘s/server_host=localhost/server_host=data-01/’ /opt/cm-5.8.2/etc/cloudera-scm-agent/config.ini
useradd –system –home=/opt/cm-5.8.2/run/cloudera-scm-server –no-create-home –shell=/bin/false –comment “Cloudera SCM User” cloudera-scm
8:安装cdh
http://10.27.229.69:7180 (admin:admin)
根据页面提示选择安装
所有节点data-01 - data-06 6台阿里云服务器8核32G内存
管理节点data-01
基础环境准备工作:
1:hosts映射及系统设置
—所有节点执行—
echo ”>> /etc/hostsecho ‘10.0.0.1 data-01’ >> /etc/hosts
echo ‘10.0.0.2 data-02’ >> /etc/hosts
echo ‘10.0.0.3 data-03’ >> /etc/hosts
echo ‘10.0.0.4 data-04’ >> /etc/hosts
echo ‘10.0.0.5 data-05’ >> /etc/hosts
echo ‘10.0.0.6 data-06’ >> /etc/hosts
echo ‘01 * * * * root ntpdate -u data-06’ >> /etc/crontab
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/’ /etc/selinux/config
setenforce 0
service iptables stop
chkconfig iptables off
echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag
echo ‘echo never > /sys/kernel/mm/redhat_transparent_hugepage/defrag’ >> /etc/rc.local
echo ‘0’ > /proc/sys/vm/swappiness
echo ‘echo 0 > /proc/sys/vm/swappiness’ >> /etc/sysctl.conf
sysctl -p
2:ssh免密码登录
—所有节点执行—
cd ~ssh-keygen
之后一路回车
cat ~/.ssh/id_rsa.pub将所有机器的id_rsa.pub内容放到一个文本内命名为authorized_keys,将authorized_keys上传到各机器/root/.ssh/ 下
验证是否实现各机器之间免密登录
3:配置cdh源(用于各节点下载安装包)—data-06执行(源节点)—
cdh.5.8.0.20161009.tar.gz此包包含了所有安装cdh要用到的东西
mkdir -p /var/www/html/tar -xvf cdh.5.8.0.20161009.tar.gz -C /var/www/html/
yum install expat-devel-* cyrus-sasl-devel-* openldap-devel-* apr-* httpd-* deltarpm-* python-deltarpm-* createrepo-*
service httpd restart
chkconfig httpd on
4:配置ntp时间服务
—data-06执行(时间服务节点)—
cp /etc/ntp.conf /etc/ntp.conf.baksed -i ‘s/restrict default kod nomodify notrap nopeer noquery/restrict default kod nomodify /’ /etc/ntp.conf
echo ” >> /etc/ntp.conf
echo ‘server 127.127.1.0’ >> /etc/ntp.conf
echo ‘fudge 127.127.1.0 stratum 10’ >> /etc/ntp.conf
service ntpd restart
chkconfig ntpd on
—所有节点执行—
yum install -y ntpservice ntpd start
chkconfig ntpd on
5:Java环境
—所有节点执行—
mkdir -p /usr/java/wget http://data-06/cdh/jdk-8u51-linux-x64.tar.gz -P ~
tar -xvf jdk-8u51-linux-x64.tar.gz -C /usr/java
vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_51
export CLASSPATH=.:JAVAHOME/jre/lib/rt.jar:JAVA_HOME/lib/dt.jar:JAVAHOME/lib/tools.jarexportPATH=PATH:$JAVA_HOME/bin
source /etc/profile
验证Java安装正确
java -version6:mysql安装
—data-01执行(mysql数据库位置)—
yum install -y mysql-server/usr/bin/mysql_install_db
service mysqld start
chkconfig mysqld on
创建数据
mysqlcreate database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database reports_manager DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
grant all on . to root@”%” Identified by “bigdata”;
use mysql;
UPDATE user SET password=PASSWORD(‘bigdata’) WHERE user=’root’;
flush privileges;
exit;
7:安装cloudera manager
—所有节点—(以下命令中安装包版本需按照自己安装版本更改)
wget http://data-06/cdh/cloudera-manager-el6-cm5.8.2_x86_64.tar.gz -P ~tar -xvf cloudera-manager-el6-cm5.8.2_x86_64.tar.gz -C /opt/
rm -f cloudera-manager-el6-cm5.8.2_x86_64.tar.gz
wget http://data-06/cdh/CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel -P /opt/cloudera/parcel-repo/
wget http://data-06/cdh/CDH-5.8.0-1.cdh5.8.0.p0.42-el6.parcel.sha -P /opt/cloudera/parcel-repo/
wget http://data-06/cdh/manifest.json -P /opt/cloudera/parcel-repo/
wget http://data-06/cdh/mysql-connector-java.jar -P /opt/cm-5.8.2/share/cmf/lib/
wget http://data-06/cdh/mysql-connector-java.jar -P /usr/share/java/
sed -i ‘s/server_host=localhost/server_host=data-01/’ /opt/cm-5.8.2/etc/cloudera-scm-agent/config.ini
useradd –system –home=/opt/cm-5.8.2/run/cloudera-scm-server –no-create-home –shell=/bin/false –comment “Cloudera SCM User” cloudera-scm
—管理节点—
连接数据库
/opt/cm-5.8.2/share/cmf/schema/scm_prepare_database.sh mysql cm -hdata-01 -uroot -pbigdata –scm-host data-01 scm scm scm启动服务
/opt/cm-5.8.2/etc/init.d/cloudera-scm-server start &—所有节点—
/opt/cm-5.8.2/etc/init.d/cloudera-scm-agent start &8:安装cdh
http://10.27.229.69:7180 (admin:admin)
根据页面提示选择安装
相关文章推荐
- 使用yum安装CDH Hadoop集群
- spark2.1.0 on yarn with CDH5.8.0 安装实战
- 第五章 更换cdh版本,hive的安装使用,原理讲解
- hadoop CDH3U5 使用tarball完整安装过程
- CDH安装和使用时碰到的各种错误记录(基于5.8)
- Cloudera Manager 5.8.1安装CDH 5.8.0权威指南
- 使用yum安装CDH Hadoop集群
- 浅析怎样使用yum安装CDH Hadoop集群
- 使用yum源安装CDH Hadoop集群
- 使用yum源安装CDH Hadoop集群
- 使用yum源安装CDH Hadoop集群
- 使用CDH Manager(及本地源)自动化安装CDH 5
- CDH使用之CM 5.3.x安装
- 使用cloudera manager 安装CDH时,界面提示修改swappiness
- OpenCV3.2+Qt5.8.0+Win10配置和使用----(1)Qt5.8.0的安装
- CDH5.7/5.8离线安装-使用CM5,基于CentOS 6.7
- 使用yum源安装CDH Hadoop集群
- 使用yum安装CDH Hadoop集群