我的Hadoop安装——使用Cloudera部署,管理Hadoop集群(离线安装CDH5.7.0)
2017-06-24 23:15
901 查看
- 概述
- 文件下载
- 系统环境搭建
- Cloudera Manager安装与集群配置
- 日志查看
- Q&A
- 参考
概述
CDH (Cloudera's Distribution, including Apache Hadoop),是Hadoop众多分支中的一种,由Cloudera维护,基于稳定版本的Apache Hadoop构建,并集成了很多补丁,可直接用于生产环境。
Cloudera Manager则是为了便于在集群中进行Hadoop等大数据处理相关的服务安装和监控管理的组件,对集群中主机、Hadoop、Hive、Spark等服务的安装配置管理做了极大简化。
文件下载
Cloudera Manager
地址:http://archive.cloudera.com/cm5/cm/5/
这里下载的是5.7.0的版本,https://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.7.0_x86_64.tar.gz
CDH安装包
地址:http://archive.cloudera.com/cdh5/parcels
本文基于CentOS6,下载的parcels包及manifest文件对应的版本为el6,如果centos7,就用el7:
CDH-5.7.0-1.cdh5.11.0.p0.34-el6.parcel
CDH-5.7.0-1.cdh5.11.0.p0.34-el6.parcel.sha1
manifest.json
JDBC
地址:http://download.softagency.net/MySQL/Downloads/Connector-J/
这里用的jdbc驱动版本是:http://download.softagency.net/MySQL/Downloads/Connector-J/mysql-connector-java-6.0.6.tar.gz
系统环境搭建
hadoop集群搭建准备
安装jdk7(cdh5.7不支持java8),链接地址:http://www.oracle.com/technetwork/java/javase/archive-139210.html
上传文件到/home/jdk
解压,配置Java环境变量:
#vi /etc/profile,在文件尾部添加
export JAVA_HOME=/usr/java/jdk1.7.0_80
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export CLASSPATH=:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
保存退出
#source /etc/profile
#java -version
关闭防火墙:查询防火墙状态:
service iptables status
停止防火墙:
service iptables stop
启动防火墙:
service iptables start
重启防火墙:
service iptables restart
永久关闭防火墙:
chkconfig iptables off
永久关闭后启用:
chkconfig iptables on 关闭selinux
A 不需要重启Linux:
[root@localhost ~]# setenforce 0
B 需要重启Linux:
vi /etc/selinux/config 将SELINUX=enforcing 改成SELINUX=disabled
修改ip,vi /etc/sysconfig/network-scripts/ifcfg-eth0,IPADDR=192.168.1.220,重启网络服务:service network restart,dns服务器设置为:114.114.115.110
修改hostname修改机器名
#hostname hadoop01
#vi /etc/sysconfig/network
#域名不能配置成hadoop_c01(不能有下划线,此问题让我无法启动agent服务,耽误了整整2天的时间)
HOSTNAME=hadoop01
保存退出,重启
修改/etc/hosts
192.168.1.220 hadoop01
192.168.1.221 hadoop02
192.168.1.222 hadoop03
192.168.1.223 hadoop04
设置ssh自动登录
从namenode到本身及各datanode免密码登录
在各机器上执行
#ssh-keygen -t rsa 一路回车
在~/.ssh/生成文件id_rsa id_rsa.pub
在namenode机器上执行:
#cd ~/.ssh/
#scp id_rsa.pub root@<各datanode的IP>:/home
在各datanode机器上执行:
#cd /home/
#cat /home/id_rsa.pub >>/root/.ssh/authorized_keys
修改linux swap空间的swappiness
Cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。
修改swappiness的值为零:
root@master:cat /proc/sys/vm/swappiness
root@master:sysctl vm.swappiness=0
root@master:echo 0 > /proc/sys/vm/swappiness
关闭告警:echo never > /sys/kernel/mm/transparent_hugepage/defrag
创建普通用户hadoop
创建普通账户,设置Sudo无密码权限
**创建hadoop用户**
[root@master /]# adduser hadoop
**为hadoop设置密码,提示输入**
[root@master /]# passwd hadoop
**为/etc/sudoers文件添加写入权限**
[root@master /]# chmod u+w /etc/sudoers
**编辑文件,加入代码**
[root@master /]# vim /etc/sudoers
hadoop ALL=(root)NOPASSWD:ALL
**将写入权限去除,才能sudo ifconfig**
[root@master /]# chmod u-w /etc/sudoers
**sudo ifconfig(验证是否成功!)**
[root@master /]# sudo ifconfig
输出以下为成功(有缩略):
eth0 Link encap:Ethernet HWaddr 00:0C:29:FA:CB:DE
lo Link encap:Local Loopback
在所有节点创建cloudera-scm用户
useradd --system --home=/opt/cm-5.7.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
#usermod -a -G root cloudera-scm
usermod -a -G cloudera-scm cloudera-scm
echo USER=\"cloudera-scm\" >> /etc/default/cloudera-scm-agent
echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin" >> /etc/sudoers
chown -R cloudera-scm:cloudera-scm /opt/cloudera/
chown -R cloudera-scm:cloudera-scm /opt/cm-5.7.0/
在每个主机上建立/opt/cloudera/parcels目录,并修改属主
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
sudo mkdir -p /var/log/cloudera-scm-headlamp
sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp
sudo mkdir -p /var/log/cloudera-scm-firehose
sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-firehose
sudo mkdir -p /var/log/cloudera-scm-alertpublisher
sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-alertpublisher
sudo mkdir -p /var/log/cloudera-scm-eventserver
sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-eventserver
sudo mkdir -p /var/lib/cloudera-scm-headlamp
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-headlamp
sudo mkdir -p /var/lib/cloudera-scm-firehose
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-firehose
sudo mkdir -p /var/lib/cloudera-scm-alertpublisher
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-alertpublisher
sudo mkdir -p /var/lib/cloudera-scm-eventserver
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-eventserver
sudo mkdir -p /var/lib/cloudera-scm-server
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server
安装Oracle Java(所有节点)
Linux可能自带OpenJDK,但运行CDH5需要使用Oracle的JDK,需要Java 7以上版本的支持
卸载自带的OpenJDK** 使用下述命令查询相关的java包
rpm -qa | grep java
使用-e选项填入包名卸载
rpm -e --nodeps packageName
安装oracle jdk 配置环境变量/etc/profile.再source使起生效这里不在赘述。
安装配置MySQL(主节点)
通过yum install mysql-server安装mysql服务器。
chkconfig mysqld on设置开机启动,并service mysqld start启动mysql服务,
设置root的初试密码:mysqladmin -u root password '123456'。
mysql -uroot -p进入mysql命令行,创建以下数据库:
#为hive建库hive
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#activity monitor
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#为oozie建库oozie
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#为hue建库hue
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database reports DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
设置root授权访问以上所有的数据库:
#授权root用户在主节点拥有所有数据库的访问权限
grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;
grant all privileges on *.* to 'root'@'hadoop01' identified by '123456' with grant option;
grant all privileges on *.* to 'root'@'127.0.0.1' identified by '123456' with grant option;
grant all privileges on *.* to 'root'@'localhost' identified by '123456' with grant option;
flush privileges;
配置NTP服务(所有节点)
集群中所有主机必须保持时间同步,如果时间相差较大会引起各种问题,例如主机运行状态不良等。 具体思路如下:
master节点作为ntp服务器与外界对时中心同步时间,随后对所有datanode节点提供时间同步服务。
所有datanode节点以master节点为基础同步时间。
所有节点安装相关组件:yum install ntp。完成后,配置开机启动:chkconfig ntpd on,检查是否设置成功:chkconfig --list ntpd其中2-5为on状态就代表成功。
主节点配置
在配置之前,先使用ntpdate手动同步一下时间,免得本机与对时中心时间差距太大,使得ntpd不能正常同步。这里选用103.226.213.30作为对时中心,ntpdate -u 103.226.213.30。
下面是国内最活跃的时间服务器地址:
server 0.cn.pool.ntp.org
server 0.asia.pool.ntp.org
server 3.asia.pool.ntp.org
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
ntp服务只有一个配置文件,配置好了就OK。 这里只给出有用的配置,不需要的配置都用#注掉,这里就不在给出:
vi /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1
restrict default nomodify notrap
server 103.226.213.30 prefer
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
配置文件完成,保存退出,启动服务,执行如下命令:service ntpd start
检查是否成功,用ntpstat命令查看同步状态,出现以下状态代表启动成功:
synchronised to NTP server () at stratum 2
time correct to within 74 ms
polling server every 128 s
如果出现异常请等待几分钟,一般等待5-10分钟才能同步。
配置ntp客户端(所有子节点)
vi /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
#这里是主节点的主机名或者ip
server hadoop01
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
ok保存退出,请求服务器前,请先使用ntpdate手动同步一下时间:ntpdate -u hadoop_c01(主节点ntp服务器)
这里可能出现同步失败的情况,请不要着急,一般是本地的ntp服务器还没有正常启动,一般需要等待5-10分钟才可以正常同步。启动服务:service ntpd start
因为是连接内网,这次启动等待的时间会比master节点快一些,但是也需要耐心等待一会儿
Cloudera Manager安装与集群配置
在上节已将环境配置完成。下面开始安装CM和搭建集群。
主节点安装CM
解压:tar xzvf cloudera-manager*.tar.gz将解压后的cm-5.11和cloudera目录放到/opt目录下。;
建立数据库:将下载好的mysql-connector-java-5.1.18-bin.jar放到/opt/cm-5.7.0/share/cmf/lib/;
初始化数据库: /opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123456 --scm-host
localhost scm scm scm(也可以直接使用mysql创建cm数据库,并授权scm用户完全控制cm数据库,scm用户默认密码为scm),
如果使用/opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hhadoop01 -uroot -p123456 --scm-host
hadoop01 scm scm scm,会报root无权限操作cm数据库,所以我此处使用MySQL直接创建cm数据库,不使用脚本,使用MySQL创建
cm数据库时,手动修改/opt/cm-5.7.0/etc/cloudera-scm-server/db.properties的配置为com.cloudera.cmf.db.host=hadoop01;
参数分别是:数据库类型 数据库名称 -h数据库主机名 -u数据库用户名 -p数据库密码--scm-host cmserver主机名 scm scm scm(数据库名称,数据库用户,数据库用户密码);
agent配置
修改/opt/cm-5.7.0/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名。
同步Agent到其他所有节点: scp -r -p /opt/cm-5.7.0 root@hadoop02:/opt/,scp -r -p /opt/cm-5.7.0 root@hadoop03:/opt/,scp -r -p /opt/cm-5.7.0 root@hadoop04:/opt/
在所有主机上启动cm agent
mkdir /opt/cm-5.7.0/run/cloudera-scm-agent
sudo chown -R cloudera-scm:cloudera-scm /opt/cm-5.7.0
启动agent:
/opt/cm-5.7.0/etc/init.d/cloudera-scm-agent start
在每个主机上建立/opt/cloudera/parcels目录,并修改属主
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /var/run/cloudera-scm-agent
chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-agent
chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-agent
准备Parcels,用以安装CDH5
On the Cloudera Manager Server host, create a parcel repository directory
sudo mkdir -p /opt/cloudera/parcel-repo
sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
On each cluster host, create a parcels directory
sudo mkdir -p /opt/cloudera/parcels
sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中(parcel-repo需要手动创建)。
相关的文件如下:
- CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel
- CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha1
- manifest.json
最后将CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha1,重命名为CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha,这点必须注意,否则,系统会重新下载CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel文件。
相关启动脚本
主节点:通过/opt/cm-5.7.0/etc/init.d/cloudera-scm-server start启动服务端。
(如果安装失败,则sudo /opt/cm-5.7.0/etc/init.d/cloudera-scm-server stop,删除数据库cm,再重试)
所有节点(包括主节点):通过sudo /opt/cm-5.7.0/etc/init.d/cloudera-scm-agent start启动Agent服务。
我们启动的其实是个service脚本,需要停止服务将以上的start参数改为stop就可以了,重启是restart。
CDH5安装配置
Cloudera Manager Server和Agent都启动以后,就可以进行CDH5的安装配置了。
这时可以通过浏览器访问主节点的7180端口测试一下了(由于CM Server的启动需要花点时间,这里可能要等待一会才能访问),默认的用户名和密码均为admin:
登录:
协议条款:
选择版本:
完成选择:
输入主机名:
群集安装:
安装中:
Q&A
访问需要密码的错误:
ERROR com.cloudera.enterprise.dbutil.DbProvisioner - Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://localhost/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: Access denied for user 'root'@'localhost' (using password: YES)
解决方案:
1、配置/etc/hosts文件如下:
127.0.0.1 localhost.hadoop_cloudrea01
注意:如果执行的时候报一下错误,可能是配置host的有问题,127.0.0.1 localhost.hadoop_cloudrea01 这个localhost不能少
2、
需要修改mysql登录密码:
解决:
修改MySQL的root用户的密码以及打开远程连接
mysql>use mysql;
mysql>desc user;
mysql>GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY "123456"; //为root添加远程连接的能力。
mysql>update user set Password = password('123456') where User='root';
mysql>select Host,User,Password from user where User='root';
mysql>flush privileges;
mysql>exit
用户不存在的错误:
[root@hadoop_cloudrea01 lib]# /opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hhadoop_cloudrea01 -uroot -p123456 --scm-host hadoop_cloudrea01 scm scm scm
JAVA_HOME=/home/jdk/jdk1.8.0_131
Verifying that we can write to /opt/cm-5.7.0/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cm-5.7.0/etc/cloudera-scm-server
groups: cloudera-scm: No such user
Executing: /home/jdk/jdk1.8.0_131/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.7.0/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor
/opt/cm-5.7.0/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Unable to login using supplied username/password.
[ main] DbCommandExecutor ERROR Error when connecting to database.
java.sql.SQLException: Access denied for user 'scm'@'hadoop_cloudrea01' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:943)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4113)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1308)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2336)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java:253)
at com.cloudera.enterprise.dbutil.DbCommandExecutor.main(DbCommandExecutor.java:138)
[ main] DbCommandExecutor ERROR Exiting with exit code 8
--> Error 8, giving up (use --force if you wish to ignore the error)
解决方法:
[root@master /]# adduser cloudera-scm
**为hadoop设置密码,提示输入**
[root@master /]# passwd cloudera-scm
数据用户scm不存在的错误:
[root@hadoop_cloudrea01 lib]# /opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost-uroot
-p123456 --scm-host localhost scm scm 123456
JAVA_HOME=/home/jdk/jdk1.8.0_131
Verifying that we can write to /opt/cm-5.7.0/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cm-5.7.0/etc/cloudera-scm-server
Executing: /home/jdk/jdk1.8.0_131/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.7.0/share/cmf/schema/../lib/*
com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cm-5.7.0/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Unable to login using supplied username/password.
[ main] DbCommandExecutor ERROR Error when connecting to database.
java.sql.SQLException: Access denied for user 'scm'@'localhost'
(using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:943)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4113)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1308)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2336)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java:253)
at com.cloudera.enterprise.dbutil.DbCommandExecutor.main(DbCommandExecutor.java:138)
[ main] DbCommandExecutor ERROR Exiting with exit code 8
--> Error 8, giving up (use --force if you wish to ignore the error)
解决方法:
mysql -uroot -p123456
mysql> use mysql;
mysql>CREATE USER 'scm'@'%' IDENTIFIED BY '123456';
mysql>GRANT ALL PRIVILEGES ON *.* TO scm@"%" IDENTIFIED BY "123456";
mysql>update user set Password = password('123456') where User='scm';
mysql>select Host,User,Password from user where User='scm';
mysql>flush privileges;
mysql>quit;
host mornitor 无法启动,mv /usr/bin/host /usr/bin/hostbake rm -rf - /opt/cm-5.7.0/lib/cloudera-scm-agent/
agent授权错误,无法启动,可能是由于新启动的agent服务连接到了老的supervisor,解决办法是:ps -ef | grep scm,得到进程号后,使用kill -9 进程号杀死进程,ps -ef | grep supervisor,得到进程号后,使用kill -9 进程号杀死进程,删除老agent启动记录,rm -rf /var/lib/cloudera-scm-agent/*
rm -rf - /opt/cm-5.7.0/lib/cloudera-scm-agent/,进入mysql,删除cm数据库,重新初始化数据库,启动cloudera-scm-server,能够访问web登录页面后,再启动agent,授权错误问题即可解决
cloudera-scm-server在最后进行数据库配置时挂死,原因是内存分配太少,使用strace跟踪打印如下:
参考:https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#install_cdh_file_system的Cloudera
Manager Resource Requirements,RAM至少为4G,此次因为是学习使用的hadoop环境,故将default/cloudera-scm-server的export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"改为export CMF_JAVA_OPTS="-Xmx1G
-XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp",同时将虚拟机的内存分配为3.5G,重新安装即可(经过我的测试,修改为-Xmx1G会无法启动server服务,我直接将虚拟机内存提升到了5G,最终安装成功,所以如果是为了学习而搭建的环境,使用cdh4会更好些),因为内存不够用,每次安装到配置数据库信息时,server服务就down了,找了2天才跟踪到是内存不足导致的。
参考:
http://www.cnblogs.com/codedevelop/p/6762555.html
http://www.thebigdata.cn/Hadoop/29063.html
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
http://blog.csdn.net/yhl27/article/details/37726639 (本地yum源安装CDH)
http://blog.csdn.net/wzy0623/article/details/51601989
http://blog.csdn.net/pickinfo/article/details/50488231
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_c.html
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b.html
https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_jdk(cdh安装依赖)
- 文件下载
- 系统环境搭建
- Cloudera Manager安装与集群配置
- 日志查看
- Q&A
- 参考
概述
CDH (Cloudera's Distribution, including Apache Hadoop),是Hadoop众多分支中的一种,由Cloudera维护,基于稳定版本的Apache Hadoop构建,并集成了很多补丁,可直接用于生产环境。
Cloudera Manager则是为了便于在集群中进行Hadoop等大数据处理相关的服务安装和监控管理的组件,对集群中主机、Hadoop、Hive、Spark等服务的安装配置管理做了极大简化。
文件下载
Cloudera Manager
地址:http://archive.cloudera.com/cm5/cm/5/
这里下载的是5.7.0的版本,https://archive.cloudera.com/cm5/cm/5/cloudera-manager-el6-cm5.7.0_x86_64.tar.gz
CDH安装包
地址:http://archive.cloudera.com/cdh5/parcels
本文基于CentOS6,下载的parcels包及manifest文件对应的版本为el6,如果centos7,就用el7:
CDH-5.7.0-1.cdh5.11.0.p0.34-el6.parcel
CDH-5.7.0-1.cdh5.11.0.p0.34-el6.parcel.sha1
manifest.json
JDBC
地址:http://download.softagency.net/MySQL/Downloads/Connector-J/
这里用的jdbc驱动版本是:http://download.softagency.net/MySQL/Downloads/Connector-J/mysql-connector-java-6.0.6.tar.gz
系统环境搭建
hadoop集群搭建准备
机器名 | 机器ip | 用途 |
hadoop01 | 192.168.1.220 | namenode&secondarynamenode&jobtrack |
hadoop02 | 192.168.1.221 | datanode&tasktrack |
hadoop03 | 192.168.1.222 | datanode&tasktrack |
hadoop04 | 192.168.1.223 | datanode&tasktrac |
上传文件到/home/jdk
解压,配置Java环境变量:
#vi /etc/profile,在文件尾部添加
export JAVA_HOME=/usr/java/jdk1.7.0_80
export JRE_HOME=$JAVA_HOME/jre
export PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export CLASSPATH=:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
保存退出
#source /etc/profile
#java -version
关闭防火墙:查询防火墙状态:
service iptables status
停止防火墙:
service iptables stop
启动防火墙:
service iptables start
重启防火墙:
service iptables restart
永久关闭防火墙:
chkconfig iptables off
永久关闭后启用:
chkconfig iptables on 关闭selinux
A 不需要重启Linux:
[root@localhost ~]# setenforce 0
B 需要重启Linux:
vi /etc/selinux/config 将SELINUX=enforcing 改成SELINUX=disabled
修改ip,vi /etc/sysconfig/network-scripts/ifcfg-eth0,IPADDR=192.168.1.220,重启网络服务:service network restart,dns服务器设置为:114.114.115.110
修改hostname修改机器名
#hostname hadoop01
#vi /etc/sysconfig/network
#域名不能配置成hadoop_c01(不能有下划线,此问题让我无法启动agent服务,耽误了整整2天的时间)
HOSTNAME=hadoop01
保存退出,重启
修改/etc/hosts
192.168.1.220 hadoop01
192.168.1.221 hadoop02
192.168.1.222 hadoop03
192.168.1.223 hadoop04
设置ssh自动登录
从namenode到本身及各datanode免密码登录
在各机器上执行
#ssh-keygen -t rsa 一路回车
在~/.ssh/生成文件id_rsa id_rsa.pub
在namenode机器上执行:
#cd ~/.ssh/
#scp id_rsa.pub root@<各datanode的IP>:/home
在各datanode机器上执行:
#cd /home/
#cat /home/id_rsa.pub >>/root/.ssh/authorized_keys
修改linux swap空间的swappiness
Cloudera 建议将 /proc/sys/vm/swappiness 设置为 0。
修改swappiness的值为零:
root@master:cat /proc/sys/vm/swappiness
root@master:sysctl vm.swappiness=0
root@master:echo 0 > /proc/sys/vm/swappiness
关闭告警:echo never > /sys/kernel/mm/transparent_hugepage/defrag
创建普通用户hadoop
创建普通账户,设置Sudo无密码权限
**创建hadoop用户**
[root@master /]# adduser hadoop
**为hadoop设置密码,提示输入**
[root@master /]# passwd hadoop
**为/etc/sudoers文件添加写入权限**
[root@master /]# chmod u+w /etc/sudoers
**编辑文件,加入代码**
[root@master /]# vim /etc/sudoers
hadoop ALL=(root)NOPASSWD:ALL
**将写入权限去除,才能sudo ifconfig**
[root@master /]# chmod u-w /etc/sudoers
**sudo ifconfig(验证是否成功!)**
[root@master /]# sudo ifconfig
输出以下为成功(有缩略):
eth0 Link encap:Ethernet HWaddr 00:0C:29:FA:CB:DE
lo Link encap:Local Loopback
在所有节点创建cloudera-scm用户
useradd --system --home=/opt/cm-5.7.0/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
#usermod -a -G root cloudera-scm
usermod -a -G cloudera-scm cloudera-scm
echo USER=\"cloudera-scm\" >> /etc/default/cloudera-scm-agent
echo "Defaults secure_path = /sbin:/bin:/usr/sbin:/usr/bin" >> /etc/sudoers
chown -R cloudera-scm:cloudera-scm /opt/cloudera/
chown -R cloudera-scm:cloudera-scm /opt/cm-5.7.0/
在每个主机上建立/opt/cloudera/parcels目录,并修改属主
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
sudo mkdir -p /var/log/cloudera-scm-headlamp
sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-headlamp
sudo mkdir -p /var/log/cloudera-scm-firehose
sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-firehose
sudo mkdir -p /var/log/cloudera-scm-alertpublisher
sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-alertpublisher
sudo mkdir -p /var/log/cloudera-scm-eventserver
sudo chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-eventserver
sudo mkdir -p /var/lib/cloudera-scm-headlamp
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-headlamp
sudo mkdir -p /var/lib/cloudera-scm-firehose
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-firehose
sudo mkdir -p /var/lib/cloudera-scm-alertpublisher
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-alertpublisher
sudo mkdir -p /var/lib/cloudera-scm-eventserver
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-eventserver
sudo mkdir -p /var/lib/cloudera-scm-server
sudo chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-server
安装Oracle Java(所有节点)
Linux可能自带OpenJDK,但运行CDH5需要使用Oracle的JDK,需要Java 7以上版本的支持
卸载自带的OpenJDK** 使用下述命令查询相关的java包
rpm -qa | grep java
使用-e选项填入包名卸载
rpm -e --nodeps packageName
安装oracle jdk 配置环境变量/etc/profile.再source使起生效这里不在赘述。
安装配置MySQL(主节点)
通过yum install mysql-server安装mysql服务器。
chkconfig mysqld on设置开机启动,并service mysqld start启动mysql服务,
设置root的初试密码:mysqladmin -u root password '123456'。
mysql -uroot -p进入mysql命令行,创建以下数据库:
#为hive建库hive
create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#activity monitor
create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#为oozie建库oozie
create database oozie DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
#为hue建库hue
create database hue DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
create database reports DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
设置root授权访问以上所有的数据库:
#授权root用户在主节点拥有所有数据库的访问权限
grant all privileges on *.* to 'root'@'%' identified by '123456' with grant option;
grant all privileges on *.* to 'root'@'hadoop01' identified by '123456' with grant option;
grant all privileges on *.* to 'root'@'127.0.0.1' identified by '123456' with grant option;
grant all privileges on *.* to 'root'@'localhost' identified by '123456' with grant option;
flush privileges;
配置NTP服务(所有节点)
集群中所有主机必须保持时间同步,如果时间相差较大会引起各种问题,例如主机运行状态不良等。 具体思路如下:
master节点作为ntp服务器与外界对时中心同步时间,随后对所有datanode节点提供时间同步服务。
所有datanode节点以master节点为基础同步时间。
所有节点安装相关组件:yum install ntp。完成后,配置开机启动:chkconfig ntpd on,检查是否设置成功:chkconfig --list ntpd其中2-5为on状态就代表成功。
主节点配置
在配置之前,先使用ntpdate手动同步一下时间,免得本机与对时中心时间差距太大,使得ntpd不能正常同步。这里选用103.226.213.30作为对时中心,ntpdate -u 103.226.213.30。
下面是国内最活跃的时间服务器地址:
server 0.cn.pool.ntp.org
server 0.asia.pool.ntp.org
server 3.asia.pool.ntp.org
server 0.centos.pool.ntp.org iburst
server 1.centos.pool.ntp.org iburst
server 2.centos.pool.ntp.org iburst
server 3.centos.pool.ntp.org iburst
ntp服务只有一个配置文件,配置好了就OK。 这里只给出有用的配置,不需要的配置都用#注掉,这里就不在给出:
vi /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1
restrict default nomodify notrap
server 103.226.213.30 prefer
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
配置文件完成,保存退出,启动服务,执行如下命令:service ntpd start
检查是否成功,用ntpstat命令查看同步状态,出现以下状态代表启动成功:
synchronised to NTP server () at stratum 2
time correct to within 74 ms
polling server every 128 s
如果出现异常请等待几分钟,一般等待5-10分钟才能同步。
配置ntp客户端(所有子节点)
vi /var/lib/ntp/drift
restrict 127.0.0.1
restrict -6 ::1
restrict default kod nomodify notrap nopeer noquery
restrict -6 default kod nomodify notrap nopeer noquery
#这里是主节点的主机名或者ip
server hadoop01
includefile /etc/ntp/crypto/pw
keys /etc/ntp/keys
ok保存退出,请求服务器前,请先使用ntpdate手动同步一下时间:ntpdate -u hadoop_c01(主节点ntp服务器)
这里可能出现同步失败的情况,请不要着急,一般是本地的ntp服务器还没有正常启动,一般需要等待5-10分钟才可以正常同步。启动服务:service ntpd start
因为是连接内网,这次启动等待的时间会比master节点快一些,但是也需要耐心等待一会儿
Cloudera Manager安装与集群配置
在上节已将环境配置完成。下面开始安装CM和搭建集群。
主节点安装CM
解压:tar xzvf cloudera-manager*.tar.gz将解压后的cm-5.11和cloudera目录放到/opt目录下。;
建立数据库:将下载好的mysql-connector-java-5.1.18-bin.jar放到/opt/cm-5.7.0/share/cmf/lib/;
初始化数据库: /opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost -uroot -p123456 --scm-host
localhost scm scm scm(也可以直接使用mysql创建cm数据库,并授权scm用户完全控制cm数据库,scm用户默认密码为scm),
如果使用/opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hhadoop01 -uroot -p123456 --scm-host
hadoop01 scm scm scm,会报root无权限操作cm数据库,所以我此处使用MySQL直接创建cm数据库,不使用脚本,使用MySQL创建
cm数据库时,手动修改/opt/cm-5.7.0/etc/cloudera-scm-server/db.properties的配置为com.cloudera.cmf.db.host=hadoop01;
参数分别是:数据库类型 数据库名称 -h数据库主机名 -u数据库用户名 -p数据库密码--scm-host cmserver主机名 scm scm scm(数据库名称,数据库用户,数据库用户密码);
agent配置
修改/opt/cm-5.7.0/etc/cloudera-scm-agent/config.ini中的server_host为主节点的主机名。
同步Agent到其他所有节点: scp -r -p /opt/cm-5.7.0 root@hadoop02:/opt/,scp -r -p /opt/cm-5.7.0 root@hadoop03:/opt/,scp -r -p /opt/cm-5.7.0 root@hadoop04:/opt/
在所有主机上启动cm agent
mkdir /opt/cm-5.7.0/run/cloudera-scm-agent
sudo chown -R cloudera-scm:cloudera-scm /opt/cm-5.7.0
启动agent:
/opt/cm-5.7.0/etc/init.d/cloudera-scm-agent start
在每个主机上建立/opt/cloudera/parcels目录,并修改属主
mkdir -p /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
chown cloudera-scm:cloudera-scm /var/run/cloudera-scm-agent
chown cloudera-scm:cloudera-scm /var/lib/cloudera-scm-agent
chown cloudera-scm:cloudera-scm /var/log/cloudera-scm-agent
准备Parcels,用以安装CDH5
On the Cloudera Manager Server host, create a parcel repository directory
sudo mkdir -p /opt/cloudera/parcel-repo
sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
On each cluster host, create a parcels directory
sudo mkdir -p /opt/cloudera/parcels
sudo chown cloudera-scm:cloudera-scm /opt/cloudera/parcels
将CHD5相关的Parcel包放到主节点的/opt/cloudera/parcel-repo/目录中(parcel-repo需要手动创建)。
相关的文件如下:
- CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel
- CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha1
- manifest.json
最后将CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha1,重命名为CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel.sha,这点必须注意,否则,系统会重新下载CDH-5.7.0-1.cdh5.7.0.p0.45-el6.parcel文件。
相关启动脚本
主节点:通过/opt/cm-5.7.0/etc/init.d/cloudera-scm-server start启动服务端。
(如果安装失败,则sudo /opt/cm-5.7.0/etc/init.d/cloudera-scm-server stop,删除数据库cm,再重试)
所有节点(包括主节点):通过sudo /opt/cm-5.7.0/etc/init.d/cloudera-scm-agent start启动Agent服务。
我们启动的其实是个service脚本,需要停止服务将以上的start参数改为stop就可以了,重启是restart。
CDH5安装配置
Cloudera Manager Server和Agent都启动以后,就可以进行CDH5的安装配置了。
这时可以通过浏览器访问主节点的7180端口测试一下了(由于CM Server的启动需要花点时间,这里可能要等待一会才能访问),默认的用户名和密码均为admin:
登录:
协议条款:
选择版本:
完成选择:
输入主机名:
群集安装:
安装中:
Q&A
访问需要密码的错误:
ERROR com.cloudera.enterprise.dbutil.DbProvisioner - Exception when creating/dropping database with user 'root' and jdbc url 'jdbc:mysql://localhost/?useUnicode=true&characterEncoding=UTF-8'
java.sql.SQLException: Access denied for user 'root'@'localhost' (using password: YES)
解决方案:
1、配置/etc/hosts文件如下:
127.0.0.1 localhost.hadoop_cloudrea01
注意:如果执行的时候报一下错误,可能是配置host的有问题,127.0.0.1 localhost.hadoop_cloudrea01 这个localhost不能少
2、
需要修改mysql登录密码:
解决:
修改MySQL的root用户的密码以及打开远程连接
mysql>use mysql;
mysql>desc user;
mysql>GRANT ALL PRIVILEGES ON *.* TO root@"%" IDENTIFIED BY "123456"; //为root添加远程连接的能力。
mysql>update user set Password = password('123456') where User='root';
mysql>select Host,User,Password from user where User='root';
mysql>flush privileges;
mysql>exit
用户不存在的错误:
[root@hadoop_cloudrea01 lib]# /opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hhadoop_cloudrea01 -uroot -p123456 --scm-host hadoop_cloudrea01 scm scm scm
JAVA_HOME=/home/jdk/jdk1.8.0_131
Verifying that we can write to /opt/cm-5.7.0/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cm-5.7.0/etc/cloudera-scm-server
groups: cloudera-scm: No such user
Executing: /home/jdk/jdk1.8.0_131/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.7.0/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor
/opt/cm-5.7.0/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Unable to login using supplied username/password.
[ main] DbCommandExecutor ERROR Error when connecting to database.
java.sql.SQLException: Access denied for user 'scm'@'hadoop_cloudrea01' (using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:943)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4113)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1308)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2336)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java:253)
at com.cloudera.enterprise.dbutil.DbCommandExecutor.main(DbCommandExecutor.java:138)
[ main] DbCommandExecutor ERROR Exiting with exit code 8
--> Error 8, giving up (use --force if you wish to ignore the error)
解决方法:
[root@master /]# adduser cloudera-scm
**为hadoop设置密码,提示输入**
[root@master /]# passwd cloudera-scm
数据用户scm不存在的错误:
[root@hadoop_cloudrea01 lib]# /opt/cm-5.7.0/share/cmf/schema/scm_prepare_database.sh mysql cm -hlocalhost-uroot
-p123456 --scm-host localhost scm scm 123456
JAVA_HOME=/home/jdk/jdk1.8.0_131
Verifying that we can write to /opt/cm-5.7.0/etc/cloudera-scm-server
Creating SCM configuration file in /opt/cm-5.7.0/etc/cloudera-scm-server
Executing: /home/jdk/jdk1.8.0_131/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/opt/cm-5.7.0/share/cmf/schema/../lib/*
com.cloudera.enterprise.dbutil.DbCommandExecutor /opt/cm-5.7.0/etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[ main] DbCommandExecutor INFO Unable to login using supplied username/password.
[ main] DbCommandExecutor ERROR Error when connecting to database.
java.sql.SQLException: Access denied for user 'scm'@'localhost'
(using password: YES)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3609)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3541)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:943)
at com.mysql.jdbc.MysqlIO.secureAuth411(MysqlIO.java:4113)
at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1308)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2336)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at com.cloudera.enterprise.dbutil.DbCommandExecutor.testDbConnection(DbCommandExecutor.java:253)
at com.cloudera.enterprise.dbutil.DbCommandExecutor.main(DbCommandExecutor.java:138)
[ main] DbCommandExecutor ERROR Exiting with exit code 8
--> Error 8, giving up (use --force if you wish to ignore the error)
解决方法:
mysql -uroot -p123456
mysql> use mysql;
mysql>CREATE USER 'scm'@'%' IDENTIFIED BY '123456';
mysql>GRANT ALL PRIVILEGES ON *.* TO scm@"%" IDENTIFIED BY "123456";
mysql>update user set Password = password('123456') where User='scm';
mysql>select Host,User,Password from user where User='scm';
mysql>flush privileges;
mysql>quit;
host mornitor 无法启动,mv /usr/bin/host /usr/bin/hostbake rm -rf - /opt/cm-5.7.0/lib/cloudera-scm-agent/
agent授权错误,无法启动,可能是由于新启动的agent服务连接到了老的supervisor,解决办法是:ps -ef | grep scm,得到进程号后,使用kill -9 进程号杀死进程,ps -ef | grep supervisor,得到进程号后,使用kill -9 进程号杀死进程,删除老agent启动记录,rm -rf /var/lib/cloudera-scm-agent/*
rm -rf - /opt/cm-5.7.0/lib/cloudera-scm-agent/,进入mysql,删除cm数据库,重新初始化数据库,启动cloudera-scm-server,能够访问web登录页面后,再启动agent,授权错误问题即可解决
cloudera-scm-server在最后进行数据库配置时挂死,原因是内存分配太少,使用strace跟踪打印如下:
参考:https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#install_cdh_file_system的Cloudera
Manager Resource Requirements,RAM至少为4G,此次因为是学习使用的hadoop环境,故将default/cloudera-scm-server的export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp"改为export CMF_JAVA_OPTS="-Xmx1G
-XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp",同时将虚拟机的内存分配为3.5G,重新安装即可(经过我的测试,修改为-Xmx1G会无法启动server服务,我直接将虚拟机内存提升到了5G,最终安装成功,所以如果是为了学习而搭建的环境,使用cdh4会更好些),因为内存不够用,每次安装到配置数据库信息时,server服务就down了,找了2天才跟踪到是内存不足导致的。
参考:
http://www.cnblogs.com/codedevelop/p/6762555.html
http://www.thebigdata.cn/Hadoop/29063.html
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html
http://blog.csdn.net/yhl27/article/details/37726639 (本地yum源安装CDH)
http://blog.csdn.net/wzy0623/article/details/51601989
http://blog.csdn.net/pickinfo/article/details/50488231
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_c.html
https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_install_path_b.html
https://www.cloudera.com/documentation/enterprise/release-notes/topics/rn_consolidated_pcm.html#pcm_jdk(cdh安装依赖)
相关文章推荐
- 使用Cloudera部署,管理Hadoop集群
- 离线部署 Cloudera Manager 5 和 CDH 5.12.1 及使用 CDH 部署 Hadoop 集群服务
- 使用Cloudera部署,管理Hadoop集群
- Hadoop系列之(三):使用Cloudera部署,管理Hadoop集群
- Hadoop-Linux集群批量部署管理工具parallel-ssh(PSSH)的安装与使用
- 使用Cloudera部署,管理Hadoop集群
- hadoop基础----hadoop实战(七)-----hadoop管理工具---使用Cloudera Manager安装Hadoop---Cloudera Manager和CDH5.8离线安装
- hadoop基础----hadoop实战(七)-----hadoop管理工具---使用Cloudera Manager安装Hadoop---Cloudera Manager和CDH5.8离线安装
- 离线部署 Cloudera Manager 5 和 CDH 5.12.1 及使用 CDH 部署 Hadoop 集群服务
- Hadoop建设工具Ambari的安装部署及完整使用(五)——Ambari使用之集群卸载
- 离线部署 CDH 5.12.1 及使用 CDH 部署 Hadoop 大数据平台集群服务
- Cloudera Manager 5 和 CDH5 本地(离线)安装指南(Hadoop集群搭建)
- Cloudera Manager 5 和 CDH5 本地(离线)安装指南(Hadoop集群搭建)
- Linux 下Redis集群安装部署及使用详解(在线和离线两种安装+相关错误解决方案)
- 离线部署 CDH 5.12.1 及使用 CDH 部署 Hadoop 大数据平台集群服务
- Cloudera Hadoop 4 实战课程(Hadoop 2.0、集群界面化管理、电商在线查询+日志离线分析
- 使用expect命令实现远程管理集群和一键安装Hadoop集群
- Cloudera Hadoop 4 实战课程(Hadoop 2.0、集群界面化管理、电商在线查询+日志离线分析)
- Cloudera Manager 5 和 CDH5 本地(离线)安装指南(Hadoop集群搭建)
- 离线安装cloudera manager server和hadoop集群