hadoop的HA实现,超详细(一)
2016-09-22 09:04
447 查看
网上的转载,但超详细http://blog.csdn.net/xiaojin21cen/article/details/42611073我主要关注第三个问题,他们是主、备关系,但是datanode必须要求一致的吧问题导读:
1、hadoop ha是通过什么配置实现自动切换的?
2、配置中mapred与mapreduce的区别是什么?
3、hadoop ha两个namenode之间的关系是什么?
-- hadoop 版本:2.4.0
-- 安装包名:
hadoop-2.4.0.tar.gz 或者源码版本 hadoop-2.4.0-src.tar.gz(我hadoop、hbase、hive均是用的源码编译安装)
-- 安装参考: http://www.netfoucs.com/article/book_mmicky/79985.html http://www.byywee.com/page/M0/S934/934356.html http://www.itpub.net/thread-1631536-1-1.html http://demo.netfoucs.com/u014393917/article/details/25913363 http://www.aboutyun.com/thread-8294-1-1.html
-- 找不到本地库
参考:http://www.ercoppa.org/Linux-Com ... -hadoop-library.htm
-- lzo支持,
参考:http://blog.csdn.net/zhangzhaokun/article/details/17595325 http://slaytanic.blog.51cto.com/2057708/1162287/ http://hi.baidu.com/qingchunranzhi/item/3662ed5ed29d37a1adc85709
-- 安装以下RPM包:
yum -y install openssh*
yum -y install man*
yum -y install compat-libstdc++-33*
yum -y install libaio-0.*
yum -y install libaio-devel*
yum -y install sysstat-9.*
yum -y install glibc-2.*
yum -y install glibc-devel-2.* glibc-headers-2.*
yum -y install ksh-2*
yum -y install libgcc-4.*
yum -y install libstdc++-4.*
yum -y install libstdc++-4.*.i686*
yum -y install libstdc++-devel-4.*
yum -y install gcc-4.*x86_64*
yum -y install gcc-c++-4.*x86_64*
yum -y install elfutils-libelf-0*x86_64* elfutils-libelf-devel-0*x86_64*
yum -y install elfutils-libelf-0*i686* elfutils-libelf-devel-0*i686*
yum -y install libtool-ltdl*i686*
yum -y install ncurses*i686*
yum -y install ncurses*
yum -y install readline*
yum -y install unixODBC*
yum -y install zlib
yum -y install zlib*
yum -y install openssl*
yum -y install patch
yum -y install git
yum -y -y install lzo-devel zlib-devel gcc autoconf automake libtool
yum -y install lzop
yum -y install lrzsz
yum -y -y install lzo-devel zlib-devel gcc autoconf automake libtool
yum -y install nc
yum -y install glibc
yum -y install java-1.7.0-openjdk
yum -y install gzip
yum -y install zlib
yum -y install gcc
yum -y install gcc-c++
yum -y install make
yum -y install protobuf
yum -y install protoc
yum -y install cmake
yum -y install openssl-devel
yum -y install ncurses-devel
yum -y install unzip
yum -y install telnet
yum -y install telnet-server
yum -y install wget
yum -y install svn
yum -y install ntpdate
-- hive 安装,参考:http://kicklinux.com/hive-deploy/
-- 配置Linux、安装JDK
--参考:linux(ubuntu)安装Java jdk环境变量设置及小程序测试
linux(ubuntu)无密码互通、相互登录高可靠文档
CentOS6.4之图解SSH无验证双向登陆配置
-- Step 2. zookeeper配置(配置奇数台zk集群,我用的5台)
-- 参考:Zookeeper集群环境安装过程详解
1、hadoop ha是通过什么配置实现自动切换的?
2、配置中mapred与mapreduce的区别是什么?
3、hadoop ha两个namenode之间的关系是什么?
-- hadoop 版本:2.4.0
-- 安装包名:
hadoop-2.4.0.tar.gz 或者源码版本 hadoop-2.4.0-src.tar.gz(我hadoop、hbase、hive均是用的源码编译安装)
-- 安装参考: http://www.netfoucs.com/article/book_mmicky/79985.html http://www.byywee.com/page/M0/S934/934356.html http://www.itpub.net/thread-1631536-1-1.html http://demo.netfoucs.com/u014393917/article/details/25913363 http://www.aboutyun.com/thread-8294-1-1.html
-- 找不到本地库
参考:http://www.ercoppa.org/Linux-Com ... -hadoop-library.htm
-- lzo支持,
参考:http://blog.csdn.net/zhangzhaokun/article/details/17595325 http://slaytanic.blog.51cto.com/2057708/1162287/ http://hi.baidu.com/qingchunranzhi/item/3662ed5ed29d37a1adc85709
-- 安装以下RPM包:
yum -y install openssh*
yum -y install man*
yum -y install compat-libstdc++-33*
yum -y install libaio-0.*
yum -y install libaio-devel*
yum -y install sysstat-9.*
yum -y install glibc-2.*
yum -y install glibc-devel-2.* glibc-headers-2.*
yum -y install ksh-2*
yum -y install libgcc-4.*
yum -y install libstdc++-4.*
yum -y install libstdc++-4.*.i686*
yum -y install libstdc++-devel-4.*
yum -y install gcc-4.*x86_64*
yum -y install gcc-c++-4.*x86_64*
yum -y install elfutils-libelf-0*x86_64* elfutils-libelf-devel-0*x86_64*
yum -y install elfutils-libelf-0*i686* elfutils-libelf-devel-0*i686*
yum -y install libtool-ltdl*i686*
yum -y install ncurses*i686*
yum -y install ncurses*
yum -y install readline*
yum -y install unixODBC*
yum -y install zlib
yum -y install zlib*
yum -y install openssl*
yum -y install patch
yum -y install git
yum -y -y install lzo-devel zlib-devel gcc autoconf automake libtool
yum -y install lzop
yum -y install lrzsz
yum -y -y install lzo-devel zlib-devel gcc autoconf automake libtool
yum -y install nc
yum -y install glibc
yum -y install java-1.7.0-openjdk
yum -y install gzip
yum -y install zlib
yum -y install gcc
yum -y install gcc-c++
yum -y install make
yum -y install protobuf
yum -y install protoc
yum -y install cmake
yum -y install openssl-devel
yum -y install ncurses-devel
yum -y install unzip
yum -y install telnet
yum -y install telnet-server
yum -y install wget
yum -y install svn
yum -y install ntpdate
-- hive 安装,参考:http://kicklinux.com/hive-deploy/
5台服务器设计图
IP地址 | 主机名 | NameNode | JournalNode | DataNode | Zookeeper | Hbase | Hive |
---|---|---|---|---|---|---|---|
192.168.117.194 | funshion-hadoop194 | 是 | 是 | 否 | 是 | 是 | 否 |
192.168.117.195 | funshion-hadoop195 | 是 | 是 | 否 | 是 | 是 | 否 |
192.168.117.196 | funshion-hadoop196 | 否 | 是 | 是 | 是 | 是(Master) | 是(Mysql) |
192.168.117.197 | funshion-hadoop197 | 否 | 是 | 是 | 是 | 是 | 否 |
192.168.117.198 | funshion-hadoop198 | 否 | 是 | 是 | 是 | 是 | 否 |
--参考:linux(ubuntu)安装Java jdk环境变量设置及小程序测试
-- Step 1. 建立用户hadoop的ssh无密码登陆
--参考:linux(ubuntu)无密码互通、相互登录高可靠文档
CentOS6.4之图解SSH无验证双向登陆配置
-- Step 2. zookeeper配置(配置奇数台zk集群,我用的5台)
-- 参考:Zookeeper集群环境安装过程详解
相关文章推荐
- Apache Hadoop 2.8 完全分布式集群搭建超详细过程,实现NameNode HA、ResourceManager HA高可靠性
- hadoop的HA实现,超详细
- 【干货】Apache Hadoop 2.8 完全分布式集群搭建超详细过程,实现NameNode HA、ResourceManager HA高可靠性...
- 【干货】Apache Hadoop 2.8 完全分布式集群搭建超详细过程,实现NameNode HA、ResourceManager HA高可靠性
- 详细讲解hadoop2的automatic HA+Federation+Yarn配置的教程
- Hadoop-2.7.0 HA实现细节分析之HealthMonitor
- 基于zookeeper的hadoop HA实现
- Hadoop2.2.0 HA + Jdk1.8.0 + Zookeeper3.4.5 + Hbase0.98 集群搭建详细过程(服务器集群)
- Hadoop-2.7.0中HDFS NameNode HA实现之DFSZKFailoverController、ZKFailoverController(一)
- hadoop2.x通过Zookeeper来实现namenode的HA方案以及ResourceManager单点故障的解决方案
- Hadoop2.2.0+HA+zookeeper3.4.5详细配置过程+错误处理(二)
- 详细探究Spark的shuffle实现和hadoop mapreduce shuffle原理
- hadoop2.x通过Zookeeper来实现namenode的HA方案以及ResourceManager单点故障的解决方案
- 【Hadoop】Hadoop HA 部署 详细过程(架构、机器规划、配置文件、部署步骤)
- hadoop 2.X HA详细配置
- 最详细的hadoop2.2.0集群的HA高可靠的最简单配置
- 国内最全最详细的hadoop2.2.0集群的HA高可靠的最简单配置
- Hadoop2.6.4 HA 高可用分布式集群安装配置详细步骤
- Hadoop HDFS通过QJM实现高可用HA环境搭建
- Centos 6.5 下hadoop2.5.2的HA集群原理讲解以及详细配置(手动切换)