hbase+hadoop完全分布式环境搭建
2014-12-28 14:52
1141 查看
1、环境准备
linux系统:Centos6.4 release (Final) 下载地址:http://www.centos.org/jdk:1.7.0_45 下载地址:http://www.oracle.com/technetwork/java/javase/downloads/index.html
hadoop:hadoop-2.6.0 下载地址:http://apache.claz.org/hadoop/common/stable2/
hbase:hbase-0.98.8-hadoop2 下载地址:http://mirrors.gigenet.com/apache/hbase/stable/
2、配置步骤:
2.1 安装jdk,别忘了修改 vim /etc/profile。
2.2 ssh免密码登陆
说明:由于jdk的环境配置以及ssh面密码登陆一般不出什么问题,网上资料较多,故在此不做赘述2.3 hadoop环境配置
本机分布式服务器ip30:lingcloud30 (由于本台服务器配置相对较低,故用其作为NameNode,也是SecondaryNameNode。也是下面Hbase配置的HMaster)
ip29:lingcloud29 (本台以及下面的两台作为DateNode。以及下面Hbase配置的HRegionServer)
ip31:lingcloud31
ip32:lingcloud32
2.3.1{hadoop}/etc/hadoop/core-site.xml配置
下面是我的lingcloud30的配置,有的有注释,方便理解<code=xml>
<prename="code" class="html"><configuration> <!-- this file must be the same with the $hbase$/conf/ "hbase.rootdir"property --> <property> <name>fs.defaultFS</name> <value>hdfs://lingcloud30:9000</value> </property> <!-- if the HMaster start servel seconds andthen HMaster server abort ,you should remove the file hadooptmp and restart thehbase --> <property> <name>hadoop.tmp.dir</name> <value>/usr/qhl/hadoopWorkspace/hadooptmp</value> </property> <property> <name>io.file.buffer.size</name> <value>131072</value> </property> <!--添加 httpfs 的 选项--> <property> <name>hadoop.proxyuser.root.hosts</name> <value>lingcloud30</value> </property> <property> <name>hadoop.proxyuser.root.groups</name> <value>*</value> </property> </configuration>
</code>
2.3.2{hadoop}/etc/hadoop/hdfs-site.xml配置
<code=xml><prename="code" class="html"><configuration> <property> <name>dfs.datanode.handler.count</name> <value>5</value> <description>The number ofserver threads for the datanode.</description> </property> <property> <name>dfs.namenode.handler.count</name> <value>5</value> <description>The number ofserver threads for the namenode.</description> </property> <property> <name>dfs.replication</name> <value>3</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>file:/usr/qhl/hadoopWorkspace/hdfs/name</value> <final>true</final> </property> <property> <name>dfs.permissions</name> <value>false</value> </property> <property> <name>dfs.federation.nameservice.id</name> <value>ns1</value> </property> <property> <name>dfs.namenode.backup.address.ns1</name> <value>lingcloud30:50100</value> </property> <property> <name>dfs.namenode.backup.http-address.ns1</name> <value>lingcloud30:50105</value> </property> <property> <name>dfs.federation.nameservices</name> <value>ns1</value> </property> <property> <name>dfs.namenode.rpc-address.ns1</name> <value>lingcloud30:9000</value> </property> <property> <name>dfs.namenode.rpc-address.ns2</name> <value>lingcloud30:9000</value> </property> <property> <name>dfs.namenode.http-address.ns1</name> <value>lingcloud30:23001</value> <!-- this is the web browsre port if you browsre http://lingcloud30:23001. you'll see some information --> </property> <property> <name>dfs.namenode.http-address.ns2</name> <value>lingcloud30:13001</value> </property> <property> <name>dfs.dataname.data.dir</name> <value>file:/usr/qhl/hadoopWorkspace/hdfs/data</value> <final>true</final> </property> <property> <name>dfs.namenode.secondary.http-address.ns1</name> <value>lingcloud30:23002</value> </property> <property> <name>dfs.namenode.secondary.http-address.ns2</name> <value>lingcloud30:23002</value> </property> <property> <name>dfs.namenode.secondary.http-address.ns1</name> <value>lingcloud30:23003</value> </property> <property> <name>dfs.namenode.secondary.http-address.ns2</name> <value>lingcloud30:23003</value> </property> <property> <name>dfs.datanode.max.xcievers</name> <value>8192</value> </property> </configuration>
</code>
2.3.3{hadoop}/etc/hadoop/yarn-site.xml配置
<code=xml><configuration> <property> <name>yarn.resourcemanager.address</name> <value>lingcloud30:18040</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>lingcloud30:18030</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>lingcloud30:18088</value> <!-- this is the web browsre port, ifyou browsre http://lingcloud30:18088, you'll see some information --> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>lingcloud30:18025</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>lingcloud30:18141</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce.shuffle</value> </property> </configuration>
</code>
2.3.4{hadoop}/etc/hadoop/mapred-env.sh配置
添加<code=xml>
exportHADOOP_MAPRED_PID_DIR=/usr/qhl/hadoopWorkspace/haddopMapredPidDir # The pidfiles are stored. /tmp by default.
</code>
2.3.5{hadoop}/etc/hadoop/slaves配置
<code=xml>lingcloud32 lingcloud31 lingcloud29
</code>
2.3.6 小结
以上配置复制到其他node节点,按照相同配置即可。3.hbase配置
3.1 habs-env.sh环境配置
<code=xml>exportJAVA_HOME=/usr/lib/jdk/jdk1.7.0_45/ #你的jdk安装目录 exportHBASE_PID_DIR=/usr/qhl/hbaseWorkspace/pids # The directory where pid files arestored. /tmp by default. exportHBASE_MANAGES_ZK=true #使用hhase自带的zookeeper
</code>
3.2 hbase-site.xml配置
<code=xml><configuration> <property> <name>hbase.rootdir</name> <value>hdfs://lingcloud30:9000/hbase</value> <!-- this must be ths same with the{hadoop}/etc/hadoop:core-site.xml <property> <name>fs.defaultFS</name> <value>hdfs://lingcloud30:9000</value> </property> --> </property> <property> <name>hbase.zookeeper.property.dataD c135 ir</name> <value>/usr/qhl/zookeeper</value> </property> <property> <name>hbase.cluster.distributed</name> <value>true</value> <description>which directsHBase to run in distributed mode, with one JVM instance perdaemon.</description> </property> <property> <name>hbase.tmp.dir</name> <value>/usr/qhl/hbaseWorkspace/hbasetmp</value> <!-- if the HMaster start servel seconds andthen HMaster server abort ,you should remove the file hadooptmp and restart thehbase --> </property> <property> <name>hbase.zookeeper.quorum</name> <value>lingcloud29,lingcloud31,lingcloud32</value> </property> <property> <name>hbase.master</name> <value>lingcloud30:60000</value> </property> <property> <name>hbase.master.port</name> <value>60000</value> <description>The portmaster should bind to.</description> </property> <property> <name>hbase.master.maxclockskew</name> <value>200000</value> <description>Time difference ofregionserver from master</description> </property> </configuration>
</code>
3.3 regionserver 配置
<code=xml>lingcloud29 lingcloud31 lingcloud32
</code>
4.配置成功后浏览器截图
我在{hadoop}etc/hadoop下的hdfs-site.xml中配置了端口为23001. 配置项是dfs.namenode.http-address.ns1访问18088端口会出现如下界面:
hbase默认端口为60010
5.总结和参考文献
本人必须承认,很多东西都是从网上学到的,参考了很多的博文,也遇到了很多的错误,但是由于日志中间没有保存,错误就不贴了。同时由于参考了很多的博文,在此也不能一一列举了,本篇博文是我的第一篇,有什么不对之处,还请批评指正,共同学习。下面是参考文献:
1.hadoop的API,个人感觉资料还不错:http://hadoop.apache.org/docs/current/api/overview-summary.html#overview_description
2.hbase官方配置。 http://hbase.apache.org/book/configuration.html 3. hadoop官方配置。 http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html 4.参考博文。http://www.cnblogs.com/scotoma/archive/2012/09/18/2689902.html
5.参考博文。 http://www.cnblogs.com/ringwang/p/3623149.html 6.参考博文。http://www.linuxidc.com/Linux/2012-12/76947p6.htm
7.参考博文。http://dongxicheng.org/mapreduce-nextgen/apache-hadoop-2-0-alpha/
8.参考博文。http://www.cnblogs.com/xia520pi/archive/2012/05/16/2503949.html
9.hbase官方文档汉语版。http://abloz.com/hbase/book.html#hbase_default_configurations
相关文章推荐
- hadoop+hbase+zookeeper完全分布式环境搭建
- Hadoop-04-HBase完全分布式环境搭建
- HBase0.98 + Hadoop2.6 Fully Distributed 完全分布式环境搭建
- hadoop完全分布式环境搭建,整合zookeeper,hbase,spark,hive,hue
- Hadoop Hbase完全分布式环境搭建
- Hadoop-1.1.2、HBase-0.94.7完全分布式集群搭建
- hadoop+hive-完全分布式环境搭建
- (csdn内转载)hbase完全分布式环境搭建
- hbase完全分布式环境搭建
- Hadoop-2.4.1完全分布式环境搭建
- hadoop+hbase完全分布式环境搭建
- [hadoop]hadoop2.6完全分布式环境搭建
- hadoop,zookeeper,hbase搭建完全分布式集群回忆录
- hadoop完全分布式环境搭建
- 基于hadoop+nutch+solr的搜索引擎环境搭载<一>hadoop完全分布式环境搭建
- VMWare下快速搭建Hadoop完全分布式环境
- Hadoop-2.3.0-cdh5.0.1完全分布式环境搭建(NameNode,ResourceManager HA)
- 基于WindowsXP环境的Hadoop完全分布式环境的搭建
- 在oracle Virtual Box 虚拟机中搭建hadoop1.2.1完全分布式环境(转自康哥的博客)
- 分布式集群环境hadoop1.2.0、hbase0.94.4、zookeeper、elasticsearch搭建2