您的位置：首页 > 运维架构

Hadoop2.5.1集群安装

2014-11-27 17:07 316 查看

首先准备环境测试使用3台rebhatlinux5.4VM中安装Linux 计算机名分别为LinuxMainLinuxNode1LinuxNode2Hadoop2.5.1版本第一步：配置三台linux的IP地址。不懂可以参照下面链接

http://blog.163.com/xiao_long/blog/static/2176511742014928111355143

第二步：修改计算机名

可能有人会问为什么要修改计算机名称，因为每一个登录的时候都是叫localhost这样不方便区分是那台机器hal/host.confhostshosts.allowhosts.deny[root@linuxmain~]#vi/etc/hosts#Donotremovethefollowingline,orvariousprograms#thatrequirenetworkfunctionalitywillfail.127.0.0.1localhost.localdomainlocalhost::1localhost6.localdomain6localhost6192.168.1.216LinuxMain192.168.1.217LinuxNode1192.168.1.218LinuxNode2~Type:quit<Enter>toexitVim把你的集群的所有IP和计算机名称都写在这完了之后使用wq!保存reboot重新启动第三步：安装jdk不会可以参照

http://blog.163.com/xiao_long/blog/static/2176511742014101152635471

第四步：解压hadoop粗体的都是需要配置的

[root@linuxmainusr]#tar-zxvfhadoop-2.5.1.tar.gz[root@linuxmainusr]#cdhadoop/etc/hadoop/[root@linuxmainhadoop]#[root@linuxmainhadoop]#lscapacity-scheduler.xmlhttpfs-site.xmlconfiguration.xsllog4j.propertiescontainer-executor.cfgmapred-env.cmdcore-site.xmlmapred-env.shhadoop-env.cmdmapred-queues.xml.templatehadoop-env.shmapred-site.xmlhadoop-metrics2.propertiesmapred-site.xml.templatehadoop-metrics.propertiesslaveshadoop-policy.xmlssl-client.xml.examplehdfs-site.xmlssl-server.xml.examplehttpfs-env.shyarn-env.cmdhttpfs-log4j.propertiesyarn-env.shhttpfs-signature.secretyarn-site.xml[root@linuxmainhadoop]#

配置hadoop-env.sh

#settotherootofyourJavainstallation#你的jdk安装的路径exportJAVA_HOME=/usr/java/latest#Assumingyourinstallationdirectoryis/usr/local/hadoop#你的hadoop安装路径exportHADOOP_PREFIX=/usr/local/hadoop

配置yarn-env.sh

#settotherootofyourJavainstallation#你的jdk安装的路径exportJAVA_HOME=/usr/java/latest

使用sourcehadoop-env.sh生效一下sourceyarn-env.sh生效一下

配置core-site.xml

<name>hadoop.tmp.dir</name><value>/root/hadoop2.5</value><description>Abaseforothertemporarydirectories.</description></property><property><name>fs.defaultFS</name><value>hdfs://LinuxMain:9000</value><description>hdfs://host:port/</description></property><property><name>io.native.lib.available</name><value>true</value><description>Shouldnativehadooplibraries,ifpresent,beused.</description></property><property><name>io.file.buffer.size</name><value>4096</value></property>配置hdfs-site.xml

<property><name>dfs.namenode.name.dir</name><value>file://${hadoop.tmp.dir}/dfs/name</value></property><property><name>dfs.datanode.data.dir</name><value>file://${hadoop.tmp.dir}/dfs/data</value></property><property><name>dfs.replication</name><value>3</value></property><property><name>dfs.namenode.secondary.http-address</name><value>LinuxMain:50090</value><description>Thesecondarynamenodehttpserveraddressandport.</description></property><property><name>dfs.namenode.secondary.https-address</name><value>LinuxMain:50091</value><description>ThesecondarynamenodeHTTPSserveraddressandport.</description></property><property><name>dfs.datanode.address</name><value>LinuxMain:50010</value><description>Thedatanodeserveraddressandportfordatatransfer.</description></property><property><name>dfs.datanode.http.address</name><value>LinuxMain:50075</value><description>Thedatanodehttpserveraddressandport.</description></property><property><name>dfs.datanode.ipc.address</name><value>LinuxMain:50020</value><description>Thedatanodeipcserveraddressandport.</description></property>

配置mapred-site.xml但是会发现没有这个文件

使用命令cpmapred-site.xml.templatemapred-site.xml重新复制命名一个文件就可以了

<property><name>mapreduce.framework.name</name><value>yarn</value><description>TheruntimeframeworkforexecutingMapReducejobs.Canbeoneoflocal,classicoryarn.</description></property><property><name>mapreduce.jobhistory.address</name><value>LinuxMain:10020</value><description>MapReduceJobHistoryServerIPChost:port</description></property><property><name>mapreduce.jobhistory.webapp.address</name><value>LinuxMain:19888</value><description>MapReduceJobHistoryServerWebUIhost:port</description></property><property><name>mapreduce.jobtracker.http.address</name><value>LinuxMain:50030</value><description>Thejobtrackerhttpserveraddressandporttheserverwilllistenon.Iftheportis0thentheserverwillstartonafreeport.</description></property>

配置yarn-site.xml

<property><description>thevalidservicenameshouldonlycontaina-zA-Z0-9_andcannotstartwithnumbers</description><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property><property><description>ThehostnameoftheRM.</description><name>yarn.resourcemanager.hostname</name><value>LinuxMain</value></property><property><description>TheaddressoftheapplicationsmanagerinterfaceintheRM.</description><name>yarn.resourcemanager.address</name><value>${yarn.resourcemanager.hostname}:8032</value></property><property><description>Theaddressoftheschedulerinterface.</description><name>yarn.resourcemanager.scheduler.address</name><value>${yarn.resourcemanager.hostname}:8030</value></property><property><description>TheaddressoftheRMadmininterface.</description><name>yarn.resourcemanager.admin.address</name><value>${yarn.resourcemanager.hostname}:8033</value></property><property><description>ThehttpaddressoftheRMwebapplication.</description><name>yarn.resourcemanager.webapp.address</name><value>${yarn.resourcemanager.hostname}:8088</value></property><property><description>ThehttpsadddressoftheRMwebapplication.</description><name>yarn.resourcemanager.webapp.https.address</name><value>${yarn.resourcemanager.hostname}:8090</value></property><property><name>yarn.resourcemanager.resource-tracker.address</name><value>${yarn.resourcemanager.hostname}:8031</value></property>

配置slaves

LinuxMainLinuxNode1LinuxNode2

配置完了之后

格式化文件命令bin/hdfsnamenode-format

14/11/2622:43:26INFOblockmanagement.BlockManager:encryptDataTransfer=false14/11/2622:43:26INFOblockmanagement.BlockManager:maxNumBlocksToLog=100014/11/2622:43:26INFOnamenode.FSNamesystem:fsOwner=root(auth:SIMPLE)14/11/2622:43:26INFOnamenode.FSNamesystem:supergroup=supergroup14/11/2622:43:26INFOnamenode.FSNamesystem:dfs.namenode.safemode.min.datanodes=014/11/2622:43:26INFOnamenode.FSNamesystem:dfs.namenode.safemode.extension=3000014/11/2622:43:26INFOnamenode.FSNamesystem:Retrycacheonnamenodeisenabled14/11/2622:43:26INFOnamenode.FSNamesystem:Retrycachewilluse0.03oftotalheapandretrycacheentryexpirytimeis600000millis14/11/2622:43:26INFOutil.GSet:ComputingcapacityformapNameNodeRetryCache14/11/2622:43:26INFOutil.GSet:VMtype=32-bit14/11/2622:43:26INFOutil.GSet:0.029999999329447746%maxmemory966.7MB=297.0KB14/11/2622:43:26INFOutil.GSet:capacity=2^16=65536entries14/11/2622:43:26INFOnamenode.NNConf:ACLsenabled?false14/11/2622:43:26INFOnamenode.NNConf:XAttrsenabled?true14/11/2622:43:26INFOnamenode.NNConf:Maximumsizeofanxattr:1638414/11/2622:43:26INFOnamenode.FSImage:AllocatednewBlockPoolId:BP-1894983963-192.168.1.217-141707060667714/11/2622:43:26INFOcommon.Storage:Storagedirectory/root/hadoop2.5/dfs/namehasbeensuccessfullyformatted.14/11/2622:43:26INFOnamenode.NNStorageRetentionManager:Goingtoretain1imageswithtxid>=014/11/2622:43:26INFOutil.ExitUtil:Exitingwithstatus014/11/2622:43:26INFOnamenode.NameNode:SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG:ShuttingdownNameNodeatlinuxnode1/192.168.1.217************************************************************/看到这句就成了

接下来复制文件，这是一个比较漫长的过程

scp-rhadooproot@linuxnode1:/usr/

scp-rhadooproot@linuxnode2:/usr/

完了之后再两个节点上也格式化文件系统

bin/hdfsnamenode-format

之后启动

sbin/start-all.sh全部启动

linuxMain[root@linuxmainhadoop]#jps5781NodeManager5679ResourceManager5506SecondaryNameNode5170NameNode6222Jps5294DataNode[root@linuxmainhadoop]#

打开浏览器输入http://192.168.1.216:50070看看页面能不能出来

http://192.168.1.216:8088看看集群节点

如果这两个画面都有那就成了亲测可以运行

hadoop还带了一个单词统计的小例子呢

明天再写吧

内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理

标签： hadoop2.5.1集群安装 hadoop安装 hadoop集群安装

相关文章推荐

新的分享

章节导航