Hadoop2.6.0 mac 安装
2015-07-08 12:05
495 查看
1. 单机模式
(1) ssh设置远程登录:
xiaolan:~ xiaolan$@ssh-keygen -t rsa -P ""
xiaolan:~ xiaolan$cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
登录测试
xiaolan:~ xiaolan$ssh localhost
xiaolan:~ xiaolan$exit
(2) 安装jdk(忽略)
(3) 配置环境变量
xiaolan:~ xiaolan$vim ~/.bash_profile
输入内容:
export HADOOP_HOME=/Users/xiaolan/Applications/hadoop-2.6.0
export YARN_HOME=/Users/xiaolan/Applications/hadoop-2.6.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$YARN_HOME/etc/hadoop
:wq
让环境变量生效
xiaolan:~ xiaolan$source ~/.bash_profile
(4) 配置hadoop
进入Applications/hadoop-2.6.0/etc/hadoop
xiaolan:~ xiaolan$vim hadoop-env.sh
输入内容:
export J***A_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
export HADOOP_HOME=/Users/xiaolan/Applications/hadoop-2.6.0
export PATH=$PATH:/Users/xiaolan/Applications/hadoop-2.6.0/bin
:wq
让环境变量生效
xiaolan:~ xiaolan$source hadoop-env.sh
单机模式安装成功
(5) 测试
xiaolan:~ xiaolan$mkdir input
xiaolan:~ xiaolan$cp **/** input
注:随便拷贝点文件
xiaolan:~ xiaolan$hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount input output
xiaolan:~ xiaolan$cat output/*
注:成功看到结果
2. 伪分布模式
(1) 配置core-site.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
(2) 配置yarn-site.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>yarn.noCHdemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
(3) 创建和配置mapred-site.xml
默认/etc/hadoop文件夹下有mapred.xml.template文件,复制更改用户名mapred.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
(4) 配置hdfs-site.xml
指定主机上作为namenode和datanode的目录:
/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/name
/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/data
<configuration></configuration>之间增加如下内容:
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/data</value>
</property>
(5) 格式化hdfs
xiaolan:~ xiaolan$hdfs
namenode -format
(6) 启动hadoop
xiaolan:~ xiaolan$cd Applications/hadoop-2.6.0/sbin
xiaolan:sbin xiaolan$./start-dfs.sh
xiaolan:sbin xiaolan$./start-yarn.sh
浏览器打开 http://localhost:50070/,会看到hdfs管理页面
浏览器打开 http://localhost:8088/,会看到hadoop进程管理页面
(7) WordCount验证
创建input目录
xiaolan:hadoop-2.6.0 xiaolan$hadooop fs -mkdir input
fs:hadoop下的文件命令,和普通命令基本一致
拷贝文件
xiaolan:hadoop-2.6.0 xiaolan$hadooop fs -copyFromLocal README.txt input
xiaolan:hadoop-2.6.0 xiaolan$hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount input output
xiaolan:hadoop-2.6.0 xiaolan$cat
output/*
注:成功看到结果
linux:图文教程,版本较老: http://www.linuxidc.com/Linux/2013-01/77681p4.htm http://www.cnblogs.com/kinglau/p/3794433.html
官网安装教程:http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SingleCluster.html
Eclipse配置:
http://andy-ghg.iteye.com/blog/1165453
注:
据说可以忽略:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable:
http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos
转帖:http://wollt1992.blog.163.com/blog/static/57013456201411244394378
(1) ssh设置远程登录:
xiaolan:~ xiaolan$@ssh-keygen -t rsa -P ""
xiaolan:~ xiaolan$cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
登录测试
xiaolan:~ xiaolan$ssh localhost
xiaolan:~ xiaolan$exit
(2) 安装jdk(忽略)
(3) 配置环境变量
xiaolan:~ xiaolan$vim ~/.bash_profile
输入内容:
export HADOOP_HOME=/Users/xiaolan/Applications/hadoop-2.6.0
export YARN_HOME=/Users/xiaolan/Applications/hadoop-2.6.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_CONF_DIR=$YARN_HOME/etc/hadoop
:wq
让环境变量生效
xiaolan:~ xiaolan$source ~/.bash_profile
(4) 配置hadoop
进入Applications/hadoop-2.6.0/etc/hadoop
xiaolan:~ xiaolan$vim hadoop-env.sh
输入内容:
export J***A_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home
export HADOOP_HOME=/Users/xiaolan/Applications/hadoop-2.6.0
export PATH=$PATH:/Users/xiaolan/Applications/hadoop-2.6.0/bin
:wq
让环境变量生效
xiaolan:~ xiaolan$source hadoop-env.sh
单机模式安装成功
(5) 测试
xiaolan:~ xiaolan$mkdir input
xiaolan:~ xiaolan$cp **/** input
注:随便拷贝点文件
xiaolan:~ xiaolan$hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount input output
xiaolan:~ xiaolan$cat output/*
注:成功看到结果
2. 伪分布模式
(1) 配置core-site.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
(2) 配置yarn-site.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>yarn.noCHdemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name><value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
(3) 创建和配置mapred-site.xml
默认/etc/hadoop文件夹下有mapred.xml.template文件,复制更改用户名mapred.xml
<configuration></configuration>之间增加如下内容:
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
(4) 配置hdfs-site.xml
指定主机上作为namenode和datanode的目录:
/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/name
/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/data
<configuration></configuration>之间增加如下内容:
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/name</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/Users/xiaolan/Applications/hadoop-2.6.0/hdfs/data</value>
</property>
(5) 格式化hdfs
xiaolan:~ xiaolan$hdfs
namenode -format
(6) 启动hadoop
xiaolan:~ xiaolan$cd Applications/hadoop-2.6.0/sbin
xiaolan:sbin xiaolan$./start-dfs.sh
xiaolan:sbin xiaolan$./start-yarn.sh
浏览器打开 http://localhost:50070/,会看到hdfs管理页面
浏览器打开 http://localhost:8088/,会看到hadoop进程管理页面
(7) WordCount验证
创建input目录
xiaolan:hadoop-2.6.0 xiaolan$hadooop fs -mkdir input
fs:hadoop下的文件命令,和普通命令基本一致
拷贝文件
xiaolan:hadoop-2.6.0 xiaolan$hadooop fs -copyFromLocal README.txt input
xiaolan:hadoop-2.6.0 xiaolan$hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar wordcount input output
xiaolan:hadoop-2.6.0 xiaolan$cat
output/*
注:成功看到结果
linux:图文教程,版本较老: http://www.linuxidc.com/Linux/2013-01/77681p4.htm http://www.cnblogs.com/kinglau/p/3794433.html
官网安装教程:http://hadoop.apache.org/docs/r2.6.0/hadoop-project-dist/hadoop-common/SingleCluster.html
Eclipse配置:
http://andy-ghg.iteye.com/blog/1165453
注:
据说可以忽略:WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable:
http://stackoverflow.com/questions/19943766/hadoop-unable-to-load-native-hadoop-library-for-your-platform-error-on-centos
转帖:http://wollt1992.blog.163.com/blog/static/57013456201411244394378
相关文章推荐
- dede 开源 中文分词 可用于 ecshop
- Linux文件权限属性后面的点的含义(ls -l)
- 拷贝80关关卡copy_maps.py
- ELK最新版实战配置检索nginx日志
- Linux编程手册
- Nginx 反向代理、负载均衡、页面缓存、URL重写及读写分离详解
- log4cxx在linux下的编译使用
- Linux编程手册
- Linux编程手册
- centos 7.1 yum
- linux系统工程师面试题(附答案)
- OpenWRT UCI命令实现无线中继
- 花生壳如何发布tomcat网站到外网?
- SSH linux 常用命令
- Nginx反向代理实现负载均衡配置图解
- 每天一个linux命令(18):locate 命令
- 每天一个linux命令(17):whereis 命令
- Apache Commons Codec 与消息摘要算法(hash算法)
- 使用 Apache JMeter 测试基于云的应用程序
- [Study] 通过游戏学编程的网站