Hadoop学习(一): Ubuntu上安装Hadoop
2016-06-21 13:45
253 查看
Hadoop学习(一): Ubuntu上安装Hadoop
1.安装ssh
2.查看JAVA_HOME变量值
3.安装hadoop-2.7.2
从官网(http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/)下载,解压到hadoop-2.7.2
4.修改hadoop-2.7.2的etc/hadoop/hadoop-env.sh文件,设置JAVA_HOME
输入以下命令,弹出hadoop的用法,则配置成功
hadoop支持以下三种模式:
5.Standalone Operation(单机模式)
开启ssh服务
免密码登陆
测试:The following example copies the unpacked conf directory to use as input and then finds and displays every match of the given regular expression. Output is written to the given output directory.
6.Pseudo-Distributed Operation(单机伪分布模式)
修改两处文件:
etc/hadoop/core-site.xml:
etc/hadoop/hdfs-site.xml:
ssh登陆:
执行:
Browse the web interface for the NameNode; by default it is available at: NameNode - http://localhost:50070/
1.安装ssh
$ sudo apt-get install openssh-client $ sudo apt-get install openssh-server
2.查看JAVA_HOME变量值
/opt/jdk1.8.0_91
3.安装hadoop-2.7.2
从官网(http://mirror.bit.edu.cn/apache/hadoop/common/hadoop-2.7.2/)下载,解压到hadoop-2.7.2
4.修改hadoop-2.7.2的etc/hadoop/hadoop-env.sh文件,设置JAVA_HOME
export JAVA_HOME=/opt/jdk1.8.0_91
输入以下命令,弹出hadoop的用法,则配置成功
$ bin/hadoop
hadoop支持以下三种模式:
5.Standalone Operation(单机模式)
开启ssh服务
$ sudo /etc/init.d/ssh start
免密码登陆
#client端产生密钥: $ ssh-keygen -t rsa #server端: $ cp id_rsa.pub authorized_keys $ chmod 600 authorized_keys
测试:The following example copies the unpacked conf directory to use as input and then finds and displays every match of the given regular expression. Output is written to the given output directory.
$ mkdir input $ cp etc/hadoop/*.xml input $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep input output 'dfs[a-z.]+' $ cat output/*
6.Pseudo-Distributed Operation(单机伪分布模式)
修改两处文件:
etc/hadoop/core-site.xml:
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://localhost:9000</value> </property> </configuration>
etc/hadoop/hdfs-site.xml:
<configuration> <property> <name>dfs.replication</name> <value>1</value> </property> </configuration>
ssh登陆:
$ ssh localhost
执行:
/ 4000 /Format the filesystem: $ bin/hdfs namenode -format //Start NameNode daemon and DataNode daemon: $ sbin/start-dfs.sh
Browse the web interface for the NameNode; by default it is available at: NameNode - http://localhost:50070/
//Make the HDFS directories required to execute MapReduce jobs: $ bin/hdfs dfs -mkdir /user $ bin/hdfs dfs -mkdir /user/<username> //Copy the input files into the distributed filesystem: $ bin/hdfs dfs -put etc/hadoop input //Run some of the examples provided: $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar grep input output 'dfs[a-z.]+' //Examine the output files: Copy the output files from the distributed filesystem to the local filesystem and examine them: $ bin/hdfs dfs -get output output $ cat output/* //or view the output files on the distributed filesystem: $ bin/hdfs dfs -cat output/* //When you’re done, stop the daemons with: $ sbin/stop-dfs.sh
相关文章推荐
- Linux USB 驱动开发实例(二)—— USB 鼠标驱动注解及测试
- tomcat的配置
- Linux USB 驱动开发实例(一) —— USB摄像头驱动实现源码分析
- Linux USB 驱动开发(五)—— USB驱动程序开发过程简单总结
- centos6.5搭建Nginx(Web Server)+Tomcat+MySql
- 第19课:Spark Streaming架构设计和运行机制大总结
- centos中当不小心删除了用户目录后如何恢复
- Linux USB 驱动开发(四)—— 热插拔那点事
- Linux USB 驱动开发(一)—— USB设备基础概念
- Linux USB 驱动开发(三)—— 编写USB 驱动程序
- 如何从应用程序进入linux内核
- Linux raid信息 查看
- OpenGL texture黑边问题
- Nginx反向代理实现Tomcat多个应用80端口访问
- 12、RedHat6 监控集群状态
- 网站架构系列:消息队列
- 浅谈Web网站架构演变过程
- 饿了么 openapi demo
- 高可用性系统在大众点评的实践与经验
- centos 7: 迁移MySQL目录