Hadoop自带例子学习
2014-05-01 16:43
253 查看
用简单例子学习Hadoop,统计字符出现次数。
环境:OS:Centos 6.5 x64 & Soft:Hadoop 1.2.1
1、创建文件夹,写入字符到文本文件。
2、写到hdfs中
3、运行自带例子
4、查看结果
环境:OS:Centos 6.5 x64 & Soft:Hadoop 1.2.1
1、创建文件夹,写入字符到文本文件。
[huser@master hadoop-1.2.1]$ mkdir input [huser@master hadoop-1.2.1]$ echo "hello world" >test1.txt [huser@master hadoop-1.2.1]$ echo "hello hadoop" >test2.txt
2、写到hdfs中
[huser@master hadoop-1.2.1]$ bin/hadoop fs -put ../input ./in [huser@master hadoop-1.2.1]$ bin/hadoop fs -ls Found 1 items drwxr-xr-x - huser supergroup 0 2014-04-16 19:00 /user/huser/in [huser@master hadoop-1.2.1]$ bin/hadoop fs -ls ./in/* -rw-r--r-- 1 huser supergroup 12 2014-04-16 19:00 /user/huser/in/test1.txt -rw-r--r-- 1 huser supergroup 13 2014-04-16 19:00 /user/huser/in/test2.txt
3、运行自带例子
[huser@master hadoop-1.2.1]$ bin/hadoop jar hadoop-examples-1.2.1.jar wordcount in out 14/04/16 19:02:53 INFO input.FileInputFormat: Total input paths to process : 2 14/04/16 19:02:53 INFO util.NativeCodeLoader: Loaded the native-hadoop library 14/04/16 19:02:53 WARN snappy.LoadSnappy: Snappy native library not loaded 14/04/16 19:02:54 INFO mapred.JobClient: Running job: job_201404161850_0001 14/04/16 19:02:55 INFO mapred.JobClient: map 0% reduce 0% 14/04/16 19:03:10 INFO mapred.JobClient: map 100% reduce 0% 14/04/16 19:03:19 INFO mapred.JobClient: map 100% reduce 33% 14/04/16 19:03:21 INFO mapred.JobClient: map 100% reduce 100% 14/04/16 19:03:23 INFO mapred.JobClient: Job complete: job_201404161850_0001 14/04/16 19:03:23 INFO mapred.JobClient: Counters: 30 14/04/16 19:03:23 INFO mapred.JobClient: Job Counters 14/04/16 19:03:23 INFO mapred.JobClient: Launched reduce tasks=1 14/04/16 19:03:23 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=19368 14/04/16 19:03:23 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 14/04/16 19:03:23 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 14/04/16 19:03:23 INFO mapred.JobClient: Rack-local map tasks=1 14/04/16 19:03:23 INFO mapred.JobClient: Launched map tasks=2 14/04/16 19:03:23 INFO mapred.JobClient: Data-local map tasks=1 14/04/16 19:03:23 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=11082 14/04/16 19:03:23 INFO mapred.JobClient: File Output Format Counters 14/04/16 19:03:23 INFO mapred.JobClient: Bytes Written=25 14/04/16 19:03:23 INFO mapred.JobClient: FileSystemCounters 14/04/16 19:03:23 INFO mapred.JobClient: FILE_BYTES_READ=55 14/04/16 19:03:23 INFO mapred.JobClient: HDFS_BYTES_READ=239 14/04/16 19:03:23 INFO mapred.JobClient: FILE_BYTES_WRITTEN=169887 14/04/16 19:03:23 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=25 14/04/16 19:03:23 INFO mapred.JobClient: File Input Format Counters 14/04/16 19:03:23 INFO mapred.JobClient: Bytes Read=25 14/04/16 19:03:23 INFO mapred.JobClient: Map-Reduce Framework 14/04/16 19:03:23 INFO mapred.JobClient: Map output materialized bytes=61 14/04/16 19:03:23 INFO mapred.JobClient: Map input records=2 14/04/16 19:03:23 INFO mapred.JobClient: Reduce shuffle bytes=61 14/04/16 19:03:23 INFO mapred.JobClient: Spilled Records=8 14/04/16 19:03:23 INFO mapred.JobClient: Map output bytes=41 14/04/16 19:03:23 INFO mapred.JobClient: Total committed heap usage (bytes)=415633408 14/04/16 19:03:23 INFO mapred.JobClient: CPU time spent (ms)=4060 14/04/16 19:03:23 INFO mapred.JobClient: Combine input records=4 14/04/16 19:03:23 INFO mapred.JobClient: SPLIT_RAW_BYTES=214 14/04/16 19:03:23 INFO mapred.JobClient: Reduce input records=4 14/04/16 19:03:23 INFO mapred.JobClient: Reduce input groups=3 14/04/16 19:03:23 INFO mapred.JobClient: Combine output records=4 14/04/16 19:03:23 INFO mapred.JobClient: Physical memory (bytes) snapshot=402755584 14/04/16 19:03:23 INFO mapred.JobClient: Reduce output records=3 14/04/16 19:03:23 INFO mapred.JobClient: Virtual memory (bytes) snapshot=2173550592 14/04/16 19:03:23 INFO mapred.JobClient: Map output records=4
4、查看结果
[huser@master hadoop-1.2.1]$ bin/hadoop fs -ls ./out/* -rw-r--r-- 1 huser supergroup 0 2014-04-16 19:03 /user/huser/out/_SUCCESS drwxr-xr-x - huser supergroup 0 2014-04-16 19:02 /user/huser/out/_logs/history -rw-r--r-- 1 huser supergroup 25 2014-04-16 19:03 /user/huser/out/part-r-00000 [huser@master hadoop-1.2.1]$ bin/hadoop fs -cat ./out/part-r-00000 hadoop 1 hello 2 world 1
相关文章推荐
- Caffe 深度学习入门教程 运行caffe自带的两个简单例子
- 学习Hadoop MapReduce与WordCount例子分析
- 在linux下eclipse中运行hadoop自带的WordCount例子出现的两个错误
- Caffe学习系列(9):运行caffe自带的两个简单例子
- 通过lua自带例子学习lua 08 (36-38)
- hadoop自带RandomWriter例子解析
- 创建第一个JBPM6项目并且运行自带的helloword例子(JBPM6学习之三)
- Caffe 深度学习入门教程 运行caffe自带的两个简单例子
- hadoop学习第四天-Writable和WritableComparable序列化接口的使用&&MapReduce中传递javaBean的简单例子
- Caffe 深度学习入门教程 运行caffe自带的两个简单例子
- hadoop反编译安装包自带的例子
- hadoop学习之HDFS(2.5):windows下eclipse远程连接linux下的hadoop集群并测试wordcount例子
- hadoop自带例子wordcount的具体运行步骤
- 【深度学习】笔记3_caffe自带的第一个例子,Mnist手写数字识别所使用的LeNet网络模型的详细解释
- (原创)学习NotesList(Robotium自带的例子)
- vegaprime自带例子学习笔记(1)
- 分析Hadoop自带WordCount例子的执行过程
- 通过lua自带例子学习lua 01
- hadoop mahout 运行自带的例子
- R学习:通过REmap向Echarts中的地图写入数据(运行自带例子)