spark1.6从hdfs上读取文件运行wordcount
2016-06-02 00:00
483 查看
从hdfs上读取文件并运行wordcount
![](https://oscdn.geek-share.com/Uploads/Images/Content/202012/08/a920bd0c0e0783ffa9ee6fa6b78771c7)
scala> val count=file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
结果:
[root@hadoop14 app]# hadoop fs -put word.txt /yuhui
scala> val file=sc.textFile("hdfs://hadoop14:9000/yuhui/word.txt")
scala> val count=file.flatMap(line => line.split(" ")).map(word => (word,1)).reduceByKey(_+_)
scala> count.collect()
结果:
res1: Array[(String, Int)] = Array((cccc,2), (aaaa,2), (bbb,2), (ddd,1))
相关文章推荐
- Writing from Flume to HDFS
- 基于HDFS的SparkStreaming案例实战
- hdfs 命令行操作
- 【漫画解读】HDFS存储原理
- 通过Nifi 导入csv文件到HDFS
- HDFS EC:将纠删码技术融入HDFS
- hdfs 使用
- hadoop yarn container
- Hadoop Map 数目决定因素
- SparkStreaming0nHDFS实战
- HDFS内存存储
- flume-ng+Kafka+Storm+HDFS 实时系统搭建
- 第85课:基于HDFS的SparkStreaming案例实战和内幕源码解密
- HDFS文件上传:8020端口拒绝连接问题解决!
- HDFS学习之FileSystem
- spark读写hdfs后出现的异常错误
- hdfs文件误删恢复
- hdfs块丢失导致的异常问题排查解决
- sparkStreming on HDFS
- hdfs 常用命令