[MapReduce]Top N 任务的mapper
2015-07-27 18:55
337 查看
这是Udacity的课程 intro to hadoop and mapReduce里面Lesson4的练习
求总体的Top N。
首先在Mapper中求出局部的Top N,求Top N不能像word count那样来一句print一句,要把所有的line都读完,计数,排序,输入topN
然后再Reducer中求出全局的 Top N。
以下是Mapper 代码
求总体的Top N。
首先在Mapper中求出局部的Top N,求Top N不能像word count那样来一句print一句,要把所有的line都读完,计数,排序,输入topN
然后再Reducer中求出全局的 Top N。
以下是Mapper 代码
#!/usr/bin/python """ Your mapper function should print out 10 lines containing longest posts, sorted in ascending order from shortest to longest. Please do not use global variables and do not change the "main" function. """ import sys import csv def mapper(): reader = csv.reader(sys.stdin, delimiter='\t') writer = csv.writer(sys.stdout, delimiter='\t', quotechar='"', quoting=csv.QUOTE_ALL) lines = [] for line in reader: lines.append(line) # YOUR CODE HERE lines.sort(key = lambda x: len(x[4]), reverse = True) for i in range(9, -1, -1): writer.writerow(lines[i]) test_text = """\"\"\t\"\"\t\"\"\t\"\"\t\"333\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"88888888\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"1\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"11111111111\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"1000000000\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"22\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"4444\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"666666\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"55555\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"999999999\"\t\"\" \"\"\t\"\"\t\"\"\t\"\"\t\"7777777\"\t\"\" """ # This function allows you to test the mapper with the provided test string def main(): import StringIO sys.stdin = StringIO.StringIO(test_text) mapper() sys.stdin = sys.__stdin__ main()
相关文章推荐
- cocos2dx飞机大战开发记录(1)
- 基于ZXing的二维码,你可以这样改造它
- Android 广播机制 详解
- 深入浅出Android App耗电量统计
- Fragment的startActivityForResult方法
- Android - DownloadManager的使用
- App Store 审核指南中文翻译
- iOS学习笔记--(c基础题9)
- 我的Android进阶之旅------>android:drawableLeft的用法
- Object-C 面向对象的三大特征
- android沉浸式状态栏
- Android中Handler与Thread的区别
- Android开发之SQLite的使用方法
- 发布iOS应用(Xcode5)到App Store详细解析
- 百度地图用城市名称切换城市更新地图
- 简单实现Android平台多语言
- Android中Intent传递Object和ArrayList<Object>对象---笔记
- Android中Intent传递Object和ArrayList对象---笔记
- android学习笔记---63-PopupWindow,泡泡窗口的实现
- 美团Android自动化之旅—生成渠道包