Python+hadoop Streaming编写的Map-Reduce程序与调试运行
2015-03-24 11:56
381 查看
Hadoop的 Streaming介绍
Hadoop本身是用Java开发的,程序也需要用Java编写,但是通过Hadoop Streaming,可以使 用任意语言(python、ruby、C++等)来编写程序,让Hadoop运行。Hadoop提供了MapReduce的API,并允许使用非java的其他语言来写自己的map和reduce函数。而Streaming使用Unix标准流作为Hadoop和应用程序之间的接口,所以允许我们可以使用任何编程语言通过标准输入/输出来写MapReduce程序。Streaming的相关源码可以在hadoop的源码中查看。
Streaming天生适合用于文本处理,在文本模式下使用时,它有一个数据的行视图。map的输入数据通过标准输入流传递给map函数,并且是一行一行地传输,最后结果写入标准输出。map输出的键值对以一个制表符分割的行,它以这样的形式写到标准输出。reduce函数的输入格式相同——通过制表符来分割的键值对——并通过标准输入流进行传输。reduce函数从标准输入流中读取输入行,该输入已由hadoop框架根据键排序,最后将结果写入标准输出。
评价Hadoop的 Streaming
优点可以使用自己喜欢的语言来编写MapReduce程序
不需要像写Java的MR程序那样import一大堆库,在代码里做一大堆配置,很多东西都抽象到了stdio上,代码量显著减少
因为没有库的依赖,调试方便,并且可以脱离Hadoop先在本地用管道pipes模拟调试
缺点
只能通过命令行参数来控制MapReduce框架,不像Java的程序那样可以在代码里使用API,控制力比较弱,有些东西鞭长莫及
因为中间隔着一层处理,效率会比较慢
所以Hadoop Streaming比较适合做一些简单的任务,比如用python写只有一两百行的脚本。如果项目比较复杂,或者需要进行比较细致的优化,使用Streaming就容易出现一些束手束脚的地方。
Python编写的MapReduce程序
1) 输入的数据为books.json:["milton-paradise.txt", "[ Paradise Lost by John Milton 1667 ] Book I Of Man ' s first disobedience , and the fruit Of that forbidden tree whose mortal taste Brought death into the World , and all our woe , With loss of Eden , till one greater Man Restore us , and regain the blissful seat , Sing , Heavenly Muse , that , on the secret top Of Oreb , or of Sinai , didst inspire That shepherd who first taught the chosen seed In the beginning how the heavens and earth Rose out of Chaos : or , if Sion hill Delight thee more , and Siloa ' s brook that flowed Fast by the oracle of God , I thence Invoke thy aid to my adventurous song , That with no middle flight intends to soar Above th ' Aonian mount , while it pursues Things unattempted yet in prose or rhyme ."] ["edgeworth-parents.txt", "[ The Parent ' s Assistant , by Maria Edgeworth ] THE ORPHANS . Near the ruins of the castle of Rossmore , in Ireland , is a small cabin , in which there once lived a widow and her four children . As long as she was able to work , she was very industrious , and was accounted the best spinner in the parish ; but she overworked herself at last , and fell ill , so that she could not sit to her wheel as she used to do , and was obliged to give it up to her eldest daughter , Mary ."] ["austen-emma.txt", "[ Emma by Jane Austen 1816 ] VOLUME I CHAPTER I Emma Woodhouse , handsome , clever , and rich , with a comfortable home and happy disposition , seemed to unite some of the best blessings of existence ; and had lived nearly twenty - one years in the world with very little to distress or vex her . She was the youngest of the two daughters of a most affectionate , indulgent father ; and had , in consequence of her sister ' s marriage , been mistress of his house from a very early period . Her mother had died too long ago for her to have more than an indistinct remembrance of her caresses ; and her place had been supplied by an excellent woman as governess , who had fallen little short of a mother in affection ."] ["chesterton-ball.txt", "[ The Ball and The Cross by G . K . Chesterton 1909 ] I . A DISCUSSION SOMEWHAT IN THE AIR The flying ship of Professor Lucifer sang through the skies like a silver arrow ; the bleak white steel of it , gleaming in the bleak blue emptiness of the evening . That it was far above the earth was no expression for it ; to the two men in it , it seemed to be far above the stars . The professor had himself invented the flying machine , and had also invented nearly everything in it ."] ["bible-kjv.txt", "[ The King James Bible ] The Old Testament of the King James Bible The First Book of Moses : Called Genesis 1 : 1 In the beginning God created the heaven and the earth . 1 : 2 And the earth was without form , and void ; and darkness was upon the face of the deep . And the Spirit of God moved upon the face of the waters . 1 : 3 And God said , Let there be light : and there was light . 1 : 4 And God saw the light , that it was good : and God divided the light from the darkness . 1 : 5 And God called the light Day , and the darkness he called Night . And the evening and the morning were the first day ."] ["chesterton-thursday.txt", "[ The Man Who Was Thursday by G . K . Chesterton 1908 ] To Edmund Clerihew Bentley A cloud was on the mind of men , and wailing went the weather , Yea , a sick cloud upon the soul when we were boys together . Science announced nonentity and art admired decay ; The world was old and ended : but you and I were gay ; Round us in antic order their crippled vices came -- Lust that had lost its laughter , fear that had lost its shame . Like the white lock of Whistler , that lit our aimless gloom , Men showed their own white feather as proudly as a plume . Life was a fly that faded , and death a drone that stung ; The world was very old indeed when you and I were young ."] ["blake-poems.txt", "[ Poems by William Blake 1789 ] SONGS OF INNOCENCE AND OF EXPERIENCE and THE BOOK of THEL SONGS OF INNOCENCE INTRODUCTION Piping down the valleys wild , Piping songs of pleasant glee , On a cloud I saw a child , And he laughing said to me : \" Pipe a song about a Lamb !\" So I piped with merry cheer . \" Piper , pipe that song again ;\" So I piped : he wept to hear . \" Drop thy pipe , thy happy pipe ; Sing thy songs of happy cheer :!\" So I sang the same again , While he wept with joy to hear . \" Piper , sit thee down and write In a book , that all may read .\" So he vanish ' d from my sight ; And I pluck ' d a hollow reed , And I made a rural pen , And I stain ' d the water clear , And I wrote my happy songs Every child may joy to hear ."] ["shakespeare-caesar.txt", "[ The Tragedie of Julius Caesar by William Shakespeare 1599 ] Actus Primus . Scoena Prima . Enter Flauius , Murellus , and certaine Commoners ouer the Stage . Flauius . Hence : home you idle Creatures , get you home : Is this a Holiday ? What , know you not ( Being Mechanicall ) you ought not walke Vpon a labouring day , without the signe Of your Profession ? Speake , what Trade art thou ? Car . Why Sir , a Carpenter Mur . Where is thy Leather Apron , and thy Rule ? What dost thou with thy best Apparrell on ? You sir , what Trade are you ? Cobl . Truely Sir , in respect of a fine Workman , I am but as you would say , a Cobler Mur . But what Trade art thou ? Answer me directly Cob . A Trade Sir , that I hope I may vse , with a safe Conscience , which is indeed Sir , a Mender of bad soules Fla ."] ["whitman-leaves.txt", "[ Leaves of Grass by Walt Whitman 1855 ] Come , said my soul , Such verses for my Body let us write , ( for we are one ,) That should I after return , Or , long , long hence , in other spheres , There to some group of mates the chants resuming , ( Tallying Earth ' s soil , trees , winds , tumultuous waves ,) Ever with pleas ' d smile I may keep on , Ever and ever yet the verses owning -- as , first , I here and now Signing for Soul and Body , set to them my name , Walt Whitman [ BOOK I . INSCRIPTIONS ] } One ' s - Self I Sing One ' s - self I sing , a simple separate person , Yet utter the word Democratic , the word En - Masse ."] ["melville-moby_dick.txt", "[ Moby Dick by Herman Melville 1851 ] ETYMOLOGY . ( Supplied by a Late Consumptive Usher to a Grammar School ) The pale Usher -- threadbare in coat , heart , body , and brain ; I see him now . He was ever dusting his old lexicons and grammars , with a queer handkerchief , mockingly embellished with all the gay flags of all the known nations of the world . He loved to dust his old grammars ; it somehow mildly reminded him of his mortality ."]把文本数据放入fs建立好的input文件夹中
$ hadoop fs -mkdir input
$ hadoop fs -ls
$ hadoop fs -copyFromLocal <PATH TO LOCAL FILE(S)> input/
2)统计每个单词在哪些文档里出现
map函数~invertedMapper.py文件
#!/usr/bin/env python import sys import json for line in sys.stdin: line = line.strip() record = json.loads(line) fileName = record[0]; value = record[1]; words = value.split() for word in words: print '%s\t%s' % (word, fileName)
reduce函数~invertedReducer.py文件
#!/usr/bin/env python import sys # maps words to their counts media = {} wordInMedia = {} # input comes from STDIN for line in sys.stdin: (word,fileName)=line.strip().split('\t',1) media.setdefault(word,[]) media[word].append(fileName) for word in media: wordInMedia.setdefault(word,list(set(media[word]))) # write the results to STDOUT (standard output) for word in wordInMedia: print '%s\t%s' % (word, wordInMedia[word])
运行Python编写的MapReduce程序
在hadoop2.5.1中测试运行。1)确保输出文件夹output不存在
hadoop dfs -rm -r /user/hadoop/pytest/output
2)调试方法
hadoop命令运行MR程序
格式如下 :
$hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-2.5.1.jar \ -mapper 'python mapper.py' -file mapper.py\ -reducer 'python reducer.py' -file reducer.py \ -input input/* -output output运行程序:
hadoop jar /home/hadoop/hadoop/share/hadoop/tools/lib/hadoop-streaming-2.5.1.jar -mapper "python invertedMapper.py" -reducer "python invertedReducer.py" -input /user/hadoop/pytest/input/* -output /user/hadoop/pytest/output -file /home/hadoop/invertedMapper.py -file /home/hadoop/invertedReducer.py
本地进行测试
模式如下:
cat <input path> | python <path to mapper script> | sort -t $'\t' -k1,1 | python <path to reducer script> > <output path>本地测试运行:
cat /home/hadoop/books.json | python /home/hadoop/invertedMapper.py | sort | python /home/hadoop/invertedReducer.py
这里有几点需要注意:
Hadoop默认按照tab(\t)来分割key和value,以第一个分割出的部分为key,按key进行排序,因此这里使用
sort -t $'\t' -k1,1 来模拟。
如果你有其他需求,在交给Hadoop Streaming执行时可以通过命令行参数调,本地调试也可以进行相应的调整,主要是调整sort的参数。因此为了能够熟练进行本地调试,建议先掌握sort命令的用法。
如果你在python脚本里加上了shabang标记(
#! /usr/bin/env python),并且为它们添加了执行权限,也可以用类似于
./mapper.py 代替 python mapper.py
结果
在output中生成了结果文件。all ['milton-paradise.txt', 'blake-poems.txt', 'melville-moby_dick.txt'] Rossmore ['edgeworth-parents.txt'] Consumptive ['melville-moby_dick.txt'] Self ['whitman-leaves.txt'] Night ['bible-kjv.txt'] eldest ['edgeworth-parents.txt'] four ['edgeworth-parents.txt'] Caesar ['shakespeare-caesar.txt'] winds ['whitman-leaves.txt'] Moses ['bible-kjv.txt'] children ['edgeworth-parents.txt'] seemed ['chesterton-ball.txt', 'austen-emma.txt'] whose ['milton-paradise.txt'] Parent ['edgeworth-parents.txt'] father ['austen-emma.txt'] young ['chesterton-thursday.txt'] vex ['austen-emma.txt'] to ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'blake-poems.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] th ['milton-paradise.txt'] smile ['whitman-leaves.txt'] Day ['bible-kjv.txt'] hill ['milton-paradise.txt'] me ['blake-poems.txt', 'shakespeare-caesar.txt'] woman ['austen-emma.txt'] Whitman ['whitman-leaves.txt'] song ['milton-paradise.txt', 'blake-poems.txt'] far ['chesterton-ball.txt'] void ['bible-kjv.txt'] rhyme ['milton-paradise.txt'] Oreb ['milton-paradise.txt'] Pipe ['blake-poems.txt'] Where ['shakespeare-caesar.txt'] heaven ['bible-kjv.txt'] Bentley ['chesterton-thursday.txt'] -- ['whitman-leaves.txt', 'melville-moby_dick.txt', 'chesterton-thursday.txt'] sang ['chesterton-ball.txt', 'blake-poems.txt'] 1667 ['milton-paradise.txt'] pleas ['whitman-leaves.txt'] Paradise ['milton-paradise.txt'] Fast ['milton-paradise.txt'] VOLUME ['austen-emma.txt'] Shakespeare ['shakespeare-caesar.txt'] earth ['milton-paradise.txt', 'bible-kjv.txt', 'chesterton-ball.txt'] Jane ['austen-emma.txt'] emptiness ['chesterton-ball.txt'] blue ['chesterton-ball.txt'] what ['shakespeare-caesar.txt'] darkness ['bible-kjv.txt'] Cobler ['shakespeare-caesar.txt'] supplied ['austen-emma.txt'] Delight ['milton-paradise.txt'] above ['chesterton-ball.txt'] ; ['austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] ever ['whitman-leaves.txt', 'melville-moby_dick.txt'] piped ['blake-poems.txt'] Apparrell ['shakespeare-caesar.txt'] body ['melville-moby_dick.txt'] Emma ['austen-emma.txt'] Professor ['chesterton-ball.txt'] men ['chesterton-ball.txt', 'chesterton-thursday.txt'] here ['whitman-leaves.txt'] water ['blake-poems.txt'] let ['whitman-leaves.txt'] crippled ['chesterton-thursday.txt'] sing ['whitman-leaves.txt'] [ ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] invented ['chesterton-ball.txt'] daughter ['edgeworth-parents.txt'] my ['milton-paradise.txt', 'blake-poems.txt', 'whitman-leaves.txt'] spheres ['whitman-leaves.txt'] DISCUSSION ['chesterton-ball.txt'] Blake ['blake-poems.txt'] heart ['melville-moby_dick.txt'] Things ['milton-paradise.txt'] Actus ['shakespeare-caesar.txt'] till ['milton-paradise.txt'] Book ['milton-paradise.txt', 'bible-kjv.txt'] 1908 ['chesterton-thursday.txt'] 1909 ['chesterton-ball.txt'] Answer ['shakespeare-caesar.txt'] cloud ['blake-poems.txt', 'chesterton-thursday.txt'] from ['bible-kjv.txt', 'austen-emma.txt', 'blake-poems.txt'] would ['shakespeare-caesar.txt'] give ['edgeworth-parents.txt'] two ['chesterton-ball.txt', 'austen-emma.txt'] more ['milton-paradise.txt', 'austen-emma.txt'] Piper ['blake-poems.txt'] Leather ['shakespeare-caesar.txt'] about ['blake-poems.txt'] Rule ['shakespeare-caesar.txt'] So ['blake-poems.txt'] excellent ['austen-emma.txt'] known ['melville-moby_dick.txt'] fly ['chesterton-thursday.txt'] Signing ['whitman-leaves.txt'] word ['whitman-leaves.txt'] middle ['milton-paradise.txt'] this ['shakespeare-caesar.txt'] work ['edgeworth-parents.txt'] soul ['whitman-leaves.txt', 'chesterton-thursday.txt'] hope ['shakespeare-caesar.txt'] INNOCENCE ['blake-poems.txt'] laughter ['chesterton-thursday.txt'] admired ['chesterton-thursday.txt'] shepherd ['milton-paradise.txt'] lock ['chesterton-thursday.txt'] Edmund ['chesterton-thursday.txt'] Aonian ['milton-paradise.txt'] sir ['shakespeare-caesar.txt'] sit ['blake-poems.txt', 'edgeworth-parents.txt'] divided ['bible-kjv.txt'] Who ['chesterton-thursday.txt'] Carpenter ['shakespeare-caesar.txt'] ,) ['whitman-leaves.txt'] machine ['chesterton-ball.txt'] how ['milton-paradise.txt'] Truely ['shakespeare-caesar.txt'] Why ['shakespeare-caesar.txt'] embellished ['melville-moby_dick.txt'] A ['chesterton-ball.txt', 'shakespeare-caesar.txt', 'chesterton-thursday.txt'] may ['blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt'] ORPHANS ['edgeworth-parents.txt'] after ['whitman-leaves.txt'] :!" ['blake-poems.txt'] consequence ['austen-emma.txt'] waves ['whitman-leaves.txt'] Hence ['shakespeare-caesar.txt'] bleak ['chesterton-ball.txt'] a ['austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] short ['austen-emma.txt'] light ['bible-kjv.txt'] One ['whitman-leaves.txt'] so ['edgeworth-parents.txt'] order ['chesterton-thursday.txt'] signe ['shakespeare-caesar.txt'] What ['shakespeare-caesar.txt'] indeed ['shakespeare-caesar.txt', 'chesterton-thursday.txt'] Supplied ['melville-moby_dick.txt'] years ['austen-emma.txt'] brain ['melville-moby_dick.txt'] !" ['blake-poems.txt'] through ['chesterton-ball.txt'] same ['blake-poems.txt'] existence ['austen-emma.txt'] its ['chesterton-thursday.txt'] Body ['whitman-leaves.txt'] 1 ['bible-kjv.txt'] industrious ['edgeworth-parents.txt'] group ['whitman-leaves.txt'] decay ['chesterton-thursday.txt'] chosen ['milton-paradise.txt'] glee ['blake-poems.txt'] brook ['milton-paradise.txt'] , ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] Sir ['shakespeare-caesar.txt'] Sing ['milton-paradise.txt', 'blake-poems.txt', 'whitman-leaves.txt'] Tallying ['whitman-leaves.txt'] them ['whitman-leaves.txt'] evening ['chesterton-ball.txt', 'bible-kjv.txt'] Grammar ['melville-moby_dick.txt'] greater ['milton-paradise.txt'] died ['austen-emma.txt'] Car ['shakespeare-caesar.txt'] thee ['milton-paradise.txt', 'blake-poems.txt'] safe ['shakespeare-caesar.txt'] Murellus ['shakespeare-caesar.txt'] She ['austen-emma.txt'] not ['shakespeare-caesar.txt', 'edgeworth-parents.txt'] now ['whitman-leaves.txt', 'melville-moby_dick.txt'] silver ['chesterton-ball.txt'] gloom ['chesterton-thursday.txt'] name ['whitman-leaves.txt'] arrow ['chesterton-ball.txt'] went ['chesterton-thursday.txt'] CHAPTER ['austen-emma.txt'] Mur ['shakespeare-caesar.txt'] Rose ['milton-paradise.txt'] taught ['milton-paradise.txt'] our ['milton-paradise.txt', 'chesterton-thursday.txt'] out ['milton-paradise.txt'] INSCRIPTIONS ['whitman-leaves.txt'] ' ['milton-paradise.txt', 'blake-poems.txt', 'whitman-leaves.txt', 'austen-emma.txt', 'edgeworth-parents.txt'] day ['bible-kjv.txt', 'shakespeare-caesar.txt'] Muse ['milton-paradise.txt'] Genesis ['bible-kjv.txt'] William ['blake-poems.txt', 'shakespeare-caesar.txt'] flowed ['milton-paradise.txt'] waters ['bible-kjv.txt'] announced ['chesterton-thursday.txt'] Stage ['shakespeare-caesar.txt'] disobedience ['milton-paradise.txt'] G ['chesterton-ball.txt', 'chesterton-thursday.txt'] pleasant ['blake-poems.txt'] small ['edgeworth-parents.txt'] ETYMOLOGY ['melville-moby_dick.txt'] disposition ['austen-emma.txt'] Mechanicall ['shakespeare-caesar.txt'] mistress ['austen-emma.txt'] beginning ['milton-paradise.txt', 'bible-kjv.txt'] En ['whitman-leaves.txt'] created ['bible-kjv.txt'] Being ['shakespeare-caesar.txt'] could ['edgeworth-parents.txt'] keep ['whitman-leaves.txt'] fruit ['milton-paradise.txt'] place ['austen-emma.txt'] mortal ['milton-paradise.txt'] hence ['whitman-leaves.txt'] castle ['edgeworth-parents.txt'] fallen ['austen-emma.txt'] first ['milton-paradise.txt', 'bible-kjv.txt', 'whitman-leaves.txt'] Lamb ['blake-poems.txt'] flying ['chesterton-ball.txt'] There ['whitman-leaves.txt'] Speake ['shakespeare-caesar.txt'] one ['milton-paradise.txt', 'whitman-leaves.txt', 'austen-emma.txt'] Scoena ['shakespeare-caesar.txt'] directly ['shakespeare-caesar.txt'] hear ['blake-poems.txt'] Ever ['whitman-leaves.txt'] Sinai ['milton-paradise.txt'] owning ['whitman-leaves.txt'] Chesterton ['chesterton-ball.txt', 'chesterton-thursday.txt'] little ['austen-emma.txt'] " ['blake-poems.txt'] indulgent ['austen-emma.txt'] twenty ['austen-emma.txt'] top ['milton-paradise.txt'] their ['chesterton-thursday.txt'] 2 ['bible-kjv.txt'] too ['austen-emma.txt'] Such ['whitman-leaves.txt'] white ['chesterton-ball.txt', 'chesterton-thursday.txt'] cabin ['edgeworth-parents.txt'] that ['milton-paradise.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'edgeworth-parents.txt'] 1599 ['shakespeare-caesar.txt'] aimless ['chesterton-thursday.txt'] SOMEWHAT ['chesterton-ball.txt'] Democratic ['whitman-leaves.txt'] herself ['edgeworth-parents.txt'] than ['austen-emma.txt'] steel ['chesterton-ball.txt'] loved ['melville-moby_dick.txt'] showed ['chesterton-thursday.txt'] Come ['whitman-leaves.txt'] tree ['milton-paradise.txt'] nations ['melville-moby_dick.txt'] idle ['shakespeare-caesar.txt'] Usher ['melville-moby_dick.txt'] coat ['melville-moby_dick.txt'] threadbare ['melville-moby_dick.txt'] indistinct ['austen-emma.txt'] and ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] handkerchief ['melville-moby_dick.txt'] God ['milton-paradise.txt', 'bible-kjv.txt'] labouring ['shakespeare-caesar.txt'] mind ['chesterton-thursday.txt'] comfortable ['austen-emma.txt'] Late ['melville-moby_dick.txt'] say ['shakespeare-caesar.txt'] mates ['whitman-leaves.txt'] have ['austen-emma.txt'] blissful ['milton-paradise.txt'] saw ['bible-kjv.txt', 'blake-poems.txt'] Invoke ['milton-paradise.txt'] self ['whitman-leaves.txt'] able ['edgeworth-parents.txt'] soules ['shakespeare-caesar.txt'] lit ['chesterton-thursday.txt'] also ['chesterton-ball.txt'] forbidden ['milton-paradise.txt'] drone ['chesterton-thursday.txt'] which ['shakespeare-caesar.txt', 'edgeworth-parents.txt'] flags ['melville-moby_dick.txt'] With ['milton-paradise.txt'] Spirit ['bible-kjv.txt'] pursues ['milton-paradise.txt'] soar ['milton-paradise.txt'] Drop ['blake-poems.txt'] most ['austen-emma.txt'] Piping ['blake-poems.txt'] Enter ['shakespeare-caesar.txt'] The ['chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'shakespeare-caesar.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] ] ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] Called ['bible-kjv.txt'] SONGS ['blake-poems.txt'] Restore ['milton-paradise.txt'] clear ['blake-poems.txt'] face ['bible-kjv.txt'] pipe ['blake-poems.txt'] accounted ['edgeworth-parents.txt'] Earth ['whitman-leaves.txt'] } ['whitman-leaves.txt'] came ['chesterton-thursday.txt'] inspire ['milton-paradise.txt'] wailing ['chesterton-thursday.txt'] queer ['melville-moby_dick.txt'] utter ['whitman-leaves.txt'] fear ['chesterton-thursday.txt'] heavens ['milton-paradise.txt'] fine ['shakespeare-caesar.txt'] ( ['shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt'] plume ['chesterton-thursday.txt'] should ['whitman-leaves.txt'] rich ['austen-emma.txt'] stain ['blake-poems.txt'] handsome ['austen-emma.txt'] mortality ['melville-moby_dick.txt'] do ['edgeworth-parents.txt'] his ['melville-moby_dick.txt', 'austen-emma.txt'] Prima ['shakespeare-caesar.txt'] get ['shakespeare-caesar.txt'] Leaves ['whitman-leaves.txt'] trees ['whitman-leaves.txt'] While ['blake-poems.txt'] nearly ['chesterton-ball.txt', 'austen-emma.txt'] THE ['chesterton-ball.txt', 'blake-poems.txt', 'edgeworth-parents.txt'] him ['melville-moby_dick.txt'] child ['blake-poems.txt'] Maria ['edgeworth-parents.txt'] ended ['chesterton-thursday.txt'] EXPERIENCE ['blake-poems.txt'] morning ['bible-kjv.txt'] overworked ['edgeworth-parents.txt'] bad ['shakespeare-caesar.txt'] antic ['chesterton-thursday.txt'] she ['edgeworth-parents.txt'] Austen ['austen-emma.txt'] ruins ['edgeworth-parents.txt'] wrote ['blake-poems.txt'] widow ['edgeworth-parents.txt'] set ['whitman-leaves.txt'] art ['shakespeare-caesar.txt', 'chesterton-thursday.txt'] Chaos ['milton-paradise.txt'] seat ['milton-paradise.txt'] ouer ['shakespeare-caesar.txt'] see ['melville-moby_dick.txt'] Lucifer ['chesterton-ball.txt'] are ['shakespeare-caesar.txt', 'whitman-leaves.txt'] John ['milton-paradise.txt'] best ['shakespeare-caesar.txt', 'austen-emma.txt', 'edgeworth-parents.txt'] said ['blake-poems.txt', 'whitman-leaves.txt', 'bible-kjv.txt'] That ['milton-paradise.txt', 'chesterton-ball.txt', 'whitman-leaves.txt'] INTRODUCTION ['blake-poems.txt'] 3 ['bible-kjv.txt'] woe ['milton-paradise.txt'] marriage ['austen-emma.txt'] K ['chesterton-ball.txt', 'chesterton-thursday.txt'] King ['bible-kjv.txt'] we ['whitman-leaves.txt', 'chesterton-thursday.txt'] Bible ['bible-kjv.txt'] Commoners ['shakespeare-caesar.txt'] joy ['blake-poems.txt'] were ['bible-kjv.txt', 'chesterton-thursday.txt'] pluck ['blake-poems.txt'] merry ['blake-poems.txt'] AND ['blake-poems.txt'] youngest ['austen-emma.txt'] last ['edgeworth-parents.txt'] Yea ['chesterton-thursday.txt'] ill ['edgeworth-parents.txt'] thence ['milton-paradise.txt'] called ['bible-kjv.txt'] s ['milton-paradise.txt', 'whitman-leaves.txt', 'austen-emma.txt', 'edgeworth-parents.txt'] mildly ['melville-moby_dick.txt'] Sion ['milton-paradise.txt'] expression ['chesterton-ball.txt'] Yet ['whitman-leaves.txt'] Conscience ['shakespeare-caesar.txt'] THEL ['blake-poems.txt'] distress ['austen-emma.txt'] Her ['austen-emma.txt'] Clerihew ['chesterton-thursday.txt'] simple ['whitman-leaves.txt'] period ['austen-emma.txt'] hollow ['blake-poems.txt'] gleaming ['chesterton-ball.txt'] Ireland ['edgeworth-parents.txt'] Cobl ['shakespeare-caesar.txt'] respect ['shakespeare-caesar.txt'] Holiday ['shakespeare-caesar.txt'] Was ['chesterton-thursday.txt'] Flauius ['shakespeare-caesar.txt'] Apron ['shakespeare-caesar.txt'] Mender ['shakespeare-caesar.txt'] dusting ['melville-moby_dick.txt'] been ['austen-emma.txt'] . ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] secret ['milton-paradise.txt'] feather ['chesterton-thursday.txt'] affectionate ['austen-emma.txt'] lived ['edgeworth-parents.txt', 'austen-emma.txt'] flight ['milton-paradise.txt'] gay ['melville-moby_dick.txt', 'chesterton-thursday.txt'] am ['shakespeare-caesar.txt'] deep ['bible-kjv.txt'] World ['milton-paradise.txt'] Whistler ['chesterton-thursday.txt'] Round ['chesterton-thursday.txt'] And ['blake-poems.txt', 'bible-kjv.txt'] Old ['bible-kjv.txt'] mount ['milton-paradise.txt'] thou ['shakespeare-caesar.txt'] while ['milton-paradise.txt'] seed ['milton-paradise.txt'] ought ['shakespeare-caesar.txt'] lexicons ['melville-moby_dick.txt'] wild ['blake-poems.txt'] aid ['milton-paradise.txt'] Creatures ['shakespeare-caesar.txt'] soil ['whitman-leaves.txt'] is ['shakespeare-caesar.txt', 'edgeworth-parents.txt'] Dick ['melville-moby_dick.txt'] it ['chesterton-ball.txt', 'bible-kjv.txt', 'melville-moby_dick.txt', 'milton-paradise.txt', 'edgeworth-parents.txt'] AIR ['chesterton-ball.txt'] shame ['chesterton-thursday.txt'] grammars ['melville-moby_dick.txt'] good ['bible-kjv.txt'] Siloa ['milton-paradise.txt'] in ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] prose ['milton-paradise.txt'] return ['whitman-leaves.txt'] You ['shakespeare-caesar.txt'] vanish ['blake-poems.txt'] if ['milton-paradise.txt'] 1816 ['austen-emma.txt'] ) ['shakespeare-caesar.txt', 'melville-moby_dick.txt'] parish ['edgeworth-parents.txt'] reminded ['melville-moby_dick.txt'] unite ['austen-emma.txt'] chants ['whitman-leaves.txt'] Of ['milton-paradise.txt', 'shakespeare-caesar.txt'] Workman ['shakespeare-caesar.txt'] daughters ['austen-emma.txt'] wheel ['edgeworth-parents.txt'] used ['edgeworth-parents.txt'] I ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt'] upon ['bible-kjv.txt', 'chesterton-thursday.txt'] Soul ['whitman-leaves.txt'] Cob ['shakespeare-caesar.txt'] IN ['chesterton-ball.txt'] dust ['melville-moby_dick.txt'] thy ['milton-paradise.txt', 'blake-poems.txt', 'shakespeare-caesar.txt'] Is ['shakespeare-caesar.txt'] didst ['milton-paradise.txt'] nonentity ['chesterton-thursday.txt'] person ['whitman-leaves.txt'] cheer ['blake-poems.txt'] without ['bible-kjv.txt', 'shakespeare-caesar.txt'] resuming ['whitman-leaves.txt'] In ['milton-paradise.txt', 'blake-poems.txt', 'bible-kjv.txt'] very ['edgeworth-parents.txt', 'austen-emma.txt', 'chesterton-thursday.txt'] the ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] 1789 ['blake-poems.txt'] - ['whitman-leaves.txt', 'austen-emma.txt'] verses ['whitman-leaves.txt'] Tragedie ['shakespeare-caesar.txt'] Heavenly ['milton-paradise.txt'] yet ['milton-paradise.txt', 'whitman-leaves.txt'] dost ['shakespeare-caesar.txt'] vices ['chesterton-thursday.txt'] death ['milton-paradise.txt', 'chesterton-thursday.txt'] Lust ['chesterton-thursday.txt'] mother ['austen-emma.txt'] had ['chesterton-ball.txt', 'chesterton-thursday.txt', 'austen-emma.txt'] intends ['milton-paradise.txt'] adventurous ['milton-paradise.txt'] spinner ['edgeworth-parents.txt'] book ['blake-poems.txt'] 4 ['bible-kjv.txt'] skies ['chesterton-ball.txt'] On ['blake-poems.txt'] certaine ['shakespeare-caesar.txt'] read ['blake-poems.txt'] James ['bible-kjv.txt'] stung ['chesterton-thursday.txt'] early ['austen-emma.txt'] know ['shakespeare-caesar.txt'] world ['melville-moby_dick.txt', 'chesterton-thursday.txt', 'austen-emma.txt'] Grass ['whitman-leaves.txt'] Or ['whitman-leaves.txt'] d ['blake-poems.txt', 'whitman-leaves.txt'] loss ['milton-paradise.txt'] like ['chesterton-ball.txt'] lost ['chesterton-thursday.txt'] OF ['blake-poems.txt'] Profession ['shakespeare-caesar.txt'] laughing ['blake-poems.txt'] governess ['austen-emma.txt'] Milton ['milton-paradise.txt'] Vpon ['shakespeare-caesar.txt'] valleys ['blake-poems.txt'] old ['melville-moby_dick.txt', 'chesterton-thursday.txt'] obliged ['edgeworth-parents.txt'] proudly ['chesterton-thursday.txt'] somehow ['melville-moby_dick.txt'] Woodhouse ['austen-emma.txt'] Primus ['shakespeare-caesar.txt'] vse ['shakespeare-caesar.txt'] sight ['blake-poems.txt'] home ['shakespeare-caesar.txt', 'austen-emma.txt'] pale ['melville-moby_dick.txt'] Mary ['edgeworth-parents.txt'] Life ['chesterton-thursday.txt'] for ['chesterton-ball.txt', 'whitman-leaves.txt', 'austen-emma.txt'] 1851 ['melville-moby_dick.txt'] Men ['chesterton-thursday.txt'] 1855 ['whitman-leaves.txt'] everything ['chesterton-ball.txt'] pen ['blake-poems.txt'] Let ['bible-kjv.txt'] ? ['shakespeare-caesar.txt'] He ['melville-moby_dick.txt'] regain ['milton-paradise.txt'] be ['chesterton-ball.txt', 'bible-kjv.txt'] School ['melville-moby_dick.txt'] who ['milton-paradise.txt', 'austen-emma.txt'] Eden ['milton-paradise.txt'] ;" ['blake-poems.txt'] Near ['edgeworth-parents.txt'] by ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] First ['bible-kjv.txt'] wept ['blake-poems.txt'] on ['milton-paradise.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'chesterton-thursday.txt'] sister ['austen-emma.txt'] of ['milton-paradise.txt', 'austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] Trade ['shakespeare-caesar.txt'] blessings ['austen-emma.txt'] or ['milton-paradise.txt', 'austen-emma.txt'] own ['chesterton-thursday.txt'] into ['milton-paradise.txt'] down ['blake-poems.txt'] weather ['chesterton-thursday.txt'] But ['shakespeare-caesar.txt'] remembrance ['austen-emma.txt'] Edgeworth ['edgeworth-parents.txt'] your ['shakespeare-caesar.txt'] Ball ['chesterton-ball.txt'] her ['edgeworth-parents.txt', 'austen-emma.txt'] Assistant ['edgeworth-parents.txt'] ." ['blake-poems.txt'] there ['bible-kjv.txt', 'edgeworth-parents.txt'] long ['whitman-leaves.txt', 'austen-emma.txt', 'edgeworth-parents.txt'] Every ['blake-poems.txt'] caresses ['austen-emma.txt'] stars ['chesterton-ball.txt'] house ['austen-emma.txt'] : ['milton-paradise.txt', 'bible-kjv.txt', 'shakespeare-caesar.txt', 'chesterton-thursday.txt', 'blake-poems.txt'] was ['austen-emma.txt', 'chesterton-ball.txt', 'bible-kjv.txt', 'chesterton-thursday.txt', 'melville-moby_dick.txt', 'edgeworth-parents.txt'] Herman ['melville-moby_dick.txt'] happy ['blake-poems.txt', 'austen-emma.txt'] himself ['chesterton-ball.txt'] form ['bible-kjv.txt'] Science ['chesterton-thursday.txt'] but ['shakespeare-caesar.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt'] boys ['chesterton-thursday.txt'] Moby ['melville-moby_dick.txt'] reed ['blake-poems.txt'] Fla ['shakespeare-caesar.txt'] with ['milton-paradise.txt', 'austen-emma.txt', 'blake-poems.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'melville-moby_dick.txt'] mockingly ['melville-moby_dick.txt'] he ['bible-kjv.txt', 'blake-poems.txt'] unattempted ['milton-paradise.txt'] made ['blake-poems.txt'] up ['edgeworth-parents.txt'] us ['milton-paradise.txt', 'whitman-leaves.txt', 'chesterton-thursday.txt'] Walt ['whitman-leaves.txt'] BOOK ['blake-poems.txt', 'whitman-leaves.txt'] faded ['chesterton-thursday.txt'] professor ['chesterton-ball.txt'] affection ['austen-emma.txt'] walke ['shakespeare-caesar.txt'] taste ['milton-paradise.txt'] moved ['bible-kjv.txt'] Cross ['chesterton-ball.txt'] Thursday ['chesterton-thursday.txt'] an ['austen-emma.txt'] Masse ['whitman-leaves.txt'] To ['chesterton-thursday.txt'] as ['austen-emma.txt', 'shakespeare-caesar.txt', 'whitman-leaves.txt', 'chesterton-thursday.txt', 'edgeworth-parents.txt'] clever ['austen-emma.txt'] at ['edgeworth-parents.txt'] Above ['milton-paradise.txt'] tumultuous ['whitman-leaves.txt'] some ['whitman-leaves.txt', 'austen-emma.txt'] ship ['chesterton-ball.txt'] again ['blake-poems.txt'] no ['milton-paradise.txt', 'chesterton-ball.txt'] when ['chesterton-thursday.txt'] other ['whitman-leaves.txt'] 5 ['bible-kjv.txt'] sick ['chesterton-thursday.txt'] Poems ['blake-poems.txt'] you ['shakespeare-caesar.txt', 'chesterton-thursday.txt'] Man ['milton-paradise.txt', 'chesterton-thursday.txt'] Like ['chesterton-thursday.txt'] Lost ['milton-paradise.txt'] separate ['whitman-leaves.txt'] Julius ['shakespeare-caesar.txt'] fell ['edgeworth-parents.txt'] Testament ['bible-kjv.txt'] Melville ['melville-moby_dick.txt'] ago ['austen-emma.txt'] together ['chesterton-thursday.txt'] Brought ['milton-paradise.txt'] As ['edgeworth-parents.txt'] write ['blake-poems.txt', 'whitman-leaves.txt'] oracle ['milton-paradise.txt'] rural ['blake-poems.txt'] once ['edgeworth-parents.txt'] songs ['blake-poems.txt']
相关文章推荐
- Python利用hadoop Streaming编写的Map-Reduce程序命令运行和本地调试运行
- 用hadoop-streaming 运行python map-reduce程序
- 用python + hadoop streaming 编写分布式程序的本地调试方法
- 用python + hadoop streaming 编写分布式程序(一) -- 原理介绍,样例程序与本地调试
- 用python + hadoop streaming 编写分布式程序(一) -- 原理介绍,样例程序与本地调试
- 用python + hadoop streaming 编写分布式程序(二) -- 在集群上运行与监控
- 用python + hadoop streaming 编写分布式程序(三) -- 自定义功能
- Hadoop Streaming运行Python脚本程序
- 使用python+hadoop-streaming编写hadoop处理程序
- hadoop——在命令行下编译并运行map-reduce程序
- Python+Hadoop Streaming实现MapReduce(如何给map和reduce的脚本传递参数)
- Python+Hadoop Streaming实现MapReduce(如何给map和reduce的脚本传递参数)
- Python+Hadoop Streaming实现MapReduce(如何给map和reduce的脚本传递参数)
- 用python + hadoop streaming 分布式编程(一) -- 原理介绍,样例程序与本地调试
- 用python + hadoop streaming 分布式编程(一) -- 原理介绍,样例程序与本地调试
- Hadoop Streaming: 使用Java以外的语言去实现Map/Reduce
- Python编写Hadoop MapReduce程序
- Hadoop 使用Combiner提高Map/Reduce程序效率
- Eclipse重写Wordcount类实现处理中文字符,利用hadoop Eclipse插件远程调试hadoop运行WordCount程序
- python 运行 hadoop 2.0 mapreduce 程序