Streaming（C++）实现WordCount

Streaming 提供来Mapreduce的API,允许我们用非JAVA语言编写map和reduce函数，这是我第一次使用Streaming，也是第一次在linux写shell脚本，值得记录一下（2015/12/16）！

我以前习惯用C++写程序，所以这里用C++实现map和reduce。

先介绍下使用Streaming实现的步骤：

1、写map和reduce函数

2、在linux下测试map和reduce函数

3、用hadoop实现

步骤一：

//mapper function
#include<string>
#include<iostream>

using namespace std;

int main()
{
string str;
while(cin>>str)
{
cout<<str<<"\t"<<1<<endl;
}
return 0;
}
//reducer function
#include<iostream>
#include<string>

using namespace std;

int main()
{
string key1,value,key2;
int cnt = 1;
cin>>key1>>value;
while(cin>>key2>>value)
{
if(key1 == key2)
{
cnt++;
}
else
{
cout<<key1<<"\t"<<cnt<<endl;
cnt = 1;
key1 = key2;
}
}
cout<<key2<<"\t"<<cnt<<endl;
return 0;
}

步骤2和3放在wordcount.sh中

#!/bin/bash

#在linux下测试
:<<cc
g++ ./map.cpp -o map
g++ ./reduce.cpp -o reduce
cat /home/hadoop/input/wordcount/wordcount.txt | ./map | sort | ./reduce
cc

#在hadoop下执行
hadoop fs -rmr /user/hadoop/output
hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.2.1.jar \
-input /user/hadoop/input/wordcount/ \
-output /user/hadoop/output/wordcount \
-mapper /home/hadoop/c++/streaming/map \
-reducer /home/hadoop/c++/streaming/reduce \
-file /home/hadoop/c++/streaming/map \
-file /home/hadoop/c++/streaming/reduce