您的位置:首页 > 其它

两个mapreduce程序实现不同的功能 而mapreduce1的输出将要作为mapreduce2的输入

2013-04-17 17:37 351 查看
example:

JobConf conf1 = new JobConf(YourClass.class);

//set configurations

...

//set inputformat

conf1.setInputFormat(SomeInputFormatExtendsFromInputFormat.class)

conf1.setOutputFormat(SomeOutputFormatExtendsFromOutputFormat.class)

//set input path

FileInputFormat.setInputPaths(conf1, "/your_input_dir");

FileOutputFormat.setOutputPaths(conf1, "/your_first_output_dir");

JobClient.runJob(conf1);

//at this point, the job should have finished. Use submitJob(conf1) to submit it asynchronisely.

JobConf conf2 = new JobConf();

//do the same for conf2, except the input path

FileInputFormat.setInputPaths(conf1, "/your_first_output_dir");

FileOutputFormat.setOutputPaths(conf1, "/your_first_input_dir");

JobClient.runJob(conf);

自己继承InputFormat, OutputFormat来定义合适的分割,读,写文件方式。mapreduce有一些实现好的,比如FileInputFormat, SequenceFileInputFormat。
内容来自用户分享和网络整理,不保证内容的准确性,如有侵权内容,可联系管理员处理 点击这里给我发消息
标签:  MapReduce
相关文章推荐