solution:No job file jar和ClassNotFoundException(hadoop,mapreduce)

hadoop-1.2.1伪分布式搭建好了，也只是用命令跑过hadoop-example.jar包的wordcount，这一切看起来so easy。

但没想到的是，自己的mr程序，运行起来却遇到了No job file jar和ClassNotFoundException的问题。

经过一番周折，自己写的mapreduce 终于成功运行了。

我没有将第三方jar包（hadoop-core，commons-cli，commons-xxx等6个jar包）和自己的代码的jar包全部都添加到远程集群上，在本地也没有将第三方jar包打成third-party.jar，也没有用到“-libjars”参数，连GenericOptionsParser也没使用（网上很多solution都说这个用来解析hadoop的命令参数），，

关键代码：

Job job = new Job(getConf());

job.setJarByClass(WordCountJob.class)；

和

int res = ToolRunner.run(new WordCountJob(),args);

source code：

package wordcount2;

import java.io.IOException;

import java.util.StringTokenizer;

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.conf.Configured;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.util.GenericOptionsParser;

import org.apache.hadoop.util.Tool;

import org.apache.hadoop.util.ToolRunner;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class WordCountJob extends Configured implements Tool {



public static class TokenizerMapper extends Mapper<Object,Text,Text,IntWritable>{

private final static IntWritable one = new IntWritable(1);

private Text word = new Text();



public void map(Object key,Text value,Context context) throws IOException,InterruptedException{

StringTokenizer itr = new StringTokenizer(value.toString());



while(itr.hasMoreTokens()){

word.set(itr.nextToken());

context.write(word,one);

}

}

}







public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable>{

private IntWritable result = new IntWritable();

public void reduce(Text key,Iterable<IntWritable> values,Context context) throws IOException, InterruptedException {

int sum = 0;

for(IntWritable val:values){

sum += val.get();

}

result.set(sum);

context.write(key, result);

}

}











@Override

public int run(String[] args) throws Exception {

// TODO Auto-generated method stub





// Configuration conf = new Configuration();

// String[] otherArgs = new GenericOptionsParser(conf,args).getRemainingArgs();

if(args.length !=2){

System.err.println("Usage:wordcount <in> <out>");

System.exit(2);

}





// Job job = new Job(conf,"wordcountmr");

Job job = new Job(getConf());



job.setJarByClass(WordCountJob.class);

job.setMapperClass(TokenizerMapper.class);



job.setCombinerClass(IntSumReducer.class);

job.setReducerClass(IntSumReducer.class);



job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);



FileInputFormat.addInputPath(job, new Path(args[0]));

FileOutputFormat.setOutputPath(job, new Path(args[1]));



System.exit(job.waitForCompletion(true)?0:1);

return 0;

}



public static void main(String[] args) throws Exception{

int res = ToolRunner.run(new WordCountJob(),args);

System.exit(res);

}

}

编译成jar包，可以使用命令（javac -classpath /home/lzc/hadoop-1.2.1/hadoop-core-1.2.1.jar:/home/lzc/hadoop-1.2.1/lib/commons-cli-1.2.jar -d ./classes/ ./src/WordCountJob.java以及jar -cvfm wordcountjob.jar -C ./classes/两个命令），最简单的方式是使用eclipse的导出jar文件功能，单独将该class生成一个jar文件。

把生成的jar包cp到hadoop_home下，执行以下命令。

hadoop121@ubuntu:~/Dolphin/hadoop-1.2.1$ bin/hadoop jar wc2.jar wordcount2.WordCountJob input/file*.txt output

14/12/10 15:48:59 INFO input.FileInputFormat: Total input paths to process : 2

14/12/10 15:48:59 INFO util.NativeCodeLoader: Loaded the native-hadoop library

14/12/10 15:48:59 WARN snappy.LoadSnappy: Snappy native library not loaded

14/12/10 15:49:00 INFO mapred.JobClient: Running job: job_201412080836_0026

14/12/10 15:49:01 INFO mapred.JobClient: map 0% reduce 0%

14/12/10 15:49:06 INFO mapred.JobClient: map 100% reduce 0%

14/12/10 15:49:13 INFO mapred.JobClient: map 100% reduce 33%

14/12/10 15:49:15 INFO mapred.JobClient: map 100% reduce 100%

14/12/10 15:49:15 INFO mapred.JobClient: Job complete: job_201412080836_0026

14/12/10 15:49:15 INFO mapred.JobClient: Counters: 29

14/12/10 15:49:15 INFO mapred.JobClient: Job Counters

14/12/10 15:49:15 INFO mapred.JobClient: Launched reduce tasks=1

14/12/10 15:49:15 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=7921

14/12/10 15:49:15 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

14/12/10 15:49:15 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0

14/12/10 15:49:15 INFO mapred.JobClient: Launched map tasks=2

14/12/10 15:49:15 INFO mapred.JobClient: Data-local map tasks=2

14/12/10 15:49:15 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=9018

14/12/10 15:49:15 INFO mapred.JobClient: File Output Format Counters

14/12/10 15:49:15 INFO mapred.JobClient: Bytes Written=48

14/12/10 15:49:15 INFO mapred.JobClient: FileSystemCounters

14/12/10 15:49:15 INFO mapred.JobClient: FILE_BYTES_READ=102

14/12/10 15:49:15 INFO mapred.JobClient: HDFS_BYTES_READ=284

14/12/10 15:49:15 INFO mapred.JobClient: FILE_BYTES_WRITTEN=190665

14/12/10 15:49:15 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=48

14/12/10 15:49:15 INFO mapred.JobClient: File Input Format Counters

14/12/10 15:49:15 INFO mapred.JobClient: Bytes Read=48

14/12/10 15:49:15 INFO mapred.JobClient: Map-Reduce Framework

14/12/10 15:49:15 INFO mapred.JobClient: Map output materialized bytes=108

14/12/10 15:49:15 INFO mapred.JobClient: Map input records=2

14/12/10 15:49:15 INFO mapred.JobClient: Reduce shuffle bytes=108

14/12/10 15:49:15 INFO mapred.JobClient: Spilled Records=16

14/12/10 15:49:15 INFO mapred.JobClient: Map output bytes=80

14/12/10 15:49:15 INFO mapred.JobClient: CPU time spent (ms)=2420

14/12/10 15:49:15 INFO mapred.JobClient: Total committed heap usage (bytes)=390004736

14/12/10 15:49:15 INFO mapred.JobClient: Combine input records=8

14/12/10 15:49:15 INFO mapred.JobClient: SPLIT_RAW_BYTES=236

14/12/10 15:49:15 INFO mapred.JobClient: Reduce input records=8

14/12/10 15:49:15 INFO mapred.JobClient: Reduce input groups=6

14/12/10 15:49:15 INFO mapred.JobClient: Combine output records=8

14/12/10 15:49:15 INFO mapred.JobClient: Physical memory (bytes) snapshot=436707328

14/12/10 15:49:15 INFO mapred.JobClient: Reduce output records=6

14/12/10 15:49:15 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1908416512

14/12/10 15:49:15 INFO mapred.JobClient: Map output records=8

hadoop121@ubuntu:~/Dolphin/hadoop-1.2.1$ bin/hadoop fs -ls output

Found 3 items

-rw-r--r-- 2 hadoop121 supergroup 0 2014-12-10 15:49 /user/hadoop121/output/_SUCCESS

drwxr-xr-x - hadoop121 supergroup 0 2014-12-10 15:49 /user/hadoop121/output/_logs

-rw-r--r-- 2 hadoop121 supergroup 48 2014-12-10 15:49 /user/hadoop121/output/part-r-00000

hadoop121@ubuntu:~/Dolphin/hadoop-1.2.1$ bin/hadoop fs -cat output/part-r-00000

Hadoop 1

Hello 2

Word 1

hadoop 1

hello 2

word 1

有人说hdfs不能访问本地文件，有权限问题，但我特意试了下，本地一样成功执行。

hadoop121@ubuntu:~/Dolphin/hadoop-1.2.1$ bin/hadoop jar /home/lzc/workspace/wordcount1/wc2.jar wordcount2.WordCountJob input/file*.txt output

14/12/10 16:08:26 INFO input.FileInputFormat: Total input paths to process : 2

14/12/10 16:08:26 INFO util.NativeCodeLoader: Loaded the native-hadoop library

14/12/10 16:08:26 WARN snappy.LoadSnappy: Snappy native library not loaded

14/12/10 16:08:27 INFO mapred.JobClient: Running job: job_201412080836_0027

14/12/10 16:08:28 INFO mapred.JobClient: map 0% reduce 0%

14/12/10 16:08:33 INFO mapred.JobClient: map 100% reduce 0%

14/12/10 16:08:40 INFO mapred.JobClient: map 100% reduce 33%

14/12/10 16:08:41 INFO mapred.JobClient: map 100% reduce 100%

14/12/10 16:08:42 INFO mapred.JobClient: Job complete: job_201412080836_0027

14/12/10 16:08:42 INFO mapred.JobClient: Counters: 29

14/12/10 16:08:42 INFO mapred.JobClient: Job Counters

14/12/10 16:08:42 INFO mapred.JobClient: Launched reduce tasks=1

14/12/10 16:08:42 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=7221

14/12/10 16:08:42 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0

14/12/10 16:08:42 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0

14/12/10 16:08:42 INFO mapred.JobClient: Launched map tasks=2

14/12/10 16:08:42 INFO mapred.JobClient: Data-local map tasks=2

14/12/10 16:08:42 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=8677

14/12/10 16:08:42 INFO mapred.JobClient: File Output Format Counters

14/12/10 16:08:42 INFO mapred.JobClient: Bytes Written=48

14/12/10 16:08:42 INFO mapred.JobClient: FileSystemCounters

14/12/10 16:08:42 INFO mapred.JobClient: FILE_BYTES_READ=102

14/12/10 16:08:42 INFO mapred.JobClient: HDFS_BYTES_READ=284

14/12/10 16:08:42 INFO mapred.JobClient: FILE_BYTES_WRITTEN=190665

14/12/10 16:08:42 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=48

14/12/10 16:08:42 INFO mapred.JobClient: File Input Format Counters

14/12/10 16:08:42 INFO mapred.JobClient: Bytes Read=48

14/12/10 16:08:42 INFO mapred.JobClient: Map-Reduce Framework

14/12/10 16:08:42 INFO mapred.JobClient: Map output materialized bytes=108

14/12/10 16:08:42 INFO mapred.JobClient: Map input records=2

14/12/10 16:08:42 INFO mapred.JobClient: Reduce shuffle bytes=108

14/12/10 16:08:42 INFO mapred.JobClient: Spilled Records=16

14/12/10 16:08:42 INFO mapred.JobClient: Map output bytes=80

14/12/10 16:08:42 INFO mapred.JobClient: CPU time spent (ms)=2280

14/12/10 16:08:42 INFO mapred.JobClient: Total committed heap usage (bytes)=373489664

14/12/10 16:08:42 INFO mapred.JobClient: Combine input records=8

14/12/10 16:08:42 INFO mapred.JobClient: SPLIT_RAW_BYTES=236

14/12/10 16:08:42 INFO mapred.JobClient: Reduce input records=8

14/12/10 16:08:42 INFO mapred.JobClient: Reduce input groups=6

14/12/10 16:08:42 INFO mapred.JobClient: Combine output records=8

14/12/10 16:08:42 INFO mapred.JobClient: Physical memory (bytes) snapshot=433147904

14/12/10 16:08:42 INFO mapred.JobClient: Reduce output records=6

14/12/10 16:08:42 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1911033856

14/12/10 16:08:42 INFO mapred.JobClient: Map output records=8

hadoop121@ubuntu:~/Dolphin/hadoop-1.2.1$

references：

1.http://dongxicheng.org/mapreduce/run-hadoop-job-problems/

2.http://lucene.472066.n3.nabble.com/Trouble-with-Word-Count-example-td4023269.html

3.http://stackoverflow.com/questions/22850532/warn-mapred-jobclient-no-job-jar-file-set-user-classes-may-not-be-found