Hadoop on Mac with IntelliJ IDEA - 9 解决Type mismatch in value from map问题
2014-10-07 13:34
801 查看
修改陆喜恒. Hadoop实战(第2版)5.3排序的代码时遇到IO异常。
环境:Mac OS X 10.9.5, IntelliJ IDEA 13.1.5, Hadoop 1.2.1
异常具体信息如下
相关代码如下
Map的输出与Reduce的输入类型相符。但是,根据错误信息,Map的value值预期为IntWritable,接受到的却是Text,两者类型不匹配。另外,错误提示与代码定义也不同,因为定义声明了map阶段输出值为<IntWritable, Text>。
造成这个问题的原因等以后阅读源码时再分析,先处理问题。配置作业的MapOutputKeyClass和MapOutputValueClass参数,将其设置成Map输出类型即可。
环境:Mac OS X 10.9.5, IntelliJ IDEA 13.1.5, Hadoop 1.2.1
异常具体信息如下
14/10/06 03:08:51 INFO mapred.JobClient: Task Id : attempt_201410021756_0043_m_000000_0, Status : FAILED java.io.IOException: Type mismatch in value from map: expected org.apache.hadoop.io.IntWritable, recieved org.apache.hadoop.io.Text at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1024) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:690) at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80) at main.ch5.ReSort$Map.map(ReSort.java:51) at main.ch5.ReSort$Map.map(ReSort.java:43) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) at org.apache.hadoop.mapred.Child.main(Child.java:249)
相关代码如下
public static class Map extends Mapper<LongWritable, Text, IntWritable, Text> { // ... } public static class Reduce extends Reducer<IntWritable, Text, IntWritable, IntWritable> { // ... } public static void main(String[] args){ // ... job.setOutputFormatClass(TextOutputFormat.class); job.setOutputKeyClass(IntWritable.class); job.setOutputValueClass(IntWritable.class); // ... }
Map的输出与Reduce的输入类型相符。但是,根据错误信息,Map的value值预期为IntWritable,接受到的却是Text,两者类型不匹配。另外,错误提示与代码定义也不同,因为定义声明了map阶段输出值为<IntWritable, Text>。
造成这个问题的原因等以后阅读源码时再分析,先处理问题。配置作业的MapOutputKeyClass和MapOutputValueClass参数,将其设置成Map输出类型即可。
job.setMapOutputKeyClass(/*K2*/IntWritable.class); job.setMapOutputValueClass(/*V2*/Text.class);
相关文章推荐
- hadoop问题Type mismatch in value from map解决方法
- hadoop问题Type mismatch in value from map解决方法
- Hadoop on Mac with IntelliJ IDEA - 6 解决KeyValueTextInputFormat读取时只有key值问题
- hadoop问题Type mismatch in value from map解决方法
- hadoop-1.0.4 Type mismatch in value from map解决方法
- Hadoop on Mac with IntelliJ IDEA - 3 解决MRUnit - No applicable class implementing Serialization问题
- Hadoop on Mac with IntelliJ IDEA - 5 解决java heap space问题
- Hadoop on Mac with IntelliJ IDEA - 7 解决failed to report status for 600 seconds. Killing!问题
- hadoop下 Type mismatch in value from map的问题
- Hadoop on Mac with IntelliJ IDEA - 2 解决URI错误导致Permission denied
- hadoop中mapreduce的 Type mismatch in value from map错误
- Hadoop on Mac with IntelliJ IDEA - 1 解决input path does not exist问题
- 问题: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org.apache.hadoop.io.LongWritable 解决方案
- hadoop中mapreduce的 Type mismatch in value from map错误
- java.io.IOException: Type mismatch in key from map:解决方法
- Type mismatch in key from map: expected org.apache.hadoop.io.LongWritable, received org.apache.hadoo
- Hadoop on Mac with IntelliJ IDEA - 8 单表关联NullPointerException
- Hadoop on Mac with IntelliJ IDEA - 4 制作jar包
- Hadoop on Mac with IntelliJ IDEA - 10 陆喜恒. Hadoop实战(第2版)6.4.1(Shuffle和排序)Map端 内容整理
- java.io.IOException: Type mismatch in key from map: expected org.apache.hadoop.io.Text, recieved org