hive-错误-处理解压文件gz出错

gz压缩文件上传到hdfs，hive读取处理

Task with the most failures(4):

-----

Task ID:

task_1456816082333_1354_m_000339

URL:
http://xxxx:8088/taskdetails.jsp?jobid=job_1456816082333_1354&tipid=task_1456816082333_1354_m_000339
-----

Diagnostic Messages for this Task:

Error: java.io.IOException: java.io.EOFException: Unexpected end of input stream

at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)

at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)

at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:344)

at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)

at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)

at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:122)

at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:197)

at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:183)

at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)

Caused by: java.io.EOFException: Unexpected end of input stream

at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:145)

at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)

at java.io.InputStream.read(InputStream.java:101)

at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:211)

at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)

at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:206)

at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:45)

at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:339)

... 13 more

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

MapReduce Jobs Launched:

Job 0: Map: 358 Cumulative CPU: 5067.56 sec HDFS Read: 2749408714 HDFS Write: 2747850422 FAIL

Total MapReduce CPU Time Spent: 0 days 1 hours 24 minutes 27 seconds 560 msec

出现这个错误，明细是压缩文件损坏，处理方式应是删除hdfs对应的文件，重新上传

可hive表或者分区内，有几万个文件，识别不了是哪个压缩文件，想办法找出来，寻找方式：

执行hive任务的时候，进入到8088的map详细进度列表，即是RUNNING MAP attempts in job_1456816082333_1354，查看最后出错的map是哪个节点或者在页面直接点击logs进入详细log日志查看，或者进入到节点的hadoop的logs/userlogs目录，根据jobid找到对应的目录： application_1456816082333_1354，里面有错误的文件id，然后删除掉hdfs的对应的损坏文件。