hadoop streaming 编程参数设置

运行python hadoop streaming 参数如下时：

hadoop jar
$HADOOP_HOME/contrib/hadoop-streaming.jar

-mapper /usr/local/mapper.py                   //本地mapper.py文件地址

-reducer /usr/local/reducer.py                    //本地reducer.py文件地址

-input /user/root/test/word                          //hdfs文件地址

-output /user/root/test/output                      //hdfs 文件地址

会出现如下错误：

Error: java.lang.RuntimeException: Error in configuring object

        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)

        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)

        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)

        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)

        at java.security.AccessController.doPrivileged(Native Method)

        at javax.security.auth.Subject.doAs(Subject.java:415)

        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)

        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)

        ... 9 more

Caused by: java.lang.RuntimeException: Error in configuring object

        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)

        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)

        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)

        at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)

        ... 14 more

Caused by: java.lang.reflect.InvocationTargetException

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)

        ... 17 more

Caused by: java.lang.RuntimeException: configuration exception

        at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:222)

        at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66)

        ... 22 more

Caused by: java.io.IOException: Cannot run program "/usr/local/mapper.py": error=2, No such file or directory

        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1041)

        at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:209)

        ... 23 more

Caused by: java.io.IOException: error=2, No such file or directory

        at java.lang.UNIXProcess.forkAndExec(Native Method)

        at java.lang.UNIXProcess.<init>(UNIXProcess.java:186)

        at java.lang.ProcessImpl.start(ProcessImpl.java:130)

        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1022)

正确参数设置如下：

hadoop jar $HADOOP_HOME/contrib/hadoop-streaming.jar
-files  /usr/local/mapper.py,/usr/local/reducer.py  //此处也可以为 -file /usr/local/mapper.py -file /usr/local/reducer.py 
-mapper mapper.py                    //本地mapper.py文件地址
-reducer reducer.py                    //本地reducer.py文件地址
-input /user/root/test/word           //hdfs文件地址
-output /user/root/test/output      //hdfs 文件地址
如果想要设置map和reduce任务数据，在-input前添加-D mapred.map.tasks=num -D mapred.reduce.tasks=num

注意：-D操作设置在-input和之前。