win7上eclipse提交mr至hadoop2.6分布式运行错误集锦
2015-01-14 17:43
399 查看
1、下载hadoop2.6,编译插件
2、Eclipse安装插件
3、安装maven插件,配置jar包,实现自动将工程打包jar
4、创建mapreduce工程,运行wordcount
5、在其中遇到各种错误
(1)java.lang.NullPointerException
修复方法:
下载hadoop-common-2.2.0-bin-master,解压将其中的winutils.exe和hadoop.dll拷贝到HADOOP_HOME/bin下
(2)Exception in thread "main" java.lang.UnsatisfiedLinkError:ConfiguredFailoverProxyProvider
修复方法:
在继承的抽象类ConfiguredFailoverProxyProvider中增加抽象方法
public abstract boolean useLogicalURI();
public abstract ProxyInfo<T> getProxy();
public abstract Class<T> getInterface();
public abstract void performFailover(T currentProxy);
public abstract void close() throws IOException;;
(3)Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method)
修复方法:在hadoop2.6源码里找到NativeCrc32.java,创建与源码一样的包名,拷贝NativeCrc32.java到该包工程目录下。
(4)org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control
修改代码
http://www.imooo.com/windows/windows/1330494.htm
YarnRunner.java的390行 (Apache Hadoop2.6的源码)
// Setup the command to run the AM List<String> vargs = new ArrayList<String>(8); vargs.add(Environment.JAVA_HOME.$() + "/bin/java");
改为
vargs.add("$JAVA_HOME/bin/java");
在YarnRunner.java类中,新增一个路径转换的方法
private void replaceEnvironment(Map<String, String> environment) { String tmpClassPath = environment.get("CLASSPATH"); tmpClassPath=tmpClassPath.replaceAll(";", ":"); tmpClassPath=tmpClassPath.replaceAll("%PWD%", "\\$PWD"); tmpClassPath=tmpClassPath.replaceAll("%HADOOP_MAPRED_HOME%", "\\$HADOOP_MAPRED_HOME"); tmpClassPath= tmpClassPath.replaceAll("\\\\", "/" ); environment.put("CLASSPATH",tmpClassPath); }
在YarnRunner.java的在466行添加:
replaceEnvironment(environment);
(5)Application application_1420601118686_0001 failed 2 times due to AM Container for appattempt_1420601118686_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://master:8088/proxy/application_1420601118686_0001/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1420601118686_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
添加路径:
Mapreduce-site.xml
<property>
<name>mapreduce.application.classpath</name>
<value>
/opt/app/hadoop-2.6.0/etc/hadoop,
/opt/app/hadoop-2.6.0/share/hadoop/common/*,
/opt/app/hadoop-2.6.0/share/hadoop/common/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/lib/*
</value>
</property>
Yarn-site.xml
<property>
<name>yarn.application.classpath</name>
<value>
/opt/app/hadoop-2.6.0/etc/hadoop,
/opt/app/hadoop-2.6.0/share/hadoop/common/*,
/opt/app/hadoop-2.6.0/share/hadoop/common/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/lib/*
</value>
</property>
2、Eclipse安装插件
3、安装maven插件,配置jar包,实现自动将工程打包jar
4、创建mapreduce工程,运行wordcount
5、在其中遇到各种错误
(1)java.lang.NullPointerException
修复方法:
下载hadoop-common-2.2.0-bin-master,解压将其中的winutils.exe和hadoop.dll拷贝到HADOOP_HOME/bin下
(2)Exception in thread "main" java.lang.UnsatisfiedLinkError:ConfiguredFailoverProxyProvider
修复方法:
在继承的抽象类ConfiguredFailoverProxyProvider中增加抽象方法
public abstract boolean useLogicalURI();
public abstract ProxyInfo<T> getProxy();
public abstract Class<T> getInterface();
public abstract void performFailover(T currentProxy);
public abstract void close() throws IOException;;
(3)Exception in thread "main" java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(II[BI[BIILjava/lang/String;JZ)V
at org.apache.hadoop.util.NativeCrc32.nativeComputeChunkedSumsByteArray(Native Method)
修复方法:在hadoop2.6源码里找到NativeCrc32.java,创建与源码一样的包名,拷贝NativeCrc32.java到该包工程目录下。
(4)org.apache.hadoop.util.Shell$ExitCodeException: /bin/bash: line 0: fg: no job control
修改代码
http://www.imooo.com/windows/windows/1330494.htm
YarnRunner.java的390行 (Apache Hadoop2.6的源码)
// Setup the command to run the AM List<String> vargs = new ArrayList<String>(8); vargs.add(Environment.JAVA_HOME.$() + "/bin/java");
改为
vargs.add("$JAVA_HOME/bin/java");
在YarnRunner.java类中,新增一个路径转换的方法
private void replaceEnvironment(Map<String, String> environment) { String tmpClassPath = environment.get("CLASSPATH"); tmpClassPath=tmpClassPath.replaceAll(";", ":"); tmpClassPath=tmpClassPath.replaceAll("%PWD%", "\\$PWD"); tmpClassPath=tmpClassPath.replaceAll("%HADOOP_MAPRED_HOME%", "\\$HADOOP_MAPRED_HOME"); tmpClassPath= tmpClassPath.replaceAll("\\\\", "/" ); environment.put("CLASSPATH",tmpClassPath); }
在YarnRunner.java的在466行添加:
replaceEnvironment(environment);
(5)Application application_1420601118686_0001 failed 2 times due to AM Container for appattempt_1420601118686_0001_000002 exited with exitCode: 1
For more detailed output, check application tracking page:http://master:8088/proxy/application_1420601118686_0001/Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1420601118686_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
添加路径:
Mapreduce-site.xml
<property>
<name>mapreduce.application.classpath</name>
<value>
/opt/app/hadoop-2.6.0/etc/hadoop,
/opt/app/hadoop-2.6.0/share/hadoop/common/*,
/opt/app/hadoop-2.6.0/share/hadoop/common/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/lib/*
</value>
</property>
Yarn-site.xml
<property>
<name>yarn.application.classpath</name>
<value>
/opt/app/hadoop-2.6.0/etc/hadoop,
/opt/app/hadoop-2.6.0/share/hadoop/common/*,
/opt/app/hadoop-2.6.0/share/hadoop/common/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/*,
/opt/app/hadoop-2.6.0/share/hadoop/hdfs/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/*,
/opt/app/hadoop-2.6.0/share/hadoop/mapreduce/lib/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/*,
/opt/app/hadoop-2.6.0/share/hadoop/yarn/lib/*
</value>
</property>
相关文章推荐
- 在linux下eclipse中运行hadoop自带的WordCount例子出现的两个错误
- Win7下eclipse提交Job到hadoop集群
- Permission denied: win7下面eclipse上传本地文件到DFS && 运行M/R程序时出现的同样的错误解决方法
- win7环境Eclipse中运行JAVA项目读取HDFS(hadoop-2.7.2)中的文件实例(2)
- mac电脑的eclipse把mapreduce程序提交到hadoop2.x集群虚拟机上运行
- Win7下快速搭建hadoopEclipse运行环境--方案
- win7下安装hadoop 2.6.0 的eclipse插件并编写运行WordCount程序
- windows eclipse远程连接hadoop集群并提交任务运行
- win7环境Eclipse中运行JAVA项目读取HDFS(hadoop-2.7.2)中的文件实例与问题汇总
- win7使用eclipse连接hadoop集群,运行mapreduce报错之:org.apache.hadoop.security.AccessControlException
- 在Hadoop 2.3上运行C++程序各种疑难杂症(Hadoop Pipes选择、错误集锦、Hadoop2.3编译等)
- 基于Hive的hadoop日志的分析及错误排解(eclipse上运行出现错误:Access denied for user ... (using password: YES))
- Eclipse调用hadoop2运行MR程序
- 从window中的eclipse中提交jar包到yarn框架运行,出现Exception from container-launch: org.apache.hadoop.util.Shell$
- 在Eclipse下开发Hadoop程序并提交到集群中运行
- 在Hadoop 2.3上运行C++程序各种疑难杂症(Hadoop Pipes选择、错误集锦、Hadoop2.3编译等)
- Eclipse打包mapreduce程序并提交至hadoop集群运行
- win7下Eclipse4.4.0+JDK1.8配置hadoop2.5.1插件遇到的各种错误
- 在win7上的eclipse向hadoop提交作业异常-权限/设置调度器
- Hadoop中MR程序的几种提交运行模式