spark on yarn运行产生缺jar包错误及解决办法

spark on yarn运行产生缺jar包错误及解决办法

2014-12-10 19:57:44http://blog.csdn.net/mach_learn/article/details/41824737--点击数: 1246



<iframe id="cproIframe_u848856_2" height="280" marginheight="0" src="http://pos.baidu.com/acom?adn=3&at=231&aurl=&cad=1&ccd=32&cec=utf-8&cfv=0&ch=0&col=zh-cn&conBW=0&conOP=1&cpa=1&dai=2&dis=0&ltr=http%3A%2F%2Fwww.baidu.com%2Flink%3Furl%3DqT_bdfmrTLnKqp_Yn875Q-vKjcOP35XdSNSiXNpTuSPDLhtFcq6axCHat2Q6YI7umyjx6tBUEYIoll_P73eQ5q%26wd%3D%26eqid%3Df4937c8a00078fa500000004560f46e4&ltu=http%3A%2F%2Fwww.th7.cn%2FProgram%2Fjava%2F201412%2F329531.shtml&lu_161=10&lunum=6&n=smileking_cpr&pcs=1012x561&pis=10000x10000&ps=-2x-2&psr=1920x1080&pss=1012x561&qn=52a58d0dd1e46970&rad=&rsi0=336&rsi1=280&rsi5=4&rss0=%23FFFFFF&rss1=%23FFFFFF&rss2=%230000ff&rss3=%23444444&rss4=%23444444&rss5=&rss6=%23e10900&rss7=&scale=&skin=tabcloud_skin_3&stid=5&td_id=848856&titFF=%E5%AE%8B%E4%BD%93&titFS=12&titTA=left&tn=text_default_336_280&tpr=1443842207406&ts=1&version=2.0&xuanting=0&dtm=BAIDU_DUP2_SETJSONADSLOT&dc=2&di=u848856&ti=spark%20on%20yarn%E8%BF%90%E8%A1%8C%E4%BA%A7%E7%94%9F%E7%BC%BAjar%E5%8C%85%E9%94%99%E8%AF%AF%E5%8F%8A%E8%A7%A3%E5%86%B3%E5%8A%9E%E6%B3%95-Java-%E7%AC%AC%E4%B8%83%E5%9F%8E%E5%B8%82&tt=1443842207312.516.516.516" frameborder="0" width="336" allowtransparency="" marginwidth="0" scrolling="no" align="center,center" _xhe_src="http://pos.baidu.com/acom?adn=3&at=231&aurl=&cad=1&ccd=32&cec=utf-8&cfv=0&ch=0&col=zh-cn&conBW=0&conOP=1&cpa=1&dai=2&dis=0&ltr=http%3A%2F%2Fwww.baidu.com%2Flink%3Furl%3DqT_bdfmrTLnKqp_Yn875Q-vKjcOP35XdSNSiXNpTuSPDLhtFcq6axCHat2Q6YI7umyjx6tBUEYIoll_P73eQ5q%26wd%3D%26eqid%3Df4937c8a00078fa500000004560f46e4&ltu=http%3A%2F%2Fwww.th7.cn%2FProgram%2Fjava%2F201412%2F329531.shtml&lu_161=10&lunum=6&n=smileking_cpr&pcs=1012x561&pis=10000x10000&ps=-2x-2&psr=1920x1080&pss=1012x561&qn=52a58d0dd1e46970&rad=&rsi0=336&rsi1=280&rsi5=4&rss0=%23FFFFFF&rss1=%23FFFFFF&rss2=%230000ff&rss3=%23444444&rss4=%23444444&rss5=&rss6=%23e10900&rss7=&scale=&skin=tabcloud_skin_3&stid=5&td_id=848856&titFF=%E5%AE%8B%E4%BD%93&titFS=12&titTA=left&tn=text_default_336_280&tpr=1443842207406&ts=1&version=2.0&xuanting=0&dtm=BAIDU_DUP2_SETJSONADSLOT&dc=2&di=u848856&ti=spark%20on%20yarn%E8%BF%90%E8%A1%8C%E4%BA%A7%E7%94%9F%E7%BC%BAjar%E5%8C%85%E9%94%99%E8%AF%AF%E5%8F%8A%E8%A7%A3%E5%86%B3%E5%8A%9E%E6%B3%95-Java-%E7%AC%AC%E4%B8%83%E5%9F%8E%E5%B8%82&tt=1443842207312.516.516.516"></iframe>

1、本地运行出错及解决办法

当运行如下命令时：

./bin/spark-submit /--class org.apache.spark.examples.mllib.JavaALS /--master local[*] //opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop-yarn/lib/spark-examples_2.10-1.0.0-cdh5.1.2.jar //user/data/netflix_rating 10 10 /user/data/result

会出现如下错误：

Exception in thread "main" java.lang.RuntimeException: java.io.IOException: No FileSystem for scheme: hdfs	at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:657)	at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:389)	at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:362)	at org.apache.spark.SparkContext$$anonfun$22.apply(SparkContext.scala:546)	at org.apache.spark.SparkContext$$anonfun$22.apply(SparkContext.scala:546)	at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:145)	at org.apache.spark.rdd.HadoopRDD$$anonfun$getJobConf$1.apply(HadoopRDD.scala:145)	at scala.Option.map(Option.scala:145)	at org.apache.spark.rdd.HadoopRDD.getJobConf(HadoopRDD.scala:145)	at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:168)	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)	at scala.Option.getOrElse(Option.scala:120)	at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)	at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)	at scala.Option.getOrElse(Option.scala:120)	at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)	at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:204)	at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:202)	at scala.Option.getOrElse(Option.scala:120)	at org.apache.spark.rdd.RDD.partitions(RDD.scala:202)	at org.apache.spark.mllib.recommendation.ALS.run(ALS.scala:167)	at org.apache.spark.mllib.recommendation.ALS$.train(ALS.scala:599)	at org.apache.spark.mllib.recommendation.ALS.train(ALS.scala)	at org.apache.spark.examples.mllib.JavaALS.main(JavaALS.java:80)	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)	at java.lang.reflect.Method.invoke(Method.java:597)	at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: java.io.IOException: No FileSystem for scheme: hdfs	at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2385)	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)	at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:653)	... 34 more

出现此错误的原因为spark执行过程中缺少hadoop-hdfs的jar包，使用spark-submit中的--jar或者--driver-class-path参数可以解决此问题。当使用hadoop-hdfs时路径指的就是hdfs路径。

正确的执行方式如下：

./bin/spark-submit /--class org.apache.spark.examples.mllib.JavaALS /--driver-class-path /opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop-hdfs/hadoop-hdfs-2.3.0-cdh5.1.2.jar /--master local[*] //opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop-yarn/lib/spark-examples_2.10-1.0.0-cdh5.1.2.jar //user/data/netflix_rating 10 10 /user/data/result或者./bin/spark-submit /--class org.apache.spark.examples.mllib.JavaALS /--jars /opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop-hdfs/hadoop-hdfs-2.3.0-cdh5.1.2.jar /--master local[*] //opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop-yarn/lib/spark-examples_2.10-1.0.0-cdh5.1.2.jar //user/data/netflix_rating 10 10 /user/data/result

2、spark在yarn上运行错误及解决办法

当运行如下命令时：

./bin/spark-submit /--class org.apache.spark.examples.mllib.JavaALS /--master yarn-cluster //opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop-yarn/lib/spark-examples_2.10-1.0.0-cdh5.1.2.jar //user/data/netflix_rating 10 10 /user/data/result

会出现如下错误：

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/client/api/impl/YarnClientImpl	at java.lang.ClassLoader.defineClass1(Native Method)	at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)	at java.lang.ClassLoader.defineClass(ClassLoader.java:615)	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)	at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)	at java.net.URLClassLoader.access$000(URLClassLoader.java:58)	at java.net.URLClassLoader$1.run(URLClassLoader.java:197)	at java.security.AccessController.doPrivileged(Native Method)	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)	at java.lang.Class.forName0(Native Method)	at java.lang.Class.forName(Class.java:247)	at org.apache.spark.util.Utils$$anonfun$classIsLoadable$1.apply(Utils.scala:143)	at org.apache.spark.util.Utils$$anonfun$classIsLoadable$1.apply(Utils.scala:143)	at scala.util.Try$.apply(Try.scala:161)	at org.apache.spark.util.Utils$.classIsLoadable(Utils.scala:143)	at org.apache.spark.deploy.SparkSubmit$.createLaunchEnv(SparkSubmit.scala:158)	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:54)	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.yarn.client.api.impl.YarnClientImpl	at java.net.URLClassLoader$1.run(URLClassLoader.java:202)	at java.security.AccessController.doPrivileged(Native Method)	at java.net.URLClassLoader.findClass(URLClassLoader.java:190)	at java.lang.ClassLoader.loadClass(ClassLoader.java:306)	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)	at java.lang.ClassLoader.loadClass(ClassLoader.java:247)	... 21 more

产生此错误的原因是缺少hadoop-yarn目录下的jar包，解决此问题的方法只能使用--driver-class-path参数，原因是执行spark on yarn时，需要提前将
hadoop-yarn目录下的jar包导入 。

正确的执行方式如下：

./bin/spark-submit /--class org.apache.spark.examples.mllib.JavaALS /--master yarn-cluster /--driver-class-path $(echo /opt/cloudera/parcels/CDH/lib/hadoop-yarn/*.jar |sed 's/ /:/g'):/opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop-hdfs/hadoop-hdfs-2.3.0-cdh5.1.2.jar //opt/cloudera/parcels/CDH-5.1.2-1.cdh5.1.2.p0.3/lib/hadoop-yarn/lib/spark-examples_2.10-1.0.0-cdh5.1.2.jar //user/data/netflix_rating 10 10 /user/data/result

执行结果集如下图所示


/user/data/result/productFeatures/part-00000数据格式为：

22,[5.561720883259194, 1.8295046510786157, 1.456597387276617, 0.8851233321058966, -0.6750794769961516, 0.2105431165110079, 1.868136268816477, -0.7426684616337039, -0.5856268982634872, 2.2788288132587358] 31,[0.9093801231293616, 0.31519780093777366, 1.1875509370524693,
0.40381375438624073, 2.518833489342341, 1.4242427194658087, 2.0950977044322574, 0.9012256614215569, 1.1604700989497398, 0.15791920617498328] 76,[1.8285525546730474, 0.6330058247735413, 2.5686801366906984, 1.4128062599776998, 1.401816974160943, 0.1596137900376602,
1.5625150218484072, -0.9678843308247949, 2.682242352514027, 1.0599465865866935] 152,[0.014905493368344078, 0.43308346940343456, 0.2351848253710811, 0.26220235713374834, 0.055210836978533295, 0.21723689234341548, 0.09391052568889097, 0.7231946368850907, 0.02497671848923523,
0.5022350772242716] 206,[-0.5501117679008718, 0.4105849318486638, 1.0876481291363873, 2.233025299808942, 2.1038565118723387, 1.662798954470802, 1.575332336431819, 0.8167712158963146, 1.4536436809654083, -0.5224582242822096]

/user/data/result/userFeatures/part-00000数据格式为：

22,[0.18595332423070562, 0.26223861694267697, 0.2220917583718615, 0.015729079507204886, 0.4450456773474982, 0.12287125816024044, 0.4644319181495295, 0.38377345920108646, 0.28428991637647794, 0.17875507467819415] 31,[0.15133710263843259, -0.02354886937021699,
0.10618787396390789, 0.03258147800653979, 0.3556889855610244, 1.021110467423965, 0.3701959855785832, 0.1524124835894395, 0.23381646690418442, -0.012011907243505829] 76,[0.2344438657777155, 0.03821305024729112, 0.230093903321136, 0.48888224387617607, 0.30121869825786685,
0.48198504753122795, 0.29543641416718835, 0.39299434584620146, 0.27798068299013984, 0.15611605797193095] 121,[0.2038917971256244, 0.7576071991072084, 0.30603993855416245, 0.41995044224403344, 0.06550681386608997, 0.20395370870960078, 0.3444359097858106, 0.4935457123179016,
0.2041119263872145, 0.3518582534508109] 130,[0.042995762604581524, -0.21177745644812881, 0.7047019111940551, 0.44978429350262916, 0.18912686527984246, 0.6349887274906566, 0.29651737861710675, 0.49758500548973844, 0.02699514514764544, 0.39330900998421187] 152,[1.9989336762046868,
1.2185456627280438, -0.14465791504370654, 0.32972894935630664, -0.6316151112173617, -0.5568528040594881, 0.007477525352213408, -0.012087520291972442, 0.4184613236246099, -0.24669307203702268]

3、对于spark on hadoop执行中出行的问题需要去hadoop yarn对应的job日志中去看