Zeppelin spark.executor.extraClassPath 和 --driver-class-path 配置冲突问题
2017-06-27 17:35
716 查看
报错如下:
WARN [2017-06-27 15:47:59,777] ({pool-2-thread-2} Logging.scala[logWarning]:66) -
SPARK_CLASSPATH was detected (set to '/home/raini/spark/lib/mysql-connector-java-5.1.38-bin.jar:').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --driver-class-path to augment the driver classpath
- spark.executor.extraClassPath to augment the executor classpath
WARN [2017-06-27 15:47:59,778] ({pool-2-thread-2} Logging.scala[logWarning]:66) - Setting 'spark.executor.extraClassPath' to '/home/raini/spark/lib/mysql-connector-java-5.1.38-bin.jar:'
as a work-around.
ERROR [2017-06-27 15:47:59,780] ({pool-2-thread-2} Logging.scala[logError]:91) - Error initializing SparkContext.
org.apache.spark.SparkException: Found both spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former.
at org.apache.spark.SparkConf$$anonfun$validateSettings$7$$anonfun$apply$8.apply(SparkConf.scala:543)
at org.apache.spark.SparkConf$$anonfun$validateSettings$7$$anonfun$apply$8.apply(SparkConf.scala:541)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.SparkConf$$anonfun$validateSettings$7.apply(SparkConf.scala:541)
at org.apache.spark.SparkConf$$anonfun$validateSettings$7.apply(SparkConf.scala:529)
解决:
1.提交的作业可以通过加入--driver-class-path参数来设置driver的classpath。
$ bin/spark-submit --master local[2] --driver-class-path lib/mysql-connector-java-5.1.35.jar --class
spark.SparkToJDBC ./spark-test_2.10-1.0.jar
2.其实,我们还可以在spark安装包的conf/spark-env.sh通过配置SPARK_CLASSPATH来设置driver的环境变量,如下:
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/iteblog/com/mysql-connector-java-5.1.35.jar
这样也可以解决上面出现的异常。但是,我们不能同时在conf/spark-env.sh里面配置SPARK_CLASSPATH和提交作业加上–driver-class-path参数,否则会出现以上异常。
所以,删掉一个配置即可,这里删掉了spark配置项:export SPARK_CLASSPATH=...
WARN [2017-06-27 15:47:59,777] ({pool-2-thread-2} Logging.scala[logWarning]:66) -
SPARK_CLASSPATH was detected (set to '/home/raini/spark/lib/mysql-connector-java-5.1.38-bin.jar:').
This is deprecated in Spark 1.0+.
Please instead use:
- ./spark-submit with --driver-class-path to augment the driver classpath
- spark.executor.extraClassPath to augment the executor classpath
WARN [2017-06-27 15:47:59,778] ({pool-2-thread-2} Logging.scala[logWarning]:66) - Setting 'spark.executor.extraClassPath' to '/home/raini/spark/lib/mysql-connector-java-5.1.38-bin.jar:'
as a work-around.
ERROR [2017-06-27 15:47:59,780] ({pool-2-thread-2} Logging.scala[logError]:91) - Error initializing SparkContext.
org.apache.spark.SparkException: Found both spark.driver.extraClassPath and SPARK_CLASSPATH. Use only the former.
at org.apache.spark.SparkConf$$anonfun$validateSettings$7$$anonfun$apply$8.apply(SparkConf.scala:543)
at org.apache.spark.SparkConf$$anonfun$validateSettings$7$$anonfun$apply$8.apply(SparkConf.scala:541)
at scala.collection.immutable.List.foreach(List.scala:381)
at org.apache.spark.SparkConf$$anonfun$validateSettings$7.apply(SparkConf.scala:541)
at org.apache.spark.SparkConf$$anonfun$validateSettings$7.apply(SparkConf.scala:529)
解决:
1.提交的作业可以通过加入--driver-class-path参数来设置driver的classpath。
$ bin/spark-submit --master local[2] --driver-class-path lib/mysql-connector-java-5.1.35.jar --class
spark.SparkToJDBC ./spark-test_2.10-1.0.jar
2.其实,我们还可以在spark安装包的conf/spark-env.sh通过配置SPARK_CLASSPATH来设置driver的环境变量,如下:
export SPARK_CLASSPATH=$SPARK_CLASSPATH:/iteblog/com/mysql-connector-java-5.1.35.jar
这样也可以解决上面出现的异常。但是,我们不能同时在conf/spark-env.sh里面配置SPARK_CLASSPATH和提交作业加上–driver-class-path参数,否则会出现以上异常。
所以,删掉一个配置即可,这里删掉了spark配置项:export SPARK_CLASSPATH=...
相关文章推荐
- Cannot find class: ${jdbc.driver}——配置了sqlSessionFactoryBeanName也报错之问题分析
- sessionFactory' defined in class path /mappingDirectoryLocations配置问题
- 解决配置数据库时Could not load JDBC driver class问题
- web.xml 配制加载spring上下文及 Spring context-param param-value classpath 配置加载问题
- linux安装多tomcat的配置方法(常见问题:Cannot find //bin/setclasspath.sh)
- IBatis配置tomcat的JNDI问题解决Cannot create JDBC driver of class '' for connect URL 'null'
- weblogic 配置中the jdbc driver is not on the classpath解决
- linux安装多tomcat的配置方法(常见问题:Cannot find //bin/setclasspath.sh)
- 使用JNDI配置数据池问题Cannot create JDBC driver of class 'com.microsoft.sqlserver.jdbc.SQLServerDriver' f解决方法
- java classpath配置需要注意的问题
- 有关java环境配置的 JAVA_HOME path classpath 问题
- jbuilder中的Class.forName("com.microsoft.jdbc.sqlserver.SQLServerDriver");问题
- 解决 RNI接口调用出现的异常UnsatisfiedLinkError: no jtokyocabinet in java.library.path 及 wrong ELF class: ELFCLASS64问题
- Spring 配置问题还是代码问题?Error configuring application listener of class org.springframework.web.context.ContextLoaderListener
- 解决com.sun.tools.javac.Main is not on the classpath的问题
- 遇到的问题:Cannot load JDBC driver class 'oracle.jdbc.OracleDriver'
- java 运行环境配置 path classpath
- jbpm4.4和tomcat6包冲突问题(javax.el.ELException: expression factory class not found )
- Tomcat + SQL Server 2005 + jsp : ClassNotFoundException:com.microsoft.sqlserver.jdbc.SQLServerDriver 问题的解决