sparksql 提交到yarn出现 java.lang.NoClassDefFoundError: Lorg/apache/hadoop/hive/ql/plan/TableDesc
2015-06-23 12:12
1036 查看
出现这个问题,是因为spark的excutor执行的时候,缺少hive的依赖包,添加依赖包的参数是:
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*"
下面是是一个例子:
spark-submit --class com.simple.spark.Test \
--master yarn-client \
--num-executors 2 \
--driver-memory 600m \
--executor-memory 600m \
--executor-cores 1 \
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*" \
--jars /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/hbase-common.jar test.jar
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*"
下面是是一个例子:
spark-submit --class com.simple.spark.Test \
--master yarn-client \
--num-executors 2 \
--driver-memory 600m \
--executor-memory 600m \
--executor-cores 1 \
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/*" \
--jars /opt/cloudera/parcels/CDH-5.3.1-1.cdh5.3.1.p0.5/lib/hive/lib/hbase-common.jar test.jar
相关文章推荐
- 错误处理:java.lang.NoSuchMethodException: org.apache.catalina.deploy.WebXml addFilter
- Apache Jakarta 项目介绍
- WARN org.apache.hadoop.hdfs.servåer.namenode.FSNamesystem: Get corrupt file b
- Apache Shiro介绍
- 安全认证框架-apache shiro研究心得
- apache2.2 虚拟主机配置
- Apache2.4在Windows下配置出错(Starting 64 worker threads)
- Apache Windows下无权限访问Server文件目录解决办法
- 【Java.Shiro】让Apache Shiro保护你的应用
- CentOS6.5安装LAMP环境APACHE的安装
- apache2.2 虚拟主机配置
- Tips & Tricks:Apache log4j简明教程(一)
- Mac OS X 系统配置 Apache+MySql+PHP 详细教程
- 重启apache
- Apache配置虚拟主机
- apache chiro 使用手册
- org.apache.jasper.JasperException: Unable to compile class for JSP 解决方法
- org.apache.jasper.JasperException The value for the useBean class attribute class.MyBean is invalid.
- Apache--Override参数详解
- thinkphp项目在apache服务器中“去掉”index.php后出现找不到url的问题