spark-sql部署实现与Hive交互
2016-04-11 15:14
375 查看
spark-sql部署
版本
Hadoop-2.5.0-cdh5.3.2
Hive-0.13.1-cdh5.3.2
Spark-1.5.1
以CNSH001节点为例
spark master在CNSH001上:spark://CNSH001:7077
spark HistoryServer在CNSH001上:CNSH001:8032
spark eventLog在hdfs上:hdfs://testenv/spark/eventLog
1. 拷贝$HIVE_HOME/conf/hive-site.xml, hive-log4j.properties 到 $SPARK_HOME/conf/目录
2.修改spark-defaults.conf
[plain] view
plain copy
spark.eventLog.enabled true
spark.eventLog.dir hdfs://testenv/spark/eventLog
spark.eventLog.compress true
spark.yarn.historyServer.address=CNSH001:8032
spark.sql.hive.metastore.version=0.13.1
spark.port.maxRetries=100
spark.sql.hive.metastore.jars=/opt/apps/hadoop/share/hadoop/mapreduce/*:/opt/apps/hadoop/share/hadoop/mapreduce/lib/*:/opt/apps/hadoop/share/hadoop/common/*:/opt/apps/hadoop/share/hadoop/common/lib/*:/opt/apps/hadoop/share/hadoop/hdfs/*
:/opt/apps/hadoop/share/hadoop/hdfs/lib/*:/opt/apps/hadoop/share/hadoop/yarn/*:/opt/apps/hadoop/share/hadoop/yarn/lib/*:/opt/apps/hive/lib/*:/opt/apps/spark/lib/*
spark.driver.extraLibraryPath=/opt/apps/hadoop/share/hadoop/mapreduce/*:/opt/apps/hadoop/share/hadoop/mapreduce/lib/*:/opt/apps/hadoop/share/hadoop/common/*:/opt/apps/hadoop/share/hadoop/common/lib/*:/opt/apps/hadoop/share/hadoop/hdfs/*
:/opt/apps/hadoop/share/hadoop/hdfs/lib/*:/opt/apps/hadoop/share/hadoop/yarn/*:/opt/apps/hadoop/share/hadoop/yarn/lib/*:/opt/apps/hive/lib/*:/opt/apps/spark/lib/*
spark.executor.extraLibraryPath=/opt/apps/hadoop/share/hadoop/mapreduce/*:/opt/apps/hadoop/share/hadoop/mapreduce/lib/*:/opt/apps/hadoop/share/hadoop/common/*:/opt/apps/hadoop/share/hadoop/common/lib/*:/opt/apps/hadoop/share/hadoop/hdfs
/*:/opt/apps/hadoop/share/hadoop/hdfs/lib/*:/opt/apps/hadoop/share/hadoop/yarn/*:/opt/apps/hadoop/share/hadoop/yarn/lib/*:/opt/apps/hive/lib/*:/opt/apps/spark/lib/*
3.修改spark-env.sh
[plain] view
plain copy
#set Hadoop path
export HDFS_YARN_LOGS_DIR=/data1/hadooplogs
export HADOOP_PREFIX=/opt/apps/hadoop
export HADOOP_HOME=$HADOOP_PREFIX
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_MAPRED_PID_DIR=$HADOOP_HOME/pids
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR=$HDFS_YARN_LOGS_DIR/logs
export HADOOP_PID_DIR=$HADOOP_HOME/pids
export HADOOP_SECURE_DN_PID_DIR=$HADOOP_PID_DIR
export YARN_HOME=$HADOOP_HOME
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_LOG_DIR=$HDFS_YARN_LOGS_DIR/logs
export YARN_PID_DIR=$HADOOP_HOME/pids
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_CONF_DIR:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
export CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
[plain] view
plain copy
### sparkSQL and hive
export HIVE_HOME=/opt/apps/hive
export SPARK_CLASSPATH=$SPARK_HOME/lib:$HIVE_HOME/lib:$HADOOP_CLASSPATH
4.[ERROR] Terminal initialization failed; falling back to unsupported
解决问题java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
需要删除/opt/apps/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar文件,在/etc/profile中配置export HADOOP_USER_CLASSPATH_FIRST=true 并source操作
见:https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark:+Getting+Started
5.使用spark-shell
[plain] view
plain copy
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext.sql("select * from test limit 2 ").collect().foreach(println)
6.使用spark-sql启动local模式
里面可以直接输入HiveQL语句执行
7. 启动standalone集群模式的spark-sql
[plain] view
plain copy
spark-sql --master spark://CNSH001:7077
(该主机地址应该对应spark的ALIVE节点)
8.启动spark on yarn模式的spark-sql
[plain] view
plain copy
spark-sql --master yarn-client
[plain] view
plain copy
或者
[plain] view
plain copy
spark-sql --master yarn-cluster(暂时还不支持)
版本
Hadoop-2.5.0-cdh5.3.2
Hive-0.13.1-cdh5.3.2
Spark-1.5.1
以CNSH001节点为例
spark master在CNSH001上:spark://CNSH001:7077
spark HistoryServer在CNSH001上:CNSH001:8032
spark eventLog在hdfs上:hdfs://testenv/spark/eventLog
分步指南
1. 拷贝$HIVE_HOME/conf/hive-site.xml, hive-log4j.properties 到 $SPARK_HOME/conf/目录
2.修改spark-defaults.conf
[plain] view
plain copy
spark.eventLog.enabled true
spark.eventLog.dir hdfs://testenv/spark/eventLog
spark.eventLog.compress true
spark.yarn.historyServer.address=CNSH001:8032
spark.sql.hive.metastore.version=0.13.1
spark.port.maxRetries=100
spark.sql.hive.metastore.jars=/opt/apps/hadoop/share/hadoop/mapreduce/*:/opt/apps/hadoop/share/hadoop/mapreduce/lib/*:/opt/apps/hadoop/share/hadoop/common/*:/opt/apps/hadoop/share/hadoop/common/lib/*:/opt/apps/hadoop/share/hadoop/hdfs/*
:/opt/apps/hadoop/share/hadoop/hdfs/lib/*:/opt/apps/hadoop/share/hadoop/yarn/*:/opt/apps/hadoop/share/hadoop/yarn/lib/*:/opt/apps/hive/lib/*:/opt/apps/spark/lib/*
spark.driver.extraLibraryPath=/opt/apps/hadoop/share/hadoop/mapreduce/*:/opt/apps/hadoop/share/hadoop/mapreduce/lib/*:/opt/apps/hadoop/share/hadoop/common/*:/opt/apps/hadoop/share/hadoop/common/lib/*:/opt/apps/hadoop/share/hadoop/hdfs/*
:/opt/apps/hadoop/share/hadoop/hdfs/lib/*:/opt/apps/hadoop/share/hadoop/yarn/*:/opt/apps/hadoop/share/hadoop/yarn/lib/*:/opt/apps/hive/lib/*:/opt/apps/spark/lib/*
spark.executor.extraLibraryPath=/opt/apps/hadoop/share/hadoop/mapreduce/*:/opt/apps/hadoop/share/hadoop/mapreduce/lib/*:/opt/apps/hadoop/share/hadoop/common/*:/opt/apps/hadoop/share/hadoop/common/lib/*:/opt/apps/hadoop/share/hadoop/hdfs
/*:/opt/apps/hadoop/share/hadoop/hdfs/lib/*:/opt/apps/hadoop/share/hadoop/yarn/*:/opt/apps/hadoop/share/hadoop/yarn/lib/*:/opt/apps/hive/lib/*:/opt/apps/spark/lib/*
3.修改spark-env.sh
[plain] view
plain copy
#set Hadoop path
export HDFS_YARN_LOGS_DIR=/data1/hadooplogs
export HADOOP_PREFIX=/opt/apps/hadoop
export HADOOP_HOME=$HADOOP_PREFIX
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_MAPRED_PID_DIR=$HADOOP_HOME/pids
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HDFS_CONF_DIR=$HADOOP_HOME/etc/hadoop
export HADOOP_LOG_DIR=$HDFS_YARN_LOGS_DIR/logs
export HADOOP_PID_DIR=$HADOOP_HOME/pids
export HADOOP_SECURE_DN_PID_DIR=$HADOOP_PID_DIR
export YARN_HOME=$HADOOP_HOME
export YARN_CONF_DIR=$HADOOP_HOME/etc/hadoop
export YARN_LOG_DIR=$HDFS_YARN_LOGS_DIR/logs
export YARN_PID_DIR=$HADOOP_HOME/pids
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$HADOOP_CONF_DIR:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*
export CLASSPATH=$HADOOP_CLASSPATH:$CLASSPATH
[plain] view
plain copy
### sparkSQL and hive
export HIVE_HOME=/opt/apps/hive
export SPARK_CLASSPATH=$SPARK_HOME/lib:$HIVE_HOME/lib:$HADOOP_CLASSPATH
4.[ERROR] Terminal initialization failed; falling back to unsupported
解决问题java.lang.IncompatibleClassChangeError: Found class jline.Terminal, but interface was expected
需要删除/opt/apps/hadoop/share/hadoop/yarn/lib/jline-0.9.94.jar文件,在/etc/profile中配置export HADOOP_USER_CLASSPATH_FIRST=true 并source操作
见:https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark:+Getting+Started
5.使用spark-shell
[plain] view
plain copy
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
sqlContext.sql("select * from test limit 2 ").collect().foreach(println)
6.使用spark-sql启动local模式
里面可以直接输入HiveQL语句执行
7. 启动standalone集群模式的spark-sql
[plain] view
plain copy
spark-sql --master spark://CNSH001:7077
(该主机地址应该对应spark的ALIVE节点)
8.启动spark on yarn模式的spark-sql
[plain] view
plain copy
spark-sql --master yarn-client
[plain] view
plain copy
或者
[plain] view
plain copy
spark-sql --master yarn-cluster(暂时还不支持)
相关文章推荐
- SqlServer发送邮件
- mysql使用的语句总结
- pl/sql 过程、函数
- NoSql database——Redis
- MySQL 入门(五)—— 表的操作
- T-Sql访问远程数据库
- 有关SQL*PLUS命令使用大全
- SQL SERVER 2008单表分页
- centos 6.7 搭建redis 主从服务
- mysql xtrabackup在线搭建主从
- Oracle比较时间大小
- mysql存储过程详解
- MySQLlimit分页查询rest接口
- 两台redis + keepalived主备
- mysql中文乱码
- Oracle内处理字符串
- oracle删除锁表进程
- mysqldump备份数据下篇-mysqldump自动备份数据库
- oracle 数据库使用记录,mybatis
- Redis 配置