spark初体验
2016-03-31 23:31
190 查看
现在这个美好的时代,作为一个技术人,如果连Spark的大名都不知道,那显然是说不过去的,话说大数据的时候,必提Hadoop、Spark。我跟进Spark也好些日子,这次用最新搭建的Hadoop实验集群用最新的Spark来体验了一把酸甜苦辣!
环境信息:
centos 6.7 64位
Hadoop-2.7.2(QJM HA)
Spark-1.6.1
scala 2.10.5
我用的是Standalone模式,hadoop-2.7.2已经安装好,并且支持LZO压缩,Hive-1.2.1运行良好,spark和hadoop都运行在同一个用户和用户组下(hadoop:hadoop),HADOOP_HOME=/home/hadoop/hadoop-2.7.2,spark和Hadoop混合部署,在有hadoop的节点上都部署spark。
直接下载:http://www.apache.org/dyn/closer.lua/spark/spark-1.6.1/spark-1.6.1-bin-without-hadoop.tgz,解压缩后做如下变更即可试运行spark了。
(1)变更spark-env.sh文件内容
(2)变更slave文件内容
(4)追加如下内容到 spark-defaults.conf文件
java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:231)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:139)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:221)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:172)
at org.apache.spark.rdd.RDD
(5)读取hdfs中的lzo文件,并且分片来执行
参考:
http://stackoverflow.com/questions/33174386/accessing-hdfs-ha-from-spark-job-unknownhostexception-error
环境信息:
centos 6.7 64位
Hadoop-2.7.2(QJM HA)
Spark-1.6.1
scala 2.10.5
我用的是Standalone模式,hadoop-2.7.2已经安装好,并且支持LZO压缩,Hive-1.2.1运行良好,spark和hadoop都运行在同一个用户和用户组下(hadoop:hadoop),HADOOP_HOME=/home/hadoop/hadoop-2.7.2,spark和Hadoop混合部署,在有hadoop的节点上都部署spark。
直接下载:http://www.apache.org/dyn/closer.lua/spark/spark-1.6.1/spark-1.6.1-bin-without-hadoop.tgz,解压缩后做如下变更即可试运行spark了。
(1)变更spark-env.sh文件内容
#SYSTEM JAVA_HOME=/usr/local/jdk SCALA_HOME=/usr/local/scala HADOOP_HOME=/home/hadoop/hadoop-2.7.2 HADOOP_CONF_DIR=/home/hadoop/hadoop-2.7.2/etc/hadoop SPARK_DIST_CLASSPATH=$(/home/hadoop/hadoop-2.7.2/bin/hadoop classpath) #SPARK_DIST_CLASSPATH=$(hadoop --config /home/hadoop/hadoop-2.7.2/etc/hadoop classpath) #export SPARK_DIST_CLASSPATH="$SPARK_DIST_CLASSPATH:/home/hadoop/hadoop-2.7.2/share/hadoop/tools/lib/*" #spark SPARK_HOME=/home/hadoop/spark SPARK_MASTER_IP=lrts5 SPARK_WORKER_CORES=4 SPARK_WORKER_INSTANCES=1 SPARK_WORKER_MEMORY=4g SPARK_EXECUTOR_CORES=1 SPARK_EXECUTOR_MEMORY=1g #spark SPARK_WORKER_DIR=/home/hadoop/spark/work SPARK_LOG_DIR=/home/hadoop/spark/logs SPARK_PID_DIR=/home/hadoop/spark/pid #LZO export SPARK_CLASSPATH=/home/hadoop/hadoop-2.7.2/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar export SPARK_CLASSPATH=$SPARK_CLASSPATH:$CLASSPATH export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$HADOOP_HOME/lib/native export HADOOP_CONF_DIR=/home/hadoop/hadoop-2.7.2/etc/hadoop
(2)变更slave文件内容
lrts5 lrts6 lrts7 lrts8(3)将Hadoop的hdfs-site.xml 和core-site.xml文件复制到spark/conf目录下
(4)追加如下内容到 spark-defaults.conf文件
spark.files file:///home/hadoop/spark/conf/hdfs-site.xml,file:///home/hadoop/spark/conf/core-site.xml如果不加这个,会有如下问题发生:
java.lang.IllegalArgumentException: java.net.UnknownHostException: mycluster
at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:418)
at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:231)
at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:139)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:510)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:453)
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:136)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:221)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:172)
at org.apache.spark.rdd.RDD
(5)读取hdfs中的lzo文件,并且分片来执行
import org.apache.hadoop.io._ import com.hadoop.mapreduce._ val data = sc.newAPIHadoopFile[LongWritable, Text, LzoTextInputFormat]("hdfs://mycluster/user/hive/warehouse/logs_app_nginx/logdate=20160322/loghost=70/var.log.nginx.access_20160322.log.70.lzo") data.count()(6)读取hdfs中的通配符表示的目录和子目录下文件,并且分片来执行
import org.apache.hadoop.io._ import com.hadoop.mapreduce._ val dirdata = sc.newAPIHadoopFile[LongWritable, Text, LzoTextInputFormat]("hdfs://mycluster/user/hive/warehouse/logs_app_nginx/logdate=20160322/loghost=*/") dirdata.count()
参考:
http://stackoverflow.com/questions/33174386/accessing-hdfs-ha-from-spark-job-unknownhostexception-error
相关文章推荐
- Socket编程—TcpListener和TcpClient的使用
- Runtime.exec() sucks!!!!
- Python学习笔记day11
- 如何解决maven搭建项目的时候,src/main/java无法建立的问题,提示信息The folder is already a source folder.
- 搞不清FastCgi与PHP-fpm之间是个什么样的关系(转载)
- SPOJ QTREE4 Query on a tree IV(边分治)
- [BZOJ3048][Usaco2013 Jan]Cow Lineup(单调队列)
- python 多继承(新式类) 三
- eclipse与cmake配合使用
- 分布式服务框架DUBBO(一)dubbo 简介
- 跟我一起学kafka(一)
- 旧发票要保留SIRET等信息,或者整个PDF
- Node初学者入门,一本全面的NodeJS教程(转载)
- 一、php环境
- C++作业2
- 第61课:SparkSQl数据加载和保存内幕深度解密实战学习笔记
- baiyun
- 解决SVN客户端修改log message失败的问题
- eclipse 安装svn插件
- Could not resolve archetype org.apache.maven.archetypes:maven-archetype-quickstart