Spark_2.0 on hadoop_2.7.2
2016-08-10 10:47
375 查看
参考文档:
http://spark.apache.org/docs/latest/running-on-yarn.html
http://blog.csdn.net/qq_35613461/article/details/52125795
hadoop2.7.x与spark2.0兼容
1、获取spark2.0:
http://spark.apache.org/docs/latest/running-on-yarn.html
一、在安装hadoop2.7.2
参考:http://blog.csdn.net/qq_35613461/article/details/52125795
二、安装Spark2.0
hadoop2.6.x与spark1.6兼容hadoop2.7.x与spark2.0兼容
1、获取spark2.0:
http://spark.apache.org/downloads.html http://d3kbcqa49mib13.cloudfront.net/spark-2.0.0-bin-hadoop2.7.tgz[/code]
2、获取scala:http://downloads.lightbend.com/scala/2.11.8/scala-2.11.8.tgz
3、上传安装文件到服务器,用winscp这个工具非常方便
4、解压scala、spark[hadoop@master ~]$ pwd /home/hadoop [hadoop@master ~]$ tar xvf scala-2.11.8.tgz [hadoop@master ~]$ tar xvzf spark-2.0.0-bin-hadoop2.7.tgz [hadoop@master ~]$ ln -s scala-2.11.8 scala [hadoop@master ~]$ ln -s spark-2.0.0-bin-hadoop2.7 spark [hadoop@master ~]$ ls -l [hadoop@master ~]$ ll total 12 lrwxrwxrwx 1 hadoop hadoop 13 Aug 4 00:41 hadoop2 -> hadoop-2.7.2/ drwxr-xr-x 10 hadoop hadoop 4096 Aug 4 02:28 hadoop-2.7.2 lrwxrwxrwx 1 hadoop hadoop 12 Aug 9 18:13 scala -> scala-2.11.8 drwxrwxr-x 6 hadoop hadoop 4096 Mar 4 07:30 scala-2.11.8 lrwxrwxrwx 1 hadoop hadoop 25 Aug 9 18:20 spark -> spark-2.0.0-bin-hadoop2.7 drwxr-xr-x 13 hadoop hadoop 4096 Aug 9 18:56 spark-2.0.0-bin-hadoop2.7
5、修改spark-en.sh 、slaves、log4j.properties[hadoop@master ~]$ mv spark-env.sh.template spark-env.sh [hadoop@master ~]$ mv slaves.template slaves [hadoop@master ~]$ mv log4j.properties.template log4j.properties[hadoop@master ~]$ vi spark/conf/spark-env.sh ## 添加如下内容: export HADOOP_CONF_DIR=/home/hadoop/hadoop2/etc/hadoop export SCALA_HOME=/home/hadoop/scala export HADOOP_HOME=/home/hadoop/hadoop2[hadoop@master conf]$ more slaves # A Spark Worker will be started on each of the machines listed below. slave1 slave2
6、启动Spark2.0
[hadoop@master ~]$ /home/hadoop/spark/sbin/start-all.sh[hadoop@master ~]$ jps 5334 Master 4657 NameNode 5012 ResourceManager 4818 SecondaryNameNode 6141 Jps [hadoop@master ~]$[hadoop@slave1 ~]$ jps 2868 Jps 2820 Worker 2649 NodeManager 2529 DataNode [hadoop@slave1 ~]$[hadoop@slave2 ~]$ jps 4588 NodeManager 4469 DataNode 4811 Jps 4763 Worker [hadoop@slave2 ~]$
7、监控界面
http://192.168.1 4000
50.30:8080/
8、测试spark[hadoop@master ~]$ /home/hadoop/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue default ../examples/jars/spark-examples_2.11-2.0.0.jar[hadoop@master ~]$ /home/hadoop/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue default /home/hadoop/spark/examples/jars/spark-examples_2.11-2.0.0.jar [hadoop@master ~]$ /home/hadoop/spark/bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn --deploy-mode cluster --driver-memory 1g --executor-memory 1g --executor-cores 1 --queue default /home/hadoop/spark/examples/jars/spark-examples_2.11-2.0.0.jar 16/08/09 20:23:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/08/09 20:23:40 INFO RMProxy: Connecting to ResourceManager at master/192.168.150.30:8032 16/08/09 20:23:40 INFO Client: Requesting a new application from cluster with 2 NodeManagers 16/08/09 20:23:40 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (2000 MB per container) 16/08/09 20:23:40 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead 16/08/09 20:23:40 INFO Client: Setting up container launch context for our AM 16/08/09 20:23:40 INFO Client: Setting up the launch environment for our AM container 16/08/09 20:23:40 INFO Client: Preparing resources for our AM container 16/08/09 20:23:41 WARN Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 16/08/09 20:23:43 INFO Client: Uploading resource file:/tmp/spark-d3674319-c85c-4256-b8ab-f0728fc9611e/__spark_libs__8560551026056083403.zip -> hdfs://master:9000/user/hadoop/.sparkStaging/application_1470793690302_0008/__spark_libs__8560551026056083403.zip 16/08/09 20:23:48 INFO Client: Uploading resource file:/home/hadoop/spark/examples/jars/spark-examples_2.11-2.0.0.jar -> hdfs://master:9000/user/hadoop/.sparkStaging/application_1470793690302_0008/spark-examples_2.11-2.0.0.jar 16/08/09 20:23:48 INFO Client: Uploading resource file:/tmp/spark-d3674319-c85c-4256-b8ab-f0728fc9611e/__spark_conf__5039562237723981219.zip -> hdfs://master:9000/user/hadoop/.sparkStaging/application_1470793690302_0008/__spark_conf__.zip 16/08/09 20:23:48 INFO SecurityManager: Changing view acls to: hadoop 16/08/09 20:23:48 INFO SecurityManager: Changing modify acls to: hadoop 16/08/09 20:23:48 INFO SecurityManager: Changing view acls groups to: 16/08/09 20:23:48 INFO SecurityManager: Changing modify acls groups to: 16/08/09 20:23:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set() 16/08/09 20:23:48 INFO Client: Submitting application application_1470793690302_0008 to ResourceManager 16/08/09 20:23:48 INFO YarnClientImpl: Submitted application application_1470793690302_0008 16/08/09 20:23:49 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:49 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1470799428761 final status: UNDEFINED tracking URL: http://master:8088/proxy/application_1470793690302_0008/ user: hadoop 16/08/09 20:23:50 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:51 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:52 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:53 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:54 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:55 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:56 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:57 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:58 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:23:59 INFO Client: Application report for application_1470793690302_0008 (state: ACCEPTED) 16/08/09 20:24:00 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:00 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.150.32 ApplicationMaster RPC port: 0 queue: default start time: 1470799428761 final status: UNDEFINED tracking URL: http://master:8088/proxy/application_1470793690302_0008/ user: hadoop 16/08/09 20:24:01 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:02 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:03 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:04 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:05 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:06 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:07 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:08 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:09 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:10 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:11 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:12 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:13 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:14 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:15 INFO Client: Application report for application_1470793690302_0008 (state: RUNNING) 16/08/09 20:24:16 INFO Client: Application report for application_1470793690302_0008 (state: FINISHED) 16/08/09 20:24:16 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: 192.168.150.32 ApplicationMaster RPC port: 0 queue: default start time: 1470799428761 final status: SUCCEEDED tracking URL: http://master:8088/proxy/application_1470793690302_0008/ user: hadoop 16/08/09 20:24:16 INFO ShutdownHookManager: Shutdown hook called 16/08/09 20:24:16 INFO ShutdownHookManager: Deleting directory /tmp/spark-d3674319-c85c-4256-b8ab-f0728fc9611e [hadoop@master ~]$
相关文章推荐
- Spark2.0 + Hadoop2.7.2 + Centos7 集群部署<一>
- ubuntu 14.04 下单机安装 hadoop 2.7.2+scala 2.11.8+spark 2.0伪分布式教程
- Linux搭建Hive On Spark环境(spark-1.6.3-without-hive+hadoop2.8.0+hive2.1.1)
- 基于Ubuntu/Debian的hadoop2.7.2+spark1.6实验环境快速部署
- Install Single Node Hadoop(2.7.2) on Mac
- Distributed configure (hadoop 2.7.2 & spark 2.1.0)
- 大数据基础(五)从零开始安装配置Hadoop 2.7.2+Spark 2.0.0到Ubuntu 16.04
- [置顶] CentOS7 基于Hadoop2.7 的Spark2.0集群搭建
- 基于hadoop2.7集群的Spark2.0,Sqoop1.4.6,Mahout0.12.2完全分布式安装
- CentOS7 基于Hadoop2.7 的Spark2.0集群搭建
- spark学习1--centOS7.2下基于hadoop2.7.3的spark2.0集群环境搭建
- Spark2.1.0 + CarbonData1.0.0+hadoop2.7.2集群模式部署及使用入门
- 搭建Hive On Spark 编译Hive源码错误解决方法(spark2.1.0,hadoop2.7.2)
- 从零开始安装配置Hadoop 2.7.2+Spark 2.0.0到Ubuntu 16.04
- Hadoop2.0/YARN深入浅出(Hadoop2.0、Spark、Storm和Tez)
- spark 1.1.0 on hadoop 2.4.1 安装笔记 (resourcemanager ha 支持)
- maven编译spark1.2 on hadoop-2.6.0
- spark2.0+hadoop2.7.1简单部署
- Hadoop2.0/YARN深入浅出(Hadoop2.0、Spark、Storm和Tez)
- spark2.0 on mesos1.0.1