CentOS上安装spark standalone mode
2013-09-04 16:36
260 查看
参考:
http://spark.incubator.apache.org/docs/latest/
http://spark.incubator.apache.org/docs/latest/spark-standalone.html
http://www.yanjiuyanjiu.com/blog/20130617/
1.安装JDK
2.安装scala 2.9.3
Spark 0.7.2 依赖 Scala 2.9.3, 我们必须要安装Scala 2.9.3.
下载 scala-2.9.3.tgz 并 保存到home目录(已经在sg206上).
$ tar -zxf scala-2.9.3.tgz
$ sudo mv scala-2.9.3 /usr/lib
$ sudo vim /etc/profile
# add the following lines at the end
export SCALA_HOME=/usr/lib/scala-2.9.3
export PATH=$PATH:$SCALA_HOME/bin
# save and exit vim
#make the bash profile take effect immediately
source /etc/profile
# test
$ scala -version
3.building spark
cd /home
tar -zxf spark-0.7.3-sources.gz
cd spark-0.7.3
sbt/sbt package (需要git环境 yum install git)
4.配置文件
spark-env.sh
############
export SCALA_HOME=/usr/lib/scala-2.9.3
export SPARK_MASTER_IP=172.16.48.202
export SPARK_WORKER_MEMORY=10G
#############
slaves
将从节点IP添加至slaves配置文件
5.启动和停止
bin/start-master.sh - Starts a master instance on the machine the script is executed on.
bin/start-slaves.sh - Starts a slave instance on each machine specified in the conf/slaves file.
bin/start-all.sh - Starts both a master and a number of slaves as described above.
bin/stop-master.sh - Stops the master that was started via the bin/start-master.sh script.
bin/stop-slaves.sh - Stops the slave instances that were started via bin/start-slaves.sh.
bin/stop-all.sh - Stops both the master and the slaves as described above.
6. 浏览master的web UI(默认http://localhost:8080). 这是你应该可以看到所有的word节点,以及他们的CPU个数和内存等信息。
http://spark.incubator.apache.org/docs/latest/
http://spark.incubator.apache.org/docs/latest/spark-standalone.html
http://www.yanjiuyanjiu.com/blog/20130617/
1.安装JDK
2.安装scala 2.9.3
Spark 0.7.2 依赖 Scala 2.9.3, 我们必须要安装Scala 2.9.3.
下载 scala-2.9.3.tgz 并 保存到home目录(已经在sg206上).
$ tar -zxf scala-2.9.3.tgz
$ sudo mv scala-2.9.3 /usr/lib
$ sudo vim /etc/profile
# add the following lines at the end
export SCALA_HOME=/usr/lib/scala-2.9.3
export PATH=$PATH:$SCALA_HOME/bin
# save and exit vim
#make the bash profile take effect immediately
source /etc/profile
# test
$ scala -version
3.building spark
cd /home
tar -zxf spark-0.7.3-sources.gz
cd spark-0.7.3
sbt/sbt package (需要git环境 yum install git)
4.配置文件
spark-env.sh
############
export SCALA_HOME=/usr/lib/scala-2.9.3
export SPARK_MASTER_IP=172.16.48.202
export SPARK_WORKER_MEMORY=10G
#############
slaves
将从节点IP添加至slaves配置文件
5.启动和停止
bin/start-master.sh - Starts a master instance on the machine the script is executed on.
bin/start-slaves.sh - Starts a slave instance on each machine specified in the conf/slaves file.
bin/start-all.sh - Starts both a master and a number of slaves as described above.
bin/stop-master.sh - Stops the master that was started via the bin/start-master.sh script.
bin/stop-slaves.sh - Stops the slave instances that were started via bin/start-slaves.sh.
bin/stop-all.sh - Stops both the master and the slaves as described above.
6. 浏览master的web UI(默认http://localhost:8080). 这是你应该可以看到所有的word节点,以及他们的CPU个数和内存等信息。
相关文章推荐
- CentOS上安装spark standalone mode(转载)
- Spark 1.2 Standalone Mode 单机安装
- CentOS6.5下安装ZooKeeper3.4.8单机模式(Standalone mode)
- Spark Standalone Mode 安装
- Spark Standalone Mode安装配置
- 安装spark standalone mode
- linux环境部署Spark集群(Standalone Mode)
- Spark Standalone Mode
- Spark Standalone Mode
- spark standalone mode
- Spark Standalone Mode 多机启动 -- 分布式计算系统spark学习(二)(更新一键启动slavers)
- Spark Standalone Mode 在 Win10 下搭建开发调试环境
- Spark Standalone Mode 单机启动Spark -- 分布式计算系统spark学习(一)
- Spark Standalone Mode
- Spark官方文档——独立集群模式(Standalone Mode)
- Spark standalone集群安装
- Spark学习(三)---Spark Standalone Mode说明及参数配置详解
- MongoDB安装(Standalone on SUSE、CentOS)
- Spark standalone安装(最小化集群部署)
- Spark Standalone Mode 单机启动Spark -- 分布式计算系统spark学习(一)