Spark1.6.0 on Hadoop-2.6.3 安装配置

安装配置Scala

移动scala到指定目录

mkdir /usr/local/scala

mv scala-2.10.4 /usr/local/scala

export SCALA_HOME=/usr/local/scala/scala-2.10.4

export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${SCALA_HOME}/bin:$PATH

输入scala进入scala环境：

测试：12*12 回车

安装配置Spark1.6.0

附：下载链接
 http://spark.apache.org/downloads.html[/code] 
将下载解压后的spark移动到指定目录(/usr/local/spark)

mkdir /usr/local/spark

mv spark-1.6.0-bin-hadoop2.6 /usr/local/spark




2 配置Spark环境变量

export SPARK_HOME=/usr/local/spark/spark-1.6.0-bin-hadoop2.6

export PATH=${JAVA_HOME}/bin:${HADOOP_HOME}/bin:${SCALA_HOME}/bin:${SPARK_HOME}/bin:$PATH




注：source .bashrc 使配置生效

3 配置Spark

cp spark-env.sh.template spark-env.sh

vim spark-env.sh






添加SPARK配置信息

export JAVA_HOME=/usr/lib/java8/jdk1.8.0_65
export SCALA_HOME=/usr/local/scala/scala-2.10.4
export SPARK_MASTER_IP=hjr
export SPARK_WORKER_CORES=2
export SPARK_WORKER_MEMORY=1g
export HADOOP_CONF_DIR=/usr/local/hadoop/hadoop-2.6.3/etc/hadoop




cp slaves.template slaves

vim slaves


注意 IP 地址：



添加节点：
hjr 或者 127.0.1.1




4 启动Spark，查看集群状况

cd /usr/local/spark/spark-1.6.0-bin-hadoop2.6

启动：
./start-all.sh


jps查看进程：多了一个Master和Worker进程




启动：spark-shell






测试运行：

val file=sc.textFile("hdfs:/hjr/hdfs-site.xml")




val count=file.flatMap(line=>line.split(" ")).map(word=>(word,1)).reduceByKey(_+_)




count.collect






Spark UI：