用spark自带的示例SparkPi测试scala和spark集群

在按照王家林的文档安装完scala，spark集群和idea-IC开发工具后，用spark自带的示例SparkPi测试scala和spark集群

1、按照王家林文档中的方法把spark自带的SparkPi示例的代码复制到idea-IC开发工具中创建的项目中后，把SparkPi代码的修改为：

/*

* Licensed to the Apache Software Foundation (ASF) under one or more

* contributor license agreements. See the NOTICE file distributed with

* this work for additional information regarding copyright ownership.

* The ASF licenses this file to You under the Apache License, Version 2.0

* (the "License"); you may not use this file except in compliance with

* the License. You may obtain a copy of the License at

*

* http://www.apache.org/licenses/LICENSE-2.0
*

* Unless required by applicable law or agreed to in writing, software

* distributed under the License is distributed on an "AS IS" BASIS,

* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

* See the License for the specific language governing permissions and

* limitations under the License.

*/

import scala.math.random

import org.apache.spark._

/** Computes an approximation to pi */

object SparkPi {

def main(args: Array[String]) {

val conf = new SparkConf().setAppName("Spark Pi").setMaster("spark://192.168.91.128:7077")//改成自己master主机的ip地址

val spark = new SparkContext(conf)

spark.addJar("/home/cy/workspace/untitled/out/artifacts/sparkpi_jar/untitled.jar")//把写的代码打包成jar包，存放打包后jar包的路径

val slices = if (args.length > 0) args(0).toInt else 2

val n = math.min(100000L * slices, Int.MaxValue).toInt // avoid overflow

val count = spark.parallelize(1 until n, slices).map { i =>

val x = random * 2 - 1

val y = random * 2 - 1

if (x*x + y*y < 1) 1 else 0

}.reduce(_ + _)

println("Pi is roughly " + 4.0 * count / n)

spark.stop()

}

}

2、点击idea-IC开发工具左上角的File->Project Structure->Artifcats

3、添加jar包：



4、选择Main Class，点击OK：



5、修改jar包的名字、保存的路径和类型，注意上述代码中jar包的路径要与这边jar包的路径一致：



6、完成上述的操作后，先通过start-all.sh启动hadoop集群，再进入spark安装目录的sbin目录下，通过./start-all.sh命令启动spark集群

7、运行SparkPi代码，出现以下结果就说明成功了：



8、要关掉集群的时候，先进入spark安装目录下的sbin目录，通过./stop-all.sh命令先关掉spark集群，再通过stop-all.sh关掉hadoop集群