spark从入门到放弃四: worldcount-scala submit
2018-03-07 23:54
211 查看
文章地址:http://www.haha174.top/article/details/252432
项目源码:https://github.com/haha174/spark.git
提交到spark 集群上面去运行 与本地有点不同
将world-count.txt 上传到hdfs 如果不会可以参考 http://www.haha174.top/article/details/254272
脚本
/spark/spark-2.2.1-bin-hadoop2.7/bin/spark-submit –class spark.core.WorldCountLocal –num-executors 3 –driver-memory 1g –executor-memory 1g –executor-cores 3 –master spark://192.168.1.221:7077 /data/spark-submit/scala/world-count-scala/spark-world-count-scala-submit-1.0-SNAPSHOT-jar-with-dependencies.jar
项目源码:https://github.com/haha174/spark.git
提交到spark 集群上面去运行 与本地有点不同
/** * 如果提交集群只需要修改两个地方 * 第一删除setMaster * 第二 hadf 上面的文件 */ val sparkConf = new SparkConf().setAppName("WorldCount");
/** * * 1 将world-count.txt 上传到hdfs * 2 maven 打包 * 3 编写提交脚本 */
将world-count.txt 上传到hdfs 如果不会可以参考 http://www.haha174.top/article/details/254272
脚本
/spark/spark-2.2.1-bin-hadoop2.7/bin/spark-submit –class spark.core.WorldCountLocal –num-executors 3 –driver-memory 1g –executor-memory 1g –executor-cores 3 –master spark://192.168.1.221:7077 /data/spark-submit/scala/world-count-scala/spark-world-count-scala-submit-1.0-SNAPSHOT-jar-with-dependencies.jar
相关文章推荐
- spark从入门到放弃二: worldcount-java submit
- spark从入门到放弃三: worldcount-scala
- spark从入门到放弃一: worldcount-java
- IDEA搭建spark-scala的sbt编辑环境实现WorldCount练习
- Spark Streaming开发入门——WordCount(Java&Scala)
- spark入门知识讲解和基础数据操作编程(统一用scala编程实例)
- pyspark源码之SparkSubmit学习( SparkSubmit.scala)
- [C#从入门到放弃]程序生命起点--Hello World
- SPARK-Shell 用Scala执行WordCount
- maven构建Scala程序,实现spark的wordcount
- spark快速入门与WordCount程序机制深度解析 spark研习第二季
- Spark Streaming WordCount 入门
- spark入门wordcount详解(JAVA)
- 带你走进Spark编程之Scala入门
- Spark2.x学习笔记:16、Spark Streaming入门实例NetworkWordCount
- SparkStreaming快速入门程序----WordCount
- scala-eclipse 编写spark简单程序 WordCount
- Spark:用Java和Scala实现WordCount
- spark入门wordcount详解(JAVA)
- spark helloworld (wordCount实现并按照词频排序)