Maven spark project on Linux

Linux 下Maven交互式初始化项目

1）mkdir myspark; cd myspark

2）mvn命令初始化，创建相应目录与文件

mvn archetype:generate //创建，下载jars

默认类型: org.apache.maven.archetypes:maven-archetype-quickstart

手动输入 groupId:com.eric; artifactId:myspark; package:com.xxx

生成java项目,包含main/test 及相关文件

hadoop@slave1:~/myspk/myspark/myspark$ find .

...

./src/main/java/com/eric/App.java

...

./src/test/java/com/eric/AppTest.java

./pom.xml

或者指令提供所有信息:

mvn archetype:generate -DgroupId=com.eric -DartifactId=myspark -DarchetypeArtifactId=maven-archetype-quickstart //-DinteractiveMode=false

3) java源码 /myspark/src/main/java/com/eric

4）根据依赖关系，pom中指定依赖

...

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_2.10</artifactId>

<version>1.2.0</version>

...

5) mvn package 完成编译

6） 运行：如果报链接master错误，则需要设置sparkconf.setMaster("local");

java -cp myspark-1.0-SNAPSHOT.jar:/home/hadoop/work/spark-1.5.2/assembly/target/scala-2.10/spark-assembly-1.5.2-hadoop2.6.0.jar com.eric.JavaWordCount ~/test/exp.txt

7）插件运行,需要先edit pom.xml

<plugin>

<groupId>org.codehaus.mojo</groupId>

<artifactId>exec-maven-plugin</artifactId>

<version>1.2.1</version>

</plugin>

执行命令：

mvn exec:java -Dexec.mainClass="com.eric.JavaWordCount" -Dexec.args="/home/hadoop/test/exp.txt"

8) scalac 命令行编译运行

在maven生成的项目下添加.scala 文件并编译

scalac -d target/classes -classpath target/classes:/home/hadoop/work/spark-1.5.2/assembly/target/scala-2.10/spark-assembly-1.5.2-hadoop2.6.0.jar src/main/java/com/eric/myscala.scala

运行： -cp //classpath

java -cp target/classes:/home/hadoop/work/spark-1.5.2/assembly/target/scala-2.10/spark-assembly-1.5.2-hadoop2.6.0.jar myscala ~/test/exp.txt

9) maven 插件编译执行scala

edit pom.xml

<dependencies>

<dependency>

<groupId>junit</groupId>

<artifactId>junit</artifactId>

<version>3.8.1</version>

<scope>test</scope>

</dependency>

<dependency>

<groupId>org.scala-lang</groupId>

<artifactId>scala-library</artifactId>

<version>2.10.4</version>

</dependency>

<dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-core_2.10</artifactId>

<version>1.5.2</version>

</dependency>

</dependencies>

<build>

<plugins>

<plugin>

<groupId>net.alchim31.maven</groupId>

<artifactId>scala-maven-plugin</artifactId>

<version>3.2.2</version>

</plugin>

<plugin>

<groupId>org.codehaus.mojo</groupId>

<artifactId>exec-maven-plugin</artifactId>

<version>1.2.1</version>

</plugin>

</plugins>

</build>

编译

mvn scala:compile

运行：

mvn exec:java -Dexec.mainClass="myscala" -Dexec.args="/home/hadoop/test/log.txt"

Windows project之后用IDEA试试.