spark学习9之在window下进行源码编译打包

1.配置基本环境：

jdk1.7

scala-2.10.4

maven 3.3.9
具体请看【2】和系列文章

2.编译运行

（1）下载：https://github.com/apache/spark
（2）编译：

D:\1win7\java\spark-1.5.2>set MAVEN_OPTS=-Xmx2g -XX:MaxPermSize=512M -XX:Reserve
dCodeCacheSize=512m

D:\1win7\java\spark-1.5.2>mvn -DskipTests clean package

3.编译成功：

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [04:40 min]
[INFO] Spark Project Launcher ............................. SUCCESS [03:19 min]
[INFO] Spark Project Networking ........................... SUCCESS [ 24.102 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 8.934 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 40.276 s]
[INFO] Spark Project Core ................................. SUCCESS [09:48 min]
[INFO] Spark Project Bagel ................................ SUCCESS [ 29.549 s]
[INFO] Spark Project GraphX ............................... SUCCESS [01:41 min]
[INFO] Spark Project Streaming ............................ SUCCESS [02:51 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [03:24 min]
[INFO] Spark Project SQL .................................. SUCCESS [04:27 min]
[INFO] Spark Project ML Library ........................... SUCCESS [09:14 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 17.949 s]
[INFO] Spark Project Hive ................................. SUCCESS [12:06 min]
[INFO] Spark Project REPL ................................. SUCCESS [01:05 min]
[INFO] Spark Project Assembly ............................. SUCCESS [01:54 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [ 36.062 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [02:01 min]
[INFO] Spark Project External Flume ....................... SUCCESS [ 37.934 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [ 4.781 s]
[INFO] Spark Project External MQTT ........................ SUCCESS [03:22 min]
[INFO] Spark Project External MQTT Assembly ............... SUCCESS [ 26.335 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 50.894 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [02:19 min]
[INFO] Spark Project Examples ............................. SUCCESS [06:28 min]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 8.595 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:13 h
[INFO] Finished at: 2016-05-12T18:31:54+08:00
[INFO] Final Memory: 89M/1375M
[INFO] ------------------------------------------------------------------------

4. 编译记录：

2 overlapping classes:
[WARNING] - org.apache.commons.beanutils.WrapDynaBean
[WARNING] - org.apache.commons.beanutils.Converter
[WARNING] - org.apache.commons.beanutils.converters.IntegerConverter
[WARNING] - org.apache.commons.beanutils.locale.LocaleBeanUtilsBean
[WARNING] - org.apache.commons.beanutils.locale.converters.DecimalLocaleConver
ter
[WARNING] - org.apache.commons.beanutils.locale.converters.DoubleLocaleConvert
er
[WARNING] - org.apache.commons.beanutils.converters.ShortConverter
[WARNING] - org.apache.commons.beanutils.converters.StringArrayConverter
[WARNING] - org.apache.commons.beanutils.locale.LocaleConvertUtilsBean
[WARNING] - org.apache.commons.beanutils.LazyDynaClass
[WARNING] - 72 more...
[WARNING] spark-streaming-twitter_2.10-1.5.2.jar, spark-streaming-kafka_2.10-1.5
.2.jar, spark-streaming-mqtt_2.10-1.5.2.jar, spark-streaming-flume_2.10-1.5.2.ja
r, spark-streaming-zeromq_2.10-1.5.2.jar, unused-1.0.0.jar, spark-streaming-flum
e-sink_2.10-1.5.2.jar define 1 overlapping classes:
[WARNING] - org.apache.spark.unused.UnusedStubClass
[WARNING] hadoop-common-2.2.0-tests.jar, hadoop-mapreduce-client-jobclient-2.2.0
-tests.jar define 4 overlapping classes:
[WARNING] - org.apache.hadoop.util.TestReflectionUtils
[WARNING] - org.apache.hadoop.ipc.TestSocketFactory
[WARNING] - org.apache.hadoop.util.TestRunJar
[WARNING] - org.apache.hadoop.util.TestReflectionUtils$1
[WARNING] cassandra-thrift-1.2.6.jar, cassandra-all-1.2.6.jar define 2 overlappi
ng classes:
[WARNING] - org.apache.cassandra.thrift.ITransportFactory
[WARNING] - org.apache.cassandra.thrift.TFramedTransportFactory
[WARNING] jsp-2.1-6.1.14.jar, jasper-compiler-5.5.23.jar define 143 overlapping
classes:
[WARNING] - org.apache.jasper.compiler.TagLibraryInfoImpl
[WARNING] - org.apache.jasper.xmlparser.SymbolTable
[WARNING] - org.apache.jasper.compiler.Generator$FragmentHelperClass$Fragment
[WARNING] - org.apache.jasper.compiler.Generator$1TagHandlerPoolVisitor
[WARNING] - org.apache.jasper.compiler.Node$AttributeGenerator
[WARNING] - org.apache.jasper.compiler.SmapStratum$LineInfo
[WARNING] - org.apache.jasper.compiler.ScriptingVariabler
[WARNING] - org.apache.jasper.compiler.Node
[WARNING] - org.apache.jasper.compiler.Node$JspAttribute
[WARNING] - org.apache.jasper.compiler.tagplugin.TagPlugin
[WARNING] - 133 more...
[WARNING] javax.servlet-api-3.0.1.jar, javax.servlet-3.1.jar define 32 overlappi
ng classes:
[WARNING] - javax.servlet.Registration
[WARNING] - javax.servlet.DispatcherType
[WARNING] - javax.servlet.ServletRegistration$Dynamic
[WARNING] - javax.servlet.FilterRegistration
[WARNING] - javax.servlet.Registration$Dynamic
[WARNING] - javax.servlet.annotation.WebFilter
[WARNING] - javax.servlet.annotation.ServletSecurity$TransportGuarantee
[WARNING] - javax.servlet.ServletContainerInitializer
[WARNING] - javax.servlet.FilterRegistration$Dynamic
[WARNING] - javax.servlet.descriptor.JspPropertyGroupDescriptor
[WARNING] - 22 more...
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin [INFO]
[INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) @ spark-examp
les_2.10 ---
[INFO] Building jar: D:\1win7\java\spark-1.5.2\examples\target\spark-examples_2.
10-1.5.2-sources.jar
[INFO]
[INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) @ spark-
examples_2.10 ---
[INFO] Building jar: D:\1win7\java\spark-1.5.2\examples\target\spark-examples_2.
10-1.5.2-test-sources.jar
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project External Kafka Assembly 1.5.2
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ spark-streaming-kafk
a-assembly_2.10 ---
[INFO]
[INFO] --- maven-enforcer-plugin:1.4:enforce (enforce-versions) @ spark-streamin
g-kafka-assembly_2.10 ---
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-stre
aming-kafka-assembly_2.10 ---
[INFO] Add Source directory: D:\1win7\java\spark-1.5.2\external\kafka-assembly\s
rc\main\scala
[INFO] Add Test Source directory: D:\1win7\java\spark-1.5.2\external\kafka-assem
bly\src\test\scala
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-streaming
-kafka-assembly_2.10 ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ spark-stre
aming-kafka-assembly_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\1win7\java\spark-1.5.2\external\ka
fka-assembly\src\main\resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-stream
ing-kafka-assembly_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-compiler-plugin:3.3:compile (default-compile) @ spark-streaming
-kafka-assembly_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-antrun-plugin:1.8:run (create-tmp-dir) @ spark-streaming-kafka-
assembly_2.10 ---
[INFO] Executing tasks

main:
[mkdir] Created dir: D:\1win7\java\spark-1.5.2\external\kafka-assembly\targe
t\tmp
[INFO] Executed tasks
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ sp
ark-streaming-kafka-assembly_2.10 ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory D:\1win7\java\spark-1.5.2\external\ka
fka-assembly\src\test\resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ spa
rk-streaming-kafka-assembly_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-compiler-plugin:3.3:testCompile (default-testCompile) @ spark-s
treaming-kafka-assembly_2.10 ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default) @ spark-stream
ing-kafka-assembly_2.10 ---
[INFO]
[INFO] --- maven-surefire-plugin:2.18.1:test (default-test) @ spark-streaming-ka
fka-assembly_2.10 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-streaming-kafka-assemb
ly_2.10 ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.6:test-jar (prepare-test-jar) @ spark-streaming-ka
fka-assembly_2.10 ---
[INFO] Building jar: D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\sp
ark-streaming-kafka-assembly_2.10-1.5.2-tests.jar
[INFO]
[INFO] --- maven-jar-plugin:2.6:jar (default-jar) @ spark-streaming-kafka-assemb
ly_2.10 ---
[INFO] Building jar: D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\sp
ark-streaming-kafka-assembly_2.10-1.5.2.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-s
treaming-kafka-assembly_2.10 ---
[INFO]
[INFO] --- maven-shade-plugin:2.4.1:shade (default) @ spark-streaming-kafka-asse
mbly_2.10 ---
[INFO] Including org.apache.spark:spark-streaming-kafka_2.10:jar:1.5.2 in the sh
aded jar.
[INFO] Including org.apache.kafka:kafka_2.10:jar:0.8.2.1 in the shaded jar.
[INFO] Including com.yammer.metrics:metrics-core:jar:2.2.0 in the shaded jar.
[INFO] Including org.apache.kafka:kafka-clients:jar:0.8.2.1 in the shaded jar.
[INFO] Including com.101tec:zkclient:jar:0.3 in the shaded jar.
[INFO] Including com.thoughtworks.paranamer:paranamer:jar:2.6 in the shaded jar.

[INFO] Including commons-io:commons-io:jar:2.1 in the shaded jar.
[INFO] Including org.apache.avro:avro:jar:1.7.7 in the shaded jar.
[INFO] Including org.apache.commons:commons-compress:jar:1.4.1 in the shaded jar
.
[INFO] Including org.tukaani:xz:jar:1.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-client:jar:2.2.0 in the shaded ja
r.
[INFO] Including com.google.inject:guice:jar:3.0 in the shaded jar.
[INFO] Including javax.inject:javax.inject:jar:1 in the shaded jar.
[INFO] Including aopalliance:aopalliance:jar:1.0 in the shaded jar.
[INFO] Including com.sun.jersey.jersey-test-framework:jersey-test-framework-griz
zly2:jar:1.9 in the shaded jar.
[INFO] Including com.sun.jersey.jersey-test-framework:jersey-test-framework-core
:jar:1.9 in the shaded jar.
[INFO] Including javax.servlet:javax.servlet-api:jar:3.0.1 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-client:jar:1.9 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-grizzly2:jar:1.9 in the shaded jar.
[INFO] Including org.glassfish.grizzly:grizzly-http:jar:2.1.2 in the shaded jar.

[INFO] Including org.glassfish.grizzly:grizzly-framework:jar:2.1.2 in the shaded
jar.
[INFO] Including org.glassfish.gmbal:gmbal-api-only:jar:3.0.0-b023 in the shaded
jar.
[INFO] Including org.glassfish.external:management-api:jar:3.0.0-b012 in the sha
ded jar.
[INFO] Including org.glassfish.grizzly:grizzly-http-server:jar:2.1.2 in the shad
ed jar.
[INFO] Including org.glassfish.grizzly:grizzly-rcm:jar:2.1.2 in the shaded jar.
[INFO] Including org.glassfish.grizzly:grizzly-http-servlet:jar:2.1.2 in the sha
ded jar.
[INFO] Including org.glassfish:javax.servlet:jar:3.1 in the shaded jar.
[INFO] Including com.sun.jersey:jersey-json:jar:1.9 in the shaded jar.
[INFO] Including org.codehaus.jettison:jettison:jar:1.1 in the shaded jar.
[INFO] Including com.sun.xml.bind:jaxb-impl:jar:2.2.3-1 in the shaded jar.
[INFO] Including javax.xml.bind:jaxb-api:jar:2.2.2 in the shaded jar.
[INFO] Including javax.activation:activation:jar:1.1 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 in the shaded jar
.
[INFO] Including org.codehaus.jackson:jackson-xc:jar:1.9.13 in the shaded jar.
[INFO] Including com.sun.jersey.contribs:jersey-guice:jar:1.9 in the shaded jar.

[INFO] Including org.apache.hadoop:hadoop-yarn-api:jar:2.2.0 in the shaded jar.
[INFO] Including org.apache.hadoop:hadoop-yarn-common:jar:2.2.0 in the shaded ja
r.
[INFO] Including org.apache.avro:avro-ipc:jar:1.7.7 in the shaded jar.
[INFO] Including org.codehaus.jackson:jackson-core-asl:jar:1.9.13 in the shaded
jar.
[INFO] Including org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13 in the shade
d jar.
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
[WARNING] hadoop-yarn-common-2.2.0.jar, hadoop-yarn-api-2.2.0.jar define 3 overl
apping classes:
[WARNING] - org.apache.hadoop.yarn.factories.package-info
[WARNING] - org.apache.hadoop.yarn.util.package-info
[WARNING] - org.apache.hadoop.yarn.factory.providers.package-info
[WARNING] unused-1.0.0.jar, spark-streaming-kafka_2.10-1.5.2.jar define 1 overla
pping classes:
[WARNING] - org.apache.spark.unused.UnusedStubClass
[WARNING] javax.servlet-api-3.0.1.jar, javax.servlet-3.1.jar define 74 overlappi
ng classes:
[WARNING] - javax.servlet.ServletRequestWrapper
[WARNING] - javax.servlet.Registration
[WARNING] - javax.servlet.ServletContextAttributeListener
[WARNING] - javax.servlet.http.HttpSessionAttributeListener
[WARNING] - javax.servlet.ServletRegistration$Dynamic
[WARNING] - javax.servlet.http.HttpServlet
[WARNING] - javax.servlet.FilterRegistration
[WARNING] - javax.servlet.http.HttpServletRequest
[WARNING] - javax.servlet.annotation.WebFilter
[WARNING] - javax.servlet.ServletResponse
[WARNING] - 64 more...
[WARNING] maven-shade-plugin has detected that some class files are
[WARNING] present in two or more JARs. When this happens, only one
[WARNING] single version of the class is copied to the uber jar.
[WARNING] Usually this is not harmful and you can skip these warnings,
[WARNING] otherwise try to manually exclude artifacts based on
[WARNING] mvn dependency:tree -Ddetail=true and the above output.
[WARNING] See http://docs.codehaus.org/display/MAVENUSER/Shade+Plugin [INFO] Replacing original artifact with shaded artifact.
[INFO] Replacing D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\spark-
streaming-kafka-assembly_2.10-1.5.2.jar with D:\1win7\java\spark-1.5.2\external\
kafka-assembly\target\spark-streaming-kafka-assembly_2.10-1.5.2-shaded.jar
[INFO] Dependency-reduced POM written at: D:\1win7\java\spark-1.5.2\external\kaf
ka-assembly\dependency-reduced-pom.xml
[INFO] Dependency-reduced POM written at: D:\1win7\java\spark-1.5.2\external\kaf
ka-assembly\dependency-reduced-pom.xml
[INFO]
[INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) @ spark-strea
ming-kafka-assembly_2.10 ---
[INFO] Building jar: D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\sp
ark-streaming-kafka-assembly_2.10-1.5.2-sources.jar
[INFO]
[INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) @ spark-
streaming-kafka-assembly_2.10 ---
[INFO] Building jar: D:\1win7\java\spark-1.5.2\external\kafka-assembly\target\sp
ark-streaming-kafka-assembly_2.10-1.5.2-test-sources.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [04:40 min]
[INFO] Spark Project Launcher ............................. SUCCESS [03:19 min]
[INFO] Spark Project Networking ........................... SUCCESS [ 24.102 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 8.934 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [ 40.276 s]
[INFO] Spark Project Core ................................. SUCCESS [09:48 min]
[INFO] Spark Project Bagel ................................ SUCCESS [ 29.549 s]
[INFO] Spark Project GraphX ............................... SUCCESS [01:41 min]
[INFO] Spark Project Streaming ............................ SUCCESS [02:51 min]
[INFO] Spark Project Catalyst ............................. SUCCESS [03:24 min]
[INFO] Spark Project SQL .................................. SUCCESS [04:27 min]
[INFO] Spark Project ML Library ........................... SUCCESS [09:14 min]
[INFO] Spark Project Tools ................................ SUCCESS [ 17.949 s]
[INFO] Spark Project Hive ................................. SUCCESS [12:06 min]
[INFO] Spark Project REPL ................................. SUCCESS [01:05 min]
[INFO] Spark Project Assembly ............................. SUCCESS [01:54 min]
[INFO] Spark Project External Twitter ..................... SUCCESS [ 36.062 s]
[INFO] Spark Project External Flume Sink .................. SUCCESS [02:01 min]
[INFO] Spark Project External Flume ....................... SUCCESS [ 37.934 s]
[INFO] Spark Project External Flume Assembly .............. SUCCESS [ 4.781 s]
[INFO] Spark Project External MQTT ........................ SUCCESS [03:22 min]
[INFO] Spark Project External MQTT Assembly ............... SUCCESS [ 26.335 s]
[INFO] Spark Project External ZeroMQ ...................... SUCCESS [ 50.894 s]
[INFO] Spark Project External Kafka ....................... SUCCESS [02:19 min]
[INFO] Spark Project Examples ............................. SUCCESS [06:28 min]
[INFO] Spark Project External Kafka Assembly .............. SUCCESS [ 8.595 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:13 h
[INFO] Finished at: 2016-05-12T18:31:54+08:00
[INFO] Final Memory: 89M/1375M
[INFO] ------------------------------------------------------------------------

D:\1win7\java\spark-1.5.2>

参考：

【1】 http://www.jianshu.com/p/6ad9dc90b7a5

【2】http://blog.csdn.net/xubo245/article/details/51385683