Spark 编译
2015-09-13 14:46
190 查看
1.编译环境
CentOS6.6 JDK1.7.0_80 Maven3.2.5
2.下载Spark源代码并解压
3.编译
解压后的源码包的根目录下有个 pom.xml 文件,这个文件就是使用 Maven 编译 Spark 的脚步文件。
OK,现在开始编译:
编译过程中报错:
CentOS6.6 JDK1.7.0_80 Maven3.2.5
2.下载Spark源代码并解压
[yyl@vmnode ~]$ pwd /home/yyl [yyl@vmnode make]$ pwd /home/yyl/make [yyl@vmnode make]$ wget http://mirrors.cnnic.cn/apache/spark/spark-1.5.0/spark-1.5.0.tgz [yyl@vmnode make]$ tar -zxf spark-1.5.0.tgz
3.编译
解压后的源码包的根目录下有个 pom.xml 文件,这个文件就是使用 Maven 编译 Spark 的脚步文件。
OK,现在开始编译:
[yyl@vmnode spark-1.5.0]$ pwd /home/yyl/make/spark-1.5.0 [yyl@vmnode spark-1.5.0]$ export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m" [yyl@vmnode spark-1.5.0]$ mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package
编译过程中报错:
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-enforcer-plugin:1.4:enforce (enforce-versions) on project spark-parent_2.10: Some Enforcer rules have failed. Look above for specific messages explaining why the rule failed. -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[/code]
这个错误有两个解决办法:一是编译时加入 -Denforcer.skip=true 参数;二是修改 pom.xml 文件中 properties 定义的变量的值为实际环境中 maven 、java 的版本[yyl@vmnode spark-1.5.0]$ vim pom.xml <java.version>1.7</java.version> <maven.version>3.2.5</maven.version>
解决上面的错误后重新编译,结果又报错:[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Spark Project Parent POM ........................... SUCCESS [ 4.619 s] [INFO] Spark Project Launcher ............................. SUCCESS [ 11.669 s] [INFO] Spark Project Networking ........................... SUCCESS [ 11.537 s] [INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [ 6.245 s] [INFO] Spark Project Unsafe ............................... SUCCESS [ 17.217 s] [INFO] Spark Project Core ................................. SUCCESS [04:15 min] [INFO] Spark Project Bagel ................................ SUCCESS [ 22.739 s] [INFO] Spark Project GraphX ............................... SUCCESS [01:09 min] [INFO] Spark Project Streaming ............................ SUCCESS [02:04 min] [INFO] Spark Project Catalyst ............................. SUCCESS [02:43 min] [INFO] Spark Project SQL .................................. SKIPPED ...... --------------------------------------------------- java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356) Caused by: scala.reflect.internal.Types$TypeError: bad symbolic reference. A signature in WebUI.class refers to term servlet in value org.jetty which is not available. It may be completely missing from the current classpath, or the version on the classpath might be incompatible with the version used when compiling WebUI.class. at scala.reflect.internal.pickling.UnPickler$Scan.toTypeError(UnPickler.scala:847) at scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.complete(UnPickler.scala:854) at scala.reflect.internal.pickling.UnPickler$Scan$LazyTypeRef.load(UnPickler.scala:863) at scala.reflect.internal.Symbols$Symbol.typeParams(Symbols.scala:1489) ......
这是什么原因呢,查看Spark1.5官方编译文档,有这么一句话:
Building Spark using Maven requires Maven 3.3.3 or newer and Java 7+. The Spark build can supply a suitable Maven binary; see below. 果断升级 maven 到3.3.3,再次编译,OK,编译成功!
如果你想要编译兼容 Scala2.11.x 的 Spark,则使用如下命令编译(默认兼容 Scala2.10.x):[yyl@vmnode spark-1.5.0]$ ./dev/change-scala-version.sh 2.11 [yyl@vmnode spark-1.5.0]$ mvn -Pyarn -Phadoop-2.4 -Dscala-2.11 -DskipTests clean package
编译支持 Hive 和 JDBC 的 Spark[yyl@vmnode spark-1.5.0]$ mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -Phive-thriftserver -DskipTests clean package
4. 生成部署包
源码包的根目录下有个 make-distribution.sh 脚本,这个脚本可以打包Spark的发行包,make-distribution.sh 文件其实就是调用了 Maven 进行编译,可以通过下面的命令运行:[yyl@vmnode spark-1.5.0]$ ./make-distribution.sh --tgz -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -Phive-thriftserver
make-distribution.sh的语法:./make-distribution.sh [--name] [--tgz] [--mvn <mvn-command>] [--with-tachyon] <maven build options>
--tgz :在根目录下生成 spark-$VERSION-bin.tgz ,不加此参数时不生成 tgz 文件,只生成 /dist 目录
--name NAME :和 tgz 结合可以生成 spark-$VERSION-bin-$NAME.tgz 的部署包,不加此参数时 NAME 为 hadoop 的版本号
--with-tachyon :是否支持内存文件系统 Tachyon ,不加此参数时不支持 tachyon
PS:编译时如何指定 Hadoop 版本
例如,Spark要读取的是 Hadoop2.5.2 上的文件,使用 maven 该如何编译呢?答案是:mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.5.2 -Phive -Phive-thriftserver -DskipTests clean package
具体请看官网说明:
另外,如若遇到官方编译文件不兼容的新的 Hadoop 版本,只能是自行修改 pom.xml 文件,添加新版本的支持,例如添加:
<profile>
<id>hadoop-2.7</id>
<properties>
<hadoop.version>2.7.1</hadoop.version>
<jets3t.version>0.9.3</jets3t.version>
<zookeeper.version>3.4.6</zookeeper.version>
<curator.version>2.6.0</curator.version>
</properties>
</profile>
相关文章推荐
- HTTP状态码详解
- hadoop 学习笔记:mapreduce框架详解
- exchange2010安装及配置
- linux虚拟机解决校园网上网方案
- ExtJS4组件_form表单配置-属性-方法详解
- 一个完整顺序表的实现
- Android程序窗体显示:requestWindowFeature()
- django 1.8 官方文档翻译: 8-3 点击劫持保护
- 矩阵链乘法问题描述(Matrix-chain multiplication)
- 滚动视图UIScrollView、UIPageControl
- 因为yii2中jquery位置默认在最下方,可将自定义js位置放在下方
- HDU 5439 Aggregated Counting(找规律+预处理)
- 对HDFS进行操作——笔记
- 数据结构顺序表删除所有特定元素x
- 1041. 考试座位号(15)
- 携不符合安全标准充电宝禁止登机
- 3步带你快速了解特斯拉“巨大充电宝”
- 常见linux命令释义(第七天)——ulimit 与变量内容的删除替代与替换。
- IPTV视频基本概念
- 面向对象(Static关键字)(静态的应用-工具类)(帮助文档的制作javadoc)