maven编译spark1.2 on hadoop-2.6.0
2015-03-09 08:58
736 查看
1、安装maven
(1)设置MAVEN_HOME
(2)将$MAVEN_HOME/bin参加PATH变量。
(3)设置maven_opts内存参数
若不运行,编译时必定出现如下错误,因为spark编译需要很大的内存
2、编译spark
(1)下载spark
http://spark.apache.org/downloads.html
(2)解压下载的文件
(3)进入根目录
修改源码:mllib\src\main\scala\org\apache\spark\mllib\optimization\Gradient.scala
[ERROR] Failed to execute goal org.scalastyle:scalastyle-maven-plugin:0.4.0:check (default) on project spark-mllib_2.10: Failed during scalastyle execution : You have 1 Scalastyle violation(s). -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-mllib_2.10
将带Our loss function的两行删除掉,否则在编译的时候报错
(4)在根目录下执行如下命令编译
[/code]
(1)设置MAVEN_HOME
(2)将$MAVEN_HOME/bin参加PATH变量。
(3)设置maven_opts内存参数
export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
若不运行,编译时必定出现如下错误,因为spark编译需要很大的内存
[INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.10/classes... [ERROR] PermGen space -> [Help 1] [INFO] Compiling 203 Scala sources and 9 Java sources to /Users/me/Development/spark/core/target/scala-2.10/classes... [ERROR] Java heap space -> [Help 1]
2、编译spark
(1)下载spark
http://spark.apache.org/downloads.html
(2)解压下载的文件
(3)进入根目录
修改源码:mllib\src\main\scala\org\apache\spark\mllib\optimization\Gradient.scala
[ERROR] Failed to execute goal org.scalastyle:scalastyle-maven-plugin:0.4.0:check (default) on project spark-mllib_2.10: Failed during scalastyle execution : You have 1 Scalastyle violation(s). -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :spark-mllib_2.10
将带Our loss function的两行删除掉,否则在编译的时候报错
(4)在根目录下执行如下命令编译
mvn -Pyarn -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests clean package 当yarn与hadoop版本不一致时分别指定版本号 [code]mvn -Pyarn-alpha -Phadoop-2.6 -Dhadoop.version=2.6.0 -Dyarn.version=2.6.0 -DskipTests clean package 编译时间较长要耐心等待 (5) 可以跳过(4)使用./make-distribution.sh --name hadoop2.6 --tgz -Pyarn -Phive -Phive-thriftserver -Phadoop-2.6 -Dhadoop.version=2.6.0 -DskipTests 编译加打包
[/code]
相关文章推荐
- hadoop2.2.0安装spark1.1.0(maven编译)
- Spark 2.2源码编译 & 支持hadoop-2.6.0-cdh5.7.0
- 使用maven在32位centos6.7编译hadoop2.6.0
- 大数据学习20:Maven 理解 和 spark、hadoop、hive编译
- Maven安装编译Spark,搭建Spark独立集群模式(Hadoop架构之上)
- 源码编译Spark,Hadoop 2.6.0-cdh5.7.0 版本
- 使用Maven进行Hadoop源代码编译
- spark 1.1.0 on hadoop 2.4.1 安装笔记 (resourcemanager ha 支持)
- IntellJ iDEA中使用Maven 编译直接本地跑spark ,如何给算法传参数(仅作备忘记录)
- 使用Maven将Hadoop2.2.0源码编译成Eclipse项目
- hadoop2.6.0汇总:新增功能最新编译 32位、64位安装、源码包、API下载及部署文档
- hadoop2.4.1 spark1.0.1编译
- spark cdh5编译安装[spark-1.0.2 hadoop2.3.0 cdh5.1.0]
- SVN checkout hadoop源码及Maven进行Hadoop源代码编译步骤
- Spark 学习(三) maven 编译spark 源码
- error fix 4 spark application run on windows without hadoop in local model
- 使用maven在linux上对hadoop 2.2进行编译
- cloudera hadoop maven编译安装步骤详细教程
- maven 编译hadoop错误解决
- Spark 1.0.0 横空出世 Spark on Yarn 部署(Hadoop 2.4)