hadoop2.4.1 spark1.0.1编译

spark编译用的是2.4.1版本，而hadoop用的则是2.5.0版本，这样在进行rpc通信的时候，会导致失败。于是将$SPARK_HOME下的pom.xml中的protobuf版本改为2.5.0,重新进行编译

重新编译spark

export MAVEN_OPTS="-Xmx2g -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"

mvn -Pyarn -Phadoop-2.4.1 -Dhadoop.version=2.4.1 -DskipTests clean package

hadoop启动会报

WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform.

因为hadoop32位编译所以64位需要重新编译

首先重新编译hadoop2.4.1

系统安装protobuf-2.5.0.tar.bz2

首先安装这个包不然编译会出错
https://code.google.com/p/protobuf/downloads/detail?name=protobuf-2.5.0.tar.bz2&can=2&q=
安装之前还需要安装

sudo apt-get install g++ zlib1g-dev libssl-dev

./configure --prefix=/usr

make

make check

make install

重新编译hadoop

mvn package -Pdist,native -DskipTests -Dtar

编译eclipse hadoop plugin
https://github.com/winghc/hadoop2x-eclipse-plugin
进入src/contrib/eclipse-plugin

ant
jar -Dversion=2.4.1 -Declipse.home=eclipse路径

-Dhadoop.home=hadoop路径

将生成好的jar,复制到${eclipse.home}/plugins

目录下

插件运行错误修改build.xml

ClassNotFoundException:
org.apache.commons.collections.map.UnmodifiableMap

<copy
file="${hadoop.home}/share/hadoop/common/lib/slf4j-api-${slf4j-api.version}.jar"  todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.home}/share/hadoop/common/lib/guava-${guava.version}.jar"  todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.home}/share/hadoop/common/lib/hadoop-auth-2.4.1.jar"  todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.home}/share/hadoop/common/lib/commons-cli-${commons-cli.version}.jar"  todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.home}/share/hadoop/common/lib/netty-${netty.version}.jar"  todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.home}/share/hadoop/common/lib/commons-collections-3.2.1.jar"  todir="${build.dir}/lib" verbose="true"/>

<copy file="${hadoop.home}/share/hadoop/common/hadoop-common-2.4.1.jar"  todir="${build.dir}/lib" verbose="true"/>

lib/commons-configuration-1.6.jar,

lib/commons-collections-3.2.1.jar,

红色部分是修改build.xml的地方，错误就去除，该错需要查看eclipse的日志

ok最后祝大家一切顺利