Ubuntu上使用Hadoop 2.x 三 编译2.2.0 64bit版本
2014-04-17 09:08
483 查看
下载源代码：
[plain]
view plaincopyprint?
wget http://mirror.esocc.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
然后解压：
[plain]
view plaincopyprint?
tar zxvf hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src
运行下面的命令开始编译：
[plain]
view plaincopyprint?
~/code/hadoop-2.2.0-src$ mvn package -Pdist,native -DskipTests -Dtar
下载了很多maven的东东后，编译报错：
[plain]
view plaincopyprint?
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] error: cannot access AbstractLifeCycle
[ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
/home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,29] error: cannot access LifeCycle
[ERROR] class file for org.mortbay.component.LifeCycle not found
编辑hadoop-common-project/hadoop-auth/pom.xml文件，添加依赖：
[html]
view plaincopyprint?
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
再次编译，这个错误解决了。注意，2.3.0之后不需要修改pom.xml，这个问题已经解决。
之后遇到了没有安装protocol buffer库的错误，安装一下：
[html]
view plaincopyprint?
[INFO] --- hadoop-maven-plugins:2.2.0:protoc (compile-protoc) @ hadoop-common ---
[WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": error=2, No such file or directory
[html]
view plaincopyprint?
sudo apt-get install libprotobuf-dev
然后再次运行maven命令编译。发现新的错误，原来ubuntu的安装包里面没有带上protoc编译器。算了，下源代码自己编译。
[html]
view plaincopyprint?
hduser@hm1:/usr/src$ sudo wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
...
$ sudo ./configure
$ sudo make
$ sudo make check
$ sudo make install
$ sudo ldconfig
$ protoc --version
[html]
view plaincopyprint?
hduser@hm1:~$ start-dfs.sh
Starting namenodes on [localhost]
localhost: namenode running as process 983. Stop it first.
localhost: datanode running as process 1101. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 1346. Stop it first.
再来编译。等...., 说明hadoop不准备64bit的库浪费了大家多少时间。
还有错：
[html]
view plaincopyprint?
Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_LIBRARIES
安装openssl库
[html]
view plaincopyprint?
sudo apt-get install libssl-dev
再来一次，编译成功了。在目录/home/hduser/code/hadoop-2.2.0-src/hadoop-dist/target 下有文件：
hadoop-2.2.0.tar.gz
解压后，进入目录，然后复制native 目录里的东西到制定位置，覆盖32bit文件
[html]
view plaincopyprint?
sudo cp -r ./hadoop-2.2.0/lib/native/* /usr/local/hadoop/lib/native/
现在回到~目录，运行下面的命令，成功了。
[html]
view plaincopyprint?
hduser@hm1:~$ start-dfs.sh
Starting namenodes on [localhost]
localhost: namenode running as process 983. Stop it first.
localhost: datanode running as process 1101. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 1346. Stop it first.
[html]
view plaincopyprint?
hduser@hm1:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-hm1.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-hm1.out
[html]
view plaincopyprint?
hduser@hm1:~$ jps
32417 Jps
1101 DataNode
1346 SecondaryNameNode
983 NameNode
现在运行例子程序， 注意/etc/hosts中我配置了
[html]
view plaincopyprint?
127.0.0.1 hm1
[html]
view plaincopyprint?
hduser@hm1:~/code/hadoop-2.2.0-src$ cd /usr/local/hadoop/
hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/02/21 18:07:29 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/02/21 18:07:30 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumC\
fountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/02/21 18:07:31 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumC\
ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/02/21 18:07:32 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumC\
ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
需要在yarn-site.xml中添加写配置：
[html]
view plaincopyprint?
<property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:8031</value>
</property>
重新启动虚拟机，再次启动服务，现在连接问题解决了。
[html]
view plaincopyprint?
hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/02/21 18:15:53 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
14/02/21 18:15:54 INFO input.FileInputFormat: Total input paths to process : 2
14/02/21 18:15:54 INFO mapreduce.JobSubmitter: number of splits:2
14/02/21 18:15:54 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.specul\
ative
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/02/21 18:15:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1393006528872_0001
14/02/21 18:15:54 INFO impl.YarnClientImpl: Submitted application application_1393006528872_0001 to ResourceManager at /127.0.0.1:8032
14/02/21 18:15:55 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1393006528872_0001/
14/02/21 18:15:55 INFO mapreduce.Job: Running job: job_1393006528872_0001
如果重来一次的话，顺序应该是先编译64bit版本，然后用这个版本进行配置安装。
[plain]
view plaincopyprint?
wget http://mirror.esocc.com/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0-src.tar.gz
然后解压：
[plain]
view plaincopyprint?
tar zxvf hadoop-2.2.0-src.tar.gz
cd hadoop-2.2.0-src
运行下面的命令开始编译：
[plain]
view plaincopyprint?
~/code/hadoop-2.2.0-src$ mvn package -Pdist,native -DskipTests -Dtar
下载了很多maven的东东后，编译报错：
[plain]
view plaincopyprint?
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[88,11] error: cannot access AbstractLifeCycle
[ERROR] class file for org.mortbay.component.AbstractLifeCycle not found
/home/hduser/code/hadoop-2.2.0-src/hadoop-common-project/hadoop-auth/src/test/java/org/apache/hadoop/security/authentication/client/AuthenticatorTestCase.java:[96,29] error: cannot access LifeCycle
[ERROR] class file for org.mortbay.component.LifeCycle not found
编辑hadoop-common-project/hadoop-auth/pom.xml文件，添加依赖：
[html]
view plaincopyprint?
<dependency>
<groupId>org.mortbay.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>test</scope>
</dependency>
再次编译，这个错误解决了。注意，2.3.0之后不需要修改pom.xml，这个问题已经解决。
之后遇到了没有安装protocol buffer库的错误，安装一下：
[html]
view plaincopyprint?
[INFO] --- hadoop-maven-plugins:2.2.0:protoc (compile-protoc) @ hadoop-common ---
[WARNING] [protoc, --version] failed: java.io.IOException: Cannot run program "protoc": error=2, No such file or directory
[html]
view plaincopyprint?
sudo apt-get install libprotobuf-dev
然后再次运行maven命令编译。发现新的错误，原来ubuntu的安装包里面没有带上protoc编译器。算了，下源代码自己编译。
[html]
view plaincopyprint?
hduser@hm1:/usr/src$ sudo wget https://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz
...
$ sudo ./configure
$ sudo make
$ sudo make check
$ sudo make install
$ sudo ldconfig
$ protoc --version
[html]
view plaincopyprint?
hduser@hm1:~$ start-dfs.sh
Starting namenodes on [localhost]
localhost: namenode running as process 983. Stop it first.
localhost: datanode running as process 1101. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 1346. Stop it first.
再来编译。等...., 说明hadoop不准备64bit的库浪费了大家多少时间。
还有错：
[html]
view plaincopyprint?
Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the
[exec] system variable OPENSSL_ROOT_DIR (missing: OPENSSL_LIBRARIES
安装openssl库
[html]
view plaincopyprint?
sudo apt-get install libssl-dev
再来一次，编译成功了。在目录/home/hduser/code/hadoop-2.2.0-src/hadoop-dist/target 下有文件：
hadoop-2.2.0.tar.gz
解压后，进入目录，然后复制native 目录里的东西到制定位置，覆盖32bit文件
[html]
view plaincopyprint?
sudo cp -r ./hadoop-2.2.0/lib/native/* /usr/local/hadoop/lib/native/
现在回到~目录，运行下面的命令，成功了。
[html]
view plaincopyprint?
hduser@hm1:~$ start-dfs.sh
Starting namenodes on [localhost]
localhost: namenode running as process 983. Stop it first.
localhost: datanode running as process 1101. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: secondarynamenode running as process 1346. Stop it first.
[html]
view plaincopyprint?
hduser@hm1:~$ start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-hm1.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-hm1.out
[html]
view plaincopyprint?
hduser@hm1:~$ jps
32417 Jps
1101 DataNode
1346 SecondaryNameNode
983 NameNode
现在运行例子程序， 注意/etc/hosts中我配置了
[html]
view plaincopyprint?
127.0.0.1 hm1
[html]
view plaincopyprint?
hduser@hm1:~/code/hadoop-2.2.0-src$ cd /usr/local/hadoop/
hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/02/21 18:07:29 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
14/02/21 18:07:30 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is RetryUpToMaximumC\
fountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/02/21 18:07:31 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is RetryUpToMaximumC\
ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
14/02/21 18:07:32 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is RetryUpToMaximumC\
ountWithFixedSleep(maxRetries=10, sleepTime=1 SECONDS)
需要在yarn-site.xml中添加写配置：
[html]
view plaincopyprint?
<property>
<name>yarn.resourcemanager.address</name>
<value>127.0.0.1:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>127.0.0.1:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>127.0.0.1:8031</value>
</property>
重新启动虚拟机，再次启动服务，现在连接问题解决了。
[html]
view plaincopyprint?
hduser@hm1:/usr/local/hadoop$ hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5
Number of Maps = 2
Samples per Map = 5
Wrote input for Map #0
Wrote input for Map #1
Starting Job
14/02/21 18:15:53 INFO client.RMProxy: Connecting to ResourceManager at /127.0.0.1:8032
14/02/21 18:15:54 INFO input.FileInputFormat: Total input paths to process : 2
14/02/21 18:15:54 INFO mapreduce.JobSubmitter: number of splits:2
14/02/21 18:15:54 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks.speculative.execution is deprecated. Instead, use mapreduce.map.speculative
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.reduce.tasks.speculative.execution is deprecated. Instead, use mapreduce.reduce.specul\
ative
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/02/21 18:15:54 INFO Configuration.deprecation: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/02/21 18:15:54 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/02/21 18:15:54 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1393006528872_0001
14/02/21 18:15:54 INFO impl.YarnClientImpl: Submitted application application_1393006528872_0001 to ResourceManager at /127.0.0.1:8032
14/02/21 18:15:55 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1393006528872_0001/
14/02/21 18:15:55 INFO mapreduce.Job: Running job: job_1393006528872_0001
如果重来一次的话，顺序应该是先编译64bit版本，然后用这个版本进行配置安装。
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- Ubuntu上使用Hadoop 2.x 三 编译2.2.0 64bit版本
- Ubuntu上使用Hadoop 2.x 五 编译2.3.0 64bit版本
- 64位Ubuntu1404编译hadoop-2.2.0
- 使用Eclipse编译运行MapReduce程序 Hadoop2.6.0_Ubuntu/CentOS
- 使用Maven将Hadoop2.2.0源码编译成Eclipse项目
- 使用Eclipse编译运行MapReduce程序 Hadoop2.6.0_Ubuntu/CentOS
- 在Ubuntu 64位OS上运行hadoop2.2.0[重新编译hadoop]
- Ubuntu 12.04下编译hadoop 2.2.0
- Ubuntu 10.10 64bit下使用Android NDK r6编译FFmpeg 0.8.1
- Ubuntu 编译安装 hadoop 2.2.0
- hadoop2.2.0源码编译(ubuntu)
- Ubuntu 编译安装Hadoop-2.2.0
- 使用Eclipse编译运行MapReduce程序 Hadoop2.6.0/Ubuntu
- 使用Eclipse编译运行MapReduce程序_Hadoop2.6.0_Ubuntu/CentOS
- 使用Eclipse编译运行MapReduce程序 Hadoop2.6.0_Ubuntu/CentOS
- 使用Maven将Hadoop2.2.0源码编译成Eclipse项目
- 在Ubuntu 64位OS上运行hadoop2.2.0[重新编译hadoop]
- ubuntu 13 编译hadoop 2.2.0源码出现问题及解决方法
- 使用Maven将Hadoop2.2.0源码编译成Eclipse项目
- ubuntu 12.04 64bit上使用32bit编译器编译kernel