hadoop 学习笔记002（安装和启动hadoop）

linux上的hadoop操作

ls 简单的列出目录下的文件

ll 列出目录下的详细文件

tar -zxvf hadoop-2.4.1.tar.gz -C /app

1.修改hadoop的配置信息

/etc/hadoop/hadoop-env.sh

显示java安装目录

echo $JAVA_HOME

配置hadoop的java环境变量

# The java implementation to use.
export JAVA_HOME=/usr/local/java/jdk1.7.0_79

保存退出

:wq

core-site.xml配置：

<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://127.0.0.1:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop-2.4.1/tmp</value>
</property>
</configuration>

其他参考：

<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://hadoop:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp</value>
</property>
</configuration>

配置参考：

hdfs://127.0.0.1:9000/

hdfs://weekend-1206-01:9000

hdfs-site.xml配置：

<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.rpc-address</name>
<value>本机ip:8080</value>
</property>
</configuration>

注意：
修改/etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
193.168.0.103   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

(改名)移动命令

mv mapred-site.xml.template mapred-site.xml

mapred-site.xml配置：

<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>

yarn-site.xml配置：

<configuration>
<property>
<name>yarn.resourcemanager.hostname</name>
<value>127.0.0.1</value>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>

配置hadoop环境变量

sudo vi /etc/profile

export HADOOP_HOME=/app/hadoop-2.4<
4000
/span>.1
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$HADOOP_HOME/lib

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

参考例子1 start

export JAVA_HOME=/usr/local/java/jdk1.7.0_79
export HADOOP_HOME=/app/hadoop-2.4.1
export JRE_HOME=$JAVA_HOME/jre
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$HADOOP_HOME/lib
export CLASSPATH=./:$JAVA_HOME/lib:$JAVA_HOME/jre/lib

参考配置 (网上例子) start

PATH=$PATH:$HOME/bin
export HADOOP_HOME=/home/hadoop/hadoop
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_YARN_HOME=$HADOOP_HOME
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HADOOP_HOME/lib

export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_HOME/lib"

export PATH

启动hadoop并格化

hadoop namenode -format

如果出现，表示成功

xxx has been successfully formatted

进入：

cd /tmp/dfs/name/current
cat VERSION

启动dfs[sbin]

start-dfs.sh

====》start ssh

ssh: connect to host slave port 22: Connection timed out的问题

vim /etc/ssh/sshd_config
MaxSessions 1000

传送门：http://blog.csdn.net/u010503822/article/details/78255876

重启SSH服务

/etc/rc.d/init.d/sshd restart
或者
service sshd restart

====》end ssh

[sbin]查看

jps

[sbin]查看

netstat -nltp

处理org.apache.hadoop.security.AccessControlExcepti

./hadoop fs -chmod 777 /

切换用户

su - root

启动hadoop服务

start-all.sh

停止hadoop服务

stop-all.sh

验证是否成功,命令：jps

34203 ResourceManager
34051 SecondaryNameNode
39810 Jps
34292 NodeManager
33885 DataNode
33801 NameNode

访问地址：

http://localhost:50070 （HDFS管理界面） http://localhost:8088 （MR管理界面）