chukwa安装配置hadoop集群监控

chukwa安装配置 集群监控

tar -zxvf chukwa-0.6.0.tar.gz -C /opt/modules/

设置/etc/profile参数

vi /etc/profile

export CHUKWA_HOME=/opt/modules/chukwa-0.6.0

export CHUKWA_CONF_DIR=$CHUKWA_HOME/etc/chukwa

export PATH=$PATH:$CHUKWA_HOME/bin:$CHUKWA_HOME/sbin

source /etc/profile

测试是否配置成功echo $CHUKWA_HOME

将Chukwa文件复制到Hadoop中

把hadoop配置目录中的log4j.properties和hadoop-metrics2.properties文件改名备份

cd /opt/modules/hadoop-1.2.1/conf/

mv log4j.properties log4j.properties.bak

mv hadoop-metrics2.properties hadoop-metrics2.properties.bak

把chukwa配置目录中的log4j.properties和hadoop-metrics2.properties文件复制到hadoop配置目录中

cp /opt/modules/chukwa-0.6.0/etc/chukwa/hadoop-log4j.properties ./log4j.propertie

cp /opt/modules/chukwa-0.6.0/etc/chukwa/hadoop-metrics2.properties ./

将Chukwa中jar复制到Hadoop中

把chukwa中的chukwa-0.6.0-client.jar和json-simple-1.1.jar两个jar文件复制到hadoop中lib目录下

cp /opt/modules/chukwa-0.6.0/share/chukwa/chukwa-0.6.0-client.jar /opt/modules/hadoop-1.2.1/lib

cp /opt/modules/chukwa-0.6.0/share/chukwa/lib/json-simple-1.1.jar /opt/modules/hadoop-1.2.1/lib

修改$CHUKWA_HOME/libexec/chukwa-config.sh文件

export CHUKWA_HOME=/opt/modules/chukwa-0.6.0

修改$CHUKWA_HOME/etc/chukwa/chukwa-env.sh文件

export JAVA_HOME=/opt/modules/jdk1.7

export HADOOP_CONF_DIR=/opt/modules/hadoop-1.2.1/conf

export HBASE_CONF_DIR=/opt/modules/hbase-0.98.15-hadoop1/conf

修改$CHUKWA_HOME/etc/chukwa/collectors文件

该配置指定哪台机器运行收集器进程，例如修改为http://192.168.192.129:8080，指定hadoop机器运行收集器进程

修改$CHUKWA_HOME/etc/chukwa/initial_adaptors文件 可以使用默认配置（即不需要修改）

为了更好显示测试效果这里添加新建的监控服务，监控/app/chukwa-0.6.0/目录下的testing文件变化情况

add filetailer.FileTailingAdaptor FooData /opt/modules/chukwa-0.6.0/testing 0

建立被监控testing文件

cd /opt/modules/chukwa-0.6.0

touch testing

修改$CHUKWA_HOME/etc/chukwa/chukwa-collector-conf.xml文件

启用chukwaCollector.pipeline参数

<property>

<name>chukwaCollector.pipeline</name>

<value>org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter,org.apache.hadoop.chukwa.datacollection.writer.SeqFileWriter</value>

</property>

指定HDFS的位置为 hdfs://hadoop-master.dragon.org:9000/chukwa/logs

<property>

<name>writer.hdfs.filesystem</name>

<value>hdfs://hadoop-master.dragon.org:9000</value>

<description>HDFS to dump to</description>

</property>

<property>

<name>chukwaCollector.outputDir</name>

<value>/chukwa/logs/</value>

<description>Chukwa data sink directory</description>

</property>

确认默认情况下collector监听8080端口

<property>

<name>chukwaCollector.http.port</name>

<value>8080</value>

<description>The HTTP port number the collector will listen on</description>

</property>

配置Agents文件打开$CHUKWA_HOME/etc/chukwa/agents文件

编辑 hadoop-master.dragon.org

Chukwa部署验证

chukwa-0.6.0/sbin

./start-chukwa.sh

./start-collectors.sh

./start-data-processors.sh
http://192.168.192.129:4080/hicc/
默认用户名和密码是：admin

10 启动hicc

chukwa hicc
http://<server>:4080/hicc
关闭

stop-data-processors.sh

stop-collectors.sh

stop-chukwa.sh

3.2.2 准备日志数据文件和添加数据脚本

cd /opt/modules/chukwa-0.6.0

mkdir testdata

cd testdata

vi weblog

220.181.108.151 [31/Jan/2012:00:02:32] "GET /home.php?mod=space"

208.115.113.82 [31/Jan/2012:00:07:54] "GET /robots.txt"

220.181.94.221 [31/Jan/2012:00:09:24] "GET /home.php?mod=spacecp"

112.97.24.243 [31/Jan/2012:00:14:48] "GET /data/common.css?AZH HTTP/1.1"

112.97.24.243 [31/Jan/2012:00:14:48] "GET /data/auto.css?AZH HTTP/1.1"

112.97.24.243 [31/Jan/2012:00:14:48] "GET /data/display.css?AZH HTTP/1.1"

220.181.108.175 [31/Jan/2012:00:16:54] "GET /home.php"

220.181.94.221 [31/Jan/2012:00:19:15] "GET /?72 HTTP/1.1" 200 13614 "-"

218.5.72.173 [31/Jan/2012:00:21:39] "GET /forum.php?tid=89 HTTP/1.0"

65.52.109.151 [31/Jan/2012:00:24:47] "GET /robots.txt HTTP/1.1"

220.181.94.221 [31/Jan/2012:00:26:12] "GET /?67 HTTP/1.1"

218.205.245.7 [31/Jan/2012:00:27:16] "GET /forum-58-1.html HTTP/1.0"

vi weblogadd.sh

cat /opt/modules/chukwa-0.6.0/testdata/weblog >> /opt/modules/chukwa-0.6.0/chukwa-0.6.0/testing

查看HDFS的文件

启动chukwa的agents和collector，然后运行weblogadd.sh脚本，往weblog文件中添加数据，最后查看HDFS的/chukwa/logs目录下监听生成的数据文件

cd /opt/modules/chukwa-0.6.0/testdata

sudo sh ./weblogadd.sh

hadoop fs -ls /chukwa/logs