flume安装

flume安装
1 .flume的官方网站 http://flume.apache.org/ 下载flume安装包 apache-flume-1.6.0-bin.tar.gz
2. 将下载的flume包，解压到/usr/softinstall/目录下

tar -zxvf apache-flume-1.6.0-bin.tar.gz

3. 在/usr/softinstall/flume/conf目录下 更名
flume-env.sh.template 文件

mv flume-env.sh.template flume-env.sh

4.修改flume-env.sh主要是JAVA_HOME变量设置

# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # If this file is placed at FLUME_CONF_DIR/flume-env.sh, it will be sourced # during Flume startup. # Enviroment variables can be set here. export JAVA_HOME=/usr/softinstall/jdk # Give Flume more memory and pre-allocate, enable remote monitoring via JMX # export JAVA_OPTS="-Xms100m -Xmx2000m -Dcom.sun.management.jmxremote" # Note that the Flume conf directory is always included in the classpath. #FLUME_CLASSPATH=""

6.验证是否安装成功

[root@hadoop0 bin]# /usr/softinstall/flume/bin/flume-ng version Flume 1.6.0 Source code repository: https://git-wip-us.apache.org/repos/asf/flume.git Revision: 2561a23240a71ba20bf288c7c2cda88f443c2080 Compiled by hshreedharan on Mon May 11 11:15:44 PDT 2015 From source with checksum b29e416802ce9ece3269d34233baf43f

实例Spool

Spool监测配置的目录下新增的文件，并将文件中的数据读取出来。需要注意两点：
1) 拷贝到spool目录下的文件不可以再打开编辑。
2) spool目录下不可包含相应的子目录
a)创建agent配置文件

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

[root@hadoop0 bin]# vi /usr/softinstall/flume/conf/spool.conf a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type= spooldir a1.sources.r1.channels = c1 #指定监控的目录地址 a1.sources.r1.spoolDir =/usr/softinstall/flume/logs a1.sources.r1.fileHeader =true # Describe the sink a1.sinks.k1.type= logger # Use a channel which buffers events in memory a1.channels.c1.type= memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1

b)启动flume agent a1

1	[root@hadoop0 conf]# /usr/softinstall/flume/bin/flume-ng agent -c . -f /usr/softinstall/flume/conf/spool.conf -n a1 -Dflume.root.logger=INFO,console

-n 指定agent名称 -c 指定配置文件目录 -f 指定配置文件 -Dflume.root.logger=DEBUG,console 因此完整的启动命令应该这么写 bin/flume-ng agent –n a1 –c conf –f conf/spool.conf –Dflume.root.logger=DEBUG,console

c)追加文件到/home/hadoop/flume-1.5.0-bin/logs目录

1	[root@hadoop0 bin]# echo “spool test1″ > /usr/softinstall/flume/logs/spool_text.log

d)在hadoop0的控制台，可以看到以下相关信息：

2015-12-16 01:19:06,890 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:94)] Event: { headers:{file=/usr/softinstall/flume/logs/spool_text.log} body: E2 80 9C 73 70 6F 6F 6C 20 68 65 6C 6C 6F 20 77 ...spool hello w }

更多实例参考http://www.niubua.com/?p=724