flume-ng 日志收集系统单机测试demo

a.单机测试

flume-ng启动agent的命令为：

$ bin/flume-ng agent -n $agent_name -c conf -f conf/flume-conf.properties.template

添加如下配置文件

example.conf

# example.conf: A single-node Flume configuration

# Name the components on this agent

agent1.sources = source1

agent1.sinks = sink1

agent1.channels = channel1

# Describe/configure source1

agent1.sources.source1.type = netcat

agent1.sources.source1.bind = localhost

agent1.sources.source1.port = 44444

# Describe sink1

agent1.sinks.sink1.type = logger

# Use a channel which buffers events in memory

agent1.channels.channel1.type = memory

agent1.channels.channel1.capacity = 1000

agent1.channels.channel1.transactionCapactiy = 100

# Bind the source and sink to the channel

agent1.sources.source1.channels = channel1

agent1.sinks.sink1.channel = channel1

以上配置信息描述的是：

该配置文件中 配置了一个代理agent1

在代理agent中配置了一个source（源）一个sink（接收器）和一个channel（通道），分别为：source1，sink1，channel1

source1的类型定义为netcat，对应该类型参数为bind和port 分别为localhost和44444

sink1的类型定义为logger，直接输出到日志文件中

channel的类型定义为内存方式，设置其参数capactiy和transactionCapacity分别为1000和100

指定source1和sink1的channel为channel1

现在我们可以启动看看我们的flume-ng是否成功。

在flume-ng的home目录下

$ bin/flume-ng agent --conf-file example.conf --name agent1 -Dflume.root.logger=INFO,console --conf = conf

启动之后日志信息如下：

2012-07-30 15:42:30,708 (main) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 1

2012-07-30 15:42:30,713 (main) [INFO - org.apache.flume.node.FlumeNode.start(FlumeNode.java:54)] Flume node starting - agent1

2012-07-30 15:42:30,715 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.start(DefaultLogicalNodeManager.java:187)] Node manager starting

2012-07-30 15:42:30,716 (lifecycleSupervisor-1-2) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider.start(AbstractFileConfigurationProvider.java:67)] Configuration provider starting

2012-07-30 15:42:30,718 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.start(LifecycleSupervisor.java:67)] Starting lifecycle supervisor 11

2012-07-30 15:42:30,720 (conf-file-poller-0) [INFO - org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:195)] Reloading configuration file:example.conf

2012-07-30 15:42:30,726 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:sink1

2012-07-30 15:42:30,726 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:988)] Processing:sink1

2012-07-30 15:42:30,726 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty(FlumeConfiguration.java:902)] Added sinks: sink1 Agent: agent1

2012-07-30 15:42:30,747 (conf-file-poller-0) [INFO - org.apache.flume.conf.FlumeConfiguration.validateConfiguration(FlumeConfiguration.java:122)] Post-validation flume configuration contains configuration for agents: [agent1]

2012-07-30 15:42:30,747 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:249)] Creating channels

2012-07-30 15:42:30,808 (conf-file-poller-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.<init>(MonitoredCounterGroup.java:68)] Monitoried counter group for type: CHANNEL, name: channel1, registered successfully.

2012-07-30 15:42:30,808 (conf-file-poller-0) [INFO - org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadChannels(PropertiesFileConfigurationProvider.java:273)] created channel channel1

2012-07-30 15:42:30,828 (conf-file-poller-0) [INFO - org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:70)] Creating instance of sink: sink1, type: logger

2012-07-30 15:42:30,832 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:92)] Starting new configuration:{ sourceRunners:{source1=EventDrivenSourceRunner: { source

rg.apache.flume.source.NetcatSource@18f1d7e
}} sinkRunners:{sink1=SinkRunner: { policy

rg.apache.flume.sink.DefaultSinkProcessor@d9660d counterGroup:{ name:null counters:{} } }} channels:{channel1=org.apache.flume.channel.MemoryChannel@bb0d0d}
}

2012-07-30 15:42:30,832 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:99)] Starting Channel channel1

2012-07-30 15:42:30,835 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:82)] Component type: CHANNEL, name: channel1 started

2012-07-30 15:42:30,835 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:114)] Waiting for channel: channel1 to start. Sleeping for 500 ms

2012-07-30 15:42:31,338 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:127)] Starting Sink sink1

2012-07-30 15:42:31,338 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.startAllComponents(DefaultLogicalNodeManager.java:138)] Starting Source source1

2012-07-30 15:42:31,339 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:147)] Source starting

2012-07-30 15:42:31,358 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:161)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]

我们打开另外一个console：执行 telnet localhost 44444

Trying 127.0.0.1...

Connected to localhost.localdomain (127.0.0.1).

Escape character is '^]'.

haha

OK

在日志控制台，我们看到如下信息

2012-07-30 15:42:40,780 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:70)] Event: { headers:{} body: 68 61 68 61 0D haha. }

说明我们的flume-ng运行良好。可以继续下一步操作…………