[置顶] STORM入门之（Flume Kafka集成架构）

Flume

Flume作用：收集日志

组件下载地址：http://archive.apache.org/dist/flume/1.6.0/ 

版本：1.6.0以上集成Kafka

Flume配置

a1.sources = r1
a1.sinks = k1
a1.channels = c1

a1.sources.r1.type = syslogudp
a1.sources.r1.port = 514
a1.sources.r1.host = 0.0.0.0

a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.brokerList=10.2.4.12:9092,10.2.4.13:9092,10.2.4.14:9092
a1.sinks.k1.topic=test_rce_yjd
a1.sinks.k1.batchSize=20

a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000

a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

启动命令

/flume/apache-flume-1.6.0-bin/bin/flume-ng agent --conf conf --conf-file /flume/apache-flume-1.6.0-bin/conf/syslog.conf --name a1 -Dflume.root.logger=INFO,console

Kafka

Kafka作用：缓存日志  

监听是否发送至集群中

（1）shell 传送门：http://blog.csdn.net/yl3395017/article/details/73650250

（2）消费 传送门：http://blog.csdn.net/yl3395017/article/details/77547881

Storm

Strom作用：流式处理

（1）消费Kafka集群信息 传送门：http://blog.csdn.net/yl3395017/article/details/77477716

效果