flume采集数据导入elasticsearch 配置
2016-02-25 11:37
627 查看
Flume启动一般会报两种错,一种是log4j没有配置,另外一种就是缺少各种jar包。SO:
[root@laiym ~]# cp /usr/local/elasticsearch/lib/*/usr/local/flume/lib/
如果有相同的jar包不用覆盖
下述为flume到elasticsearch的一个配置文件,字段用法详情大家看官方给出的定义。
#文件名称为flume-es.conf
#定义sources,channel和sinks的名称
agent.sources = tail
agent.sinks = elasticsearch
agent.channels = memoryChannel
#配置source的详情
agent.sources.tail.type = exec
agent.sources.tail.command = tail -F /var/log/secure
agent.sources.tail.interceptors=i1 i2 i3
agent.sources.tail.interceptors.i1.type=regex_extractor
agent.sources.tail.interceptors.i1.regex =(\\w+\\s+\\w+\\s+\\d+\\\:\\d+\\\:\\d+)\\s+(\\w+)\\s+(\\w+)
agent.sources.tail.interceptors.i1.serializers = s1 s2s3
agent.sources.tail.interceptors.i1.serializers.s1.name= time
agent.sources.tail.interceptors.i1.serializers.s2.name= hostname
agent.sources.tail.interceptors.i1.serializers.s3.name= service
agent.sources.tail.interceptors.i2.type=org.apache.flume.interceptor.TimestampInterceptor$Builder
agent.sources.tail.interceptors.i3.type=org.apache.flume.interceptor.HostInterceptor$Builder
agent.sources.tail.interceptors.i3.hostHeader = host
#配置channel的详情
agent.channels.memoryChannel.type = memory
agentes.channels.channel1.capacity = 1000000
agentes.channels.channel1.transactionCapacity = 5000
#agentes.channels.channel1.keep-alive = 10
#配置sink的详情
agent.sinks.elasticsearch.type=org.apache.flume.sink.elasticsearch.ElasticSearchSink
agent.sinks.elasticsearch.batchSize=100
agent.sinks.elasticsearch.hostNames=127.0.0.1:9300
agent.sinks.elasticsearch.indexName=linux_secure
agent.sinks.elasticsearch.indexType=message
agent.sinks.elasticsearch.clusterName=elasticsearch
agent.sinks.elasticsearch.serializer=org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer
#配置source、sink和channel的详情
agent.sources.tail.channels = memoryChannel
agent.sinks.elasticsearch.channel = memoryChannel
样本日志为linux的secure日志。
Feb 23 17:38:20 laiym sshd[1591]:pam_unix(sshd:session): session closed for user root
Feb 23 17:38:20 laiym sshd[1616]:pam_unix(sshd:session): session closed for user root
Feb 23 17:38:38 laiym sshd[1954]: reverse mappingchecking getaddrinfo for bogon [192.168.141.1] failed - POSSIBLE BREAK-INATTEMPT!
Feb 23 17:38:38 laiym sshd[1954]: Accepted passwordfor root from 192.168.141.1 port 61857 ssh2
Feb 23 17:38:38 laiym sshd[1954]:pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 23 17:50:19 laiym sshd[2019]: reverse mappingchecking getaddrinfo for bogon [192.168.141.1] failed - POSSIBLE BREAK-INATTEMPT!
Feb 23 17:50:19 laiym sshd[2019]: Accepted passwordfor root from 192.168.141.1 port 50289 ssh2
Feb 23 17:50:20 laiym sshd[2019]:pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 24 09:40:51 laiym sshd[1585]:pam_unix(sshd:session): session closed for user root
启动时打开INFO日志和console日志,查看启动状态。
[root@laiym ~]# cd /usr/local/flume/
[root@laiym flume]# ./bin/flume-ng agent -c ./conf/ -f./conf/flume-es.conf -n agent -Dflume.root.logger=INFO,console
在ES中的数据截图:
在kibana中的数据截图:
ok,完美!!!
[root@laiym ~]# cp /usr/local/elasticsearch/lib/*/usr/local/flume/lib/
如果有相同的jar包不用覆盖
下述为flume到elasticsearch的一个配置文件,字段用法详情大家看官方给出的定义。
#文件名称为flume-es.conf
#定义sources,channel和sinks的名称
agent.sources = tail
agent.sinks = elasticsearch
agent.channels = memoryChannel
#配置source的详情
agent.sources.tail.type = exec
agent.sources.tail.command = tail -F /var/log/secure
agent.sources.tail.interceptors=i1 i2 i3
agent.sources.tail.interceptors.i1.type=regex_extractor
agent.sources.tail.interceptors.i1.regex =(\\w+\\s+\\w+\\s+\\d+\\\:\\d+\\\:\\d+)\\s+(\\w+)\\s+(\\w+)
agent.sources.tail.interceptors.i1.serializers = s1 s2s3
agent.sources.tail.interceptors.i1.serializers.s1.name= time
agent.sources.tail.interceptors.i1.serializers.s2.name= hostname
agent.sources.tail.interceptors.i1.serializers.s3.name= service
agent.sources.tail.interceptors.i2.type=org.apache.flume.interceptor.TimestampInterceptor$Builder
agent.sources.tail.interceptors.i3.type=org.apache.flume.interceptor.HostInterceptor$Builder
agent.sources.tail.interceptors.i3.hostHeader = host
#配置channel的详情
agent.channels.memoryChannel.type = memory
agentes.channels.channel1.capacity = 1000000
agentes.channels.channel1.transactionCapacity = 5000
#agentes.channels.channel1.keep-alive = 10
#配置sink的详情
agent.sinks.elasticsearch.type=org.apache.flume.sink.elasticsearch.ElasticSearchSink
agent.sinks.elasticsearch.batchSize=100
agent.sinks.elasticsearch.hostNames=127.0.0.1:9300
agent.sinks.elasticsearch.indexName=linux_secure
agent.sinks.elasticsearch.indexType=message
agent.sinks.elasticsearch.clusterName=elasticsearch
agent.sinks.elasticsearch.serializer=org.apache.flume.sink.elasticsearch.ElasticSearchLogStashEventSerializer
#配置source、sink和channel的详情
agent.sources.tail.channels = memoryChannel
agent.sinks.elasticsearch.channel = memoryChannel
样本日志为linux的secure日志。
Feb 23 17:38:20 laiym sshd[1591]:pam_unix(sshd:session): session closed for user root
Feb 23 17:38:20 laiym sshd[1616]:pam_unix(sshd:session): session closed for user root
Feb 23 17:38:38 laiym sshd[1954]: reverse mappingchecking getaddrinfo for bogon [192.168.141.1] failed - POSSIBLE BREAK-INATTEMPT!
Feb 23 17:38:38 laiym sshd[1954]: Accepted passwordfor root from 192.168.141.1 port 61857 ssh2
Feb 23 17:38:38 laiym sshd[1954]:pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 23 17:50:19 laiym sshd[2019]: reverse mappingchecking getaddrinfo for bogon [192.168.141.1] failed - POSSIBLE BREAK-INATTEMPT!
Feb 23 17:50:19 laiym sshd[2019]: Accepted passwordfor root from 192.168.141.1 port 50289 ssh2
Feb 23 17:50:20 laiym sshd[2019]:pam_unix(sshd:session): session opened for user root by (uid=0)
Feb 24 09:40:51 laiym sshd[1585]:pam_unix(sshd:session): session closed for user root
启动时打开INFO日志和console日志,查看启动状态。
[root@laiym ~]# cd /usr/local/flume/
[root@laiym flume]# ./bin/flume-ng agent -c ./conf/ -f./conf/flume-es.conf -n agent -Dflume.root.logger=INFO,console
在ES中的数据截图:
在kibana中的数据截图:
ok,完美!!!
相关文章推荐
- React插件-双向绑定辅助工具
- Benchmark库的建立03:空指针异常Bug
- C#中正则表达式在replace中的应用!
- 解决mac os x下 tomcat启动报 java.net.BindException: Permission denied <null>:80 错误
- matlab plot一点小细节
- ansible 系列教程
- Java基础:Day10笔记内容 (多态概述、抽象类)
- Android Launcher 解决BubbleTextView 点击事件只在ICON上面触发,防止误触
- 对 oc 学习的 阶段反思
- I/O复用——poll系统调用
- sql server查询技巧
- 怎样推断手机用户是移动,电信,联通?
- JavaScript中call,apply,bind方法的总结
- PHP字符串函数(4)
- 用户认证
- STM32 USB学习笔记4
- android 自定义选项卡tabhoust
- MongoDB数据自动同步到ElasticSearch(实现中文全文检索)
- 微商逆袭?指数开始暴涨
- iOS9适配注意点(转)