elk 日志处理的一点思路

zjtest7-frontend:/usr/local/logstash-2.3.4/bin# ./logstash  -f ../config/logstash_agent.conf

zjtest7-frontend:/usr/local/logstash-2.3.4/bin# ./logstash  -f ../config/logstash_indexer.conf

你可以在每一个input插件设置一个type，根据type配置不同的filter....这样能节省点资源。logstash起多了，还是不太好的。毕竟每一个都是一个jvm进程....

/*** 写入redis
[elk@zjtest7-frontend config]$ cat logstash_agent.conf
input {
file {
type => "zj_nginx_access"
path => ["/usr/local/nginx/logs/zj_access.log"]
}

file {
type => "wj_nginx_access"
path => ["/usr/local/nginx/logs/wj_access.log"]
}
}

filter {
grok {
match => {
"message" => "%{IPORHOST:clientip} \[%{HTTPDATE:time}\] \"%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}\" %{NUMBER:http_status_code} %{NUMBER:bytes} \"(?

<http_referer>\S+)\" \"(?<http_user_agent>\S+)\" \"(?<http_x_forwarded_for>\S+)\""
}
}
}
output {
redis {
host => "192.168.32.67"
data_type => "list"
key => "logstash:redis"
port=>"6379"
password => "1234567"
}
}

output {
if [type] == "xxx" {
redis {xxxxx}
}
}

根除不同的type 写入到redis

/***从redis读取,发送到elasticsearch

[elk@zjtest7-frontend config]$ cat logstash_indexer.conf
input {
redis {
host => "192.168.32.67"
data_type => "list"
key => "logstash:redis"
type => "redis-input"
password => "1234567"
port =>"6379"
}
}

output {
elasticsearch {
hosts => "192.168.32.80:9200"
index => "logstash-nginx-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}

不同的 redis key 怎么发送到不同的elasticsearch

比如 我同步应用日志到logstash 的  /log/zjzc 下面有nginx-192.168.32.1.log  nginx-192.168.32.2.log  nginx-192.168.32.3.log

那么我    file {
type => "zj_nginx_access"
path => ["/log/zjzc/nginx-*.log"]
}