Kibana+Logstash+Elasticsearch+Redis安装部署

最近做日志分析，发现logstash较符合自己的需求，
Logstash：做系统log收集，转载的工具。同时集成各类日志插件,对日志查询和分析的效率有很大的帮助.一般使用shipper作为log收集、indexer作为log转载.

Logstash shipper收集log 并将log转发给redis 存储

Logstash indexer从redis中读取数据并转发给elasticsearch

redis：是一个db，logstash shipper将log转发到redis数据库中存储。Logstash indexer从redis中读取数据并转发给elasticsearch。

Elasticsearch：elasticsearch是基于lucene的开源搜索引擎,用来做索引。

Kibana： 开源web展现,界面很漂亮,是一个功能强大的elasticsearch数据显示客户端，logstash已经内置了kibana，你也可以单独部署kibana，最新版的kibana3是纯html+js客户端.








软件下载目录
http://www.elasticsearch.org/downloads/

我的环境如下
os：centos6.3_x86-64

redis-2.8.7.tar.gz

kibana-3.0.0
java version "1.7.0_51"
elasticsearch-0.90.12

一，安装java
yum -y install java
二,安装redis

cd ~/src
wget http://download.redis.io/releases/redis-2.8.7.tar.gz tar -zxf redis-2.8.7.tar.gz
cd redis-2.8.7.tar.gz
make
sudo make install

安装完毕后

/etc/init.d/redis_6379 start

测试是否正常

[root@file1 ~]# redis-cli ping
PONG
[root@file1 ~]#

[root@file1 ~]# netstat -tanpu|grep redis
tcp        0      0 0.0.0.0:6379                0.0.0.0:*                   LISTEN      1391/redis-server *

三，安装Elasticsearch

cd /search
sudo mkdir elasticsearch
cd elasticsearch
sudo wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.12.zip sudo unzip elasticsearch-0.90.12.zip

备注：当开始使用的是1.x.x java报错，后来用的0.9.。
https://groups.google.com/forum/#!topic/logstash-users/fvFT7pgQTEM

Are you using elasticsearch_http for your output?  If not, 1.3.3 is based on 0.90.x elasticsearch, and won’t play nice with 1.0.x elasticsearch with just the “elasticsearch” output.

启动ES服务器
切换到elasticsearch目录运行

bin/elasticsearch -f

默认端口是9200

curl -X GET http://localhost:9200 [root@file1 ~]# curl -X GET http://localhost:9200 {
"ok" : true,
"status" : 200,
"name" : "Master Pandemonium",
"version" : {
"number" : "0.90.12",
"build_hash" : "26feed79983063ae83bfa11bd4ce214b1f45c884",
"build_timestamp" : "2014-02-25T15:38:23Z",
"build_snapshot" : false,
"lucene_version" : "4.6"
},
"tagline" : "You Know, for Search"
}

四.安装logstash

cd /search
sudo mkdir logstash
cd logstash
sudo wget http://download.elasticsearch.org/logstash/logstash/logstash-1.2.1-flatjar.jar[/code] 
新建配置文件index.conf
# This is the logstash server index configuration.
# This file will be put in the same folder with logtash.jar file in the
# /etc/logtash/
# This takes information straight from redis and loads it into elasticsearch.
input {
redis {
host => "127.0.0.1"
type => "syslog"
threads => 4
# these settings should match the output of the agent
data_type => "list"
key => "logstash"
# We use json_event here since the sender is a logstash agent
format => "json_event"
}
}
output {
elasticsearch {
host => "127.0.0.1"
}
}


新建shiper.conf
input {
stdin {
type => "test"
}
}
output {
stdout { codec => rubydebug }
redis { host => "127.0.0.1" data_type => "list" key => "logstash" }
}

运行配置
java -jar logstash.jar agent -f shipper.conf
java -jar logstash.jar agent -f index.conf


五，配置kibana
logstash的最新版已经内置kibana，你也可以单独部署kibana。kibana3是纯粹JavaScript+html的客户端，所以可以部署到任意http服务器上。
https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0.zip
解压到web目录
http://127.0.0.1/kibana/index.html


资料来源参考：

1./article/5683961.html

2.http://michael.bouvy.net/blog/en/2013/11/19/collect-visualize-your-logs-logstash-elasticsearch-redis-kibana/

3.http://my.oschina.net/guol/blog/179848

4.http://tinytub.github.io/logstash-install.html

5.install in Ubuntu server
 http://tips4admin.com/blog/2013/10/how-to-centralize-your-log-with-logstash-elasticsearch-redis-kibana-in-ubuntu-server/ 6.logstash官方文档 the logstash book

本文出自 “婚格线” 博客，请务必保留此出处http://motor.blog.51cto.com/729413/1411199