ELK Stack 日志分析监控平台
2016-05-06 11:02
369 查看
ELK Stack
最近因工作需要部署一个日志分析监控平台, 最后选型ELK Stack:elasticsearch-2.1.1
logstash-2.1.1
kibana-4.3.1
redis-3.0.6
Deployment Diagram
说明:
AppServer: 应用(Application)部署的服务器,应用运行会在本地写日志, 这里的Logstash负责收集这些日志并将日志输出到Broker中
Broker: 把AppServer的日志传给LogServer过程中的缓冲队列, 官方推荐使用Redis
LogServer: 汇总各个AppServer的日志进行分析处理展示.
过程:
AppServer中的Logstash收集本地的日志,将日志发送到Broker中Redis.
LogServer中的Logstash收集Broker(Redis)中的日志发送给Elasticsearch.
Kibana将Elasticsearch处理的结果进行展示.
Installation
Downloadlogstash2.1.1.tar.gz
elasticsearch-2.1.1.tar.gz
kibana-4.3.1-linux-x64.tar.gz
Installation
安装Reids, 略.
LogServer
解压上面下载的三个文件分别到
/usr/local/logstash,
/usr/local/elasticsearch,
/usr/local/kibana
启动elasticsearch
/usr/local/ealsticsearch/bin/ealsticsearch
启动kibana
/usr/local/kibana/bin/kibana
默认配置下已经可以通过127.0.0.1:5601访问了
创建logstash的配置文件:
<code class="hljs lua has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vim /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/logstash/conf/logstash.conf</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
conf目录是没有的, 新建. logstash.conf文件也没有的, 新建.
配置文件的内容:
<code class="hljs php has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">input { redis{ host => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"127.0.0.1"</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#Broker的IP,当前示例中Broker与LogServer为同一台机</span> port => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"6379"</span> type => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"stagefalcon"</span> data_type => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"list"</span> key => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"stagefalcon"</span> } } output { stdout {codec=>rubydebug} elasticsearch{ hosts =>[<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"127.0.0.1:9200"</span>] } } </code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li></ul>
表示从127.0.0.1中的redis中取出数据,输出到elasticsearch中. elasticsearch默认通讯端口9200.
配置文件检查命令:
<code class="hljs lua has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/logstash/bin/logstash -f /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/logstash/conf/logstash.conf <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">--configtest</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
启动logstash:
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/logstash/bin/logstash <span class="hljs-attribute" style="box-sizing: border-box;">-f</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/logstash/conf/logstash<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>conf</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
AppServer
解压logstash2.1.1.tar.gz到
/usr/local/logstash下
创建logstash的配置文件:
<code class="hljs lua has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">vim /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/logstash/conf/logstash.conf</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
conf目录是没有的, 新建. logstash.conf文件也没有的, 新建.
配置文件的内容:
<code class="hljs php has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">input { file { type => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"stagefalcon"</span> tags => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"falcon"</span> path => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/var/log/falcon/access.log"</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># 日志文件路径</span> start_position => beginning <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># 文件开始位置</span> sincedb_path => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"/usr/local/logstash/conf/access.sincedb"</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;"># 如果传输文件过程记录位置,下次启动时从上次终端位置开始传输,否则文件开头开始传输</span> } } output { stdout {} redis { host => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"10.0.72.50"</span> <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">#Broker的IP</span> port => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"6379"</span> data_type => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"list"</span> key => <span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;">"stagefalcon"</span> } }</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li><li style="box-sizing: border-box; padding: 0px 5px;">2</li><li style="box-sizing: border-box; padding: 0px 5px;">3</li><li style="box-sizing: border-box; padding: 0px 5px;">4</li><li style="box-sizing: border-box; padding: 0px 5px;">5</li><li style="box-sizing: border-box; padding: 0px 5px;">6</li><li style="box-sizing: border-box; padding: 0px 5px;">7</li><li style="box-sizing: border-box; padding: 0px 5px;">8</li><li style="box-sizing: border-box; padding: 0px 5px;">9</li><li style="box-sizing: border-box; padding: 0px 5px;">10</li><li style="box-sizing: border-box; padding: 0px 5px;">11</li><li style="box-sizing: border-box; padding: 0px 5px;">12</li><li style="box-sizing: border-box; padding: 0px 5px;">13</li><li style="box-sizing: border-box; padding: 0px 5px;">14</li><li style="box-sizing: border-box; padding: 0px 5px;">15</li><li style="box-sizing: border-box; padding: 0px 5px;">16</li><li style="box-sizing: border-box; padding: 0px 5px;">17</li><li style="box-sizing: border-box; padding: 0px 5px;">18</li></ul>
配置文件检查命令:
<code class="hljs lua has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/logstash/bin/logstash -f /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">local</span>/logstash/conf/logstash.conf <span class="hljs-comment" style="color: rgb(136, 0, 0); box-sizing: border-box;">--configtest</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
启动logstash:
<code class="hljs lasso has-numbering" style="display: block; padding: 0px; color: inherit; box-sizing: border-box; font-family: 'Source Code Pro', monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal; background: transparent;">/usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/logstash/bin/logstash <span class="hljs-attribute" style="box-sizing: border-box;">-f</span> /usr/<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">local</span>/logstash/conf/logstash<span class="hljs-built_in" style="color: rgb(102, 0, 102); box-sizing: border-box;">.</span>conf</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right-width: 1px; border-right-style: solid; border-right-color: rgb(221, 221, 221); list-style: none; text-align: right; background-color: rgb(238, 238, 238);"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
参考资料
相关文章推荐
- 如何禁止DELETE、PUT、OPTIONS、TRACE、HEAD等协议访问应用程序
- OPENGL设备坐标系(dns)是左手坐标系,屏幕坐标系原点在左下角向上向右增加
- 详解OpenGL的坐标系、投影和几何变换-矩阵压栈思想/矩阵列式存储
- win7 VS2012+openCV-2.4.11 配置
- squid实现反向代理!!!
- CentOS7中关闭selinux
- 本地yum库制作及本地安装Docker
- 1.2 大型网站架构演化发展过程[读书敲录]
- 【Linux程序员福音】在Visual Studio上用C++写Linux
- Linux进程的睡眠和唤醒
- Nginx的工作模式和一些理解
- Linux中,shell脚本调用java程序模板
- 解决vmware上linux虚拟机的eth0不存在的问题
- Linux 文件描述符
- linux 文件归档
- Linux-37-linux常用快捷键总结(L005-19)
- Linux-36-linux基础重要命令13(L005-18)
- 1.1 大型网站软件系统的特点[读书敲录]
- Linux进程管理之“四大名捕”
- 第一章 大型网站架构演化[手工敲录]