Logstash安装和设置(图文详解)(多节点的ELK集群安装在一个节点就好)
2017-03-25 14:49
489 查看
[b]前提[/b]
我这里的机器集群情况分别是:
HadoopMaster(192.168.80.10)、HadoopSlave1(192.168.80.11)和HadoopSlave2(192.168.80.12)。
1、上传logstash-2.4.1.tar.gz压缩包
-rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip
drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73
-rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip
drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6
[hadoop@HadoopMaster app]$
2、解压
第三步:删除安装包,并修改所属组和用户
第四步:认识目录结构
Filebeat啊,根据input来监控数据,根据output来使用数据!!!
对应于,Logstash啊,有input、filter和output。
[b]最简单的Logstash测试(即,输入什么,直接在console打印输出)[/b]
以上是最简单的Logstash测试(即,输入什么,直接在console打印输出)。
[b]Logstash可以以指定某种格式来输入[/b]。比如如下:
我们可以看到,我们输入什么内容logstash按照某种格式输出,其中-e参数参数允许Logstash直接通过命令行接受设置。这点尤其快速的帮助我们反复的测试配置是否正确而不用写配置文件。使用Ctrl + C命令可以退出之前运行的Logstash。
使用-e参数在命令行中指定配置是很常用的方式,不过如果需要配置更多设置则需要很长的内容。这种情况,我们首先创建一个简单的配置文件,并且指定logstash使用这个配置文件。例如:在logstash安装目录下创建一个“基本配置”测试文件logstash-simple.conf。
[b]Logstash使用-f参数替换命令行中的-e参数(既可以写到配置文件里,为了方便)[/b]
[b]
[/b]
推荐用这个!!!
因为,在调试,每次都要重启。加这个,不需每次去重启Logstash,即自己会加载。
Elasticsearch-2.4.3的下载(图文详解)
Elasticsearch-2.4.3的单节点安装(多种方式图文详解)
Elasticsearch-2.4.3的3节点安装(多种方式图文详解)
Logstash-2.4.1的下载(图文详解)
Logstash是一个管理日志和事件的工具。我这里的机器集群情况分别是:
HadoopMaster(192.168.80.10)、HadoopSlave1(192.168.80.11)和HadoopSlave2(192.168.80.12)。
1、上传logstash-2.4.1.tar.gz压缩包
[hadoop@HadoopMaster app]$ ll total 16832 drwxrwxr-x. 9 hadoop hadoop 4096 Feb 22 06:05 elasticsearch-2.4.3 -rw-r--r--. 1 hadoop hadoop 908862 Jan 10 11:38 elasticsearch-head-master.zip -rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz -rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73 -rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6 [hadoop@HadoopMaster app]$ rz [hadoop@HadoopMaster app]$ ll total 98864 drwxrwxr-x. 9 hadoop hadoop 4096 Feb 22 06:05 elasticsearch-2.4.3 -rw-r--r--. 1 hadoop hadoop 908862 Jan 10 11:38 elasticsearch-head-master.zip -rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 -rw-r--r--. 1 hadoop hadoop 83999654 Feb 25 11:06 logstash-2.4.1.tar.gz -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz
-rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip
drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73
-rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip
drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6
[hadoop@HadoopMaster app]$
2、解压
[hadoop@HadoopMaster app]$ ll total 98864 drwxrwxr-x. 9 hadoop hadoop 4096 Feb 22 06:05 elasticsearch-2.4.3 -rw-r--r--. 1 hadoop hadoop 908862 Jan 10 11:38 elasticsearch-head-master.zip -rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 -rw-r--r--. 1 hadoop hadoop 83999654 Feb 25 11:06 logstash-2.4.1.tar.gz -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz -rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73 -rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6 [hadoop@HadoopMaster app]$ tar -zxvf logstash-2.4.1.tar.gz
第三步:删除安装包,并修改所属组和用户
-rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 drwxrwxr-x. 5 hadoop hadoop 4096 Mar 27 03:58 logstash-2.4.1 -rw-r--r--. 1 hadoop hadoop 83999654 Feb 25 11:06 logstash-2.4.1.tar.gz -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz -rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73 -rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6 [hadoop@HadoopMaster app]$ rm logstash-2.4.1.tar.gz [hadoop@HadoopMaster app]$ ll total 16836 drwxrwxr-x. 9 hadoop hadoop 4096 Feb 22 06:05 elasticsearch-2.4.3 -rw-r--r--. 1 hadoop hadoop 908862 Jan 10 11:38 elasticsearch-head-master.zip -rw-r--r--. 1 hadoop hadoop 2228252 Jan 10 11:38 elasticsearch-kopf-master.zip drwxr-xr-x. 2 hadoop hadoop 4096 Mar 27 01:36 filebeat-1.3.1-x86_64 drwxr-xr-x. 10 hadoop hadoop 4096 Oct 31 17:15 hadoop-2.6.0 drwxr-xr-x. 15 hadoop hadoop 4096 Nov 14 2014 hadoop-2.6.0-src drwxrwxr-x. 8 hadoop hadoop 4096 Nov 2 18:20 hbase-1.2.3 drwxr-xr-x. 8 hadoop hadoop 4096 Apr 11 2015 jdk1.7.0_79 drwxrwxr-x. 11 hadoop hadoop 4096 Nov 4 23:24 kibana-4.6.3-linux-x86_64 drwxrwxr-x. 5 hadoop hadoop 4096 Mar 27 03:58 logstash-2.4.1 -rw-r--r--. 1 hadoop hadoop 10162116 Mar 25 10:00 marvel-2.4.4.tar.gz -rw-r--r--. 1 hadoop hadoop 2332033 Jan 16 17:25 shield-2.4.3.zip drwxrwxr-x. 9 hadoop hadoop 4096 Feb 25 19:18 tomcat-7.0.73 -rw-r--r--. 1 hadoop hadoop 1556618 Jan 16 17:22 watcher-2.4.3.zip drwxr-xr-x. 10 hadoop hadoop 4096 Nov 1 23:39 zookeeper-3.4.6
第四步:认识目录结构
[hadoop@HadoopMaster app]$ cd logstash-2.4.1/ [hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 160 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$
Filebeat啊,根据input来监控数据,根据output来使用数据!!!
对应于,Logstash啊,有input、filter和output。
[b]最简单的Logstash测试(即,输入什么,直接在console打印输出)[/b]
[hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 160 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$ bin/logstash -e 'input { stdin { } } output { stdout {} }' Settings: Default pipeline workers: 1 Pipeline main started (输入回车) 2017-03-26T21:01:02.849Z HadoopMaster (显示回车) abcd 2017-03-26T21:01:10.559Z HadoopMaster abcd
以上是最简单的Logstash测试(即,输入什么,直接在console打印输出)。
^CSIGINT received. Shutting down the agent. {:level=>:warn} stopping pipeline {:id=>"main"} Received shutdown signal, but pipeline is still waiting for in-flight events to be processed. Sending another ^C will force quit Logstash, but this may cause data loss. {:level=>:warn} ^CSIGINT received. Terminating immediately.. {:level=>:fatal} [hadoop@HadoopMaster logstash-2.4.1]$
[b]Logstash可以以指定某种格式来输入[/b]。比如如下:
[hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 160 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$ bin/logstash -e 'input { stdin { } } output { stdout { codec => json} }' Settings: Default pipeline workers: 1 Pipeline main started {"message":"","@version":"1","@timestamp":"2017-03-26T21:13:09.879Z","host":"HadoopMaster"}hjjjk {"message":"hjjjk","@version":"1","@timestamp":"2017-03-26T21:13:23.484Z","host":"HadoopMaster"}^CSIGINT received. Shutting down the agent. {:level=>:warn} stopping pipeline {:id=>"main"} ^CSIGINT received. Terminating immediately.. {:level=>:fatal} [hadoop@HadoopMaster logstash-2.4.1]$
我们可以看到,我们输入什么内容logstash按照某种格式输出,其中-e参数参数允许Logstash直接通过命令行接受设置。这点尤其快速的帮助我们反复的测试配置是否正确而不用写配置文件。使用Ctrl + C命令可以退出之前运行的Logstash。
使用-e参数在命令行中指定配置是很常用的方式,不过如果需要配置更多设置则需要很长的内容。这种情况,我们首先创建一个简单的配置文件,并且指定logstash使用这个配置文件。例如:在logstash安装目录下创建一个“基本配置”测试文件logstash-simple.conf。
[b]Logstash使用-f参数替换命令行中的-e参数(既可以写到配置文件里,为了方便)[/b]
[b]
[/b]
[hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 160 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$ vim logstash-simple.conf
input { stdin { } } output { stdout { } }
[hadoop@HadoopMaster logstash-2.4.1]$ pwd /home/hadoop/app/logstash-2.4.1 [hadoop@HadoopMaster logstash-2.4.1]$ ll total 164 drwxrwxr-x. 2 hadoop hadoop 4096 Mar 27 03:58 bin -rw-rw-r--. 1 hadoop hadoop 102879 Nov 14 10:04 CHANGELOG.md -rw-rw-r--. 1 hadoop hadoop 2249 Nov 14 10:04 CONTRIBUTORS -rw-rw-r--. 1 hadoop hadoop 5084 Nov 14 10:07 Gemfile -rw-rw-r--. 1 hadoop hadoop 23015 Nov 14 10:04 Gemfile.jruby-1.9.lock drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 lib -rw-rw-r--. 1 hadoop hadoop 589 Nov 14 10:04 LICENSE -rw-rw-r--. 1 hadoop hadoop 46 Mar 27 05:30 logstash-simple.conf -rw-rw-r--. 1 hadoop hadoop 149 Nov 14 10:04 NOTICE.TXT drwxrwxr-x. 4 hadoop hadoop 4096 Mar 27 03:58 vendor [hadoop@HadoopMaster logstash-2.4.1]$ bin/logstash -f logstash-simple.conf Settings: Default pipeline workers: 1 Pipeline main started 2017-03-26T21:32:32.782Z HadoopMaster abcd 2017-03-26T21:32:36.848Z HadoopMaster abcd ^CSIGINT received. Shutting down the agent. {:level=>:warn} stopping pipeline {:id=>"main"} ^CSIGINT received. Terminating immediately.. {:level=>:fatal} [hadoop@HadoopMaster logstash-2.4.1]$
推荐用这个!!!
bin/logstash -f logstash-simple.conf --auto-reload
因为,在调试,每次都要重启。加这个,不需每次去重启Logstash,即自己会加载。
相关文章推荐
- Filebeat-1.3.1安装和设置(图文详解)(多节点的ELK集群安装在一个节点就好)(以Console Output为例)
- Kibana安装(图文详解)(多节点的ELK集群安装在一个节点就好)
- 基于CentOS6.5或Ubuntu14.04下Suricata里搭配安装 ELK (elasticsearch, logstash, kibana)(图文详解)
- 安装Cloudera Manager集群时首次运行命令部署客户端设置失败的解决办法(图文详解)
- 图文详解zabbix的安装以及设置邮件报警
- 图文详解 Windows 2003服务器集群安装(1)
- 图文超详解zabbix的安装以及设置邮件报警 推荐
- elk集群安装配置详解
- storm的3节点集群详细启动步骤(非HA和HA)(图文详解)
- ELK(ElasticSearch / Logstash / Kibana)安装及其一个整合示例
- CentOS6.5下Cloudera安装搭建部署大数据集群(图文分五大步详解)(博主强烈推荐)
- 图文详解 Windows 2003服务器集群安装(3)
- 图文详解 Windows 2003服务器集群安装(4)
- kaa系统安装------在一个单一的Linux节点或集群环境中安装和配置kaa平台。
- 给ambari集群里的kafka安装基于web的kafka管理工具Kafka-manager(图文详解)
- CentOS6.5下Ambari安装搭建部署大数据集群(图文分五大步详解)(博主强烈推荐)
- Hadoop --MapReduce2 - 群集设置(多个节点以及大集群中数千个节点进行安装)
- Hadoop2.2.0单节点安装和配置环境图文详解
- Hadoop-2.6.4集群(三个节点)安装(详细图文)
- kafka_2.10-0.8.1.1.tgz的1或3节点集群的下载、安装和配置(图文详细教程)绝对干货