【RSYSLOG】rsyslog作为日志采集器安装配置说明
2016-06-17 16:22
393 查看
RSYSLOG is the rocket-fast system for log processing.
About
由于环境基于CentOS 6.7 x64,rsyslog本身就是OS的组件,由于本文使用rsyslog作为统一日志采集器,需要与kafka进行数据交付,而原始支持kafka组件是在v8.7.0以后版本才支持,而系统自带的rsyslog是版本是v5的,因此需要对rsyslog版本进行升级。有网的方法比较简单,通过yum命令即可,官网参考官网 http://www.rsyslog.com/rhelcentos-rpms/ 即可安装指定版本。
安装说明
由于鉴于内网环境无Internet无法使用yum进行安装,因此选择离线rpm方式升级rsyslog(rpm包下载地址:http://rpms.adiscon.com/ )。此处有坑:按习惯,一般使用官网提供的最新的稳定版/v8-stable/epel-7,然而由于CentOS 6.7 glibc版本为2.12而epel-7依赖的libgt-0.3.11-1.el7.x86_64、libfastjson-0.99.2-1.el7.x86_64、libestr-0.1.10-1.el7.x86_64几个组件需要 libc.so.6: version `GLIBC_2.14,CentOS 6.7升级后会提示bash版本有问题,因此此处选用/v8-stable/epel-6版本(ps:此坑是个人问题,centos6.x本来就应该使用epel-6,因此此坑可忽略)
需要下载的rpm
可在http://rpms.adiscon.com/v8-stable/epel-6/x86_64/RPMS/ 获取
json-c-0.11-4.el6.x86_64.rpm
libfastjson-0.99.2-1.el6.x86_64.rpm
libgt-0.3.11-1.el6.x86_64.rpm
libestr-0.1.10-1.el6.x86_64.rpm
liblogging-1.0.5-1.el6.x86_64.rpm
librdkafka1-0.8.5-0.x86_64.rpm
rsyslog-kafka-8.19.0-1.el6.x86_64.rpm
rsyslog-8.19.0-1.el6.x86_64.rpm
rpm安装
执行安装前 先给没个文件加上运行属性 chmod +x
rpm -ivh json-c-0.11-4.el6.x86_64.rpm rpm -ivh libfastjson-0.99.2-1.el6.x86_64.rpm rpm -ivh libgt-0.3.11-1.el6.x86_64.rpm rpm -ivh libestr-0.1.10-1.el6.x86_64.rpm rpm -ivh liblogging-1.0.5-1.el6.x86_64.rpm rpm -ivh librdkafka1-0.8.5-0.x86_64.rpm #这边使用 -U 参数是为了直接升级本地rsyslog rpm -Uvh rsyslog-8.19.0-1.el6.x86_64.rpm #rsyslog-kafka组件安装依赖 rsyslog-8.19.0及librdkafka,该命令最后执行 rpm -ivh rsyslog-kafka-8.19.0-1.el6.x86_64.rpm
验证安装
安装完后执行rsyslogd -v, 验证rsyslog是否升级成功。
# rsyslogd -v rsyslogd 8.19.0, compiled with: PLATFORM: x86_64-redhat-linux-gnu PLATFORM (lsb_release -d): FEATURE_REGEXP: Yes GSSAPI Kerberos 5 support: No FEATURE_DEBUG (debug build, slow code): No 32bit Atomic operations supported: Yes 64bit Atomic operations supported: Yes memory allocator: system default Runtime Instrumentation (slow code): No uuid support: Yes Number of Bits in RainerScript integers: 64 See http://www.rsyslog.com for more information.
查看/lib64/rsyslog/(32位系统为 /lib/rsyslog) 中 是否存在 omkafka.so,验证rsyslog-kafka是否安装成功。
配置说明
rsyslog官网是个大坑,能路过尽量路过。
接下来说明下配置,首先声明下本文使用rsyslog的目的是作为应用的log collector,应用使用log4x(log4net:log4net.Appender.RemoteSyslogAppender, log4j:org.apache.log4j.net.SyslogAppender),通过UDP协议将日志传输到rsyslog。
rsyslo的默认的配置文件 /etc/rsyslog.config
# rsyslog configuration file # note that most of this config file uses old-style format, # because it is well-known AND quite suitable for simple cases # like we have with the default config. For more advanced # things, RainerScript configuration is suggested. # For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html # If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html #### MODULES #### module(load="imuxsock") # provides support for local system logging (e.g. via logger command) module(load="imklog") # provides kernel logging support (previously done by rklogd) #module(load"immark") # provides --MARK-- message capability # Provides UDP syslog reception # for parameters see http://www.rsyslog.com/doc/imudp.html #module(load="imudp") # needs to be done just once #input(type="imudp" port="514") # Provides TCP syslog reception # for parameters see http://www.rsyslog.com/doc/imtcp.html #module(load="imtcp") # needs to be done just once #input(type="imtcp" port="514") #### GLOBAL DIRECTIVES #### # Use default timestamp format $ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat # File syncing capability is disabled by default. This feature is usually not required, # not useful and an extreme performance hit #$ActionFileEnableSync on # Include all config files in /etc/rsyslog.d/ $IncludeConfig /etc/rsyslog.d/*.conf #### RULES #### # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none;cron.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Log cron stuff cron.* /var/log/cron # Everybody gets emergency messages *.emerg :omusrmsg:* # Save news errors of level crit and higher in a special file. uucp,news.crit /var/log/spooler # Save boot messages also to boot.log local7.* /var/log/boot.log # ### begin forwarding rule ### # The statement between the begin ... end define a SINGLE forwarding # rule. They belong together, do NOT split them. If you create multiple # forwarding rules, duplicate the whole block! # Remote Logging (we use TCP for reliable delivery) # # An on-disk queue is created for this action. If the remote host is # down, messages are spooled to disk and sent when it is up again. #$WorkDirectory /var/lib/rsyslog # where to place spool files #$ActionQueueFileName fwdRule1 # unique name prefix for spool files #$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible) #$ActionQueueSaveOnShutdown on # save messages to disk on shutdown #$ActionQueueType LinkedList # run asynchronously #$ActionResumeRetryCount -1 # infinite retries if host is down # remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional #*.* @@remote-host:514 # ### end of the forwarding rule ###
接下来说明配置修改:
配置 input
开启接收UDP消息module(load="imudp") # 加载imudp模块 input(type="imudp" port="514") # 配置UDP监听的端口号
配置 output
开启数据转发kafkamodule(load="omkafka") # 配置加载omkafka模块 # 转发到kafka消息格式 template(name="rsyslogToKafka" type="list" option.json="on"){ constant(value="{") #constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339") constant(value="\"message\":\"") property(name="msg" format="json") constant(value="\",\"host\":\"") property(name="hostname") constant(value="\",\"severity\":\"") property(name="syslogseverity") constant(value="\",\"severity-text\":\"") property(name="syslogseverity-text") constant(value="\",\"facility\":\"") property(name="syslogfacility-text") constant(value="\",\"syslogtag\":\"") property(name="syslogtag") constant(value="\",\"type\":\"application\"") constant(value=",\"recieve-from\":\"") property(name="fromhost-ip") constant(value="\",\"recieve-at\":\"") property(name="timegenerated" dateFormat="rfc3339") constant(value="\",\"report-at\":\"") property(name="timereported" dateFormat="rfc3339") constant(value="\"}") } # 配置kafka相关信息及模板 action( type="omkafka" topic="test" broker="11.4.74.26:9092" template="rsyslogToKafka" )
另外一个坑:关于EscapeControlCharacters转换的问题
rsyslog接收的时候会对一些回车换行等敏感字符进行转换 比如 \r\n 会被转换为#015#012,而通过log4x提交的日志如果包含Exception在kibana显示就不能正常的换行,因此关闭了rsyslog中敏感字的转换
$EscapeControlCharactersOnReceive off
相关文章推荐
- 必看的nb书
- springmvc+mybaits+spring日志
- C#读取Json文件出现乱码解决方案
- C一些词汇的概念
- Crosswalk Project for Windows
- 从github下载项目到本地
- paramiko 模块
- c++(补)
- CentOS7安装Hadoop2.7完整步骤
- 503 5.5.1 Error: authentication not enabled
- 实验吧逆向工程Just Click
- Junit使用GroboUtils进行多线程测试
- 华为OJ平台——整数的二进制中1的个数
- spring mvc 项目聚合
- 迦瓦栈队 团队第六周项目总结
- Struts2 用 s:if test 判断String类型的对象属性值和单字符是否相等的问题
- 8天学通MongoDB——第四天 索引操作
- swift学习之路(十七)协议
- 日期控件的控制条件
- apktool编译和反编译apk与ecplise多渠道打包