ganglia监控自定义metric实践

Ganglia监控系统是UC Berkeley开源的一个项目，设计初衷就是要做好分布式集群的监控，监控层面包括资源层面和业务层面，资源层面包括cpu、memory、disk、IO、网络负载等，至于业务层面由于用户可以很方便的增加自定义的metric，因此可以用于做诸如服务性能、负载、出错率等的监控，例如某web服务的QPS、Http status错误率。此外，如果和Nagios集成起来还可以在某指标超过一定阈值时触发相应的报警。Ganglia相比zabbix的优势在于客户端收集agent（gmond）所带来的系统开销非常低，不会影响相关服务的性能。ganglia主要有几个模块：gmond: 部署在各个被监控机器上，用于定期将数据收集起来，进行广播或者单播。gmetad：部署在server端，定时从配置的data_source中的host去拉取gmond收集好的数据ganglia-web：将监控数据投递到web页面关于ganglia的安装本文不做过多介绍，传送门：http://www.it165.net/admin/html/201302/770.html本文主要介绍一下如何开发自定义的metric，方便监控自己关心的指标。主要有几大类的方法：1. 直接使用gmetric安装gmond的机器，会同时安装上/usr/bin/gmetric，该命令是将一个metric的namevalue等信息进行广播的工具，例如

/usr/bin/gmetric -c /etc/ganglia/gmond.conf --name=test --type=int32 --units=sec --value=2

具体gmetric的选项见：http://manpages.ubuntu.com/manpages/hardy/man1/gmetric.1.html

此外，除了直接命令行使用gmetric外，还可以使用常见语言的binding，例如go、Java、python等，github上都有相关的binding可以使用，只需要import进来即可。go语言   https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-goruby  https://github.com/igrigorik/gmetric/blob/master/lib/gmetric.rb   Java  https://github.com/ganglia/ganglia_contrib/tree/master/gmetric-javaPython   https://github.com/ganglia/ganglia_contrib/tree/master/gmetric-python

2. 使用基于gmetric的第三方工具本文以ganglia-logtailer举例： https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-logtailer该工具基于logtail(debain)/logcheck(centos) package, 实现对日志的定时tail，然后通过指定classname来使用相应的类进行日志的分析，根据自己关注的字段统计出自定义metric，并由gmetric广播出来。例如我们根据自己服务的nginx日志格式，修改NginxLogtailer.py如下：

# -*- coding: utf-8 -*-######  This plugin for logtailer will crunch nginx logs and produce these metrics:###    * hits per second###    * GETs per second###    * average query processing time###    * ninetieth percentile query processing time###    * number of HTTP 200, 300, 400, and 500 responses per second######  Note that this plugin depends on a certain nginx log format, documented in##   __init__.import timeimport threadingimport re# local dependenciesfrom ganglia_logtailer_helper import GangliaMetricObjectfrom ganglia_logtailer_helper import LogtailerParsingException, LogtailerStateExceptionclass NginxLogtailer(object):# only used in daemon modeperiod = 30def __init__(self):'''This function should initialize any data structures or variablesneeded for the internal state of the line parser.'''self.reset_state()self.lock = threading.RLock()# this is what will match the nginx lines#log_format ganglia-logtailer#    '$host '#    '$server_addr '#    '$remote_addr '#    '- '#    '"$time_iso8601" '#    '$status '#    '$body_bytes_sent '#    '$request_time '#    '"$http_referer" '#    '"$request" '#    '"$http_user_agent" '#    '$pid';# NOTE: nginx 0.7 doesn't support $time_iso8601, use $time_local# instead# original apache log format string:# %v %A %a %u %{%Y-%m-%dT%H:%M:%S}t %c %s %>s %B %D \"%{Referer}i\" \"%r\" \"%{User-Agent}i\" %P# host.com 127.0.0.1 127.0.0.1 - "2008-05-08T07:34:44" - 200 200 371 103918 - "-" "GET /path HTTP/1.0" "-" 23794# match keys: server_name, local_ip, remote_ip, date, status, size,#               req_time, referrer, request, user_agent, pidself.reg = re.compile('^(?P<remote_ip>[^ ]+) (?P<server_name>[^ ]+) (?P<hit>[^ ]+) \[(?P<date>[^\]]+)\] "(?P<request>[^"]+)" (?P<status>[^ ]+) (?P<size>[^ ]+) "(?P<referrer>[^"]+)" "(?P<user_agent>[^"]+)" "(?P<forward_to>[^"]+)" "(?P<req_time>[^"]+)"')# assume we're in daemon mode unless set_check_duration gets calledself.dur_override = False# example function for parse line# takes one argument (text) line to be parsed# returns nothingdef parse_line(self, line):'''This function should digest the contents of one line at a time,updating the internal state variables.'''self.lock.acquire()try:regMatch = self.reg.match(line)if regMatch:linebits = regMatch.groupdict()if '-' == linebits['request'] or 'file2get' in linebits['request']:self.lock.release()returnself.num_hits+=1# capture GETsif( 'GET' in linebits['request'] ):self.num_gets+=1# capture HTTP response coderescode = float(linebits['status'])if( (rescode >= 200) and (rescode < 300) ):self.num_two+=1elif( (rescode >= 300) and (rescode < 400) ):self.num_three+=1elif( (rescode >= 400) and (rescode < 500) ):self.num_four+=1elif( (rescode >= 500) and (rescode < 600) ):self.num_five+=1# capture request durationdur = float(linebits['req_time'])self.req_time += dur# store for 90th % calculationself.ninetieth.append(dur)else:raise LogtailerParsingException, "regmatch failed to match"except Exception, e:self.lock.release()raise LogtailerParsingException, "regmatch or contents failed with %s" % eself.lock.release()# example function for deep copy# takes no arguments# returns one objectdef deep_copy(self):'''This function should return a copy of the data structure used tomaintain state.  This copy should different from the object that iscurrently being modified so that the other thread can deal with itwithout fear of it changing out from under it.  The format of thisobject is internal to the plugin.'''myret = dict( num_hits=self.num_hits,num_gets=self.num_gets,req_time=self.req_time,num_two=self.num_two,num_three=self.num_three,num_four=self.num_four,num_five=self.num_five,ninetieth=self.ninetieth)return myret# example function for reset_state# takes no arguments# returns nothingdef reset_state(self):'''This function resets the internal data structure to 0 (savingwhatever state it needs).  This function should be calledimmediately after deep copy with a lock in place so the internaldata structures can't be modified in between the two calls.  If thetime between calls to get_state is necessary to calculate metrics,reset_state should store now() each time it's called, and get_statewill use the time since that now() to do its calculations'''self.num_hits = 0self.num_gets = 0self.req_time = 0self.num_two = 0self.num_three = 0self.num_four = 0self.num_five = 0self.ninetieth = list()self.last_reset_time = time.time()# example for keeping track of runtimes# takes no arguments# returns float number of seconds for this rundef set_check_duration(self, dur):'''This function only used if logtailer is in cron mode.  If it isinvoked, get_check_duration should use this value instead of calculatingit.'''self.duration = durself.dur_override = Truedef get_check_duration(self):'''This function should return the time since the last check.  If calledfrom cron mode, this must be set using set_check_duration().  If indaemon mode, it should be calculated internally.'''if( self.dur_override ):duration = self.durationelse:cur_time = time.time()duration = cur_time - self.last_reset_time# the duration should be within 10% of periodacceptable_duration_min = self.period - (self.period / 10.0)acceptable_duration_max = self.period + (self.period / 10.0)if (duration < acceptable_duration_min or duration > acceptable_duration_max):raise LogtailerStateException, "time calculation problem - duration (%s) > 10%% away from period (%s)" % (duration, self.period)return duration# example function for get_state# takes no arguments# returns a dictionary of (metric => metric_object) pairsdef get_state(self):'''This function should acquire a lock, call deep copy, get thecurrent time if necessary, call reset_state, then do itscalculations.  It should return a list of metric objects.'''# get the data to work withself.lock.acquire()try:mydata = self.deep_copy()check_time = self.get_check_duration()self.reset_state()self.lock.release()except LogtailerStateException, e:# if something went wrong with deep_copy or the duration, reset and continueself.reset_state()self.lock.release()raise e# crunch data to how you want to report ithits_per_second = mydata['num_hits'] / check_timegets_per_second = mydata['num_gets'] / check_timeif (mydata['num_hits'] != 0):avg_req_time = mydata['req_time'] / mydata['num_hits']else:avg_req_time = 0two_per_second = mydata['num_two'] / check_timethree_per_second = mydata['num_three'] / check_timefour_per_second = mydata['num_four'] / check_timefive_per_second = mydata['num_five'] / check_time# calculate 90th % request timeninetieth_list = mydata['ninetieth']ninetieth_list.sort()num_entries = len(ninetieth_list)if (num_entries != 0 ):ninetieth_element = ninetieth_list[int(num_entries * 0.9)]else:ninetieth_element = 0# package up the data you want to submithps_metric = GangliaMetricObject('nginx_hits', hits_per_second, units='hps')gps_metric = GangliaMetricObject('nginx_gets', gets_per_second, units='hps')avgdur_metric = GangliaMetricObject('nginx_avg_dur', avg_req_time, units='sec')ninetieth_metric = GangliaMetricObject('nginx_90th_dur', ninetieth_element, units='sec')twops_metric = GangliaMetricObject('nginx_200', two_per_second, units='hps')threeps_metric = GangliaMetricObject('nginx_300', three_per_second, units='hps')fourps_metric = GangliaMetricObject('nginx_400', four_per_second, units='hps')fiveps_metric = GangliaMetricObject('nginx_500', five_per_second, units='hps')# return a list of metric objectsreturn [ hps_metric, gps_metric, avgdur_metric, ninetieth_metric, twops_metric, threeps_metric, fourps_metric, fiveps_metric, ]

在被监控机器上部署ganglia-logtailer后，使用如下命令建立crond任务*/1 * * * * root /usr/local/bin/ganglia-logtailer --classname NginxLogtailer --log_file /usr/local/nginx-video/logs/access.log --mode cron --gmetric_options '-C test_cluster -g nginx_status'reload crond service,过一分钟后，在ganglia web上即可看到相应的metric信息：关于ganglia-logtailer的部署方法，详见：https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-logtailer3. 用支持的语言编写自己的module，本文以Python为例ganglia支持用户编写自己的Pythonmodule，以下为github上简要介绍：Writing a Python module is very simple. You just need to write it following a template and put the resulting Pythonmodule (.py) in /usr/lib(64)/ganglia/python_modules.A corresponding Python Configuration (.pyconf) file needs to reside in /etc/ganglia/conf.d/.例如，编写一个检查机器温度的示例Python文件

acpi_file = "/proc/acpi/thermal_zone/THRM/temperature"def temp_handler(name):try:f = open(acpi_file, 'r')except IOError:return 0for l in f:line = l.split()return int(line[1])def metric_init(params):global descriptors, acpi_fileif 'acpi_file' in params:acpi_file = params['acpi_file']d1 = {'name': 'temp','call_back': temp_handler,'time_max': 90,'value_type': 'uint','units': 'C','slope': 'both','format': '%u','description': 'Temperature of host','groups': 'health'}descriptors = [d1]return descriptorsdef metric_cleanup():'''Clean up the metric module.'''pass#This code is for debugging and unit testingif __name__ == '__main__':metric_init({})for d in descriptors:v = d['call_back'](d['name'])print 'value for %s is %u' % (d['name'],  v)

有了module功能文件，还需要编写一个对应的配置文件(放在/etc/ganglia/conf.d/temp.pyconf下)，格式如下：

modules {module {name = "temp"language = "python"# The following params are examples only#  They are not actually used by the temp moduleparam RandomMax {value = 600}param ConstantValue {value = 112}}}collection_group {collect_every = 10time_threshold = 50metric {name = "temp"title = "Temperature"value_threshold = 70}}

有了这两个文件，这个module就算添加成功了。更多的用户贡献的module，请查看 https://github.com/ganglia/gmond_python_modules其中包括elasticsearch、filecheck、nginx_status、MySQL等常见服务的监控metric对应的module，非常有用，只需要稍作修改，即可满足自己的需求。其他的一些比较实用的用户贡献的工具ganglia-alert ：获取gmetad数据，并报警 https://github.com/ganglia/ganglia_contrib/tree/master/ganglia-alertganglia-docker：在docker中使用ganglia，https://github.com/ganglia/ganglia_contrib/tree/master/dockergmetad-health-check：监控gmetad服务状态，如果down掉，则restart服务， https://github.com/ganglia/ganglia_contrib/tree/master/gmetad_health_checkerchef-ganglia：用chef部署ganglia， https://github.com/ganglia/chef-gangliaansible-ganglia: 使用ansible自动化部署ganglia，https://github.com/remysaissy/ansible-gangliaganglia-nagios： 集成nagios和ganglia，https://github.com/ganglia/gangliosganglia-api ： 对外提供rest api，以特定格式返回gmetad收集到的数据， https://github.com/guardian/ganglia-api如有问题，欢迎留言讨论。