openstack nova的服务心跳机制和服务状态监控机制的实现

Openstack中的服务是分布式部署，因此，各个服务的启停状态，决定了此系统的可用性。我们可以

通过openstack提供的接口来查看服务的运行状态，比如：

nova service的相关配置

Nova相关的配置项都nova/conf这个目录下，如果我们相查看service相关的配置项，可以看nova/conf/service.py。其中，注意一下这两个配置项：

service_down_time

report_interval

具体作用可以查看注释。大概意思就是说，超过service_down_time的时间没有收到的心跳 的话，就认为服务是挂 了的，report_interval是定义了心跳的周期，即多长时间

上报一次自己的心跳。

nova服务的启动

nova-compute的服务的入口在nova/cmd/compute.py:main()

其中最重要的一行代码就是高亮的部分。这里首先会初始一个service.Service的类：

def main():
config.parse_args(sys.argv)
logging.setup(CONF, 'nova')
priv_context.init(root_helper=shlex.split(utils.get_root_helper()))
utils.monkey_patch()
objects.register_all()
# Ensure os-vif objects are registered and plugins loaded
os_vif.initialize()

gmr.TextGuruMeditation.setup_autorun(version)

cmd_common.block_db_access('nova-compute')
objects_base.NovaObject.indirection_api = conductor_rpcapi.ConductorAPI()

server = service.Service.create(binary='nova-compute',
topic=CONF.compute_topic)
service.serve(server)
service.wait()

我们详细看一下service.Service()这个类：

class Service(service.Service):
"""Service object for binaries running on hosts.

A service takes a manager and enables rpc by listening to queues based
on topic. It also periodically runs tasks on the manager and reports
its state to the database services table.
"""

def __init__(self, host, binary, topic, manager, report_interval=None,
periodic_enable=None, periodic_fuzzy_delay=None,
periodic_interval_max=None, *args, **kwargs):
super(Service, self).__init__()
self.host = host
self.binary = binary
self.topic = topic
self.manager_class_name = manager
self.servicegroup_api = servicegroup.API()
manager_class = importutils.import_class(self.manager_class_name)
self.manager = manager_class(host=self.host, *args, **kwargs)
self.rpcserver = None
self.report_interval = report_interval
self.periodic_enable = periodic_enable
self.periodic_fuzzy_delay = periodic_fuzzy_delay
self.periodic_interval_max = periodic_interval_max
self.saved_args, self.saved_kwargs = args, kwargs
self.backdoor_port = None
if objects_base.NovaObject.indirection_api:
conductor_api = conductor.API()
conductor_api.wait_until_ready(context.get_admin_context())
setup_profiler(binary, self.host)

这里重点看一下高亮的部分，看到一个report_interval这个变量。还有一个servicegroup.API()。

接下来我们看一下serivce.Service的start()方法：

# Add service to the ServiceGroup membership group.
self.servicegroup_api.join(self.host, self.topic, self)

会看上面一行代码，这里调用了servicegroup_api中的join方法，具体路径在nova/servicegroup/api.py:join()

def join(self, member, group, service=None):
"""Add a new member to a service group.

:param member: the joined member ID/name
:param group: the group ID/name, of the joined member
:param service: a `nova.service.Service` object
"""
return self._driver.join(member, group, service)

从servicegroup.API()的初始化函数我们看到

class API(object):

def __init__(self, *args, **kwargs):
'''Create an instance of the servicegroup API.

args and kwargs are passed down to the servicegroup driver when it gets
created.
'''
# Make sure report interval is less than service down time
report_interval = CONF.report_interval
if CONF.service_down_time <= report_interval:
new_service_down_time = int(report_interval * 2.5)
LOG.warning(_LW("Report interval must be less than service down "
"time. Current config: <service_down_time: "
"%(service_down_time)s, report_interval: "
"%(report_interval)s>. Setting service_down_time "
"to: %(new_service_down_time)s"),
{'service_down_time': CONF.service_down_time,
'report_interval': report_interval,
'new_service_down_time': new_service_down_time})
CONF.set_override('service_down_time', new_service_down_time)

driver_class = _driver_name_class_mapping[CONF.servicegroup_driver]
self._driver = importutils.import_object(driver_class,
*args, **kwargs)

回到nova/conf/servicegrouup.py中我们可以看到，CONF.servicegroup_driver目前只支持两个类型：

SERVICEGROUP_OPTS = [
cfg.StrOpt('servicegroup_driver',
default='db',
choices=['db', 'mc'],
help="""
This option specifies the driver to be used for the servicegroup service.

ServiceGroup API in nova enables checking status of a compute node. When a
compute worker running the nova-compute daemon starts, it calls the join API
to join the compute group. Services like nova scheduler can query the
ServiceGroup API to check if a node is alive. Internally, the ServiceGroup
client driver automatically updates the compute worker status. There are
multiple backend implementations for this service: Database ServiceGroup driver
and Memcache ServiceGroup driver.

Possible Values:

* db : Database ServiceGroup driver
* mc : Memcache ServiceGroup driver

Related Options:

* service_down_time (maximum time since last check-in for up service)
"""),
]

我们假定用的是db这种方式，因此，我们要找的join方法应该在nova/securitygroup/drivers/db.py中：

def join(self, member, group, service=None):
"""Add a new member to a service group.

:param member: the joined member ID/name
:param group: the group ID/name, of the joined member
:param service: a `nova.service.Service` object
"""
LOG.debug('DB_Driver: join new ServiceGroup member %(member)s to '
'the %(group)s group, service = %(service)s',
{'member': member, 'group': group,
'service': service})
if service is None:
raise RuntimeError(_('service is a mandatory argument for DB based'
' ServiceGroup driver'))
report_interval = service.report_interval
if report_interval:
service.tg.add_timer(report_interval, self._report_state,
api.INITIAL_REPORTING_DELAY, service)

在这里我们很容易发现了他的回调函数：self._report_state()

def _report_state(self, service):
"""Update the state of this service in the datastore."""

try:
service.service_ref.report_count += 1
service.service_ref.save()

# TODO(termie): make this pattern be more elegant.
if getattr(service, 'model_disconnected', False):
service.model_disconnected = False
LOG.info(
_LI('Recovered from being unable to report status.'))

其中service是从数据库中取得的最新service数据，该函数只是仅仅把report_count加一，然后调用save方法保存到数据库中。这里需要注意的是，save方法每次都会记录更新的时间,在数据库的字段为updated_at。

由此，nova服务的心跳机制，本质就是每隔一段时间往数据库更新report_count值，并记录最后更新时间作为接收到的最新心跳时间戳。
这个值可以从数据库中查到，如下：

服务状态监控机制

我们从nova-api的分析入手。其中与service相关的代码位于： nova/api/openstack/compute/services.py。当我们调用nova service-list的命令时，实际是调用：

1. nova/api/openstack/compute/serivces.py: index()

@extensions.expected_errors(())
def index(self, req):
"""Return a list of all running services. Filter by host & service
name
"""
if api_version_request.is_supported(req, min_version='2.11'):
_services = self._get_services_list(req, ['forced_down'])
else:
_services = self._get_services_list(req)

return {'services': _services}

2. nova/api/openstack/compute/serivces.py: _get_services_list():

def _get_services_list(self, req, additional_fields=()):
_services = self._get_services(req)
return [self._get_service_detail(svc, additional_fields)
for svc in _services]

3. nova/api/openstack/compute/serivces.py: _get_services_detail():

def _get_service_detail(self, svc, additional_fields):
alive = self.servicegroup_api.service_is_up(svc)
state = (alive and "up") or "down"
active = 'enabled'
if svc['disabled']:
active = 'disabled'
service_detail = {'binary': svc['binary'],
'host': svc['host'],
'id': svc['id'],
'zone': svc['availability_zone'],
'status': active,
'state': state,
'updated_at': svc['updated_at'],
'disabled_reason': svc['disabled_reason']}

for field in additional_fields:
service_detail[field] = svc[field]

return service_detail

nova/servicegroup/api.py

def service_is_up(self, member):
"""Check if the given member is up."""
# NOTE(johngarbutt) no logging in this method,
# so this doesn't slow down the scheduler
if member.get('forced_down'):
return False

return self._driver.is_up(member)

nova/serivcegroup/drivers/db.py

def is_up(self, service_ref):
"""Moved from nova.utils
Check whether a service is up based on last heartbeat.
"""
last_heartbeat = (service_ref.get('last_seen_up') or
service_ref['created_at'])
if isinstance(last_heartbeat, six.string_types):
# NOTE(russellb) If this service_ref came in over rpc via
# conductor, then the timestamp will be a string and needs to be
# converted back to a datetime.
last_heartbeat = timeutils.parse_strtime(last_heartbeat)
else:
# Objects have proper UTC timezones, but the timeutils comparison
# below does not (and will fail)
last_heartbeat = last_heartbeat.replace(tzinfo=None)
# Timestamps in DB are UTC.
elapsed = timeutils.delta_seconds(last_heartbeat, timeutils.utcnow())
is_up = abs(elapsed) <= self.service_down_time
if not is_up:
LOG.debug('Seems service %(binary)s on host %(host)s is down. '
'Last heartbeat was %(lhb)s. Elapsed time is %(el)s',
{'binary': service_ref.get('binary'),
'host': service_ref.get('host'),
'lhb': str(last_heartbeat), 'el': str(elapsed)})
return is_up

看到高亮的部分，我们就很容易明白，nova是如何判断一个服务是up还是down的了：

首先获取service实例的最后更新时间戳，即最后心跳时间，然后计算最后心跳时间距离现在时间的间隔，如果小于等于service_down_time的值，则认为服务是up的，否则是down。

比如假设我们设置的report_interval时间为10秒，正常的话检查最后心跳到当前时间一定小于10秒，不幸的是可能中间丢了2个心跳，那检查的最后心跳距离当前时间可能为20多秒，由于小于我们的service_down_time（假设为60秒)，因此还是认为服务是up的。如果连续丢掉超过6个心跳包，则服务就会返回down了。

问题分析

当Openstack不正常工作时，首先查看下服务状态，比如执行nova service-list命令查看Nova相关的服务状态。

如果服务状态为down，根据Openstack服务的心跳机制和状态监控原理，可能有以下几种故障情形：

1. 数据库访问错误导致心跳更新失败，这种情况看日志就能发现错误日志。

2. Rabbitmq连接失败，nova-compute不能直接访问数据库，更新时是通过RPC调用nova-conductor完成的，如果rabbitmq连接失败，RPC将无法执行，导致心跳发送失败。

3. nova-conductor故障，原因同上，不过这种情况概率很低，除非人为关闭了该服务。

4. 时间不同步。这种情况排查非常困难，因为你在日志中是发现不了任何错误信息的，我们知道数据库操作由nova-conductor组件完成的，而计算心跳间隔是在nova-api服务完成的，假如这两个服务所在的主机时间不同步，将可能导致服务误判为down。对于多API节点部署时尤其容易出现这种情况，所有节点务必保证时间同步，NTP服务必须能够正常工作，否则将影响Openstack服务的心跳机制和状态监控。

nova的这种上报方式其实这种方法效率是非常低的，并且当服务众多时，数据库的压力将会非常大，因此有人提出引入Zookeeper服务发现机制维护Openstack服务状态，参考Services Heartbeat with
ZooKeeper。