为什么nova计算节点上报的剩余磁盘空间为负数？

<span style="font-family: Tahoma; text-align: -webkit-auto; background-color: rgb(255, 255, 255);">注：本文针对Kilo版本。</span>

在使用openstack时，遇到了计算节点上报的可用磁盘空间为负数的情况，这里通过代码走读来一窥究竟。

在计算节点上运行的nova-compute服务中，由一个周期任务update_available_resource来负责资源统计和上报：

@<strong>periodic_task</strong>.<strong>periodic_task</strong>
def <span style="color:#000066;">update_available_resource</span>(self, context):
"""See driver.get_available_resource()

Periodic process that keeps that the compute host's understanding of
resource availability and usage in sync with the underlying hypervisor.

:param context: security context
"""

这个函数中，调用的是ResourceTracker的接口获取可用资源：

rt = self._get_resource_tracker(nodename)
rt.<strong>update_available_resource</strong>(context)

而ResourceTracker又是实际调用libvirt driver来进行资源统计信息的获取：

def <strong>update_available_resource</strong>(self, context):
"""Override in-memory calculations of compute node resource usage based
on data audited from the hypervisor layer.

Add in resource claims in progress to account for operations that have
declared a need for resources, but not necessarily retrieved them from
the hypervisor layer yet.
"""
LOG.info(_LI("Auditing locally available compute resources for "
"node %(node)s"),
{'node': self.nodename})
resources = <strong>self.driver.get_available_resource</strong>(self.nodename)

这个获取资源统计信息的函数定义在virt\libvirt\driver.py中：

def <strong>get_available_resource</strong>(self, nodename):
"""Retrieve resource information.

This method is called when nova-compute launches, and
as part of a periodic task that records the results in the DB.

:param nodename: will be put in PCI device
:returns: dictionary containing resource info
"""

disk_info_dict = self._get_local_gb_info()
data = {}

# NOTE(dprince): calling capabilities before getVersion works around
# an initialization issue with some versions of Libvirt (1.0.5.5).
# See: https://bugzilla.redhat.com/show_bug.cgi?id=1000116 # See: https://bugs.launchpad.net/nova/+bug/1215593 
# Temporary convert supported_instances into a string, while keeping
# the RPC version as JSON. Can be changed when RPC broadcast is removed
data["supported_instances"] = jsonutils.dumps(
self._get_instance_capabilities())

data["vcpus"] = self._get_vcpu_total()
data["memory_mb"] = self._get_memory_mb_total()
data["local_gb"] = disk_info_dict['total']
data["vcpus_used"] = self._get_vcpu_used()
data["memory_mb_used"] = self._get_memory_mb_used()
data["local_gb_used"] = disk_info_dict['used']
data["hypervisor_type"] = self._host.get_driver_type()
data["hypervisor_version"] = self._host.get_version()
data["hypervisor_hostname"] = self._host.get_hostname()
# TODO(berrange): why do we bother converting the
# libvirt capabilities XML into a special JSON format ?
# The data format is different across all the drivers
# so we could just return the raw capabilities XML
# which 'compare_cpu' could use directly
#
# That said, arch_filter.py now seems to rely on
# the libvirt drivers format which suggests this
# data format needs to be standardized across drivers
data["cpu_info"] = jsonutils.dumps(self._get_cpu_info())

disk_free_gb = disk_info_dict['free']
disk_over_committed = self._get_disk_over_committed_size_total()
available_least = disk_free_gb * units.Gi - disk_over_committed
data['disk_available_least'] = available_least / units.Gi

data['pci_passthrough_devices'] = \
self._get_pci_passthrough_devices()

numa_topology = self._get_host_numa_topology()
if numa_topology:
data['numa_topology'] = numa_topology._to_json()
else:
data['numa_topology'] = None

return data

看一下跟磁盘资源相关的部分，首先是调用了libvirt driver的这个静态函数，得到total/free/used三个值，以gigabytes为单位：

@staticmethod
def get_local_gb_info():
"""Get local storage info of the compute node in GB.

:returns: A dict containing:
:total: How big the overall usable filesystem is (in gigabytes)
:free: How much space is free (in gigabytes)
:used: How much space is used (in gigabytes)
"""

if CONF.libvirt.images_type == 'lvm':
info = libvirt_utils.get_volume_group_info(CONF.libvirt.images_volume_group)
else:
info = libvirt_utils.get_fs_info(CONF.instances_path)

for (k, v) in info.iteritems():
info[k] = v / units.Gi  //注意：这里把结果的单位都换算成了GB！

return info

从get_local_gb_info这个函数中可以看到，如果存放instances用的是文件系统而非lvm，则调用下面的函数获取资源数据：

def get_fs_info(path):
"""Get free/used/total space info for a filesystem

:param path: Any dirent on the filesystem
:returns: A dict containing:

:free: How much space is free (in bytes)
:used: How much space is used (in bytes)
:total: How big the filesystem is (in bytes)
"""
hddinfo = os.statvfs(path)
total = hddinfo.f_frsize * hddinfo.f_blocks
free = hddinfo.f_frsize * hddinfo.f_bavail
used = hddinfo.f_frsize * (hddinfo.f_blocks - hddinfo.f_bfree)
return {'total': total,
'free': free,
'used': used}

get_fs_info这个函数获取到的信息和用df命令看到的结果基本是一样的：

[root@host123 ~]# python Python 2.7.5 (default, Feb 11 2014, 07:46:25) [GCC 4.8.2 20140120 (Red Hat 4.8.2-13)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> hddinfo = os.statvfs("/var/lib/nova") >>> total = hddinfo.f_frsize * hddinfo.f_blocks >>> free = hddinfo.f_frsize * hddinfo.f_bavail >>> used = hddinfo.f_frsize * (hddinfo.f_blocks - hddinfo.f_bfree) >>> >>> print total/1024/1024/1024 254 >>> print free/1024/1024/1024 194 >>> print used/1024/1024/1024 46 [root@host123 ~]# df -h Filesystem                   Size  Used Avail Use% Mounted on /dev/mapper/vg_sys-lv_root    20G  3.6G   16G  20% / devtmpfs                      11G     0   11G   0% /dev tmpfs                         12G     0   12G   0% /dev/shm tmpfs                         12G   83M   12G   1% /run tmpfs                         12G     0   12G   0% /sys/fs/cgroup /dev/sda1                    380M   96M  260M  27% /boot /dev/mapper/vg_nova-lv_nova  255G   47G  195G  20% /var/lib/nova

update_status直接利用了获取到的total和used数据项，但是注意free却没有直接使用，而是计算成了disk_available_least：

<strong>disk_free_gb </strong>= disk_info_dict['free']
<strong>disk_over_committed </strong>= self.<strong>_get_disk_over_committed_size_total</strong>()
<strong>available_least </strong>= <strong>disk_free_gb </strong>* units.Gi - <strong>disk_over_committed</strong>
data['<strong>disk_available_least</strong>'] = available_least / units.Gi

可以看到，它从操作系统给的disk_free_gb 里面又减去了disk_over_committed的值。

我们来看看get_disk_over_committed_size_total是怎么获取的，这个函数也是libvirt
driver的成员：

    def _get_disk_over_committed_size_total(self):
        """Return total over committed disk size for all instances."""
        # Disk size that all instance uses : virtual_size - disk_size
        disk_over_committed_size = 0
        for dom in self._host.list_instance_domains():
            try:
                xml = dom.XMLDesc(0)
                disk_infos = jsonutils.loads(
                        self._get_instance_disk_info(dom.name(), xml))
                for info in disk_infos:
                    disk_over_committed_size += int(
                        info['over_committed_disk_size'])
            except ……（此处略过）
            # NOTE(gtt116): give other tasks a chance.
            greenthread.sleep(0)
        return disk_over_committed_size

它是逐个获取每个instance的over_committed_disk_size，然后把它们累加起来。
意思是有的instance已经在超额使用磁盘了，那么超额在哪里呢？

对于每一个instance，是通过下面的函数获取over_committed_disk_size的：

def _get_instance_disk_info(self, instance_name, xml,
block_device_info=None):
block_device_mapping = driver.block_device_info_get_mapping(
block_device_info)

volume_devices = set()
for vol in block_device_mapping:
disk_dev = vol['mount_device'].rpartition("/")[2]
volume_devices.add(disk_dev)

disk_info = []
doc = etree.fromstring(xml)
disk_nodes = doc.findall('.//devices/disk')
path_nodes = doc.findall('.//devices/disk/source')
driver_nodes = doc.findall('.//devices/disk/driver')
target_nodes = doc.findall('.//devices/disk/target')

for cnt, path_node in enumerate(path_nodes):
disk_type = disk_nodes[cnt].get('type')
path = path_node.get('file') or path_node.get('dev')
target = target_nodes[cnt].attrib['dev']

if not path:
LOG.debug('skipping disk for %s as it does not have a path',
instance_name)
continue

if disk_type not in ['file', 'block']:
LOG.debug('skipping disk because it looks like a volume', path)
continue

if target in volume_devices:
LOG.debug('skipping disk %(path)s (%(target)s) as it is a '
'volume', {'path': path, 'target': target})
continue

# get the real disk size or
# raise a localized error if image is unavailable
<strong>            if disk_type == 'file':
dk_size = int(os.path.getsize(path))
elif disk_type == 'block':
dk_size = lvm.get_volume_size(path)

disk_type = driver_nodes[cnt].get('type')
if disk_type == "qcow2":
backing_file = libvirt_utils.get_disk_backing_file(path)
virt_size = disk.get_disk_size(path)
over_commit_size = int(virt_size) - dk_size
else:
backing_file = ""
virt_size = dk_size
over_commit_size = 0</strong>

disk_info.append({'type': disk_type,
'path': path,
'virt_disk_size': virt_size,
'backing_file': backing_file,
'disk_size': dk_size,
'over_committed_disk_size': over_commit_size})
return jsonutils.dumps(disk_info)

举个例子，对于qcow2格式的镜像，这个overcommit size等于virt_size减去dk_size：

[root@host123 ~]# ll -h /var/lib/nova/instances/109291c0-0bf0-412c-9e87-6ab01e16bc06/disk

-rw-r--r-- 1 root root 5.0G Feb 25 11:41 /var/lib/nova/instances/109291c0-0bf0-412c-9e87-6ab01e16bc06/disk

镜像文件实际大小dk_size是5.0G。我们再用qemu-img命令查看一下qcow2的详细信息：

[root@host123 ~]# qemu-img info /var/lib/nova/instances/109291c0-0bf0-412c-9e87-6ab01e16bc06/disk

image: /var/lib/nova/instances/109291c0-0bf0-412c-9e87-6ab01e16bc06/disk

file format: qcow2
virtual size: 20G (21474836480 bytes)

disk size: 4.9G

cluster_size: 65536
backing file: /var/lib/nova/instances/_base/afd631de55a9b7026775a4a1ada098a9ae6888c7

Format specific information:

    compat: 0.10

这里的virtual size减去disk size，便是over_commit_size。

可以看到，这里仅仅对qcow2格式的镜像做了overcommit处理，其它文件的over_commit_size等于0。

我们知道，在nova调度服务的DiskFilter里面，用到了disk_allocation_ratio对磁盘资源做了超分，它和这里的overcommit不是一个概念，它是从控制节点角度看到的超额使用，而计算节点看不到，overcommit是计算节点看到了磁盘qcow2压缩格式之后所得到的结果，它最终上报的剩余空间是扣除了假设qcow2镜像文件解压之后的实际结果。所以会遇到实际上报的剩余空间小于肉眼看到的空间大小。

如果管理员部署时指定了计算节点，则不走调度流程，就会把虚拟机硬塞给该计算节点，强行占用了已经归入超额分配计划的空间，则最终可能导致计算节点上报的磁盘资源为负数。并且将来随着虚拟机实际占用的磁盘空间越来越大，最终可能就导致计算节点硬盘空间不足了。