为什么KVM计算机点无故重启？

一.故障1：机器hangs

本地一台cloudstack计算节点无故连不上了，cloudstack也坏了，后查看有一台系统虚拟机在这台计算节点上，导致cs挂了。去找到这台机器后，发现这台机器卡住了，重启后卡在starting udev，等好久也不行，即使进入单用户也是一样，重启很多次也是卡在这。最后查到一篇文章，原话：

Remove

quiet

from the kernel command line and you should get enough output to see the cause of the hang.

然后进入系统菜单，在编辑kernel行，quiet静默模式。相当于设置"loglevel=4"(WARNING)。将quiet删除后，等待进入系统。大概一个小时后，进去系统了，能ssh了，问题解决了一个，得查一下机器为什么重启。

二.故障2:kvm计算机点为什么无故重启

通过查看cloudstack-agent的日志。

[root@kvm204 ~]# tail -1000f /var/log/cloudstack/agent/cloudstack-agent.out

看到了这里有报警，并且执行了reboot命令。继续往上面翻，又看到一条信息，如下：



看到很多这样的报错，但最后一次报错是下面这个，可能是它导致了kvm检测脚本执行了reboot命令，然后就出现了本文第一张图片日志里reboot the host。



现在知道了是谁重启了电脑，但kvm为什么会重启这台计算节点呢？带着疑问，我去查看了一下那个脚本，也就是kvmheartbeat.sh脚本。内容如下：

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0 #
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

help() {
printf "Usage: $0
-i nfs server ip
-p nfs server path
-m mount point
-h host
-r write/read hb log
-c cleanup
-t interval between read hb log\n"
exit 1
}
#set -x
NfsSvrIP=
NfsSvrPath=
MountPoint=
HostIP=
interval=
rflag=0
cflag=0

while getopts 'i:p:m:h:t:rc' OPTION
do
case $OPTION in
i)
NfsSvrIP="$OPTARG"
;;
p)
NfsSvrPath="$OPTARG"
;;
m)
MountPoint="$OPTARG"
;;
h)
HostIP="$OPTARG"
;;
r)
rflag=1
;;
t)
interval="$OPTARG"
;;
c)
cflag=1
;;
*)
help
;;
esac
done

if [ -z "$NfsSvrIP" ]
then
exit 1
fi

#delete VMs on this mountpoint
deleteVMs() {
local mountPoint=$1
vmPids=$(ps aux| grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
if [ $? -gt 0 ]
then
return
fi

if [ -z "$vmPids" ]
then
return
fi

for pid in $vmPids
do
kill -9 $pid &> /dev/null
done
}

#checking is there the same nfs server mounted under $MountPoint?
mounts=$(cat /proc/mounts |grep nfs|grep $MountPoint)
if [ $? -gt 0 ]
then
# remount it
mount $NfsSvrIP:$NfsSvrPath $MountPoint -o sync,soft,proto=tcp,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,noac,timeo=133,retrans=10 &> /dev/null
if [ $? -gt 0 ]
then
printf "Failed to remount $NfsSvrIP:$NfsSvrPath under $MountPoint"
exit 1
fi
if [ "$rflag" == "0" ]
then
deleteVMs $MountPoint
fi
fi

hbFolder=$MountPoint/KVMHA/
hbFile=$hbFolder/hb-$HostIP

write_hbLog() {
#write the heart beat log
stat $hbFile &> /dev/null
if [ $? -gt 0 ]
then
# create a new one
mkdir -p $hbFolder &> /dev/null
touch $hbFile &> /dev/null
if [ $? -gt 0 ]
then
printf "Failed to create $hbFile"
return 2
fi
fi

timestamp=$(date +%s)
echo $timestamp > $hbFile
return $?
}

check_hbLog() {
now=$(date +%s)
hb=$(cat $hbFile)
diff=`expr $now - $hb`
if [ $diff -gt $interval ]
then
return 1
fi
return 0
}

if [ "$rflag" == "1" ]
then
check_hbLog
if [ $? == 0 ]
then
echo "=====> ALIVE <====="
else
echo "=====> DEAD <======"
fi
exit 0
elif [ "$cflag" == "1" ]
then
reboot
exit $?
else
write_hbLog
exit $?
fi

上面这个脚本是我们现在的计算节点上的版本。从上面可以看出，有判断reboot命令，也就是知道了为什么会重启。但这很坑爹呀，你检测检测，你还重启。。。。

下面这个脚本是我在github上看到的。而且版本已经更新。距离现在8月。应该是最新的，从下面看出，这个坑爹的reboot命令已经被修改了。

#!/bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0 #
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

help() {
printf "Usage: $0
-i nfs server ip
-p nfs server path
-m mount point
-h host
-r write/read hb log
-c cleanup
-t interval between read hb log\n"
exit 1
}
#set -x
NfsSvrIP=
NfsSvrPath=
MountPoint=
HostIP=
interval=
rflag=0
cflag=0

while getopts 'i:p:m:h:t:rc' OPTION
do
case $OPTION in
i)
NfsSvrIP="$OPTARG"
;;
p)
NfsSvrPath="$OPTARG"
;;
m)
MountPoint="$OPTARG"
;;
h)
HostIP="$OPTARG"
;;
r)
rflag=1
;;
t)
interval="$OPTARG"
;;
c)
cflag=1
;;
*)
help
;;
esac
done

if [ -z "$NfsSvrIP" ]
then
exit 1
fi

#delete VMs on this mountpoint
deleteVMs() {
local mountPoint=$1
vmPids=$(ps aux| grep qemu | grep "$mountPoint" | awk '{print $2}' 2> /dev/null)
if [ $? -gt 0 ]
then
return
fi

if [ -z "$vmPids" ]
then
return
fi

for pid in $vmPids
do
kill -9 $pid &> /dev/null
done
}

#checking is there the same nfs server mounted under $MountPoint?
mounts=$(cat /proc/mounts |grep nfs|grep $MountPoint)
if [ $? -gt 0 ]
then
# remount it
mount $NfsSvrIP:$NfsSvrPath $MountPoint -o sync,soft,proto=tcp,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,noac,timeo=133,retrans=10 &> /dev/null
if [ $? -gt 0 ]
then
printf "Failed to remount $NfsSvrIP:$NfsSvrPath under $MountPoint"
exit 1
fi
if [ "$rflag" == "0" ]
then
deleteVMs $MountPoint
fi
fi

hbFolder=$MountPoint/KVMHA/
hbFile=$hbFolder/hb-$HostIP

write_hbLog() {
#write the heart beat log
stat $hbFile &> /dev/null
if [ $? -gt 0 ]
then
# create a new one
mkdir -p $hbFolder &> /dev/null
touch $hbFile &> /dev/null
if [ $? -gt 0 ]
then
printf "Failed to create $hbFile"
return 2
fi
fi

timestamp=$(date +%s)
echo $timestamp > $hbFile
return $?
}

check_hbLog() {
now=$(date +%s)
hb=$(cat $hbFile)
diff=`expr $now - $hb`
if [ $diff -gt $interval ]
then
return 1
fi
return 0
}

if [ "$rflag" == "1" ]
then
check_hbLog
if [ $? == 0 ]
then
echo "=====> ALIVE <====="
else
echo "=====> DEAD <======"
fi
exit 0
elif [ "$cflag" == "1" ]
then
/usr/bin/logger -t heartbeat "kvmheartbeat.sh rebooted system because it was unable to write the heartbeat to the storage."
sync &
sleep 5
echo b > /proc/sysrq-trigger
exit $?
else
write_hbLog
exit $?
fi