山东省枣庄市台儿庄区云平台运维故障处理一例
2018-01-12 08:14
363 查看
故障现象:
外网访问页面显示不正常,Template Error!
接到反馈后,远程到服务器,执行service nginx restart ,提示磁盘满,无法继续。
查看磁盘占用情况,磁盘/目录占用100%,磁盘空间满。
执行
查找100M以上的文件,发现主要大的是NGINX的日志:
一天的NGINX日志大到30多个G,太离谱了!!!
切割文件的最后N行到新的文件中
这样做,主要是太大的文件我用VI无法打开。
查看最新的日志:
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
对比正常的其它服务器: http://10.10.14.199/dsideal_yy/golbal/getValueByKey?key=common.rongyun.suffix {"common.rongyun.suffix":"_199"}
继续查看此服务器: http://10.24.0.7:7777/dsideal_yy/golbal/getValueByKey?key=common.rongyun.suffix {"common.rongyun.suffix":"_zztez"}
也不是404啊!!!!
访问一下http://10.24.0.7,居然可以访问的到!!!!!!还显示welcome to Nginx!!!!我们是7777的端口啊!!!
查看一下80端口是什么东东在使用:
yum install lsof
lsof -i tcp:80
[root@bogon conf]# lsof -i tcp:80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 243175 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
nginx 243176 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
nginx 243178 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
nginx 243179 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
nginx 243180 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
根据pid查文件位置:
ps 243175
[root@bogon conf]# ps 243175
PID TTY STAT TIME COMMAND
243175 ? Ss 0:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -c /usr/local/openresty/nginx/conf/nginx.conf
到这里很显示了,还是这个配置文件有问题,里面配置了两个端口:7777和80
这个东东明显就是垃圾,注释掉,并手工删除超大的日志文件,世界清静了~
回头想想,这是两个问题:
1、即然配置了7777,为什么还保留了80,这是明确的错误。
2、就算是同时配置了80和7777,那个大量的404错误从哪里来??
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
很明显,是10.24.0.6这台机器上来的,这是一台windows主机,是处理程序发过来的请求,它在干什么?为什么不去访问7777的端口??
补充:
1、在访问时,发现访问不了,直接while true了,这里应该是停止或者休息一会。
2、处理程序在访问全局变量时居然没有加上端口!!!!!造成继续访问80,这就是几个错误集中在一起,导致了问题出现,血的教训!!!
外网访问页面显示不正常,Template Error!
接到反馈后,远程到服务器,执行service nginx restart ,提示磁盘满,无法继续。
查看磁盘占用情况,磁盘/目录占用100%,磁盘空间满。
执行
find / -size +100M -exec ls -lh {} \;
查找100M以上的文件,发现主要大的是NGINX的日志:
一天的NGINX日志大到30多个G,太离谱了!!!
切割文件的最后N行到新的文件中
tail -n 1000 access.log >> 111.log
这样做,主要是太大的文件我用VI无法打开。
查看最新的日志:
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
对比正常的其它服务器: http://10.10.14.199/dsideal_yy/golbal/getValueByKey?key=common.rongyun.suffix {"common.rongyun.suffix":"_199"}
继续查看此服务器: http://10.24.0.7:7777/dsideal_yy/golbal/getValueByKey?key=common.rongyun.suffix {"common.rongyun.suffix":"_zztez"}
也不是404啊!!!!
访问一下http://10.24.0.7,居然可以访问的到!!!!!!还显示welcome to Nginx!!!!我们是7777的端口啊!!!
查看一下80端口是什么东东在使用:
yum install lsof
lsof -i tcp:80
[root@bogon conf]# lsof -i tcp:80
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
nginx 243175 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
nginx 243176 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
nginx 243178 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
nginx 243179 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
nginx 243180 root 10u IPv4 2358009951 0t0 TCP *:http (LISTEN)
根据pid查文件位置:
ps 243175
[root@bogon conf]# ps 243175
PID TTY STAT TIME COMMAND
243175 ? Ss 0:00 nginx: master process /usr/local/openresty/nginx/sbin/nginx -c /usr/local/openresty/nginx/conf/nginx.conf
到这里很显示了,还是这个配置文件有问题,里面配置了两个端口:7777和80
server { listen 80; server_name office.edusoa.com; location ^~ /yjj/ { proxy_pass http://221.194.113.150/; } location ^~ /pingli/ { proxy_pass http://61.134.47.35:9999/; } }
这个东东明显就是垃圾,注释掉,并手工删除超大的日志文件,世界清静了~
回头想想,这是两个问题:
1、即然配置了7777,为什么还保留了80,这是明确的错误。
2、就算是同时配置了80和7777,那个大量的404错误从哪里来??
{"ip":"10.24.0.6","request_method":"GET","request_uri":"/dsideal_yy/golbal/getValueByKey","args_get":"key=common.rongyun.suf
fix","args_post":"-","browser":"Mozilla/5.0 (Windows NT 5.1; rv:5.0) Gecko/20100101 Firefox/5.0","cookie":"-","request_time"
:"12/Jan/2018:07:52:15 +0800","http_status":"404"}
很明显,是10.24.0.6这台机器上来的,这是一台windows主机,是处理程序发过来的请求,它在干什么?为什么不去访问7777的端口??
补充:
1、在访问时,发现访问不了,直接while true了,这里应该是停止或者休息一会。
2、处理程序在访问全局变量时居然没有加上端口!!!!!造成继续访问80,这就是几个错误集中在一起,导致了问题出现,血的教训!!!
相关文章推荐
- 运维人员处理服务器故障的方法总结
- 远程通过监听连接报ORA-01034故障处理一例
- 运维经验分享:故障处理方法总结
- ds4700处理黄灯故障过程一例
- 关于Linux运维常见故障排查和处理的33个技巧汇总
- DNS故障处理一例(转)
- MS SQL事务日志丢失故障处理一例 推荐
- [计算机故障处理]无线网无法使用一例
- MySQL故障处理一例_Another MySQL daemon already running with the same unix socket
- ORA-27102:out of memeory 故障处理一例
- 10g异机恢复后EM无法启动故障处理一例
- webcdn故障处理一例
- 浅淡网络运维的紧急故障处理及对策 推荐
- SaaS平台上的多线程控制与故障处理
- 运维人员处理云服务器故障的方法总结
- ORA-00054 故障处理一例
- 运维人员处理服务器故障的方法总结
- ORACLE监听能正常启动,但实例监听不到故障应急处理一例
- ORA-30009 故障处理一例