Linux神奇的负载

FC-san 导致负载超大

ubuntu@git-server160:~$ top
top - 05:04:37 up 3 days, 1:26, 3 users, load average: 6078.26, 6069.95, 6048.50
Tasks: 9425 total, 1 running, 9424 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2%us, 0.5%sy, 0.0%ni, 7.4%id, 91.8%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 33000292k total, 5070652k used, 27929640k free, 145152k buffers
Swap: 70946808k total, 0k used, 70946808k free, 789796k cached

ubuntu@git-server160:~$ sudo iostat -kx
Linux 2.6.32-24-server (git-server160) 11/25/2013 _x86_64_ (16 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0.68 0.00 0.05 53.86 0.00 45.40
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
sda 0.01 1.33 0.02 0.98 0.51 8.05 17.11 0.00 1.31 0.76 0.08
dm-0 0.00 0.00 0.03 2.01 0.50 8.05 8.36 0.01 3.87 0.37 0.08
dm-1 0.00 0.00 0.00 0.00 0.00 0.00 8.00 0.00 4.07 1.33 0.00
dm-3 0.00 0.00 0.01 0.00 0.05 0.00 8.13 0.72 2095.46 62175.39 71.60
dm-4 0.00 0.00 0.01 0.00 0.05 0.00 8.18 0.72 2100.34 62497.52 71.59
dm-5 0.00 0.00 0.01 0.00 0.04 0.00 8.14 0.72 2198.67 65259.81 71.60
dm-6 0.00 0.00 0.01 0.00 0.04 0.00 8.16 0.72 2606.36 77332.14 71.60
dm-7 0.00 0.00 0.01 0.00 0.05 0.00 8.12 0.72 2092.30 62093.41 71.60
dm-8 0.00 0.00 0.01 0.00 0.04 0.00 8.19 0.72 2211.86 65820.98 71.59
sdb 0.00 0.00 0.00 0.00 0.01 0.00 8.81 0.00 0.10 0.10 0.00
sde 0.00 0.00 0.00 0.00 0.01 0.00 8.00 0.00 537.51 531.34 0.09
sdp 0.00 0.00 0.14 0.00 0.56 0.00 8.00 0.08 542.21 542.21 7.57
sdq 0.00 0.00 0.14 0.00 0.56 0.00 8.00 0.08 546.27 546.18 7.62
sdr 0.00 0.00 0.14 0.00 0.56 0.00 8.00 0.07 525.65 525.65 7.34
sds 0.00 0.00 0.14 0.00 0.56 0.00 8.00 0.07 526.57 526.57 7.35
sdt 0.00 0.00 0.14 0.00 0.56 0.00 8.00 0.07 526.77 526.67 7.35
sdu 0.00 0.00 0.14 0.00 0.56 0.00 8.00 0.07 525.91 525.91 7.34
dm-2 0.00 0.00 0.00 0.00 0.00 0.00 9.66 0.00 0.00 0.00 0.00

ubuntu@git-server160:~$ vmstat 3
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
0 6089 0 27928428 145224 793448 0 0 0 1 4 4 1 0 45 54
0 6089 0 27930776 145224 793460 0 0 0 0 73 39 0 0 22 78
0 6090 0 27928140 145224 793460 0 0 0 8 162 118 3 0 2 95