Oracle：ORA-09925 and linux Read-only file system error

今天上午有同事反映应用数据库连接不上；于是排查数据库：

[oracle@db ~]$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.4.0 - Production on Thu Dec 12 08:54:39 2013
Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.

ERROR:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 30: Read-only file system
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 30: Read-only file system
Additional information: 9925

Enter user-name:
ERROR:
ORA-01017: invalid username/password; logon denied

Enter user-name:
ERROR:
ORA-01017: invalid username/password; logon denied

SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus

上网查了，说数据库命令权限不对，解决办法如下：

-rwx------ 1 oracle oinstall      3337 Jan 15  2013 onsctl
-rwxr-xr-x 1 oracle oinstall        46 Nov  7  2000 oracg
-rwsr-s--x 1 oracle oinstall 112468376 Jan 15  2013 oracle
-rwxr-x--- 1 oracle oinstall         0 Mar 12  2008 oracleO
-r-sr-s--- 1 root   oinstall     14931 Mar 11  2008 oradism
-rwxr-x--- 1 oracle oinstall         0 Mar 12  2008 oradismO

[oracle@db bin]$ chmod 6755 /u01/app/oracle/product/10.2.0/bin/oracle
chmod: changing permissions of `/u01/app/oracle/product/10.2.0/bin/oracle': Read-only file system
[oracle@db bin]$

改变权限，但提示read-only file system。

再排查系统：

[root@db ~]# mount
/dev/mapper/LVMgroup-root on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)

mount: warning /etc/mtab is not writable (e.g. read-only filesystem).
It's possible that information reported by mount(8) is not
up to date. For actual information about system mount points
check the /proc/mounts file.

[root@db ~]# cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 ro,data=ordered 0 0
/dev /dev tmpfs rw 0 0
/proc /proc proc rw 0 0
/sys /sys sysfs rw 0 0
/proc/bus/usb /proc/bus/usb usbfs rw 0 0
devpts /dev/pts devpts rw 0 0
/dev/sda1 /boot ext3 rw,data=ordered 0 0
tmpfs /dev/shm tmpfs rw 0 0
tmpfs /dev/shm tmpfs rw 0 0
none /proc/sys/fs/binfmt_misc binfmt_misc rw 0 0
sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw 0 0
/etc/auto.misc /misc autofs rw,fd=7,pgrp=3377,timeout=300,minproto=5,maxproto=5,indirect 0 0
-hosts /net autofs rw,fd=13,pgrp=3377,timeout=300,minproto=5,maxproto=5,indirect 0 0

root根目录变成read-only，无法写文件系统，Oracle因为无法写audit文件也无法正常登陆 。

[root@db ~]# dmesg
EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted
EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted
sd 0:0:0:0: timing out command, waited 1080s
sd 0:0:0:0: SCSI error: return code = 0x06000008
end_request: I/O error, dev sda, sector 74923301
EXT3-fs error (device dm-0) in ext3_reserve_inode_write: Journal has aborted
************
***********
**********

通过dmesg发现EXT3 文件系统存在问题。

网上有资料说可以通过使用fsck命令修复文件系统，命令也执行不了（fsck可能会导致数据块损坏，最好是不做fsck，重启一下服务器）。

[root@db ~]# fsck
fsck 1.39 (29-May-2006)
e2fsck 1.39 (29-May-2006)
/dev/LVMgroup/root: recovering journal
fsck.ext3: Bad magic number in super-block while trying to re-open /dev/LVMgroup/root
e2fsck: io manager magic bad!
[root@db ~]#

通过命令重启服务器也不可行。

[root@db ~]# df -h
-bash: df: command not found
[root@db ~]# reboot
-bash: reboot: command not found

到此，问题已经很明了，Linux系统出问题了，咨询运维，服务器确实出问题。

幸好这是一个临时环境，数据库重新安装即可。

如果是正式环境，麻烦大了。切记，数据库要定期在其他系统上做备份！！