因重建oracle用户和组引起的ORA-27121错误

今天在做恢复测试的时候遇到ORA-27121的错误，具体的表现为用sqlplus登录oracle的时候提示Connected to an idle instance，但是通过ps -ef仍然能够看到oracle的进程存在。

一、故障现象

oracle@bfdx-dsg:[/oracle]rman target / nocatalog

Recovery Manager: Release 10.2.0.2.0 - Production on Fri May 16 09:52:17 2008

Copyright (c) 1982, 2005, Oracle. All rights reserved.

connected to target database (not started)

RMAN> run {
2> allocate channel t1 type 'sbt_tape';
3> send 'NB_ORA_SERV=backsrv,NB_ORA_CLIENT=sx_kf_svc';
4> allocate channel t2 type 'sbt_tape';
5> send 'NB_ORA_SERV=backsrv,NB_ORA_CLIENT=sx_kf_svc';
6> restore database;
7> release channel t1;
8> release channel t2;
9> }

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of allocate command at 05/16/2008 09:52:31
RMAN-06403: could not obtain a fully authorized session
ORA-01034: ORACLE not available
ORA-27121: unable to determine size of shared memory segment
IBM AIX RISC System/6000 Error: 13: Permission denied

RMAN> exit
Recovery Manager complete.

oracle@bfdx-dsg:[/oracle]id
uid=300(oracle) gid=300(dba) groups=1(staff)

oracle@bfdx-dsg:[/oracle]sqlplus /nolog

SQL*Plus: Release 10.2.0.2.0 - Production on Fri May 16 09:53:13 2008
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.

SQL> conn /as sysdba
Connected to an idle instance.

SQL> startup nomount
ORA-27121: unable to determine size of shared memory segment
IBM AIX RISC System/6000 Error: 13: Permission denied

SQL> exit
Disconnected

oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]ps -ef|grep ora
oracle 290896 1 0 May 14 - 0:30 ora_ckpt_sibss
oracle 430180 639092 0 09:53:27 pts/1 0:00 sqlplus
oracle 450618 1 0 May 14 - 0:02 ora_lgwr_sibss
oracle 503824 671794 0 09:55:05 pts/0 0:00 ps -ef
oracle 512110 1 0 May 14 - 0:03 ora_dbw0_sibss
oracle 520252 1 0 May 14 - 0:04 ora_mman_sibss
oracle 532528 1 0 May 14 - 0:02 ora_psp0_sibss
oracle 561394 1 0 May 14 - 0:28 ora_pmon_sibss
oracle 565438 1 0 May 14 - 0:02 ora_smon_sibss
oracle 610504 430180 0 09:53:27 - 0:00 oraclesibss (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
oracle 622730 1 0 May 14 - 0:00 ora_reco_sibss
oracle 639092 635114 0 09:53:21 pts/1 0:00 -ksh
oracle 647392 1 0 May 14 - 0:04 ora_mmon_sibss
oracle 651310 671794 0 09:55:05 pts/0 0:00 grep ora
oracle 671794 663674 0 09:52:00 pts/0 0:00 -ksh
oracle 680056 1 0 May 14 - 0:01 ora_mmnl_sibss
oracle 692432 1 0 May 14 - 0:01 ora_cjq0_sibss

oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.2.0 - Production on Fri May 16 09:55:15 2008
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
Connected to an idle instance.

SQL> shutdown immediate
ORA-01034: ORACLE not available
ORA-27121: unable to determine size of shared memory segment
IBM AIX RISC System/6000 Error: 13: Permission denied

SQL> exit
Disconnected

二、查看日志文件，没有发现报错

oracle@bfdx-dsg:[/oracle]echo $ORACLE_BASE
/oracle/app/oracle

oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]tail -f *.log
Setting recovery target incarnation to 2
Wed May 14 15:21:21 2008
Successful mount of redo thread 1, with mount id 1604448301
Wed May 14 15:21:21 2008
Database mounted in Exclusive Mode
Completed: ALTER DATABASE MOUNT
Fri May 16 09:53:33 2008
Starting ORACLE instance (normal)
Fri May 16 09:54:01 2008
Starting ORACLE instance (normal)

三、初步判断是共享内存段出现了问题

oracle@bfdx-dsg:[/oracle/app/oracle/admin/sibss/bdump]ipcs -pmb
IPC status from /dev/mem as of Fri May 16 09:56:15 BEIDT 2008
T ID KEY MODE OWNER GROUP SEGSZ CPID LPID
Shared Memory:
m 0 0x58002034 --rw-rw-rw- root system 134217728 61596 278664
m 1048577 0x52e74b4f --rw-rw-rw- imnadm imnadm 36028 422096 422096
m 1048578 0x9308e451 --rw-rw-rw- imnadm imnadm 97948 422096 422096
m 1048579 0xe4663d62 --rw-rw-rw- imnadm imnadm 96 422096 422096
m 4 0xc76283cc --rw-rw-rw- imnadm imnadm 42268 422096 422096
m 5 0x298ee665 --rw-rw-rw- imnadm imnadm 2844 422096 422096
m 6 0xffffffff --rw-rw---- root system 4096 381128 381128
m 7 0x78000019 --rw-rw-rw- root system 16777216 483570 483570
m 8 0x0d003a64 --rw-rw---- root system 1440 344086 635114
m 8388617 0xc4d142b0 --rw-r----- 205 204 6442467328 618702 638976
m 1048586 0x00005643 --rw-rw-rw- root system 1024 659660 688262
m 1048587 0x00005654 --rw-rw-rw- root system 1024 659660 688262

在做测试的时候，由于测试机的oracle用户和dba组的id和生产库不一致，所以删除后重建了。
m 8388617 0xc4d142b0 --rw-r----- 205 204 6442467328 618702 638976
这个共享段的205和204分别是删除之前的oracle用户和dba组id。

四、解决步骤

--杀掉oracle相关的进程
oracle@bfdx-dsg:[/oracle]ps -ef|grep $ORACLE_SID| /
> grep -v grep|awk '{print $2}'|xargs -i kill -9 {}

oracle@bfdx-dsg:[/oracle]ps -ef|grep ora
oracle 430180 639092 0 09:53:27 pts/1 0:00 sqlplus
oracle 639092 635114 0 09:53:21 pts/1 0:00 -ksh
oracle 671800 663674 0 10:00:38 pts/0 0:00 -ksh
oracle 680058 671800 0 10:01:05 pts/0 0:00 grep ora
oracle 692434 671800 0 10:01:05 pts/0 0:00 ps -ef

--清除数据库占用的内存段
oracle@bfdx-dsg:[/oracle]ipcrm -m 8388617
ipcrm: 0515-021 shmid(8388617): The user does not have the appropriate privileges to perform this operation.

oracle@bfdx-dsg:[/oracle]exit

bfdx-dsg:root:/acct>ipcs -pmb
IPC status from /dev/mem as of Fri May 16 10:03:14 BEIDT 2008
T ID KEY MODE OWNER GROUP SEGSZ CPID LPID
Shared Memory:
m 0 0x58002034 --rw-rw-rw- root system 134217728 61596 278664
m 1048577 0x52e74b4f --rw-rw-rw- imnadm imnadm 36028 422096 422096
m 1048578 0x9308e451 --rw-rw-rw- imnadm imnadm 97948 422096 422096
m 1048579 0xe4663d62 --rw-rw-rw- imnadm imnadm 96 422096 422096
m 4 0xc76283cc --rw-rw-rw- imnadm imnadm 42268 422096 422096
m 5 0x298ee665 --rw-rw-rw- imnadm imnadm 2844 422096 422096
m 6 0xffffffff --rw-rw---- root system 4096 381128 381128
m 7 0x78000019 --rw-rw-rw- root system 16777216 483570 483570
m 8 0x0d003a64 --rw-rw---- root system 1440 344086 635114
m 8388617 0xc4d142b0 --rw-r----- 205 204 6442467328 618702 692432
m 1048586 0x00005643 --rw-rw-rw- root system 1024 659660 688262
m 1048587 0x00005654 --rw-rw-rw- root system 1024 659660 688262

bfdx-dsg:root:/acct>ipcrm -m 8388617
bfdx-dsg:root:/acct>su - oracle
oracle@bfdx-dsg:[/oracle]sqlplus '/as sysdba'

SQL*Plus: Release 10.2.0.2.0 - Production on Fri May 16 10:03:35 2008
Copyright (c) 1982, 2005, Oracle. All Rights Reserved.
Connected to an idle instance.

SQL> startup nomount
ORACLE instance started.

Total System Global Area 6442450944 bytes
Fixed Size 2080904 bytes
Variable Size 2013269880 bytes
Database Buffers 4412407808 bytes
Redo Buffers 14692352 bytes

OK，问题解决。