RAC环境下控制文件损坏重建过程

处理过程参考了： https://blogs.oracle.com/Database4CN/entry/%E5%A6%82%E4%BD%95%E9%87%8D%E5%BB%BArac%E7%9A%84%E6%8E%A7%E5%88%B6%E6%96%87%E4%BB%B6
问题现象：

现场有学校提报 登录PL/SQL连接数据库是报错“ORA-12541: TNS：无监听程序 ”；排查日志，发现

Tue Nov 25 14:46:58 2014

Thread 2 advanced to log sequence 18098 (LGWR switch)

Current log# 4 seq# 18098 mem# 0: +DATADG/urpdb/onlinelog/group_4.266.821439273

Tue Nov 25 14:46:58 2014

Archived Log entry 50394 added for thread 2 sequence 18097 ID 0xb0471722 dest 1:

Tue Nov 25 14:54:49 2014
Read from controlfile member '+DATADG/urpdb/controlfile/current.260.821439141' has found a corrupted block (blk# 35, cf seq# 0)

Hex dump of (file 0, block 35) in trace file /u01/app/oracle/diag/rdbms/urpdb/urpdb2/trace/urpdb2_lmon_10377.trc

Corrupt block relative dba: 0x00000023 (file 0, block 35)

Completely zero block found during control file block read

Hex dump of (file 0, block 35) in trace file /u01/app/oracle/diag/rdbms/urpdb/urpdb2/trace/urpdb2_lmon_10377.trc

Corrupt block relative dba: 0x00000023 (file 0, block 35)

Completely zero block found during control file block read

Errors in file /u01/app/oracle/diag/rdbms/urpdb/urpdb2/trace/urpdb2_lmon_10377.trc:

ORA-00202: control file: '+DATADG/urpdb/controlfile/current.260.821439141'
Errors in file /u01/app/oracle/diag/rdbms/urpdb/urpdb2/trace/urpdb2_lmon_10377.trc (incident=15289):

ORA-00227: corrupt block detected in control file: (block 35, # blocks 1)

ORA-00202: control file: '+DATADG/urpdb/controlfile/current.260.821439141'

Incident details in: /u01/app/oracle/diag/rdbms/urpdb/urpdb2/incident/incdir_15289/urpdb2_lmon_10377_i15289.trc

Tue Nov 25 14:54:51 2014

Dumping diagnostic data in directory=[cdmp_20141125145451], requested by (instance=2, osid=10377 (LMON)), summary=[incident=15289].

Errors in file /u01/app/oracle/diag/rdbms/urpdb/urpdb2/trace/urpdb2_lmon_10377.trc:

ORA-00227: corrupt block detected in control file: (block 35, # blocks 1)

ORA-00202: control file: '+DATADG/urpdb/controlfile/current.260.821439141'
LMON (ospid: 10377): terminating the instance due to error 227

System state dump requested by (instance=2, osid=10377 (LMON)), summary=[abnormal instance termination].

System State dumped to trace file /u01/app/oracle/diag/rdbms/urpdb/urpdb2/trace/urpdb2_diag_10357.trc
Instance terminated by LMON, pid = 10377

从日志可以看出，控制文件损坏了，导致数据库实例终止。

解决办法：

因为日志文件、数据文件都是好的，只是唯一的一份控制文件损坏了，只能通过重建控制文件方法解决；

在RAC上重建控制文件与单实例有一些小区别：
在重建控制文件前必须设置cluster_database=false，而且只启动一个实例来执行操作，否则会报错

ORA-01503: CREATE CONTROLFILE failed

ORA-12720: operation requires database is in EXCLUSIVE mode

重建完控制文件后，需要再将cluster_database设为true.

处理步骤：

1.首先生成重建控制文件的脚本：

SQL> alter database backup controlfile to trace;

2.数据库的Alert log中会详细输出这个文件的路径和名字：

Tue Nov 25 22:32:04 2014
alter database backup controlfile to trace
Backup controlfile written to trace file /u01/app/oracle/diag/rdbms/urpdb/urpdb1/trace/urpdb1_ora_6233.trc
Completed: alter database backup controlfile to trace

3.启动一个实例

SQL> startup nomount;

4.修改cluster_database属性
SQL> alter system set cluster_database=false scope=spfile;

SQL> shutdown immediate

SQL> startup nomount;

5.从urpdb1_ora_6233.trc 中获取创建控制文件脚本（noresetlogs的）执行

SQL> CREATE CONTROLFILE REUSE DATABASE "URPDB" NORESETLOGS ARCHIVELOG

MAXLOGFILES 192

MAXLOGMEMBERS 3

MAXDATAFILES 1024

MAXINSTANCES 32

MAXLOGHISTORY 18688

LOGFILE

GROUP 1 '+DATADG/urpdb/onlinelog/group_1.261.821439143' SIZE 50M BLOCKSIZE 512,

GROUP 2 '+DATADG/urpdb/onlinelog/group_2.262.821439143' SIZE 50M BLOCKSIZE 512,

GROUP 3 '+DATADG/urpdb/onlinelog/group_3.265.821439273' SIZE 50M BLOCKSIZE 512,

GROUP 4 '+DATADG/urpdb/onlinelog/group_4.266.821439273' SIZE 50M BLOCKSIZE 512

DATAFILE

'+DATADG/urpdb/datafile/system.256.821439061',

'+DATADG/urpdb/datafile/sysaux.257.821439061',

'+DATADG/urpdb/datafile/undotbs1.258.821439061',

'+DATADG/urpdb/datafile/users.259.821439061',

'+DATADG/urpdb/datafile/undotbs2.264.821439205',

'+DATADG/urpdb/datafile/tsp_imp_d.dbf',

'+DATADG/urpdb/datafile/tsp_dpstar.dbf',

'+DATADG/urpdb/datafile/ts_portal_d.dbf',

'+DATADG/urpdb/datafile/ts_w5_d.dbf',

'+DATADG/urpdb/datafile/ts_wap_d.dbf',

'+DATADG/urpdb/datafile/ts_oa_d.dbf',

'+DATADG/urpdb/datafile/tsp_sastar.dbf',

'+DATADG/urpdb/datafile/tsp_portal.dbf'

CHARACTER SET AL32UTF8

;

6.确认redo log，thread#是正确的

SQL> select * from v$log;

7.恢复数据库、启动数据库、恢复临时文件（这些步骤，在urpdb1_ora_6233.trc 写的很清楚）
SQL> recover database;

Media recovery complete.

SQL> alter system archive log all;

SQL> alter database open;

SQL> ALTER TABLESPACE TEMP ADD TEMPFILE '+DATADG/urpdb/tempfile/temp.263.821439145' REUSE;

SQL> ALTER TABLESPACE TSP_IMP_TEMP ADD TEMPFILE '+DATADG/urpdb/tempfile/tsp_imp_temp.dbf' REUSE;

SQL> ALTER TABLESPACE TSP_DPSTAR_TEMP ADD TEMPFILE '+DATADG/urpdb/tempfile/tsp_dpstar_temp.dbf' REUSE;

SQL> ALTER TABLESPACE TS_PORTAL_TEMP ADD TEMPFILE '+DATADG/urpdb/datafile/ts_portal_temp.dbf' REUSE;

SQL> ALTER TABLESPACE TS_W5_TEMP ADD TEMPFILE '+DATADG/urpdb/datafile/ts_w5_temp.dbf' REUSE;

SQL> ALTER TABLESPACE TS_WAP_TEMP ADD TEMPFILE '+DATADG/urpdb/datafile/ts_wap_temp.dbf' REUSE;

8.还原cluster_database属性

SQL> alter system set cluster_database=true scope=spfile;

SQL> shutdown immediate

9.重启集群服务
./crsctl start cluser 问题解决。

10.登录另外一台服务器查看数据库状态，如有问题进行相应处理。

总结：

1.开启数据库控制文件自动备份很重要：
本数据库由于没有打开控制文件自动备份功能，只能采用重建方法。如果开启了，采用该控制文件，进行recover database using backup controlfile会更简单一些（有待测试验证）