12c rac 实例无法启动之磁盘组空间耗尽
2017-12-31 18:04
459 查看
一、环境描述
oracle 12c rac
二、处理过程
最近也是有客户遇到类似的问题,通过操作系统命令删除了部分归档,然后备份脚本备份失败,发现是因为这个原因引起的,优化一下脚本,添加进去crosscheck 命令,以后就不会出现类似问题了。
oracle 12c rac
二、处理过程
1、实例启动失败 SYS@orcl1> startup ORACLE instance started. Total System Global Area 1241513984 bytes Fixed Size 2923872 bytes Variable Size 620757664 bytes Database Buffers 603979776 bytes Redo Buffers 13852672 bytes Database mounted. ORA-03113: end-of-file on communication channel Process ID: 8482 Session ID: 1 Serial number: 48907
2、查看日志 部分省略............... Starting background process ARC2 Starting background process ARC3 ARC2 started with pid=44, OS id=8737 Unable to create archive log file '+DATADG' Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc: ORA-19504: failed to create file "+DATADG" ORA-17502: ksfdcre:4 Failed to create file +DATADG ORA-15041: diskgroup "DATADG" space exhausted ARCH: Error 19504 Creating archive log file to '+DATADG' Errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_ora_8659.trc: ORA-16038: log 2 sequence# 108 cannot be archived ORA-19504: failed to create file "" ORA-00312: online log 2 thread 1: '+DATADG/ORCL/ONLINELOG/group_2.263.954539539' ARC3 started with pid=45, OS id=8739 System state dump requested by (instance=1, osid=8659), summary=[abnormal instance termination]. System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_8552_20171231165944.trc USER (ospid: 8659): terminating the instance due to error 16038 Dumping diagnostic data in directory=[cdmp_20171231165944], requested by (instance=1, osid=8659), summary=[abnormal insta nce termination]. Instance terminated by USER, pid = 8659
3、确认磁盘组空间大小 <rac02:+ASM2:/home/grid>$asmcmd -p ASMCMD [+] > lsdg State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name MOUNTED EXTERN N 512 4096 1048576 10240 5763 0 5763 0 Y CRSDG/ MOUNTED EXTERN N 512 4096 1048576 6144 13 0 13 0 N DATADG/ ASMCMD [+] >
4、处理思路 1、使用rm命令清理部分归档日志。 2、启动数据库 3、使用rman crosscheck archivelog all; 4、扩磁盘组
5、注意 需要使用rman进行crosscheck ,然后再删除归档日志在控制文件中的记录。 RMAN> crosscheck archivelog all; using target database control file instead of recovery catalog allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISK validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238 validation failed for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924 部分省略.................................. validation succeeded for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_109.288.964199163 RECID=132 STAMP=964199165 validation succeeded for archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_31/thread_2_seq_110.325.964199595 RECID=135 STAMP=964199601 Crosschecked 124 objects RMAN> RMAN> delete expired archivelog all; released channel: ORA_DISK_1 allocated channel: ORA_DISK_1 channel ORA_DISK_1: SID=66 instance=orcl2 device type=DISK List of Archived Log Copies for database with db_unique_name ORCL ===================================================================== Key Thrd Seq S Low Time ------- ---- ------- - -------------------- 11 1 50 X 07-DEC-2017 15:24:55 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 13 1 51 X 07-DEC-2017 15:34:57 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 14 1 52 X 07-DEC-2017 02:55:53 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 19 1 53 X 07-DEC-2017 15:56:20 部分省略 ........................................... 51 2 66 X 08-DEC-2017 07:41:03 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.962225883 53 2 67 X 08-DEC-2017 20:57:57 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.962225885 57 2 68 X 09-DEC-2017 10:16:05 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.962273785 59 2 69 X 09-DEC-2017 10:16:23 Name: +DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133 Do you really want to delete the above objects (enter YES or NO)? yes deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_50.280.962120097 RECID=11 STAMP=962120097 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_51.282.962121355 RECID=13 STAMP=962074554 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_52.283.962121355 RECID=14 STAMP=962074555 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_1_seq_53.288.962121385 RECID=19 STAMP=962121385 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_54.289.962184681 RECID=20 STAMP=962184682 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_55.291.962185743 RECID=22 STAMP=962185742 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_56.292.962188105 RECID=23 STAMP=962188106 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_57.295.962188239 RECID=26 STAMP=962188238 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_58.297.962188923 RECID=28 STAMP=962188924 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_59.298.962206427 RECID=29 STAMP=962206427 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_60.300.962208543 RECID=31 STAMP=962208544 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_61.302.962223761 RECID=48 STAMP=962177843 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_62.305.962223817 RECID=36 STAMP=962223817 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_63.306.962223899 RECID=37 STAMP=962223899 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_64.309.962224067 RECID=40 STAMP=962224066 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_65.310.962224095 RECID=41 STAMP=962224094 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_66.311.962224101 RECID=42 STAMP=962224100 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_67.314.962224111 RECID=45 STAMP=962224111 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_68.316.962224463 RECID=47 STAMP=962224462 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_69.317.962224863 RECID=49 STAMP=962224862 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_1_seq_70.320.962225885 RECID=52 STAMP=962225884 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_71.322.962273773 RECID=54 STAMP=962273772 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_72.323.962273775 RECID=55 STAMP=962273775 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_73.324.962273779 RECID=56 STAMP=962273783 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_1_seq_74.326.962273785 RECID=58 STAMP=962273785 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_47.281.962120099 RECID=12 STAMP=962073300 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_48.284.962121379 RECID=15 STAMP=962121380 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_49.285.962121381 RECID=16 STAMP=962121380 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_07/thread_2_seq_50.286.962121383 RECID=17 STAMP=962121382 部分省略 .................... deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_66.319.962225883 RECID=51 STAMP=962225883 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_08/thread_2_seq_67.321.962225885 RECID=53 STAMP=962225884 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_68.325.962273785 RECID=57 STAMP=962273784 deleted archived log archived log file name=+DATADG/ORCL/ARCHIVELOG/2017_12_09/thread_2_seq_69.327.962274133 RECID=59 STAMP=962274134 Deleted 48 EXPIRED objects
最近也是有客户遇到类似的问题,通过操作系统命令删除了部分归档,然后备份脚本备份失败,发现是因为这个原因引起的,优化一下脚本,添加进去crosscheck 命令,以后就不会出现类似问题了。
相关文章推荐
- RAC增加ASM磁盘后某个实例无法启动
- RAC由于归档表空间满而无法启动实例的解决
- rac_第二个节点重启后无法启动实例:磁盘组dismount问题
- rac 11g_第二个节点重启后无法启动实例:磁盘组dismount问题
- linux环境下根目录磁盘空间满导致数据库无法启动的问题的解决
- ORACLE 12C 三节点 RAC 无法找到候选磁盘
- PRKP-1001,CRS-0215 rac 集群无法启动数据库和实例
- ubuntu下磁盘空间不足导致mysql无法启动的解决方法
- 磁盘空间已满导致rabbitmq无法启动
- Linux磁盘空间爆满,MySQL无法启动
- EHR 1.172无法启动数据库,提示磁盘空间不足 设置非归模式
- RAC安装完成后实例无法自动启动
- 二进制安装MySQL5.5.39,磁盘空间不足导致MySQL无法启动
- oracle 12C RAC启动实例时报ORA-00206: error in writing (block 1, # blocks 1) of control file错误处理
- rac 实例1 无法启动 ORA-17503:can open spfile,ORA-27123:unable to attach to shared memory segment
- Linux磁盘空间爆满,MySQL无法启动
- Linux磁盘空间爆满,MySQL无法启动
- 一次简单的AIX磁盘未正常激活导致RAC无法启动的故障排除
- windows2008R2 RAC环境中节点实例无法同时启动
- 12c ASM实例的参数有问题导致无法启动