ORA-15064: communication failure with ASM instance
2012-09-05 18:46
387 查看
alert_ASM2.log
Wed Sep 05 03:32:56 2012 IPC Send timeout detected.Sender: ospid 13720 Receiver: inst 1 binc 22497 ospid 1431 IPC Send timeout to 1.1 inc 56 for msg type 65518 from opid 10 Wed Sep 05 03:32:58 2012 Communications reconfiguration: instance_number 1 Wed Sep 05 03:32:58 2012 Trace dumping is performing id=[cdmp_20120905033258] Waiting for clusterware split-brain resolution Wed Sep 05 03:43:00 2012 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_13704.trc (incident=40264): ORA-29740: evicted by member 1, group incarnation 58 Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM2/incident/incdir_40264/+ASM2_lmon_13704_i40264.trc Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lmon_13704.trc: ORA-29740: evicted by member 1, group incarnation 58 LMON (ospid: 13704): terminating the instance due to error 29740 Wed Sep 05 03:43:01 2012 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_lms0_13720.trc (incident=40280): ORA-29740: evicted by member , group incarnation Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM2/incident/incdir_40280/+ASM2_lms0_13720_i40280.trc Wed Sep 05 03:43:01 2012 System state dump is made for local instance System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_13682.trc Wed Sep 05 03:43:01 2012 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_mman_13724.trc (incident=40288): ORA-29740: evicted by member , group incarnation Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM2/incident/incdir_40288/+ASM2_mman_13724_i40288.trc Wed Sep 05 03:43:02 2012 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_smon_13732.trc (incident=40320): ORA-29740: evicted by member , group incarnation Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM2/incident/incdir_40320/+ASM2_smon_13732_i40320.trc Wed Sep 05 03:43:02 2012 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_gmon_13736.trc (incident=40336): ORA-29740: evicted by member , group incarnation Incident details in: /u01/app/oracle/diag/asm/+asm/+ASM2/incident/incdir_40336/+ASM2_gmon_13736_i40336.trc Wed Sep 05 03:43:02 2012 *********************************************************************** Fatal NI connect error 12537, connecting to: (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq))) VERSION INFORMATION: TNS for Solaris: Version 11.1.0.7.0 - Production Oracle Bequeath NT Protocol Adapter for Solaris: Version 11.1.0.7.0 - Production Time: 05-SEP-2012 03:43:02 Tracing not turned on. Tns error struct: ns main err code: 12537 TNS-12537: TNS:connection closed ns secondary err code: 12560 nt main err code: 0 nt secondary err code: 0 nt OS err code: 0 Wed Sep 05 03:43:02 2012 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_13734.trc (incident=40328): ORA-29740: evicted by member , group incarnation Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_13734.trc (incident=40329): ORA-29740: evicted by member , group incarnation Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_13734.trc (incident=40330): ORA-29740: evicted by member , group incarnation ORA-29740: evicted by member , group incarnation ERROR: ORA-29740 thrown in RBAL for group number 1 Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_13734.trc: ORA-29740: evicted by member , group incarnation ORA-29740: evicted by member , group incarnation Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_13734.trc: ORA-29740: evicted by member , group incarnation ORA-29740: evicted by member , group incarnation Errors in file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_rbal_13734.trc (incident=40331): ORA-29740: evicted by member , group incarnation Trace dumping is performing id=[cdmp_20120905034301] Wed Sep 05 03:43:06 2012 ORA-1092 : opitsk aborting process Wed Sep 05 03:43:06 2012 License high water mark = 16 Instance terminated by LMON, pid = 13704 USER (ospid: 16961): terminating the instance Instance terminated by USER, pid = 16961 Wed Sep 05 03:43:12 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Picked latch-free SCN scheme 3 Using LOG_ARCHIVE_DEST_1 parameter default value as /u01/app/oracle/product/11.1.0/asm/dbs/arch Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled Starting up ORACLE RDBMS Version: 11.1.0.7.0. Using parameter settings in server-side pfile /u01/app/oracle/product/11.1.0/asm/dbs/init+ASM2.ora System parameters with non-default values: shared_pool_size = 200M large_pool_size = 12M instance_type = "asm" cluster_interconnects = "192.168.1.4" memory_target = 512M memory_max_target = 1G cluster_database = TRUE instance_number = 2 asm_diskgroups = "DATA_DG1" asm_diskgroups = "DATA_DG2" asm_diskgroups = "DATA_DG3" asm_diskgroups = "BACKUP_DG1" asm_diskgroups = "BACKUP_DG2" diagnostic_dest = "/u01/app/oracle" Cluster communication is configured to use the following interface(s) for this instance 192.168.1.4 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 Wed Sep 05 03:43:13 2012 PMON started with pid=2, OS id=17064 Wed Sep 05 03:43:13 2012 VKTM started with pid=3, OS id=17073 at elevated priority VKTM running at (20)ms precision Wed Sep 05 03:43:13 2012 DIAG started with pid=4, OS id=17078 Wed Sep 05 03:43:13 2012 PING started with pid=5, OS id=17083 Wed Sep 05 03:43:13 2012 PSP0 started with pid=6, OS id=17085 Wed Sep 05 03:43:13 2012 DIA0 started with pid=7, OS id=17088 Wed Sep 05 03:43:13 2012 LMON started with pid=8, OS id=17091 Wed Sep 05 03:43:13 2012 LMD0 started with pid=9, OS id=17093 Wed Sep 05 03:43:13 2012 LMS0 started with pid=10, OS id=17100 at elevated priority Wed Sep 05 03:43:13 2012 MMAN started with pid=11, OS id=17104 Wed Sep 05 03:43:13 2012 DBW0 started with pid=12, OS id=17109 Wed Sep 05 03:43:13 2012 LGWR started with pid=13, OS id=17112 Wed Sep 05 03:43:13 2012 CKPT started with pid=14, OS id=17114 Wed Sep 05 03:43:13 2012 SMON started with pid=15, OS id=17116 Wed Sep 05 03:43:13 2012 RBAL started with pid=16, OS id=17119 Wed Sep 05 03:43:13 2012 GMON started with pid=17, OS id=17121 DISM started, OS id=17123 lmon registered with NM - instance id 2 (internal mem no 1) Reconfiguration started (old inc 0, new inc 62) ASM instance List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 * allocate domain 1, invalid = TRUE * domain 1 valid = 1 according to instance 0 * allocate domain 2, invalid = TRUE * domain 2 valid = 1 according to instance 0 * allocate domain 3, invalid = TRUE * domain 3 valid = 1 according to instance 0 * allocate domain 4, invalid = TRUE * domain 4 valid = 1 according to instance 0 * allocate domain 5, invalid = TRUE * domain 5 valid = 1 according to instance 0 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted LMS 0: 0 GCS shadows traversed, 0 replayed Submitted all GCS remote-cache requests Post SMON to start 1st pass IR Fix write in gcs resources Reconfiguration complete Wed Sep 05 03:43:14 2012 LCK0 started with pid=18, OS id=17151 ORACLE_BASE from environment = /u01/app/oracle Wed Sep 05 03:43:15 2012 SQL> ALTER DISKGROUP ALL MOUNT NOTE: cache registered group BACKUP_DG1 number=1 incarn=0xbcfb9c9b NOTE: cache began mount (not first) of group BACKUP_DG1 number=1 incarn=0xbcfb9c9b NOTE: cache registered group BACKUP_DG2 number=2 incarn=0xbd0b9c9c NOTE: cache began mount (not first) of group BACKUP_DG2 number=2 incarn=0xbd0b9c9c NOTE: cache registered group DATA_DG1 number=3 incarn=0xbd0b9c9d NOTE: cache began mount (not first) of group DATA_DG1 number=3 incarn=0xbd0b9c9d NOTE: cache registered group DATA_DG2 number=4 incarn=0xbd0b9c9e NOTE: cache began mount (not first) of group DATA_DG2 number=4 incarn=0xbd0b9c9e NOTE: cache registered group DATA_DG3 number=5 incarn=0xbd0b9c9f NOTE: cache began mount (not first) of group DATA_DG3 number=5 incarn=0xbd0b9c9f WARNING::ASMLIB library not found. See trace file for details. NOTE: Assigning number (1,3) to disk (/dev/rdsk/c3t6006016029012200ACC05F9D295CDE11d0s6) NOTE: Assigning number (1,2) to disk (/dev/rdsk/c3t6006016029012200ABC05F9D295CDE11d0s6) NOTE: Assigning number (1,1) to disk (/dev/rdsk/c3t6006016029012200AAC05F9D295CDE11d0s6) NOTE: Assigning number (1,7) to disk (/dev/rdsk/c3t600601602901220061CD9F90295CDE11d0s6) NOTE: Assigning number (1,6) to disk (/dev/rdsk/c3t60060160290122005C4A5D75295CDE11d0s6) NOTE: Assigning number (1,5) to disk (/dev/rdsk/c3t60060160290122005B4A5D75295CDE11d0s6) NOTE: Assigning number (1,0) to disk (/dev/rdsk/c3t60060160290122005A4A5D75295CDE11d0s6) NOTE: Assigning number (1,4) to disk (/dev/rdsk/c3t60060160290122004AD4AF6E295CDE11d0s6) NOTE: Assigning number (2,5) to disk (/dev/rdsk/c3t60060160290122004DFED3A3295CDE11d0s6) NOTE: Assigning number (2,13) to disk (/dev/rdsk/c3t6006016029012200ED34DD96295CDE11d0s6) NOTE: Assigning number (2,12) to disk (/dev/rdsk/c3t6006016029012200EC34DD96295CDE11d0s6) NOTE: Assigning number (2,2) to disk (/dev/rdsk/c3t6006016029012200D2E98A82295CDE11d0s6) NOTE: Assigning number (2,1) to disk (/dev/rdsk/c3t6006016029012200D1E98A82295CDE11d0s6) NOTE: Assigning number (2,0) to disk (/dev/rdsk/c3t6006016029012200D0E98A82295CDE11d0s6) NOTE: Assigning number (2,11) to disk (/dev/rdsk/c3t6006016029012200EAE8F27B295CDE11d0s6) NOTE: Assigning number (2,7) to disk (/dev/rdsk/c3t6006016029012200E9E8F27B295CDE11d0s6) NOTE: Assigning number (2,6) to disk (/dev/rdsk/c3t6006016029012200E8E8F27B295CDE11d0s6) NOTE: Assigning number (2,4) to disk (/dev/rdsk/c3t600601602901220049D4AF6E295CDE11d0s6) NOTE: Assigning number (2,3) to disk (/dev/rdsk/c3t600601602901220048D4AF6E295CDE11d0s6) NOTE: Assigning number (2,8) to disk (/dev/rdsk/c3t600601602901220056E36868295CDE11d0s6) NOTE: Assigning number (2,9) to disk (/dev/rdsk/c3t600601602901220055E36868295CDE11d0s6) NOTE: Assigning number (2,10) to disk (/dev/rdsk/c3t600601602901220054E36868295CDE11d0s6) NOTE: Assigning number (3,3) to disk (/dev/rdsk/c3t6006016029012200AC30026A5A5BDE11d0s6) NOTE: Assigning number (3,2) to disk (/dev/rdsk/c3t6006016029012200AB30026A5A5BDE11d0s6) NOTE: Assigning number (3,1) to disk (/dev/rdsk/c3t6006016029012200AA30026A5A5BDE11d0s6) NOTE: Assigning number (3,11) to disk (/dev/rdsk/c3t6006016029012200EE774E625A5BDE11d0s6) NOTE: Assigning number (3,12) to disk (/dev/rdsk/c3t6006016029012200ED774E625A5BDE11d0s6) NOTE: Assigning number (3,15) to disk (/dev/rdsk/c3t6006016029012200EC774E625A5BDE11d0s6) NOTE: Assigning number (3,8) to disk (/dev/rdsk/c3t60060160290122008B73502D5A5BDE11d0s6) NOTE: Assigning number (3,0) to disk (/dev/rdsk/c3t60060160290122008A73502D5A5BDE11d0s6) NOTE: Assigning number (3,14) to disk (/dev/rdsk/c3t60060160290122002D9576255A5BDE11d0s6) NOTE: Assigning number (3,9) to disk (/dev/rdsk/c3t60060160290122002B9576255A5BDE11d0s6) NOTE: Assigning number (3,5) to disk (/dev/rdsk/c3t600601602901220038040AF9595BDE11d0s6) NOTE: Assigning number (3,7) to disk (/dev/rdsk/c3t600601602901220037040AF9595BDE11d0s6) NOTE: Assigning number (3,6) to disk (/dev/rdsk/c3t600601602901220036040AF9595BDE11d0s6) NOTE: Assigning number (3,13) to disk (/dev/rdsk/c3t60060160290122002DD902F1595BDE11d0s6) NOTE: Assigning number (3,10) to disk (/dev/rdsk/c3t60060160290122002BD902F1595BDE11d0s6) NOTE: Assigning number (3,4) to disk (/dev/rdsk/c3t60060160290122002AD902F1595BDE11d0s6) NOTE: Assigning number (4,2) to disk (/dev/rdsk/c3t60060160290122008973502D5A5BDE11d0s6) NOTE: Assigning number (4,3) to disk (/dev/rdsk/c3t60060160290122008873502D5A5BDE11d0s6) NOTE: Assigning number (4,0) to disk (/dev/rdsk/c3t60060160290122002C9576255A5BDE11d0s6) NOTE: Assigning number (4,1) to disk (/dev/rdsk/c3t60060160290122002CD902F1595BDE11d0s6) NOTE: Assigning number (5,6) to disk (/dev/rdsk/c3t60060160290122004EFED3A3295CDE11d0s6) NOTE: Assigning number (5,0) to disk (/dev/rdsk/c3t60060160290122004CFED3A3295CDE11d0s6) NOTE: Assigning number (5,7) to disk (/dev/rdsk/c3t6006016029012200EE34DD96295CDE11d0s6) NOTE: Assigning number (5,2) to disk (/dev/rdsk/c3t600601602901220062CD9F90295CDE11d0s6) NOTE: Assigning number (5,1) to disk (/dev/rdsk/c3t600601602901220060CD9F90295CDE11d0s6) NOTE: Assigning number (5,3) to disk (/dev/rdsk/c3t6006016029012200F6F4EF89295CDE11d0s6) NOTE: Assigning number (5,4) to disk (/dev/rdsk/c3t6006016029012200F5F4EF89295CDE11d0s6) NOTE: Assigning number (5,5) to disk (/dev/rdsk/c3t6006016029012200F4F4EF89295CDE11d0s6) kfdp_query(): 6 kfdp_queryBg(): 6 NOTE: cache opening disk 0 of grp 1: BACKUP_DG1_0000 path:/dev/rdsk/c3t60060160290122005A4A5D75295CDE11d0s6 NOTE: F1X0 found on disk 0 fcn 0.258 NOTE: cache opening disk 1 of grp 1: BACKUP_DG1_0001 path:/dev/rdsk/c3t6006016029012200AAC05F9D295CDE11d0s6 NOTE: cache opening disk 2 of grp 1: BACKUP_DG1_0002 path:/dev/rdsk/c3t6006016029012200ABC05F9D295CDE11d0s6 NOTE: cache opening disk 3 of grp 1: BACKUP_DG1_0003 path:/dev/rdsk/c3t6006016029012200ACC05F9D295CDE11d0s6 NOTE: cache opening disk 4 of grp 1: BACKUP_DG1_0004 path:/dev/rdsk/c3t60060160290122004AD4AF6E295CDE11d0s6 NOTE: cache opening disk 5 of grp 1: BACKUP_DG1_0005 path:/dev/rdsk/c3t60060160290122005B4A5D75295CDE11d0s6 NOTE: cache opening disk 6 of grp 1: BACKUP_DG1_0006 path:/dev/rdsk/c3t60060160290122005C4A5D75295CDE11d0s6 NOTE: cache opening disk 7 of grp 1: BACKUP_DG1_0007 path:/dev/rdsk/c3t600601602901220061CD9F90295CDE11d0s6 NOTE: cache mounting (not first) group 1/0xBCFB9C9B (BACKUP_DG1) kjbdomatt send to node 0 NOTE: attached to recovery domain 1 NOTE: LGWR attempting to mount thread 1 for diskgroup 1 NOTE: LGWR mounted thread 1 for disk group 1 NOTE: opening chunk 1 at fcn 0.703535333 ABA NOTE: seq=534 blk=2990 NOTE: cache mounting group 1/0xBCFB9C9B (BACKUP_DG1) succeeded NOTE: cache ending mount (success) of group BACKUP_DG1 number=1 incarn=0xbcfb9c9b kfdp_query(): 7 kfdp_queryBg(): 7 NOTE: cache opening disk 0 of grp 2: BACKUP_DG2_0000 path:/dev/rdsk/c3t6006016029012200D0E98A82295CDE11d0s6 NOTE: F1X0 found on disk 0 fcn 0.169 NOTE: cache opening disk 1 of grp 2: BACKUP_DG2_0001 path:/dev/rdsk/c3t6006016029012200D1E98A82295CDE11d0s6 NOTE: cache opening disk 2 of grp 2: BACKUP_DG2_0002 path:/dev/rdsk/c3t6006016029012200D2E98A82295CDE11d0s6 NOTE: cache opening disk 3 of grp 2: BACKUP_DG2_0003 path:/dev/rdsk/c3t600601602901220048D4AF6E295CDE11d0s6 NOTE: cache opening disk 4 of grp 2: BACKUP_DG2_0004 path:/dev/rdsk/c3t600601602901220049D4AF6E295CDE11d0s6 NOTE: cache opening disk 5 of grp 2: BACKUP_DG2_0005 path:/dev/rdsk/c3t60060160290122004DFED3A3295CDE11d0s6 NOTE: cache opening disk 6 of grp 2: BACKUP_DG2_0006 path:/dev/rdsk/c3t6006016029012200E8E8F27B295CDE11d0s6 NOTE: cache opening disk 7 of grp 2: BACKUP_DG2_0007 path:/dev/rdsk/c3t6006016029012200E9E8F27B295CDE11d0s6 NOTE: cache opening disk 8 of grp 2: BACKUP_DG2_0008 path:/dev/rdsk/c3t600601602901220056E36868295CDE11d0s6 NOTE: cache opening disk 9 of grp 2: BACKUP_DG2_0009 path:/dev/rdsk/c3t600601602901220055E36868295CDE11d0s6 NOTE: cache opening disk 10 of grp 2: BACKUP_DG2_0010 path:/dev/rdsk/c3t600601602901220054E36868295CDE11d0s6 NOTE: cache opening disk 11 of grp 2: BACKUP_DG2_0011 path:/dev/rdsk/c3t6006016029012200EAE8F27B295CDE11d0s6 NOTE: cache opening disk 12 of grp 2: BACKUP_DG2_0012 path:/dev/rdsk/c3t6006016029012200EC34DD96295CDE11d0s6 NOTE: cache opening disk 13 of grp 2: BACKUP_DG2_0013 path:/dev/rdsk/c3t6006016029012200ED34DD96295CDE11d0s6 NOTE: cache mounting (not first) group 2/0xBD0B9C9C (BACKUP_DG2) kjbdomatt send to node 0 NOTE: attached to recovery domain 2 NOTE: LGWR attempting to mount thread 1 for diskgroup 2 NOTE: LGWR mounted thread 1 for disk group 2 NOTE: opening chunk 1 at fcn 0.544141956 ABA NOTE: seq=685 blk=8752 NOTE: cache mounting group 2/0xBD0B9C9C (BACKUP_DG2) succeeded NOTE: cache ending mount (success) of group BACKUP_DG2 number=2 incarn=0xbd0b9c9c kfdp_query(): 8 kfdp_queryBg(): 8 NOTE: cache opening disk 0 of grp 3: DATA_DG1_0000 path:/dev/rdsk/c3t60060160290122008A73502D5A5BDE11d0s6 NOTE: F1X0 found on disk 0 fcn 0.152 NOTE: cache opening disk 1 of grp 3: DATA_DG1_0001 path:/dev/rdsk/c3t6006016029012200AA30026A5A5BDE11d0s6 NOTE: cache opening disk 2 of grp 3: DATA_DG1_0002 path:/dev/rdsk/c3t6006016029012200AB30026A5A5BDE11d0s6 NOTE: cache opening disk 3 of grp 3: DATA_DG1_0003 path:/dev/rdsk/c3t6006016029012200AC30026A5A5BDE11d0s6 NOTE: cache opening disk 4 of grp 3: DATA_DG1_0004 path:/dev/rdsk/c3t60060160290122002AD902F1595BDE11d0s6 NOTE: cache opening disk 5 of grp 3: DATA_DG1_0005 path:/dev/rdsk/c3t600601602901220038040AF9595BDE11d0s6 NOTE: cache opening disk 6 of grp 3: DATA_DG1_0006 path:/dev/rdsk/c3t600601602901220036040AF9595BDE11d0s6 NOTE: cache opening disk 7 of grp 3: DATA_DG1_0007 path:/dev/rdsk/c3t600601602901220037040AF9595BDE11d0s6 NOTE: cache opening disk 8 of grp 3: DATA_DG1_0008 path:/dev/rdsk/c3t60060160290122008B73502D5A5BDE11d0s6 NOTE: cache opening disk 9 of grp 3: DATA_DG1_0009 path:/dev/rdsk/c3t60060160290122002B9576255A5BDE11d0s6 NOTE: cache opening disk 10 of grp 3: DATA_DG1_0010 path:/dev/rdsk/c3t60060160290122002BD902F1595BDE11d0s6 NOTE: cache opening disk 11 of grp 3: DATA_DG1_0011 path:/dev/rdsk/c3t6006016029012200EE774E625A5BDE11d0s6 NOTE: cache opening disk 12 of grp 3: DATA_DG1_0012 path:/dev/rdsk/c3t6006016029012200ED774E625A5BDE11d0s6 NOTE: cache opening disk 13 of grp 3: DATA_DG1_0013 path:/dev/rdsk/c3t60060160290122002DD902F1595BDE11d0s6 NOTE: cache opening disk 14 of grp 3: DATA_DG1_0014 path:/dev/rdsk/c3t60060160290122002D9576255A5BDE11d0s6 NOTE: cache opening disk 15 of grp 3: DATA_DG1_0015 path:/dev/rdsk/c3t6006016029012200EC774E625A5BDE11d0s6 NOTE: cache mounting (not first) group 3/0xBD0B9C9D (DATA_DG1) kjbdomatt send to node 0 NOTE: attached to recovery domain 3 NOTE: LGWR attempting to mount thread 1 for diskgroup 3 NOTE: LGWR mounted thread 1 for disk group 3 NOTE: opening chunk 1 at fcn 0.2874008 ABA NOTE: seq=43 blk=5509 NOTE: cache mounting group 3/0xBD0B9C9D (DATA_DG1) succeeded NOTE: cache ending mount (success) of group DATA_DG1 number=3 incarn=0xbd0b9c9d kfdp_query(): 9 kfdp_queryBg(): 9 NOTE: cache opening disk 0 of grp 4: DATA_DG2_0000 path:/dev/rdsk/c3t60060160290122002C9576255A5BDE11d0s6 NOTE: F1X0 found on disk 0 fcn 0.746372 NOTE: cache opening disk 1 of grp 4: DATA_DG2_0001 path:/dev/rdsk/c3t60060160290122002CD902F1595BDE11d0s6 NOTE: cache opening disk 2 of grp 4: DATA_DG2_0002 path:/dev/rdsk/c3t60060160290122008973502D5A5BDE11d0s6 NOTE: cache opening disk 3 of grp 4: DATA_DG2_0003 path:/dev/rdsk/c3t60060160290122008873502D5A5BDE11d0s6 NOTE: cache mounting (not first) group 4/0xBD0B9C9E (DATA_DG2) kjbdomatt send to node 0 NOTE: attached to recovery domain 4 NOTE: LGWR attempting to mount thread 1 for diskgroup 4 NOTE: LGWR mounted thread 1 for disk group 4 NOTE: opening chunk 1 at fcn 0.915102 ABA NOTE: seq=31 blk=7869 NOTE: cache mounting group 4/0xBD0B9C9E (DATA_DG2) succeeded NOTE: cache ending mount (success) of group DATA_DG2 number=4 incarn=0xbd0b9c9e kfdp_query(): 10 kfdp_queryBg(): 10 NOTE: cache opening disk 0 of grp 5: DATA_DG3_0000 path:/dev/rdsk/c3t60060160290122004CFED3A3295CDE11d0s6 NOTE: F1X0 found on disk 0 fcn 0.0 NOTE: cache opening disk 1 of grp 5: DATA_DG3_0001 path:/dev/rdsk/c3t600601602901220060CD9F90295CDE11d0s6 NOTE: cache opening disk 2 of grp 5: DATA_DG3_0002 path:/dev/rdsk/c3t600601602901220062CD9F90295CDE11d0s6 NOTE: cache opening disk 3 of grp 5: DATA_DG3_0003 path:/dev/rdsk/c3t6006016029012200F6F4EF89295CDE11d0s6 NOTE: cache opening disk 4 of grp 5: DATA_DG3_0004 path:/dev/rdsk/c3t6006016029012200F5F4EF89295CDE11d0s6 NOTE: cache opening disk 5 of grp 5: DATA_DG3_0005 path:/dev/rdsk/c3t6006016029012200F4F4EF89295CDE11d0s6 NOTE: cache opening disk 6 of grp 5: DATA_DG3_0006 path:/dev/rdsk/c3t60060160290122004EFED3A3295CDE11d0s6 NOTE: cache opening disk 7 of grp 5: DATA_DG3_0007 path:/dev/rdsk/c3t6006016029012200EE34DD96295CDE11d0s6 NOTE: cache mounting (not first) group 5/0xBD0B9C9F (DATA_DG3) kjbdomatt send to node 0 NOTE: attached to recovery domain 5 NOTE: LGWR attempting to mount thread 1 for diskgroup 5 NOTE: LGWR mounted thread 1 for disk group 5 NOTE: opening chunk 1 at fcn 0.1061356 ABA NOTE: seq=37 blk=10556 NOTE: cache mounting group 5/0xBD0B9C9F (DATA_DG3) succeeded NOTE: cache ending mount (success) of group DATA_DG3 number=5 incarn=0xbd0b9c9f kfdp_query(): 11 kfdp_queryBg(): 11 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 1 SUCCESS: diskgroup BACKUP_DG1 was mounted kfdp_query(): 12 kfdp_queryBg(): 12 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 2 SUCCESS: diskgroup BACKUP_DG2 was mounted kfdp_query(): 13 kfdp_queryBg(): 13 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 3 SUCCESS: diskgroup DATA_DG1 was mounted kfdp_query(): 14 kfdp_queryBg(): 14 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 4 SUCCESS: diskgroup DATA_DG2 was mounted kfdp_query(): 15 kfdp_queryBg(): 15 NOTE: Instance updated compatible.asm to 10.1.0.0.0 for grp 5 SUCCESS: diskgroup DATA_DG3 was mounted SUCCESS: ALTER DISKGROUP ALL MOUNT Wed Sep 05 03:43:31 2012 Starting background process ASMB Wed Sep 05 03:43:31 2012 ASMB started with pid=21, OS id=17551 NOTE: ASMB process exiting due to lack of ASM file activity for 5 secondsalert_PFRM_A2.log
Wed Sep 5 01:06:39 2012 ALTER SYSTEM ARCHIVE LOG Wed Sep 5 01:06:39 2012 Thread 2 advanced to log sequence 9994 (LGWR switch) Current log# 4 seq# 9994 mem# 0: +BACKUP_DG2/pfrm_a/onlinelog/group_4.266.692725317 Current log# 4 seq# 9994 mem# 1: +DATA_DG3/pfrm_a/onlinelog/group_4.371.716207107 Wed Sep 5 01:06:43 2012 ALTER SYSTEM ARCHIVE LOG Wed Sep 5 01:06:46 2012 Thread 2 advanced to log sequence 9995 (LGWR switch) Current log# 3 seq# 9995 mem# 0: +BACKUP_DG2/pfrm_a/onlinelog/group_3.265.692725317 Current log# 3 seq# 9995 mem# 1: +DATA_DG3/pfrm_a/onlinelog/group_3.370.716207101 Wed Sep 5 01:07:00 2012 Starting control autobackup Control autobackup written to DISK device handle '+BACKUP_DG2/pfrm_a/autobackup/2012_09_05/s_793156020.4229.793156023' Wed Sep 5 03:43:02 2012 Errors in file /u01/app/oracle/product/10.2.0/db/admin/PFRM_A/bdump/pfrm_a2_asmb_14034.trc: ORA-15064: communication failure with ASM instance ORA-03113: end-of-file on communication channel Wed Sep 5 03:43:02 2012 ASMB: terminating instance due to error 15064 Wed Sep 5 03:43:02 2012 System state dump is made for local instance System State dumped to trace file /u01/app/oracle/product/10.2.0/db/admin/PFRM_A/bdump/pfrm_a2_diag_13894.trc Wed Sep 5 03:43:05 2012 Shutting down instance (abort) License high water mark = 22 Wed Sep 5 03:43:06 2012 Trace dumping is performing id=[cdmp_20120905034302] Wed Sep 5 03:43:07 2012 Instance terminated by ASMB, pid = 14034 Wed Sep 5 03:43:10 2012 Instance terminated by USER, pid = 16926alert_PFRM_C2.log
Wed Sep 5 03:43:02 2012 Errors in file /u01/app/oracle/product/10.2.0/db/admin/PFRM_C/bdump/pfrm_c2_asmb_14061.trc: ORA-15064: communication failure with ASM instance ORA-03113: end-of-file on communication channel Wed Sep 5 03:43:02 2012 ASMB: terminating instance due to error 15064 Wed Sep 5 03:43:02 2012 Errors in file /u01/app/oracle/product/10.2.0/db/admin/PFRM_C/bdump/pfrm_c2_lmon_13912.trc: ORA-15064: communication failure with ASM instance Wed Sep 5 03:43:02 2012 Errors in file /u01/app/oracle/product/10.2.0/db/admin/PFRM_C/bdump/pfrm_c2_lms0_13918.trc: ORA-15064: communication failure with ASM instance Wed Sep 5 03:43:02 2012 Errors in file /u01/app/oracle/product/10.2.0/db/admin/PFRM_C/bdump/pfrm_c2_pmon_13902.trc: ORA-15064: communication failure with ASM instance Wed Sep 5 03:43:02 2012 System state dump is made for local instance System State dumped to trace file /u01/app/oracle/product/10.2.0/db/admin/PFRM_C/bdump/pfrm_c2_diag_13906.trc Wed Sep 5 03:43:06 2012 Shutting down instance (abort) License high water mark = 37 Wed Sep 5 03:43:07 2012 Trace dumping is performing id=[cdmp_20120905034302] Wed Sep 5 03:43:07 2012 Instance terminated by ASMB, pid = 14061 Wed Sep 5 03:43:11 2012 Instance terminated by USER, pid = 16959alert_COXCALL2.log
Wed Sep 5 03:43:01 2012 Errors in file /u01/app/oracle/product/10.2.0/db/admin/COXCALL/bdump/coxcall2_asmb_14369.trc: ORA-15064: communication failure with ASM instance ORA-03113: end-of-file on communication channel Wed Sep 5 03:43:01 2012 ASMB: terminating instance due to error 15064 Wed Sep 5 03:43:01 2012 Errors in file /u01/app/oracle/product/10.2.0/db/admin/COXCALL/bdump/coxcall2_lms1_14277.trc: ORA-15064: communication failure with ASM instance Wed Sep 5 03:43:01 2012 Errors in file /u01/app/oracle/product/10.2.0/db/admin/COXCALL/bdump/coxcall2_lms0_14271.trc: ORA-15064: communication failure with ASM instance Wed Sep 5 03:43:01 2012 System state dump is made for local instance System State dumped to trace file /u01/app/oracle/product/10.2.0/db/admin/COXCALL/bdump/coxcall2_diag_14255.trc Wed Sep 5 03:43:05 2012 Shutting down instance (abort) License high water mark = 53 Wed Sep 5 03:43:07 2012 Trace dumping is performing id=[cdmp_20120905034301] Wed Sep 5 03:43:07 2012 Instance terminated by ASMB, pid = 14369 Wed Sep 5 03:43:10 2012 Instance terminated by USER, pid = 16931 Wed Sep 5 03:43:31 2012 Starting ORACLE instance (normal) LICENSE_MAX_SESSION = 0 LICENSE_SESSIONS_WARNING = 0 Picked latch-free SCN scheme 3 Autotune of undo retention is turned on. LICENSE_MAX_USERS = 0 SYS auditing is disabled ksdpec: called for event 13740 prior to event group initialization Starting up ORACLE RDBMS Version: 10.2.0.4.0. System parameters with non-default values: processes = 1500 sessions = 1800 __shared_pool_size = 469762048 __large_pool_size = 16777216 __java_pool_size = 16777216 __streams_pool_size = 16777216 spfile = +BACKUP_DG1/coxcalls/spfilecoxcall.ora cluster_interconnects = 192.168.1.4 sga_target = 3221225472 control_files = +DATA_DG1/coxcalls/controlfile/current.305.697217041, +BACKUP_DG1/coxcalls/controlfile/current.268.6972 17041 db_file_name_convert = +COXCALL_DATA/COXCALL, +DATA_DG1/COXCALLS, +COXCALL_BACKUP/COXCALL, +BACKUP_DG1/COXCALLS log_file_name_convert = +COXCALL_DATA/COXCALL, +DATA_DG1/COXCALLS, +COXCALL_BACKUP/COXCALL, +BACKUP_DG1/COXCALLS db_block_size = 8192 __db_cache_size = 2684354560 compatible = 10.2.0.2.0 log_archive_config = dg_config=(COXCALL,COXCALLS) log_archive_dest_1 = LOCATION=USE_DB_RECOVERY_FILE_DEST log_archive_dest_2 = log_archive_max_processes= 5 log_archive_format = %t_%s_%r.arc fal_client = COXCALLS fal_server = COXCALL db_file_multiblock_read_count= 16 cluster_database = TRUE cluster_database_instances= 2 db_create_file_dest = +DATA_DG1 db_recovery_file_dest = +BACKUP_DG1 db_recovery_file_dest_size= 1073741824000 standby_file_management = auto thread = 2 instance_number = 2 undo_management = AUTO undo_tablespace = UNDOTBS2 undo_retention = 20000 remote_login_passwordfile= EXCLUSIVE db_domain = service_names = RKSCALL,PHONETOOLS,BTSUSER,ITV_SVC,ACCALL,COXCALL,COXCALLS,BTA_SVC dispatchers = (PROTOCOL=TCP) (SERVICE=COXCALLXDB) local_listener = remote_listener = job_queue_processes = 10 background_dump_dest = /u01/app/oracle/product/10.2.0/db/admin/COXCALL/bdump user_dump_dest = /u01/app/oracle/product/10.2.0/db/admin/COXCALL/udump core_dump_dest = /u01/app/oracle/product/10.2.0/db/admin/COXCALL/cdump audit_file_dest = /u01/app/oracle/product/10.2.0/db/admin/COXCALL/adump db_name = COXCALL db_unique_name = COXCALLS open_cursors = 500 pga_aggregate_target = 1706033152 Cluster communication is configured to use the following interface(s) for this instance 192.168.1.4 Wed Sep 5 03:43:34 2012 cluster interconnect IPC version:Oracle UDP/IP (generic) IPC Vendor 1 proto 2 PMON started with pid=2, OS id=17625 DIAG started with pid=3, OS id=17629 PSP0 started with pid=4, OS id=17633 LMON started with pid=5, OS id=17637 LMD0 started with pid=6, OS id=17639 LMS0 started with pid=7, OS id=17643 LMS1 started with pid=8, OS id=17649 MMAN started with pid=9, OS id=17657 DBW0 started with pid=10, OS id=17661 LGWR started with pid=11, OS id=17665 CKPT started with pid=12, OS id=17669 SMON started with pid=13, OS id=17673 RECO started with pid=14, OS id=17677 CJQ0 started with pid=15, OS id=17679 MMON started with pid=16, OS id=17681 Wed Sep 5 03:43:35 2012 starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'... MMNL started with pid=17, OS id=17683 Wed Sep 5 03:43:35 2012 starting up 1 shared server(s) ... Wed Sep 5 03:43:36 2012 lmon registered with NM - instance id 2 (internal mem no 1) Wed Sep 5 03:43:37 2012 Reconfiguration started (old inc 0, new inc 48) List of nodes: 0 1 Global Resource Directory frozen * allocate domain 0, invalid = TRUE Communication channels reestablished * domain 0 valid = 1 according to instance 0 Wed Sep 5 03:43:37 2012 Master broadcasted resource hash value bitmaps Non-local Process blocks cleaned out Wed Sep 5 03:43:37 2012 LMS 1: 0 GCS shadows cancelled, 0 closed Wed Sep 5 03:43:37 2012 LMS 0: 0 GCS shadows cancelled, 0 closed Set master node info Submitted all remote-enqueue requests Dwn-cvts replayed, VALBLKs dubious All grantable enqueues granted Wed Sep 5 03:43:37 2012 LMS 1: 0 GCS shadows traversed, 0 replayed Wed Sep 5 03:43:37 2012 LMS 0: 0 GCS shadows traversed, 0 replayed Wed Sep 5 03:43:37 2012 Submitted all GCS remote-cache requests Fix write in gcs resources Reconfiguration complete LCK0 started with pid=20, OS id=17727 Wed Sep 5 03:43:38 2012 ALTER DATABASE MOUNT Wed Sep 5 03:43:39 2012 Starting background process ASMB ASMB started with pid=22, OS id=17738 Starting background process RBAL RBAL started with pid=23, OS id=17742 Wed Sep 5 03:43:42 2012 SUCCESS: diskgroup DATA_DG1 was mounted SUCCESS: diskgroup BACKUP_DG1 was mounted Wed Sep 5 03:43:46 2012 Setting recovery target incarnation to 2 Wed Sep 5 03:43:46 2012 Successful mount of redo thread 2, with mount id 819078902 Wed Sep 5 03:43:46 2012 Database mounted in Shared Mode (CLUSTER_DATABASE=TRUE) Completed: ALTER DATABASE MOUNT Wed Sep 5 03:43:47 2012 ALTER DATABASE OPEN Picked broadcast on commit scheme to generate SCNs Wed Sep 5 03:43:52 2012 LGWR: STARTING ARCH PROCESSES ARC0 started with pid=27, OS id=17912 ARC1 started with pid=28, OS id=17914 ARC2 started with pid=29, OS id=17916 ARC3 started with pid=30, OS id=17918 Wed Sep 5 03:43:52 2012 ARC0: Archival started ARC1: Archival started ARC2: Archival started ARC3: Archival started ARC4: Archival started LGWR: STARTING ARCH PROCESSES COMPLETE ARC4 started with pid=31, OS id=17920 Wed Sep 5 03:43:52 2012 Thread 2 opened at log sequence 100362 Current log# 10 seq# 100362 mem# 0: +DATA_DG1/coxcalls/onlinelog/group_10.353.697219913 Current log# 10 seq# 100362 mem# 1: +BACKUP_DG1/coxcalls/onlinelog/group_10.278.697219913 Successful open of redo thread 2 Wed Sep 5 03:43:52 2012 MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set Wed Sep 5 03:43:52 2012 SMON: enabling cache recovery Wed Sep 5 03:43:52 2012 ARC0: Becoming the 'no FAL' ARCH ARC0: Becoming the 'no SRL' ARCH Wed Sep 5 03:43:52 2012 ARC3: Becoming the heartbeat ARCH Wed Sep 5 03:43:53 2012 Successfully onlined Undo Tablespace 5 Wed Sep 5 03:43:53 2012 SMON: enabling tx recovery Wed Sep 5 03:43:53 2012 Database Characterset is WE8ISO8859P1 Opening with internal Resource Manager plan where NUMA PG = 1, CPUs = 8 replication_dependency_tracking turned off (no async multimaster replication found) Starting background process QMNC QMNC started with pid=35, OS id=17983 Wed Sep 5 03:43:57 2012 Completed: ALTER DATABASE OPEN
相关文章推荐
- ORA-15097: cannot SHUTDOWN ASM instance with connected client (process 2476)
- ORA-15097: cannot SHUTDOWN ASM instance with connected client 和 ORA-01031: insufficient privileges
- ORA-29786: SIHA attribute GET failed with Error If 11gR2 ASM instance is created manually
- oracle 10.2.0.1 ORA-15097: cannot SHUTDOWN ASM instance with connected RDBMS instance
- ORA-15097: cannot SHUTDOWN ASM instance with connected client 和 ORA-01031: insufficient privileges 处理方法
- ORA-15097: cannot SHUTDOWN ASM instance with connected client
- ORA-15097: cannot SHUTDOWN ASM instance with connected client 和 ORA-01031: insufficient privileges 处
- 10gR2 rac(asm) crash with ora-15064
- ASM INSTANCE CRASHES FREQUENTLY WITH ORA-15082 in 11g [ID 1056695.1]
- 10gR2 rac(asm) crash with ora-15064
- 【翻译自mos文章】Oracle Cluster failed to start with ASM instance getting ORA-00443 ---改动/etc/resolv.conf所致
- ORA-29786: SIHA attribute GET failed with Error If 11gR2 ASM instance is created manually [ID 976075
- ORA-15097: cannot SHUTDOWN ASM instance with connected client (process 16359)
- rac11g错误:ORA-15097: cannot SHUTDOWN ASM instance with connected client
- ORA-00099: warning: no parameter file specified for ASM instance
- ORA-00099: warning: no parameter file specified for ASM instance
- ASMCMD cp command fails with ORA-15046 (文档 ID 452158.1)
- ORA-15039: diskgroup not dropped ORA-15073: diskgroup DATA1 is mounted by another ASM instance
- 今天装完ASM时遇到的一个比较低级的问题ORA-15149: another ASM instance found running on the host
- ORA-15149: another ASM instance found running on the host