Exadata遭遇ORA-27603和ORA-27626

今天CRM反应说他们有个作业今天没跑出来报错了，一查发现Exadata的节点2上的alert日志里面在同一时间点果然报错了：

WedOct1706:22:512012

Errorsinfile/u01/app/oracle/diag/rdbms/srcbfin/SRCBFIN2/trace/SRCBFIN2_arc2_72209.trc:

ORA-27603:CellstorageI/Oerror,I/Ofailedondisko/172.11.211.9/DATA_DM01_CD_01_dm01cel01atoffset464519168fordatalength1048576

ORA-27626:Exadataerror:201(GenericI/Oerror)

WARNING:ReadFailed.group:1disk:25AU:110offset:3145728size:1048576

WARNING:failedtoreadmirrorside1ofvirtualextent29logicalextent0offile266ingroup[1.2063103479]fromdiskDATA_DM01_CD_01_DM01CEL01allocationunit110reasonerror;ifpossible,willtryanothermirrorside

NOTE:successfullyreadmirrorside2ofvirtualextent29logicalextent1offile266ingroup[1.2063103479]fromdiskDATA_DM01_CD_10_DM01CEL03allocationunit113

相应的那个trace日志文件里面也是只用相同的寥寥几行：

***2012-10-1706:22:51.655

ORA-27626:Exadataerror:201(GenericI/Oerror)

WARNING:ReadFailed.group:1disk:25AU:110offset:3145728size:1048576

path:o/172.11.211.9/DATA_DM01_CD_01_dm01cel01

incarnation:0xe9688586asynchronousresult:'I/Oerror'

subsys:OSSiop:0x2b58752a2680bufp:0x2b5879517000osderr:0xc9osderr1:0x0

Exadataerror:'GenericI/Oerror'

IOelapsedtime:12334426usecTimewaitedonI/O:12334426usec

WARNING:failedtoreadmirrorside1ofvirtualextent29logicalextent0offile266ingroup[1.2063103479]fromdiskDATA_DM01_CD_01_DM01CEL01allocationunit110reasonerror;ifpossible,willtryanothermirrorside

NOTE:successfullyreadmirrorside2ofvirtualextent29logicalextent1offile266ingroup[1.2063103479]fromdiskDATA_DM01_CD_10_DM01CEL03allocationunit113

在MOS上查了一下发现这可能是Exadata的一个Bug，无语了，上线两个月各种Bug各种宕机，官方还吹嘘的那么牛叉！！！

Bug8782572ARCHfencedinASMdiskscausinginternalerrorsatshutdown

Thisnotegivesabriefoverviewofbug8782572.

Thecontentwaslastupdatedon:17-JUN-2011

Clickherefordetailsofeachofthesectionsbelow.

Affects:

Product(Component)	OracleServer(Rdbms)
Rangeofversionsbelievedtobeaffected	VersionsBELOW12.1
Versionsconfirmedasbeingaffected	11.2.0.1
Platformsaffected	Generic(all/mostplatformsaffected)

Fixed:

Thisissueisfixedin	12.1(FutureRelease) 11.2.0.2(ServerPatchSet) 11.2.0.1BundlePatch1forExadataDatabase
Symptoms:	RelatedTo:
ProcessMayDump(ORA-7445)/Abend/Abort ErrorMayOccur InternalErrorMayOccur(ORA-600) ORA-27603 ORA-600[kfioSrMsg_send:01] Dumpinorunderkcrfatrm	AutomaticStorageManagement(ASM) Exadata PhysicalStandbyDatabase/Dataguard

Description

ThisbugcausesARCHprocessestokeepissuingIO's(orarchivelogs)

evenafteranRDBMSinstancehasbeendismounted.SuchIO'sare

fencedoffinASMdisksafterinstanceisnolongerpartofthecluster

(i.e.afterdismount),andtheASMdiskgroupcanbedismounted

asaresultoftheseIOs.

HereisanexampleexcerptfromalertlogshowingtheIOerrorsduetofence:

ORA-27603:CellstorageI/Oerror,I/Ofailedondisko/<IPAddress>/<ASMDisk>atoffset<offset#>fordatalength<length>WARNING:IOFailed.group:<group#>disk(number.incarnation):<number.inc>disk_path:o/<IPAddress>/<ASMdisk>

AU:<AU>disk_offset(bytes):<bytes>io_size:<IOsize>operation:Readtype:asynchronous

result:I/Oerrorprocess_id:<pid>

Exadataerror:221(I/Orequestfenced)

Anotherexamplefromacellalertlog:

Information:CellsrvcancelingOSSMSG_COMMAND_BREADrequestfromhost

xxxx[pid:<pid>]forfencing,sendport<port#>openfd2

Pleasenote:Theaboveisasummarydescriptiononly.Actualsymptomscanvary.Matchingtoanysymptomsheredoesnotconfirmthatyouareencounteringthisproblem.ForquestionsaboutthisbugpleaseconsultOracleSupport.

References

Bug:8782572(ThislinkwillonlyworkforPUBLISHEDbugs)

Note:245840.1Informationonthesectionsinthisarticle

这个Bug可以早到相应的Patch：



另一篇文章，不过好像不是这么一回事，应该是个Bug，不过还是把文章附上：

Exadata/Rac-Ora-27603:CellStorageI/OError[ID1445223.1]

转到底部

修改时间:2012-5-30 类型:PROBLEM 状态:PUBLISHED 优先级:3

注释(0)

Appliesto:

OracleExadataHardware-Version11.2.0.1to11.2.0.1[Release11.2]

Informationinthisdocumentappliestoanyplatform.

Symptoms

Gettingfollowingerrorsintheexadataenvironmentrunning11.2.0.1database:

Errorsinfile/u01/app/oracle/diag/rdbms/test/test2/trace/test2_ora_22485.trc(incident=242913):

ORA-00600:internalerrorcode,arguments:[kssadpm1],[],[],[],[],[],[],[],[],[],[],[]

Incidentdetailsin:/u01/app/oracle/diag/rdbms/test/test2/incident/incdir_242913/test2_ora_22485_i242913.trc

Errorsinfile/u01/app/oracle/diag/rdbms/test/test2/trace/test2_ora_22485.trc(incident=242914):

ORA-00600:internalerrorcode,arguments:[kfddsGet03],[56861],[],[],[],[],[],[],[],[],[],[]

Incidentdetailsin:/u01/app/oracle/diag/rdbms/test/test2/incident/incdir_242914/test2_ora_22485_i242914.trc

Changes

Norecentchanges

Cause

IOsarefencedevenbeforethetxnstateobjectgetsdeletedwhichneedstoperformsIOsinordertodothetxnrollback.Thisiscausingtheerror.

ORA-600[kssadpm1]israisedbecauseofbug9750033

Theseconderror,ORA-600[kfddsGet03],iscausedbyORA-600[kssadpm1].

Wecanconcludeonthebug9750033basedonthefollowingcriteria:

1.Callstackmatchesasfollows:

kssadpm

ksz_gen_reid

kfddsGet

kfioTranslateIO

kfioRqSetPrepare

2.Problematicstateobjectis'kszparent'

Thiscanbeverifiedfromthetracefile.

Example:

SO:0x6cbc55f40,type:22,owner:(nil),flag:-/FLST/-/0x00if:0x0c:0x0

proc=(nil),name=kszparent,file=ksz2.hLINE:394,pg=0

Dumpofmemoryfrom0x00000006CBC55F40to0x00000006CBC55F98

Solution

Impactofthebugistheprocessfailure.Astheerrorisfromserverprocess(notbackgroundprocess),noeffectattheinstancelevelandnocorruptionaswell.

Onlytheprocessencounteringtheerrorwillbeterminated.Alsotheerrorisduringnormalserverprocessexit.So,theimpactisveryminimal.

Fixincludedin11.2.0.1BP12.

Otheroptionistoupgradeto11.2.0.2orabovewhichincludesthefix.