ORA-30036故障处理思路

故障现象为某省结算库应用方在跑存储过程发现报错

ORA-30036: unable to extend segment by 8 in undo tablespace 'UNDOTBS1'

后进行了下面的一系列的排查分析：
查看undo表空间使用率为100%，查看alert日志中发现了大量的事物已经ORA-01555的报错。
Wed Jan 30 04:32:01 GMT+08:00 2013ORA-01555 caused by SQL statement below (SQL ID: 4ds6qq0mfac2t, Query Duration=5 sec, SCN: 0x0c00.d915c977):
Wed Jan 30 04:32:01 GMT+08:00 2013SELECT distinct FILE_NAME,to_char(file_name_time,:"SYS_B_00"),A.operation_type_grade FROM T_FILE_INFO_81 A,T_FILE_CLASS B ,T_FILE_STAT
E C WHERE ((A.RAT_FIRST_STARTTIME >=:"SYS_B_01" and A.RAT_FIRST_STARTTIME <=:"SYS_B_02" )
or (A.RAT_LAST_ENDTIME >=:"SYS_B_03" and A.RAT_LAST_ENDTIME <=:"SYS_B_04" )
or (A.RAT_FIRST_STARTTIME <=:"SYS_B_05" and A.RAT_LAST_ENDTIME >= :"SYS_B_06" ) )
and ((A.RAT_FIRST_CALLING_NBR >=:"SYS_B_07" and A.RAT_FIRST_CALLING_NBR <=:"SYS_B_08")
or (A.RAT_LAST_CALLING_NBR >= :"SYS_B_09" and A.RAT_LAST_CALLING_NBR <= :"SYS_B_10" )
or (A.RAT_FIRST_CALLING_NBR <=:"SYS_B_11" and A.RAT_LAST_CALLING_NBR >=:"SYS_B_12" ))
and A.FILE_NAME <>:"SYS_B_13" and A.File_Class_Id = B.File_Class_Id and B.operati
on_type_id=:"SYS_B_14" and (A.FILE_NAME_TIME + interval :"SYS_B_15" day ) > TO_DATE(:"SYS_B_16",:"SYS_B_17")
and A.city_id =:"SYS_B_18" and a.state in (:"SYS
Wed Jan 30 08:49:12 GMT+08:00 2013insert into STL_GX.T_all_file_TOTAL
select aa.province_id,
dd.name,
aa.BILL_DATE,
aa.OPERATION_TYPE_GRADE,
:"SYS_B_00",
aa.cj_files,
bb.pj_files,
bb.org_counts,
bb.rate_counts,
bb.inp_counts,
bb.inpc_counts,
cc.jf_files,
cc.jf_counts,
:"SYS_B_01",
:"SYS_B_02",
aa.is_rate,
aa.is_billtag,
aa.is_insert,
bb.ERR_COUNTS
from (SELECT a.province_code as province_id,
substr(A.ORG_FILENAME, :"SYS_B_03", :"SYS_B_04") as BILL_DATE,
b.operation_type_grade,
COUNT(*) as cj_files,
b.is_rate as is_rate,
b.is_billtag as is_billtag,
b.is_insert as is_insert
FROM STL_PARA.T_LOG_COLLECT_76@pub_PARA A,
stl_gx.tmp_cj_info b
WHERE substr(A.ORG_FILENAME, :"SYS_B_05", :"SYS_B_06") = :"SYS_B_07"
and b.province_id = :"S
Wed Jan 30 09:44:56 GMT+08:00 2013Thread 1 advanced to log sequence 149384 (LGWR switch)
Current log# 6 seq# 149384 mem# 0: /dev/rjs_redolog06
Wed Jan 30 09:50:00 GMT+08:00 2013ORA-01555 caused by SQL statement below (SQL ID: 7t0bjnwxt9ufv, Query Duration=180 sec, SCN: 0x0c00.e294a117):
这个很明显，是因为undo中存在大量的insert操作，导致数据库undo没有commit，由于本库的实际环境，之前做过undo_retention的调整。下面看此设定值。
SQL> show parameter undo

NAME TYPE VALUE
------------------------------------ -------------------------------- ------------------------------
undo_management string AUTO
undo_retention integer 0
undo_tablespace string UNDOTBS1

undo_management为auto、retention时间为0让Oracle自动调整保留提交后undo信息的时间。Oracle 10g之前,在自动Undo管理的模式下，我们都知道undo_retention参数的作用是用来控制当transaction被commit之后，undo信息的保留时间。这些undo信息可以用来构造consistent read以及用于一系列的闪回恢复，而且足够的undo信息还可以减少ORA-01555错误的发生，在Oracle 9R1中呢，这个value的默认值是900秒，Oracle 9R2以后这个value提高到了10800秒。即使我们设置了undo_retention这个参数，那么在默认情况下，这是一个noguarantee的限制。也就是说我将undo_retention=10800,那么原本以为在一个transaction commit之后，之前的undo还可以保存10800秒，才可以被别的transaction DML覆盖，孰不知当有其他的transaction DML处理过程中需要undo空间的时候，恰恰这个时候not enough space for undo，也就说我并没有允许undo tablespace自动扩展。由于我们的retention是noguarantee的，所以transaction DML就会忽略这种retention的时间限制直接回绕覆盖我们的undo信息，这种结果下其实在很多情况下是不希望得到的。
Oracle 10g之后，oracle提出了一个特性就是undo的guarantee，可以强制oracle来guarantee的undo信息，也就说如果一个session的transaction DML需要undo空间的时候，即使undo的空间不足，这个session也不会强制覆盖由undo_retention所保护的undo信息，那么这个transaction DML会因为undo空间的不足会而report一个error并自动退出。
SQL> select tablespace_name,block_size,extent_management
2 segment_space_management,contents,retention
3 from dba_tablespaces;

TABLESPACE_NAME BLOCK_SIZE SEGMENT_SP CONTENTS RETENTION
------------------------------ ---------- ---------- --------- -----------
SYSTEM 8192 LOCAL PERMANENT NOT APPLY
UNDOTBS1 8192 LOCAL UNDO NOGUARANTEE
SYSAUX 8192 LOCAL PERMANENT NOT APPLY
TEMP 8192 LOCAL TEMPORARY NOT APPLY
USERS 8192 LOCAL PERMANENT NOT APPLY
COMMDATA 8192 LOCAL PERMANENT NOT APPLY
SETTLEINDEX 8192 LOCAL PERMANENT NOT APPLY
SETTLEDATA 8192 LOCAL PERMANENT NOT APPLY
STATDATA 8192 LOCAL PERMANENT NOT APPLY
STATINDEX 8192 LOCAL PERMANENT NOT APPLY
COMMINDEX 8192 LOCAL PERMANENT NOT APPLY

TABLESPACE_NAME BLOCK_SIZE SEGMENT_SP CONTENTS RETENTION
------------------------------ ---------- ---------- --------- -----------
RMAN_TBS 8192 LOCAL PERMANENT NOT APPLY

12 rows selected.
之后想的既然资源无法commit是否可以重启数据库达到资源释放，所以15点20开始重启数据库，打算重新找一个数据文件，然后重新创建一个undo表空间，将undotbs1切换到undotbs2并把tbs1 offline，drop后，在切回到tbs1上面进行资源释放。

create undo tablespace UNDOTBS2 datafile '/dev/untb03.dbf' size 32700M
alter system set undo_tablespace=UNDOTBS2 scope=both;

将原来的UNDO表空间,置为脱机:

alter tablespace UNDOTBS1 offline;

删除原来的UNDO表空间:

drop tablespace UNDOTBS1 including contents AND DATAFILES CASCADE CONSTRAINTS ;
Wed Jan 30 15:33:39 GMT+08:00 2013ALTER DATABASE OPEN
重启发现，undo的利用率还是100%、也就是说undo_retention=0没有生效
Wed Jan 30 17:27:06 GMT+08:00 2013ALTER SYSTEM SET undo_retention=900 SCOPE=BOTH;
设定retention时间为15分钟，那么看看数据库中undo active使用率居然高达62GB（总共undo表空间为64GB）
发现此刻数据库中存在两个死事物
SQL> select ADDR,KTUXEUSN,KTUXESLT,KTUXESQN,KTUXESIZ from x$ktuxe where KTUXECFL='DEAD';
ADDR KTUXEUSN KTUXESLT KTUXESQN KTUXESIZ
---------------- ---------- ---------- ---------- ----------
00000001108C63F0 75 27 2368514 795545
00000001108C5D10 644 7 57597 0
由于已经报Oracle ACS服务，oracle工程师到来后，原75-27死事物已经不存在（18：35左右应用方停止了相关应用）
再次查看数据库中UNEXPIRED利用率63GB、ACTIVE利用率1GB。
那么应该是死事物得到了释放，再次查看
SQL> select ADDR,KTUXEUSN,KTUXESLT,KTUXESQN,KTUXESIZ from x$ktuxe where KTUXECFL='DEAD';

ADDR KTUXEUSN KTUXESLT KTUXESQN KTUXESIZ
---------------- ---------- ---------- ---------- ----------
0000000110845CF0 644 7 57597 0
那么绝对就是75-27得到了释放，查看现有的交易
SQL> alter session set nls_date_format='mm/dd/yy hh24:mi:ss';

Session altered.
SQL> select begin_time,end_time,UNXPSTEALCNT from v$undostat;

BEGIN_TIME END_TIME UNXPSTEALCNT
----------------- ----------------- ------------
01/30/13 20:53:28 01/31/13 11:09:42 71794
01/30/13 20:43:28 01/30/13 20:53:28 110035
01/30/13 20:33:28 01/30/13 20:43:28 15240
01/30/13 20:23:28 01/30/13 20:33:28 25489
01/30/13 20:13:28 01/30/13 20:23:28 11936
01/30/13 20:03:28 01/30/13 20:13:28 2950
01/30/13 19:53:28 01/30/13 20:03:28 707
01/30/13 19:43:28 01/30/13 19:53:28 0
01/30/13 19:33:28 01/30/13 19:43:28 0
01/30/13 19:23:28 01/30/13 19:33:28 1271
01/30/13 19:13:28 01/30/13 19:23:28 29187

BEGIN_TIME END_TIME UNXPSTEALCNT
----------------- ----------------- ------------
01/30/13 19:03:28 01/30/13 19:13:28 19976
01/30/13 18:53:28 01/30/13 19:03:28 1365
01/30/13 18:43:28 01/30/13 18:53:28 6235
01/30/13 18:33:28 01/30/13 18:43:28 24651
01/30/13 18:23:28 01/30/13 18:33:28 38220
01/30/13 18:13:28 01/30/13 18:23:28 49888
01/30/13 18:03:28 01/30/13 18:13:28 29815
01/30/13 17:53:28 01/30/13 18:03:28 43678
01/30/13 17:43:28 01/30/13 17:53:28 104834
01/30/13 17:33:28 01/30/13 17:43:28 101518
01/30/13 17:23:28 01/30/13 17:33:28 45838

BEGIN_TIME END_TIME UNXPSTEALCNT
----------------- ----------------- ------------
01/30/13 17:13:28 01/30/13 17:23:28 30964
01/30/13 17:03:28 01/30/13 17:13:28 43876
01/30/13 16:53:28 01/30/13 17:03:28 15455
01/30/13 16:43:28 01/30/13 16:53:28 7839
01/30/13 16:33:28 01/30/13 16:43:28 24606
01/30/13 16:23:28 01/30/13 16:33:28 40497
01/30/13 16:13:28 01/30/13 16:23:28 34759
01/30/13 16:03:28 01/30/13 16:13:28 118142
01/30/13 15:53:28 01/30/13 16:03:28 107958
01/30/13 15:43:28 01/30/13 15:53:28 20249
01/30/13 15:33:28 01/30/13 15:43:28 0

33 rows selected.
把疑问交给ORACLE工程师了，为什么undo_retention设定了900s未即时生效？
通过查看metalink得知一个bug问题ps：Bug 5387030 - Automatic tuning of undo_retention causes unusual extra space allocation [ID 5387030.8]

Product (Component)	Oracle Server (Rdbms)
Range of versions believed to be affected	Versions >= 10.2.0.1 but BELOW 11.1
Versions confirmed as being affected	10.2.0.3 10.2.0.2 10.2.0.1

Description
When undo tablespace is using NON-AUTOEXTEND datafiles,
V$UNDOSTAT.TUNED_UNDORETENTION may be calculated too high preventing
undo block from being expired and reused. In extreme cases the undo
tablespace could be filled to capacity by these unexpired blocks.

An alert may be posted on DBA_ALERT_HISTORY that advises to increase
the space when it is not really necessary if this fix is applied.
If the user sets their own alert thresholds for undo tablespaces the
bug may prevent alerts from being produced.

Workaround
alter system set "_smu_debug_mode" = 33554432;
This causes the v$undostat.tuned_undoretention to be calculated as
the maximum of:
maxquerylen secs + 300
undo_retention specified in init.ora

Please note: The above is a summary description only. Actual symptoms can vary. Matching to any symptoms here does not confirm that you are encountering this problem. For questions about this bug please consult Oracle Support.

References
Bug:5387030 (This link will only work for PUBLISHED bugs)

Note:245840.1 Information on the sections in this article
我们的数据库版本为10.2.0.5.0满足其Versions >= 10.2.0.1 but BELOW 11.1的条件。对于固定UNDO 表空间，将会通过表空间的剩余空间来最大限度保留UNDO 信息。如果FIXED UNDO 表空间没有对保留时间作GUARANTEE （alter tablespace xxx retention guarantee; ），则undo_retention 参数将不会起作用。（警告：如果设置UNDO 表空间为retention guarantee ，则未过期的数据不会被复写，如果表空间不够则会导致DML 操作失败或者transation 挂起）
oracle 10g有自动Automatic Undo Retention Tuning 这个特性。设置的undo_retention 参数只是一个指导值, ，Oracle 会自动调整Undo ( 会跨过undo_retention 设定的时间) 来保证不会出现Ora-1555 错误. 。通过查询V$UNDOSTAT （该视图记录4 天以内的UNDO 表空间使用情况，超过4 天可以查询DBA_HIST_UNDOSTAT 视图） 的tuned_undoretention （该字段在10G 版本才有，9I 是没有的）字段可以得到Oracle 根据事务量（如果是文件不可扩展，则会考虑剩余空间）采样后的自动计算出最佳的retenton 时间。这样对于一个事务量分布不均匀的数据库 来说, ，就会引发潜在的问题-- 在批处理的时候可能Undo 会用光， 而且这个状态将一直持续， 不会释放。
如何取消 10g 的 auto UNDO Retention Tuning ，有如下三种方法：

PS：Automatic Tuning of Undo_retention Causes Space Problems [ID 420525.1] ： Automatic Tuning of Undo_retention Causes Space Problems

1.) Set the autoextend and maxsize attribute of each datafile in the undo ts so it is autoextensible and its maxsize is equal to its current size so the undo tablespace now has the autoextend attribute but does not autoend:

SQL> alter database datafile '<datafile_flename>'

autoextend on maxsize <current_size>;

With this setting, v$undostat.tuned_undoretention is not calculated based on a percentage of the undo tablespace size, instead v$undostat.tuned_undoretention is set to the maximum of (maxquerylen secs + 300) undo_retention specified in init.ora file.

2.) Set the following hidden parameter in init.ora file:

_smu_debug_mode=33554432

or

SQL> Alter system set "_smu_debug_mode" = 33554432;

With this setting, v$undostat.tuned_undoretention is not calculated based on a percentage of the fixed size undo tablespace, instead v$undostat.tuned_undoretention is set to the maximum of (maxquerylen secs + 300) undo_retention specified in init.ora file.

3.) Set the following hidden parameter in init.ora:

_undo_autotune = false

or

SQL> Alter system set "_undo_autotune" = false; 可动态调整无需重启数据库。
Autotune of undo retention is turned off.
Wed Jan 30 20:59:26 GMT+08:00 2013ALTER SYSTEM SET _undo_autotune=FALSE SCOPE=BOTH;
SQL> show parameter undo

NAME TYPE VALUE
------------------- ----------------- ----------
_undo_autotune boolean FALSE
undo_management string AUTO
undo_retention integer 900
undo_tablespace string UNDOTBS1
在设定完_undo_autotune后，并且结果应用方跑存储过程，未发现ORA-30036: unable to extend segment by 8 in undo tablespace 'UNDOTBS1'错误（此错误并没有在alert日志中体现。）