Oracle RAC Database 11.1.0.6监听故障案例

晚上接到电话，客户的一套核心Oracle RAC数据库连接不上，连接时报无监听程序，客户的Oracle RAC版本为11.1.0.6，平台为AIX 6.1.05，使用了IBM
HACMP 5.5.0.8。

当我远程过去的时候，发现节点2已经没有任何oracle用户的进程，且concurrent的vg没有激活，HACMP的服务也offline。

另一个节点Oracle的实例是正常的，且有部分服务器进程依然在工作，但是本地监听器出现了故障，导致新的连接无法连接到实例，通过crs_stat -t看到两个实例的监听也都是OFFLINE状态。

在节点上并没有发现有LISTENER进程，且手动杀掉了所有的服务器进程，在oracle用户下启动监听时收到以下的报错：

$ lsnrctl start listener_cdfy740a

LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:09:09

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production

System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora

Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))

Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))

TNS-12542: TNS:address already in use

TNS-12560: TNS:protocol adapter error

TNS-00512: Address already in use

IBM/AIX RISC System/6000 Error: 67: Address already in use

Listener failed to start. See the error message(s) above...

10.107.64.1是该节点的vip地址，下面是RAC环境的hosts配置：

10.107.64.1 vip1

10.107.64.2 vip2

10.107.64.3 cdfy740a

10.107.64.4 cdfy740b

172.201.201.1 prv1

172.201.201.2 prv2

手动停掉该节点的nodeapps服务：

cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl stop nodeapps -n cdfy740a

成功停止后，VIP在主机层面已经消失：

cdfy740a@root[/oracle/app/11.1.0/crs/bin]ifconfig -a | more

en0: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>

inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255

tcp_sendspace 131072 tcp_recvspace
10e83
65536 rfc1323 0

en1: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>

inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255

tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0

en4: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>

inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255

tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0

lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>

inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255

inet6 ::1%1/0

tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

再次启动节点nodeapps服务：

cdfy740a@root[/oracle/app/11.1.0/crs/bin]./srvctl start nodeapps -n cdfy740a

CRS-1006: No more members to consider

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:13:07

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Copyright (c) 1991, 2007, Oracle. All rights reserved.

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12542: TNS:address already in use

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-00512: Address already in use

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: IBM/AIX RISC System/6000 Error: 67: Address already in use

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Listener failed to start. See the error message(s) above...

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:13:08

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Copyright (c) 1991, 2007, Oracle. All rights reserved.

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=vip1)(PORT=1521)))

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-00511: No listener

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: IBM/AIX RISC System/6000 Error: 79: Connection refused

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-00511: No listener

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: IBM/AIX RISC System/6000 Error: 79: Connection refused

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.3)(PORT=1521)(IP=FIRST)))

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr:TNS-12541: TNS:no listener

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-12560: TNS:protocol adapter error

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: TNS-00511: No listener

cdfy740a:ora.cdfy740a.LISTENER_CDFY740A.lsnr: IBM/AIX RISC System/6000 Error: 79: Connection refused

CRS-0215: Could not start resource 'ora.cdfy740a.LISTENER_CDFY740A.lsnr'.

之前使用lsnrctl status listener_cdfy740a查看监听器状态时也收到Connection refused的错误。

查看主机层面已经成功绑定了VIP地址：

cdfy740a@root[/oracle/app/11.1.0/crs/bin]ifconfig
-a | more

en0: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>

inet 172.200.200.1 netmask 0xffffff00 broadcast 172.200.200.255

tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0

en1: flags=1e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),LARGESEND,CHAIN>

inet 172.201.201.1 netmask 0xffffff00 broadcast 172.201.201.255

tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0

en4: flags=5e080863,c0<UP,BROADCAST,NOTRAILERS,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,CHECKSUM_OFFLOAD(ACTIVE),PSEG,LARGESEND,CHAIN>

inet 10.107.64.3 netmask 0xffffff00 broadcast 10.107.64.255

inet 10.107.64.1 netmask 0xffffff00 broadcast 10.107.64.255

tcp_sendspace 131072 tcp_recvspace 65536 rfc1323 0

lo0: flags=e08084b,c0<UP,BROADCAST,LOOPBACK,RUNNING,SIMPLEX,MULTICAST,GROUPRT,64BIT,LARGESEND,CHAIN>

inet 127.0.0.1 netmask 0xff000000 broadcast 127.255.255.255

inet6 ::1%1/0

tcp_sendspace 131072 tcp_recvspace 131072 rfc1323 1

再次尝试手动启动本地监听器：

cdfy740a@root[/]su
- oracle

$ lsnrctl start listener_cdfy740a

LSNRCTL for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production on 20-NOV-2014 20:18:37

Copyright (c) 1991, 2007, Oracle. All rights reserved.

Starting /oracle/app/oracle/product/11.1.0/db_1/bin/tnslsnr: please wait...

TNSLSNR for IBM/AIX RISC System/6000: Version 11.1.0.6.0 - Production

System parameter file is /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora

Log messages written to /oracle/app/oracle/diag/tnslsnr/cdfy740a/listener_cdfy740a/alert/log.xml

Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.107.64.1)(PORT=1521)))

Error listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=10.107.64.1)(PORT=1521)(IP=FIRST)))

TNS-12542: TNS:address already in use

TNS-12560: TNS:protocol adapter error

TNS-00512: Address already in use

IBM/AIX RISC System/6000 Error: 67: Address already in use

Listener failed to start. See the error message(s) above...

启动依然失败。

检查监听器配置文件：

$ cat listener.ora

# listener.ora.cdfy740a Network Configuration File: /oracle/app/oracle/product/11.1.0/db_1/network/admin/listener.ora.cdfy740a

# Generated by Oracle configuration tools.

LISTENER_CDFY740A =

(DESCRIPTION_LIST =

(DESCRIPTION =

(ADDRESS = (PROTOCOL = TCP)(HOST = vip1)(PORT = 1521))

(ADDRESS = (PROTOCOL = TCP)(HOST = 10.107.64.1)(PORT = 1521)(IP = FIRST))

(ADDRESS = (PROTOCOL = TCP)(HOST = 10.107.64.3)(PORT = 1521)(IP = FIRST))

)

)

在监听配置文件中，vip1和10.107.64.1是两个重复的地址，手动将10.107.64.1所在行去掉之后，监听即可正常的启动。

之后恢复节点2的HACMP服务，Oracle RAC随即恢复正常。

另外，还发现客户的监听日志已经被填得很大，大概在1.6GB左右，过大的监听日志文件也会导致监听器不稳定，这里将两个节点的监听日志进行了重命名操作。

《10g RAC监听器配置文件listener.ora中的IP=FIRST》：http://blog.itpub.net/23135684/viewspace-715967/

《IP=FIRST作用说明》：http://www.xifenfei.com/2713.html

--end--