12C RAC环境节点宕机问题
2017-09-07 08:56
1021 查看
接到用户报障,4节点RAC中有两个节点一个月前就无法正常启动。
一、问题现象
1、现场检查发现4个节点服务器中有两台(node2,node4)处于离线状态。
2、在node2 和node4上检查发现无crs相关进程,oracle crs未启动。
3、在node2和node4使用root 执行crsctl start has时无法启动成功。
4、节点间网络访问正常。
二、处理过程
1、node2上准备使用crsctl start res ora.crsd -init命令启用HAS 时有如下报错
[root@cloud4 bin]# ./crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crf' on 'cloud4'
CRS-2672: Attempting to start 'ora.cssd' on 'cloud4'
CRS-2672: Attempting to start 'ora.diskmon' on 'cloud4'
CRS-2676: Start of 'ora.diskmon' on 'cloud4' succeeded
CRS-2676: Start of 'ora.crf' on 'cloud4' succeeded
CRS-2676: Start of 'ora.cssd' on 'cloud4' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'cloud4'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'cloud4'
CRS-2676: Start of 'ora.ctssd' on 'cloud4' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'cloud4' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'cloud4'
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
CRS-2681: Clean of 'ora.asm' on 'cloud4' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'cloud4'
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/cloud4/crs/trace/ohasd_oraagent_grid.trc".
CRS-2674: Start of 'ora.asm' on 'cloud4' failed
CRS-2679: Attempting to clean 'ora.asm' on 'cloud4'
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
CRS-2681: Clean of 'ora.asm' on 'cloud4' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'cloud4'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'cloud4' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'cloud4'
CRS-2677: Stop of 'ora.ctssd' on 'cloud4' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'cloud4'
CRS-2677: Stop of 'ora.cssd' on 'cloud4' succeeded
CRS-4000: Command Start failed, or completed with errors.ORA-09925: Unable to create audit trail file
应该是文件权限问题,无法正常创建audit file。
2、在grid用户下检查$GRID_HOME/rdbms/aduit 目录权限发现,改目录的属主为root,属组也为root,且只有rx-r-----的权限
3、检查发现grid安装目录和oracle 安装目录很多目录的属主和属组错乱,对于grid 的权限错误可以使用root.sh来修复也可以在正常节点把
如在node1上拷贝对应文件权限
getfacl /u01/app/12.1.0/grid > grid.ora在异常节点做如下操作,恢复用户权限
set –restore grid.ora
4、按照上述检查方法检查node4 各目录的权限,然后进行修复。
5、在node2和node4上使用crsctl start res ora.crsd -init启动服务。
三、问题原因
oracle GI和oracle DATABASE安装完毕后最好不要随意对目录的权限进行调整,调整后会导致cluster或db启动不了,也可能导致其他各种各样的问题。
本例中用户对cluster的安装目录进行了更改属主属组的操作导致cluster无法启动。
四、管理建议
oracle 安装完毕后主要涉及几个用户
root,HA高可用性管理用户
grid,ASM管理用户
oracle,DB管理A用户
加强对用户的权限控制无关人员禁止使用root登陆,对安装目录不要自建目录和修改目录权限。
一、问题现象
1、现场检查发现4个节点服务器中有两台(node2,node4)处于离线状态。
2、在node2 和node4上检查发现无crs相关进程,oracle crs未启动。
3、在node2和node4使用root 执行crsctl start has时无法启动成功。
4、节点间网络访问正常。
二、处理过程
1、node2上准备使用crsctl start res ora.crsd -init命令启用HAS 时有如下报错
[root@cloud4 bin]# ./crsctl start res ora.crsd -init
CRS-2672: Attempting to start 'ora.crf' on 'cloud4'
CRS-2672: Attempting to start 'ora.cssd' on 'cloud4'
CRS-2672: Attempting to start 'ora.diskmon' on 'cloud4'
CRS-2676: Start of 'ora.diskmon' on 'cloud4' succeeded
CRS-2676: Start of 'ora.crf' on 'cloud4' succeeded
CRS-2676: Start of 'ora.cssd' on 'cloud4' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'cloud4'
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'cloud4'
CRS-2676: Start of 'ora.ctssd' on 'cloud4' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'cloud4' succeeded
CRS-2679: Attempting to clean 'ora.asm' on 'cloud4'
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
CRS-2681: Clean of 'ora.asm' on 'cloud4' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'cloud4'
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
CRS-5017: The resource action "ora.asm start" encountered the following error:
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
. For details refer to "(:CLSN00107:)" in "/u01/app/grid/diag/crs/cloud4/crs/trace/ohasd_oraagent_grid.trc".
CRS-2674: Start of 'ora.asm' on 'cloud4' failed
CRS-2679: Attempting to clean 'ora.asm' on 'cloud4'
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
ORA-09925: Unable to create audit trail file
Linux-x86_64 Error: 13: Permission denied
Additional information: 9925
CRS-2681: Clean of 'ora.asm' on 'cloud4' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'cloud4'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'cloud4' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'cloud4'
CRS-2677: Stop of 'ora.ctssd' on 'cloud4' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'cloud4'
CRS-2677: Stop of 'ora.cssd' on 'cloud4' succeeded
CRS-4000: Command Start failed, or completed with errors.ORA-09925: Unable to create audit trail file
应该是文件权限问题,无法正常创建audit file。
2、在grid用户下检查$GRID_HOME/rdbms/aduit 目录权限发现,改目录的属主为root,属组也为root,且只有rx-r-----的权限
3、检查发现grid安装目录和oracle 安装目录很多目录的属主和属组错乱,对于grid 的权限错误可以使用root.sh来修复也可以在正常节点把
如在node1上拷贝对应文件权限
getfacl /u01/app/12.1.0/grid > grid.ora在异常节点做如下操作,恢复用户权限
set –restore grid.ora
4、按照上述检查方法检查node4 各目录的权限,然后进行修复。
5、在node2和node4上使用crsctl start res ora.crsd -init启动服务。
三、问题原因
oracle GI和oracle DATABASE安装完毕后最好不要随意对目录的权限进行调整,调整后会导致cluster或db启动不了,也可能导致其他各种各样的问题。
本例中用户对cluster的安装目录进行了更改属主属组的操作导致cluster无法启动。
四、管理建议
oracle 安装完毕后主要涉及几个用户
root,HA高可用性管理用户
grid,ASM管理用户
oracle,DB管理A用户
加强对用户的权限控制无关人员禁止使用root登陆,对安装目录不要自建目录和修改目录权限。
相关文章推荐
- RAC环境下,创建spfile导致节点间数据库参数不一致的问题
- 11gR2 RAC启用iptables导致节点宕机问题处理
- 如何诊断RAC环境中节点重启问题(适用于10gR2和11gR1)
- 11gR2 RAC启用iptables导致节点宕机问题处理
- 如何诊断RAC环境中节点重启问题(适用于11gR2)
- 12c-rac 安装遇到的告警问题(SCAN)
- RedHat 5.6_x86_64 + ASM + RAW+ Oracle 10g RAC (七) 添加节点—初始化环境
- Hadoop集群环境下HBase集群slave节点总是掉问题
- RAC处理节点运行问题
- Oracle 10203 RAC环境添加新节点(二)
- 【翻译自mos文章】12c rac中,当脑裂发生时,哪个节点会幸存下来?
- RAC增加节点遇到的一点问题
- 记又一次解决生产环境宕机问题(业务系统)
- RAC节点服务ora.rac2.gsd的offline问题解决方法
- Oracle 11.2.0.4.0 RAC下DRM导致单节点宕机
- 如何诊断rac环境下sysdate 返回错误时间问题
- Oracle_12c_RAC_service_died问题分析处理
- RAC 环境中 gc block lost 和私网通信性能问题的诊断
- rac节点频繁重启的问题分析
- ElasticSearch集群未连接 无法发现节点(windows环境)以及windows环境下设置服务 不能自动启动的问题