rac问题思考总结

[root@m1 ~]# su - grid

[grid@m1 ~]$ ps -ef | grep ora_

grid 6385 6355 0 16:27 pts/2 00:00:00 grep ora_

[grid@m1 ~]$ crs_start -all

CRS-5702: Resource 'ora.DATA.dg' is already running on 'm1'

CRS-5702: Resource 'ora.FRA.dg' is already running on 'm1'

CRS-5702: Resource 'ora.OCRVOTING.dg' is already running on 'm1'

CRS-5702: Resource 'ora.asm' is already running on 'm1'

CRS-5702: Resource 'ora.cvu' is already running on 'm1'

CRS-2501: Resource 'ora.gsd' is disabled

CRS-5702: Resource 'ora.asm' is already running on 'm1'

CRS-2501: Resource 'ora.gsd' is disabled

CRS-5702: Resource 'ora.ons' is already running on 'm1'

CRS-5702: Resource 'ora.asm' is already running on 'm2'

CRS-2501: Resource 'ora.gsd' is disabled

CRS-5702: Resource 'ora.ons' is already running on 'm2'

CRS-5702: Resource 'ora.net1.network' is already running on 'm1'

CRS-5702: Resource 'ora.ons' is already running on 'm1'

Attempting to start `ora.m2.vip` on member `m2`

Attempting to start `ora.scan1.vip` on member `m1`

Attempting to start `ora.m1.vip` on member `m1`

Attempting to start `ora.m3.vip` on member `m2`

Attempting to start `ora.oc4j` on member `m1`

Start of `ora.scan1.vip` on member `m1` succeeded.

Start of `ora.m3.vip` on member `m2` succeeded.

Attempting to start `ora.LISTENER_SCAN1.lsnr` on member `m1`

Start of `ora.m1.vip` on member `m1` succeeded.

Attempting to start `ora.MACLEAN_LISTENER.lsnr` on member `m1`

Attempting to start `ora.LISTENER.lsnr` on member `m1`

Start of `ora.m2.vip` on member `m2` succeeded.

Attempting to start `ora.ASM_LISTENER.lsnr` on member `m2`

Attempting to start `ora.LISTENER.lsnr` on member `m2`

Attempting to start `ora.MACLEAN_LISTENER.lsnr` on member `m2`

Attempting to start `ora.NEW_ASM_LISTENER.lsnr` on member `m2`

Start of `ora.MACLEAN_LISTENER.lsnr` on member `m1` succeeded.

Start of `ora.LISTENER.lsnr` on member `m1` succeeded.

Start of `ora.LISTENER_SCAN1.lsnr` on member `m1` succeeded.

Start of `ora.NEW_ASM_LISTENER.lsnr` on member `m2` succeeded.

CRS-5702: Resource 'ora.NEW_ASM_LISTENER.lsnr' is already running on 'm2'

Start of `ora.ASM_LISTENER.lsnr` on member `m2` succeeded.

Start of `ora.MACLEAN_LISTENER.lsnr` on member `m2` succeeded.

CRS-5702: Resource 'ora.MACLEAN_LISTENER.lsnr' is already running on 'm2'

Start of `ora.LISTENER.lsnr` on member `m2` succeeded.

Attempting to stop `ora.mes.db` on member `m2`

CRS-5702: Resource 'ora.LISTENER.lsnr' is already running on 'm2'

Start of `ora.oc4j` on member `m1` succeeded.

Stop of `ora.mes.db` on member `m2` succeeded.

Attempting to start `ora.mes.db` on member `m2`

Start of `ora.mes.db` on member `m2` succeeded.

Attempting to start `ora.ASM_LISTENER.lsnr` on member `m1`

Attempting to start `ora.NEW_ASM_LISTENER.lsnr` on member `m1`

Start of `ora.ASM_LISTENER.lsnr` on member `m1` succeeded.

Start of `ora.NEW_ASM_LISTENER.lsnr` on member `m1` succeeded.

Attempting to start `ora.mes.db` on member `m1`

CRS-5702: Resource 'ora.NEW_ASM_LISTENER.lsnr' is already running on 'm1'

Start of `ora.mes.db` on member `m1` succeeded.

报错： 这个很正常 没有这方面的经验 说明已经启动了。

CRS-0223: Resource 'ora.DATA.dg' has placement error.

CRS-0223: Resource 'ora.FRA.dg' has placement error.

CRS-0223: Resource 'ora.LISTENER.lsnr m2 1' has placement error.

CRS-0223: Resource 'ora.MACLEAN_LISTENER.lsnr m2 1' has placement error.

CRS-0223: Resource 'ora.NEW_ASM_LISTENER.lsnr m1 1' has placement error.

CRS-0223: Resource 'ora.NEW_ASM_LISTENER.lsnr m2 1' has placement error.

CRS-0223: Resource 'ora.OCRVOTING.dg' has placement error.

CRS-0223: Resource 'ora.asm' has placement error.

CRS-0223: Resource 'ora.cvu' has placement error.

CRS-2660: Resource 'ora.gsd' or all of its instances are disabled

CRS-0223: Resource 'ora.m1.ASM1.asm' has placement error.

CRS-2660: Resource 'ora.m1.gsd' or all of its instances are disabled

CRS-0223: Resource 'ora.m1.ons' has placement error.

CRS-0223: Resource 'ora.m2.ASM2.asm' has placement error.

CRS-2660: Resource 'ora.m2.gsd' or all of its instances are disabled

CRS-0223: Resource 'ora.m2.ons' has placement error.

CRS-0223: Resource 'ora.net1.network' has placement error.

CRS-0223: Resource 'ora.ons' has placement error.

[grid@m1 ~]$

[grid@m1 ~]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....ER.lsnr ora....er.type ONLINE ONLINE m1

ora.DATA.dg ora....up.type ONLINE ONLINE m1

ora.FRA.dg ora....up.type ONLINE ONLINE m1

ora....ER.lsnr ora....er.type ONLINE ONLINE m1

ora....N1.lsnr ora....er.type ONLINE ONLINE m1

ora....ER.lsnr ora....er.type ONLINE ONLINE m1

ora....ER.lsnr ora....er.type ONLINE ONLINE m1

ora....TING.dg ora....up.type ONLINE ONLINE m1

ora.asm ora.asm.type ONLINE ONLINE m1

ora.cvu ora.cvu.type ONLINE ONLINE m1

ora.gsd ora.gsd.type OFFLINE OFFLINE

ora....SM1.asm application ONLINE ONLINE m1

ora....M1.lsnr application ONLINE ONLINE m1

ora....M1.lsnr application ONLINE ONLINE m1

ora....M1.lsnr application ONLINE ONLINE m1

ora....M1.lsnr application ONLINE ONLINE m1

ora.m1.gsd application OFFLINE OFFLINE

ora.m1.ons application ONLINE ONLINE m1

ora.m1.vip ora....t1.type ONLINE ONLINE m1

ora....SM2.asm application ONLINE ONLINE m2

ora....M2.lsnr application ONLINE ONLINE m2

ora....M2.lsnr application ONLINE ONLINE m2

ora....M2.lsnr application ONLINE ONLINE m2

ora....M2.lsnr application ONLINE ONLINE m2

ora.m2.gsd application OFFLINE OFFLINE

ora.m2.ons application ONLINE ONLINE m2

ora.m2.vip ora....t1.type ONLINE ONLINE m2

ora.m3.vip ora....t1.type ONLINE ONLINE m2

ora.mes.db ora....se.type ONLINE ONLINE m2

ora....network ora....rk.type ONLINE ONLINE m1

ora.oc4j ora.oc4j.type ONLINE ONLINE m1

ora.ons ora.ons.type ONLINE ONLINE m1

ora.scan1.vip ora....ip.type ONLINE ONLINE m1

[grid@m1 ~]$ ps -ef | grep ora_

oracle 6863 1 0 16:33 ? 00:00:00 ora_pmon_mes1

oracle 6865 1 0 16:33 ? 00:00:00 ora_psp0_mes1

oracle 6867 1 1 16:33 ? 00:00:06 ora_vktm_mes1

oracle 6871 1 0 16:33 ? 00:00:00 ora_gen0_mes1

oracle 6873 1 0 16:33 ? 00:00:00 ora_diag_mes1

oracle 6875 1 0 16:33 ? 00:00:00 ora_dbrm_mes1

oracle 6877 1 0 16:33 ? 00:00:00 ora_ping_mes1

oracle 6879 1 0 16:33 ? 00:00:00 ora_acms_mes1

oracle 6881 1 0 16:33 ? 00:00:02 ora_dia0_mes1

oracle 6883 1 0 16:33 ? 00:00:00 ora_lmon_mes1

oracle 6885 1 0 16:33 ? 00:00:00 ora_lmd0_mes1

oracle 6887 1 0 16:33 ? 00:00:02 ora_lms0_mes1

oracle 6891 1 0 16:33 ? 00:00:02 ora_lms1_mes1

oracle 6895 1 0 16:33 ? 00:00:00 ora_rms0_mes1

oracle 6897 1 0 16:33 ? 00:00:00 ora_lmhb_mes1

oracle 6899 1 0 16:33 ? 00:00:00 ora_mman_mes1

oracle 6901 1 0 16:33 ? 00:00:00 ora_dbw0_mes1

oracle 6903 1 0 16:33 ? 00:00:00 ora_lgwr_mes1

oracle 6905 1 0 16:33 ? 00:00:00 ora_ckpt_mes1

oracle 6907 1 0 16:33 ? 00:00:00 ora_smon_mes1

oracle 6909 1 0 16:33 ? 00:00:00 ora_reco_mes1

oracle 6911 1 0 16:33 ? 00:00:00 ora_rbal_mes1

oracle 6913 1 0 16:33 ? 00:00:00 ora_asmb_mes1

oracle 6915 1 0 16:33 ? 00:00:00 ora_mmon_mes1

oracle 6917 1 0 16:33 ? 00:00:00 ora_mmnl_mes1

oracle 6919 1 0 16:33 ? 00:00:00 ora_d000_mes1

oracle 6923 1 0 16:33 ? 00:00:00 ora_s000_mes1

oracle 6925 1 0 16:33 ? 00:00:00 ora_mark_mes1

oracle 6935 1 0 16:33 ? 00:00:00 ora_lck0_mes1

oracle 6937 1 0 16:33 ? 00:00:00 ora_rsmn_mes1

oracle 7007 1 0 16:33 ? 00:00:00 ora_o000_mes1

oracle 7014 1 0 16:33 ? 00:00:00 ora_arc0_mes1

oracle 7016 1 0 16:33 ? 00:00:00 ora_arc1_mes1

oracle 7019 1 0 16:33 ? 00:00:00 ora_arc2_mes1

oracle 7021 1 0 16:33 ? 00:00:00 ora_arc3_mes1

oracle 7034 1 0 16:33 ? 00:00:00 ora_o001_mes1

oracle 7040 1 0 16:33 ? 00:00:00 ora_gtx0_mes1

oracle 7042 1 0 16:33 ? 00:00:00 ora_rcbg_mes1

oracle 7047 1 0 16:33 ? 00:00:00 ora_qmnc_mes1

oracle 7061 1 0 16:33 ? 00:00:00 ora_pz99_mes1

oracle 7073 1 0 16:33 ? 00:00:00 ora_cjq0_mes1

oracle 7078 1 0 16:34 ? 00:00:00 ora_q000_mes1

oracle 7080 1 0 16:34 ? 00:00:00 ora_q001_mes1

oracle 7082 1 0 16:34 ? 00:00:00 ora_q002_mes1

oracle 7090 1 0 16:34 ? 00:00:00 ora_gcr0_mes1

grid 7301 6355 0 16:38 pts/2 00:00:00 grep ora_

[grid@m1 ~]$

m2

[grid@m2 ~]$ crs_stat -t

Name Type Target State Host

------------------------------------------------------------

ora....ER.lsnr ora....er.type ONLINE ONLINE m2

ora.DATA.dg ora....up.type ONLINE ONLINE m1

ora.FRA.dg ora....up.type ONLINE ONLINE m1

ora....ER.lsnr ora....er.type ONLINE ONLINE m1

ora....N1.lsnr ora....er.type ONLINE ONLINE m1

ora....ER.lsnr ora....er.type ONLINE ONLINE m1

ora....ER.lsnr ora....er.type ONLINE ONLINE m2

ora....TING.dg ora....up.type ONLINE ONLINE m1

ora.asm ora.asm.type ONLINE ONLINE m1

ora.cvu ora.cvu.type ONLINE ONLINE m1

ora.gsd ora.gsd.type OFFLINE OFFLINE

ora....SM1.asm application ONLINE ONLINE m1

ora....M1.lsnr application OFFLINE OFFLINE

ora....M1.lsnr application ONLINE ONLINE m1

ora....M1.lsnr application ONLINE ONLINE m1

ora....M1.lsnr application OFFLINE OFFLINE

ora.m1.gsd application OFFLINE OFFLINE

ora.m1.ons application ONLINE ONLINE m1

ora.m1.vip ora....t1.type ONLINE ONLINE m1

ora....SM2.asm application ONLINE ONLINE m2

ora....M2.lsnr application ONLINE ONLINE m2

ora....M2.lsnr application ONLINE ONLINE m2

ora....M2.lsnr application ONLINE ONLINE m2

ora....M2.lsnr application ONLINE ONLINE m2

ora.m2.gsd application OFFLINE OFFLINE

ora.m2.ons application ONLINE ONLINE m2

ora.m2.vip ora....t1.type ONLINE ONLINE m2

ora.m3.vip ora....t1.type ONLINE ONLINE m2

ora.mes.db ora....se.type ONLINE OFFLINE

ora....network ora....rk.type ONLINE ONLINE m1

ora.oc4j ora.oc4j.type ONLINE ONLINE m1

ora.ons ora.ons.type ONLINE ONLINE m1

ora.scan1.vip ora....ip.type ONLINE ONLINE m1

[grid@m2 ~]$ ps -ef | grep ora_

oracle 7832 1 0 16:31 ? 00:00:00 ora_pmon_mes2

oracle 7834 1 0 16:31 ? 00:00:00 ora_psp0_mes2

oracle 7836 1 1 16:31 ? 00:00:08 ora_vktm_mes2

oracle 7840 1 0 16:31 ? 00:00:00 ora_gen0_mes2

oracle 7842 1 0 16:31 ? 00:00:00 ora_diag_mes2

oracle 7844 1 0 16:31 ? 00:00:00 ora_dbrm_mes2

oracle 7846 1 0 16:31 ? 00:00:00 ora_ping_mes2

oracle 7848 1 0 16:31 ? 00:00:00 ora_acms_mes2

oracle 7850 1 0 16:31 ? 00:00:02 ora_dia0_mes2

oracle 7852 1 0 16:31 ? 00:00:00 ora_lmon_mes2

oracle 7854 1 0 16:31 ? 00:00:01 ora_lmd0_mes2

oracle 7856 1 0 16:31 ? 00:00:02 ora_lms0_mes2

oracle 7860 1 0 16:31 ? 00:00:02 ora_lms1_mes2

oracle 7864 1 0 16:31 ? 00:00:00 ora_rms0_mes2

oracle 7866 1 0 16:31 ? 00:00:00 ora_lmhb_mes2

oracle 7868 1 0 16:31 ? 00:00:01 ora_mman_mes2

oracle 7871 1 0 16:31 ? 00:00:00 ora_dbw0_mes2

oracle 7873 1 0 16:31 ? 00:00:00 ora_lgwr_mes2

oracle 7876 1 0 16:31 ? 00:00:00 ora_ckpt_mes2

oracle 7878 1 0 16:31 ? 00:00:00 ora_smon_mes2

oracle 7880 1 0 16:31 ? 00:00:00 ora_reco_mes2

oracle 7882 1 0 16:31 ? 00:00:00 ora_rbal_mes2

oracle 7884 1 0 16:31 ? 00:00:00 ora_asmb_mes2

oracle 7886 1 0 16:31 ? 00:00:00 ora_mmon_mes2

oracle 7888 1 0 16:31 ? 00:00:00 ora_mmnl_mes2

oracle 7893 1 0 16:31 ? 00:00:00 ora_d000_mes2

oracle 7895 1 0 16:31 ? 00:00:00 ora_s000_mes2

oracle 7901 1 0 16:31 ? 00:00:00 ora_mark_mes2

oracle 7911 1 0 16:31 ? 00:00:00 ora_lck0_mes2

oracle 7914 1 0 16:31 ? 00:00:00 ora_rsmn_mes2

oracle 8018 1 0 16:32 ? 00:00:00 ora_gcr0_mes2

oracle 8038 1 0 16:32 ? 00:00:00 ora_arc0_mes2

oracle 8040 1 0 16:32 ? 00:00:00 ora_arc1_mes2

oracle 8042 1 0 16:32 ? 00:00:00 ora_arc2_mes2

oracle 8044 1 0 16:32 ? 00:00:00 ora_arc3_mes2

oracle 8072 1 0 16:33 ? 00:00:00 ora_gtx0_mes2

oracle 8074 1 0 16:33 ? 00:00:00 ora_rcbg_mes2

oracle 8078 1 0 16:33 ? 00:00:00 ora_qmnc_mes2

oracle 8098 1 0 16:33 ? 00:00:00 ora_q000_mes2

oracle 8100 1 0 16:33 ? 00:00:00 ora_q001_mes2

oracle 8123 1 0 16:33 ? 00:00:01 ora_cjq0_mes2

oracle 8150 1 0 16:33 ? 00:00:00 ora_pz99_mes2

oracle 8155 1 0 16:34 ? 00:00:00 ora_q002_mes2

oracle 8284 1 0 16:38 ? 00:00:00 ora_smco_mes2

oracle 8290 1 0 16:38 ? 00:00:00 ora_w000_mes2

oracle 8299 1 0 16:38 ? 00:00:00 ora_j000_mes2

oracle 8301 1 0 16:38 ? 00:00:00 ora_j001_mes2

grid 8324 7714 0 16:39 pts/2 00:00:00 grep ora_

[grid@m2 ~]$

第三台节点无法重新启动

[root@m3 ~]# su - grid

[grid@m3 ~]$ crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

[grid@m3 ~]$ uname -r

2.6.39-300.26.1.el5uek

[grid@m3 ~]$

1、为什么操作系统关闭之后，CRS服务不能睡着系统启动。

2、m1上 crs_start -all 但是节点m3不能启动 报错：

[root@m3 ~]# su - grid

[grid@m3 ~]$ crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

[grid@m3 ~]$ uname -r

2.6.39-300.26.1.el5uek

3、启动要报错：上面的输出？？？？？？？

4、节点3不能启动启动。

crs 起不起来

通过

sqlplus / as sysdba

startup

也不行 说明RAC完全不和单实例的一样。。。。。。。。

解决：

rm -rf /var/log/.oracle /var/log/.oracle.bak

crs_stop 停止之后 不能启动起来.

crs_start -all

[grid@m2 ~]$ crs_start -all

CRS-0184: Cannot communicate with the CRS daemon.

分析思路：

1 看日志： 在你的grid安装目录下

[root@rac1 crsd]# vi /opt/app/grid/log/rac1/crsd/crsd.log

2 init6 重新启动之后出现这个情况 不知道什么原因。

[grid@m1 ~]$ crs_stat -t

11g RAC 没有这个目录

/etc/init.d/init.crs

exit

oracle用户手动startup启动

之后是这个样子

1、 为什么不能把m1 m2 m3 都启动起来？

2、 为什么操作行启动之后，不能启动数据库。

我怀疑是 直接

crs_stp -all 的问题

但是 有的时候 crs_stop了crs_start也不能启动起来。什么原因？？

是不是环境变量。

[grid@m1 ~]$ emctl stop dbconsole

Can't locate CompEMdbconsole.pm in @INC (@INC contains: %s_javapOracleHome%/sysman/admin/scripts %s_javapOracleHome%/bin /u01/app/grid/11.2.0/perl/lib/5.10.0/x86_64-linux-thread-multi /u01/app/grid/11.2.0/perl/lib/5.10.0 /u01/app/grid/11.2.0/perl/lib /u01/app/grid/11.2.0/perl/lib/site_perl/5.10.0/x86_64-linux-thread-multi /u01/app/grid/11.2.0/perl/lib/site_perl/5.10.0 /u01/app/grid/11.2.0/perl/lib/site_perl /u01/app/grid/11.2.0/perl/libwww-perl/lib /u01/app/grid/11.2.0/perl/ext/POSIX /u01/app/grid/11.2.0/perl/URI /u01/app/grid/11.2.0/perl/HTML_Parser /u01/app/grid/11.2.0/perl/HTML-Parser/lib /u01/app/grid/11.2.0/sysman/admin/scripts /u01/app/grid/11.2.0/bin /u01/app/grid/11.2.0/sysman/admin/scripts/Net-DNS-0.48/lib/ /u01/app/grid/11.2.0/sysman/admin/scripts/libnet-1.19/ .) at /u01/app/grid/11.2.0/bin/EmctlCommon.pm line 598.

[grid@m1 ~]$ emctl --help

OC4J Configuration issue. /u01/app/grid/11.2.0/sysman/j2ee not found.

[grid@m1 ~]$

如果要用OEM那么必须，启动OC4J 这个以后去折腾。

快速搭建起来 而不是以搭建为目的 搭建起来就完了 ，搭建起来之后，就要测试，搭建只是一个方面，但是能不用还是一个方面，这个周末不去玩了，就这里搞。

1、数据库重新启动之后，命令emctl stop dbconsole

2、OEM启动不起来 https://172.16.15.101:1158/em 无法显示网页。

013-11-14 09:02:42.735: [ CRSD][3216185072] ENV Logging level for Module: UiServer

2013-11-14 09:02:42.735: [ CRSMAIN][3216185072] Checking the OCR device

2013-11-14 09:02:42.736: [ CRSMAIN][3216185072] Sync-up with OCR

2013-11-14 09:02:42.736: [ CRSMAIN][3216185072] Connecting to the CSS Daemon

2013-11-14 09:02:42.739: [ CRSRTI][3216185072] CSS is not ready. Received status 3

2013-11-14 09:02:42.739: [ CRSMAIN][3216185072] Created alert : (:CRSD00109:) : Could not init the CSS context, error: 3

2013-11-14 09:02:42.739: [ CRSD][3216185072][PANIC] CRSD exiting: Could not init the CSS context, error: 3

2013-11-14 09:02:42.740: [ CRSD][3216185072] Done.

1、

ocssd.log 日志路径：

2、

crsd.log 日志路径：

-----------------出现上面的情况与意外关机init 6有关系：

以后关机：

先关闭实例，

srvctl stop

问题：

昨天的问题 是init6 之后 不启动CRS资源， 通过

mv /var/tmp/.oracle /var/tmp/.oracle.bak

int 6 之后 就启动CRS资源了，但是 crs_stop -all 之后 crs_start 就又起不起来了；

但是 init6 reboot之后 数据库mes DB 启动不起来了，但是手动启动又没有任何问题。

为什么？？？？？？？？？？？？

总结：

1、 正常关机,

2、 mv /var/tmp/。oracle /var/tmp/.oracle.bak

3、 crs_start ora.db. 不要登录进去 starup命令。

crsctl stop crs 停止掉crs集群管理软件之后，就不能手动骑起来了，不知道是怎么回事？。

但是这个情况可以通过重新启动操作系统能够解决。

重新启动之后，crs能都启动，但是还要手动启动 数据库。ora.db

1、为什么单实例 换了一个内核之后就能启动起来。