ORACLE 10G RAC 10.2.0.5 新增节点
2013-07-18 15:34
405 查看
1. 虚拟机规划 5
1.1. 新节点IP地址规划 5
1.2. 新节点裸设备规划 5
2. 安装Linux 6
2.1. 安装操作系统 6
2.1.1. 选择语言 6
2.1.2. 格式化磁盘 7
2.1.3. 修改IP地址 (不要配置DNS) 9
2.1.4. 设置时区 11
2.1.5. 选择安装组件 12
2.1.6. 开始安装 13
2.1.7. 确认安装信息 14
2.1.8. 关闭安全设置 15
2.1.9. 确认时间 16
2.1.10. 不进行新建用户 17
2.1.11. 完成安装 18
2.2. 关闭不必要的服务 19
2.3. 修改启动模式为字符界面 19
2.4. Yum安装oracle-validated 20
2.5. 确认内核的调整 21
2.6. 检查用户、组，创建安装目录 (保持与其他节点一致) 21
2.7. 修改主机名 21
2.8. 图形界面使用vnc 22
2.8.1. 第一次启动配置vnc 22
2.8.2. 停止vnc 22
2.8.3. 修改vnc图形界面为gnome模式 22
2.9. 关闭虚拟机 23
3. 配置新主机环境 24
3.1. 准备共享存储 24
3.1.1. 磁盘信息 24
3.1.2. 扫描pv 24
3.1.3. 扫描vg 24
3.1.4. 为新节点添加undo和redo文件的裸设备 25
3.1.5. 激活vg (所有节点都需要操作) 26
3.1.6. 扫描lv (所有节点) 28
3.1.7. 绑定裸设备 (所有节点都需要操作) 30
3.1.8. 修改裸设备权限 33
3.1.9. 修改启动文件使权限永久生效 34
3.2. 准备ssh认证 34
3.2.1. 修改已有主机的hosts文件 (nmacct1, nmacct2) 34
3.2.2. 新主机用户执行ssh-keygen 35
3.2.3. 在已有节点进行配置authorized_keys文件 35
3.2.4. 复制authorized_keys文件到远程节点 36
3.2.5. 修改新增节点目录、文件属性 36
3.2.6. 在所有节点测试ssh认证 36
3.3. 配置安装目录及环境变量 38
3.3.1. 配置新节点安装目录 38
3.3.2. 配置新节点环境变量 38
4. 环境备份 40
4.1. $ORACLE_HOME备份 40
4.2. $CRS_HOME备份 40
4.3. OCR备份 40
4.4. VotingDisk备份 41
4.5. 主机脚本备份 41
5. 执行安装前检测 43
5.1. 操作系统及硬件检查 43
5.2. CRS安装检查 44
6. 向新节点安装CRS软件 47
6.1. 启动vnc服务 47
6.2. 在原有节点(nmacct1)执行$CRS_HOME/oui/bin/addNode.sh脚本 47
6.3. 选择需要添加的节点 49
6.4. 确认开始安装 51
6.5. 执行root脚本 (注意脚本执行节点) 53
6.6. 完成新节点CRS安装 55
6.7. 查看CRS状态 55
6.8. 向新节点添加ONS远程端口 56
6.8.1. 查看当前集群中ONS_HOSTS配置 56
6.8.2. 查看当前ONS_HOSTS远程端口配置 56
6.8.3. 添加ONS端口 57
7. 向新节点安装Database软件 58
7.1. 在原有节点(nmacct1)执行$ORACLE_HOME/oui/bin/addNode.sh脚本 58
7.2. 选择需要添加的节点 59
7.3. 确认开始安装 61
7.4. 新节点执行root脚本 62
7.5. 完成新节点Database软件安装 63
8. 在新节点使用netca配置监听 64
8.1. 启动netca 64
8.2. 只选择新增节点 (nmacct3) 65
8.3. 进行监听配置 65
8.4. 完成监听配置 68
8.5. 查看集群中监听状态 70
9. 向新节点添加数据库实例 71
9.1. 在原有节点(nmacct1)执行dbca命令 71
9.2. 选择实例管理 72
9.3. 选择添加实例 72
9.4. 选择需要添加实例的数据库，输入用户和口令 73
9.5. 选择实例名和节点 74
9.6. 选择数据文件 74
9.7. 确认安装信息 76
9.8. 查询集群信息 77
10. 回退方案 79
192.168.56.111 nmacct1
192.168.56.112 nmacct1-vip
192.168.56.121 nmacct2
192.168.56.122 nmacct2-vip
192.168.56.131 nmacct3
192.168.56.132 nmacct3-vip
10.10.10.11 nmacct1-priv
10.10.10.12 nmacct2-priv
10.10.10.13 nmacct3-priv
表空间
占用空间
VG
裸设备
undotbs3
500M
data_vg
/dev/raw/raw29
redo3_1
100M
data_vg
/dev/raw/raw30
redo3_2
100M
data_vg
/dev/raw/raw31
redo3_3
100M
data_vg
/dev/raw/raw32
redo3_4
100M
data_vg
/dev/raw/raw33
2.1.2. 格式化磁盘
2.1.3. 修改IP地址 (不要配置DNS)
2.1.4. 设置时区
2.1.5. 选择安装组件
2.1.6. 开始安装
2.1.7. 确认安装信息
2.1.8. 关闭安全设置
2.1.9. 确认时间
2.1.10. 不进行新建用户
2.1.11. 完成安装
chkconfig --level 2345 cups off
chkconfig --level 2345 ip6tables off
chkconfig --level 2345 iptables off
chkconfig --level 2345 irqbalance off
chkconfig --level 2345 pcscd off
chkconfig --level 2345 anacron off
chkconfig --level 2345 atd off
chkconfig --level 2345 auditd off
chkconfig --level 2345 avahi-daemon off
chkconfig --level 2345 avahi-dnsconfd off
chkconfig --level 2345 cpuspeed off
chkconfig --level 2345 gpm off
chkconfig --level 2345 hidd off
chkconfig --level 2345 mcstrans off
chkconfig --level 2345 microcode_ctl off
chkconfig --level 2345 netfs off
chkconfig --level 2345 nfslock off
chkconfig --level 2345 portmap off
chkconfig --level 2345 readahead_early off
chkconfig --level 2345 readahead_later off
chkconfig --level 2345 restorecond off
chkconfig --level 2345 rpcgssd off
chkconfig --level 2345 rhnsd off
chkconfig --level 2345 rpcidmapd off
chkconfig --level 2345 sendmail off
chkconfig --level 2345 setroubleshoot off
chkconfig --level 2345 smartd off
chkconfig --level 2345 xinetd off
#
# inittab This file describes how the INIT process should set up
# the system in a certain run-level.
#
# Author: Miquel van Smoorenburg, <miquels@drinkel.nl.mugnet.org>
# Modified for RHS Linux by Marc Ewing and Donnie Barnes
#
# Default runlevel. The runlevels used by RHS are:
# 0 - halt (Do NOT set initdefault to this)
# 1 - Single user mode
# 2 - Multiuser, without NFS (The same as 3, if you do not have networking)
# 3 - Full multiuser mode
# 4 - unused
# 5 - X11
# 6 - reboot (Do NOT set initdefault to this)
#
id:3:initdefault:
修改系统为字符模式。
[root@nmacct3~]# mount /dev/cdrom /media/disk
mount: block device /dev/cdrom is write-protected, mounting read-only
[root@nmacct3~]# cp /etc/yum.repos.d/public-yum-el5.repo /etc/yum.repos.d/public-yum-el5.repo.bak
[root@nmacct3~]# vi /etc/yum.repos.d/public-yum-el5.repo
[oel5]
name = Enterprise Linux 5.9 DVD
baseurl=file:///media/disk/Server/
gpgcheck=0
enabled=1
[root@nmacct3~]# yum install oracle-validated
[root@nmacct3 ~]# umount /dev/cdrom
[root@nmacct3~]# more /etc/security/limits.conf
uid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba)
[root@nmacct3 ~]# passwd oracle
Changing password for user oracle.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[root@nmacct3~]# mkdir /oracle
[root@nmacct3~]# chown -R oracle:oinstall /oracle
127.0.0.1 localhost
192.168.56.111 nmacct1
192.168.56.112 nmacct1-vip
192.168.56.121 nmacct2
192.168.56.122 nmacct2-vip
192.168.56.131 nmacct3
192.168.56.132 nmacct3-vip
10.10.10.11 nmacct1-priv
10.10.10.12 nmacct2-priv
10.10.10.13 nmacct3-priv
You will require a password to access your desktops.
Password:
Verify:
New 'nmacct2:1 (oracle)' desktop is nmacct2:1
Creating default startup script /home/oracle/.vnc/xstartup
Starting applications specified in /home/oracle/.vnc/xstartup
Log file is /home/oracle/.vnc/nmacct2:1.log
2.8.2. 停止vnc[oracle@nmacct3~]$ vncserver -kill :1
Killing Xvnc process ID 16056
2.8.3. 修改vnc图形界面为gnome模式[oracle@nmacct3~]$ cd .vnc
[oracle@nmacct3.vnc]$ vi xstartup
#!/bin/sh
# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc
[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
gnome-session &
最后一行，修改为gnome-session，重启vnc生效。
brw-r----- 1 root disk 8, 0 Jul 17 16:10 /dev/sda
brw-r----- 1 root disk 8, 1 Jul 17 16:11 /dev/sda1
brw-r----- 1 root disk 8, 2 Jul 17 16:10 /dev/sda2
brw-r----- 1 root disk 8, 16 Jul 17 16:10 /dev/sdb
brw-r----- 1 root disk 8, 32 Jul 17 16:10 /dev/sdc
brw-r----- 1 root disk 8, 48 Jul 17 16:10 /dev/sdd
3.1.2. 扫描pv[root@nmacct3 ~]# pvscan
PV /dev/sdd VG share_vg lvm2 [2.00 GB / 1.02 GB free]
PV /dev/sdc VG data_vg lvm2 [5.00 GB / 1020.00 MB free]
PV /dev/sdb VG system_vg lvm2 [5.00 GB / 392.00 MB free]
PV /dev/sda2 VG VolGroup00 lvm2 [19.88 GB / 0 free]
Total: 4 [31.86 GB] / in use: 4 [31.86 GB] / in no VG: 0 [0 ]
[root@nmacct3 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 VolGroup00 lvm2 a-- 19.88G 0
/dev/sdb system_vg lvm2 a-- 5.00G 392.00M
/dev/sdc data_vg lvm2 a-- 5.00G 1020.00M
/dev/sdd share_vg lvm2 a-- 2.00G 1.02G
3.1.3. 扫描vg[root@nmacct3 ~]# vgscan
Reading all physical volumes. This may take a while...
Found volume group "share_vg" using metadata type lvm2
Found volume group "data_vg" using metadata type lvm2
Found volume group "system_vg" using metadata type lvm2
Found volume group "VolGroup00" using metadata type lvm2
[root@nmacct3 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
VolGroup00 1 2 0 wz--n- 19.88G 0
data_vg 1 4 0 wz--n- 5.00G 1020.00M
share_vg 1 5 0 wz--n- 2.00G 1.02G
system_vg 1 19 0 wz--n- 5.00G 392.00M
3.1.4. 为新节点添加undo和redo文件的裸设备? 新节点空间要求表空间
占用空间
VG
裸设备
undotbs3
500M
data_vg
/dev/raw/raw29
redo3_1
100M
data_vg
/dev/raw/raw30
redo3_2
100M
data_vg
/dev/raw/raw31
redo3_3
100M
data_vg
/dev/raw/raw32
redo3_4
100M
data_vg
/dev/raw/raw33
? 存储空间检查[root@nmacct3 ~]# vgdisplay data_vg
--- Volume group ---
VG Name data_vg
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 4
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 5.00 GB
PE Size 4.00 MB
Total PE 1279
Alloc PE / Size 1024 / 4.00 GB
Free PE / Size 255 / 1020.00 MB
VG UUID Oo5yzk-v3BZ-OdeY-32Qj-kHd1-1xUK-6wfFe0
? 新增undolvcreate -L 500m -n undotbs3 data_vg
? 新增redolvcreate -L 100m -n redo3_1 data_vg
lvcreate -L 100m -n redo3_2 data_vg
lvcreate -L 100m -n redo3_3 data_vg
lvcreate -L 100m -n redo3_4 data_vg
3.1.5. 激活vg (所有节点都需要操作)[root@nmacct2 ~]# lvscan
ACTIVE '/dev/share_vg/ocr1' [200.00 MB] inherit
ACTIVE '/dev/share_vg/ocr2' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting1' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting2' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting3' [200.00 MB] inherit
ACTIVE '/dev/data_vg/acct_tbs_01' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_tbs_02' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_index_01' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_index_02' [1.00 GB] inherit
inactive '/dev/data_vg/undotbs3' [500.00 MB] inherit
inactive '/dev/data_vg/redo3_1' [100.00 MB] inherit
inactive '/dev/data_vg/redo3_2' [100.00 MB] inherit
inactive '/dev/data_vg/redo3_3' [100.00 MB] inherit
inactive '/dev/data_vg/redo3_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/system' [1000.00 MB] inherit
ACTIVE '/dev/system_vg/sysaux' [1000.00 MB] inherit
ACTIVE '/dev/system_vg/undotbs1' [500.00 MB] inherit
ACTIVE '/dev/system_vg/undotbs2' [500.00 MB] inherit
ACTIVE '/dev/system_vg/temp' [500.00 MB] inherit
ACTIVE '/dev/system_vg/control1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/control2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/control3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/users' [100.00 MB] inherit
ACTIVE '/dev/system_vg/spfile' [12.00 MB] inherit
ACTIVE '/dev/system_vg/pwdfile' [12.00 MB] inherit
ACTIVE '/dev/VolGroup00/LogVol00' [15.53 GB] inherit
ACTIVE '/dev/VolGroup00/LogVol01' [4.34 GB] inherit
[root@nmacct3 ~]# vgchange -ay
5 logical volume(s) in volume group "share_vg" now active
4 logical volume(s) in volume group "data_vg" now active
19 logical volume(s) in volume group "system_vg" now active
2 logical volume(s) in volume group "VolGroup00" now active
3.1.6. 扫描lv (所有节点)[root@nmacct2 ~]# lvscan
ACTIVE '/dev/share_vg/ocr1' [200.00 MB] inherit
ACTIVE '/dev/share_vg/ocr2' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting1' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting2' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting3' [200.00 MB] inherit
ACTIVE '/dev/data_vg/acct_tbs_01' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_tbs_02' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_index_01' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_index_02' [1.00 GB] inherit
ACTIVE '/dev/data_vg/undotbs3' [500.00 MB] inherit
ACTIVE '/dev/data_vg/redo3_1' [100.00 MB] inherit
ACTIVE '/dev/data_vg/redo3_2' [100.00 MB] inherit
ACTIVE '/dev/data_vg/redo3_3' [100.00 MB] inherit
ACTIVE '/dev/data_vg/redo3_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/system' [1000.00 MB] inherit
ACTIVE '/dev/system_vg/sysaux' [1000.00 MB] inherit
ACTIVE '/dev/system_vg/undotbs1' [500.00 MB] inherit
ACTIVE '/dev/system_vg/undotbs2' [500.00 MB] inherit
ACTIVE '/dev/system_vg/temp' [500.00 MB] inherit
ACTIVE '/dev/system_vg/control1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/control2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/control3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/users' [100.00 MB] inherit
ACTIVE '/dev/system_vg/spfile' [12.00 MB] inherit
ACTIVE '/dev/system_vg/pwdfile' [12.00 MB] inherit
ACTIVE '/dev/VolGroup00/LogVol00' [15.53 GB] inherit
ACTIVE '/dev/VolGroup00/LogVol01' [4.34 GB] inherit
[root@nmacct2 ~]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
LogVol00 VolGroup00 -wi-ao 15.53G
LogVol01 VolGroup00 -wi-ao 4.34G
acct_index_01 data_vg -wi-ao 1.00G
acct_index_02 data_vg -wi-a- 1.00G
acct_tbs_01 data_vg -wi-ao 1.00G
acct_tbs_02 data_vg -wi-a- 1.00G
redo3_1 data_vg -wi-a- 100.00M
redo3_2 data_vg -wi-a- 100.00M
redo3_3 data_vg -wi-a- 100.00M
redo3_4 data_vg -wi-a- 100.00M
undotbs3 data_vg -wi-a- 500.00M
ocr1 share_vg -wi-ao 200.00M
ocr2 share_vg -wi-ao 200.00M
voting1 share_vg -wi-ao 200.00M
voting2 share_vg -wi-ao 200.00M
voting3 share_vg -wi-ao 200.00M
control1 system_vg -wi-ao 100.00M
control2 system_vg -wi-ao 100.00M
control3 system_vg -wi-ao 100.00M
pwdfile system_vg -wi-a- 12.00M
redo1_1 system_vg -wi-ao 100.00M
redo1_2 system_vg -wi-ao 100.00M
redo1_3 system_vg -wi-ao 100.00M
redo1_4 system_vg -wi-ao 100.00M
redo2_1 system_vg -wi-ao 100.00M
redo2_2 system_vg -wi-ao 100.00M
redo2_3 system_vg -wi-ao 100.00M
redo2_4 system_vg -wi-ao 100.00M
spfile system_vg -wi-a- 12.00M
sysaux system_vg -wi-ao 1000.00M
system system_vg -wi-ao 1000.00M
temp system_vg -wi-ao 500.00M
undotbs1 system_vg -wi-ao 500.00M
undotbs2 system_vg -wi-ao 500.00M
users system_vg -wi-ao 100.00M
3.1.7. 绑定裸设备 (所有节点都需要操作)[root@nmacct3 ~]# vi /etc/sysconfig/rawdevices
# raw device bindings
# format: <rawdev> <major> <minor>
# <rawdev> <blockdev>
# example: /dev/raw/raw1 /dev/sda1
# /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/share_vg/ocr1
/dev/raw/raw2 /dev/share_vg/ocr2
/dev/raw/raw3 /dev/share_vg/voting1
/dev/raw/raw4 /dev/share_vg/voting2
/dev/raw/raw5 /dev/share_vg/voting3
/dev/raw/raw6 /dev/data_vg/acct_tbs_01
/dev/raw/raw7 /dev/data_vg/acct_tbs_02
/dev/raw/raw8 /dev/data_vg/acct_index_01
/dev/raw/raw9 /dev/data_vg/acct_index_02
/dev/raw/raw10 /dev/system_vg/system
/dev/raw/raw11 /dev/system_vg/sysaux
/dev/raw/raw12 /dev/system_vg/undotbs1
/dev/raw/raw13 /dev/system_vg/undotbs2
/dev/raw/raw14 /dev/system_vg/temp
/dev/raw/raw15 /dev/system_vg/control1
/dev/raw/raw16 /dev/system_vg/control2
/dev/raw/raw17 /dev/system_vg/control3
/dev/raw/raw18 /dev/system_vg/redo1_1
/dev/raw/raw19 /dev/system_vg/redo1_2
/dev/raw/raw20 /dev/system_vg/redo1_3
/dev/raw/raw21 /dev/system_vg/redo1_4
/dev/raw/raw22 /dev/system_vg/redo2_1
/dev/raw/raw23 /dev/system_vg/redo2_2
/dev/raw/raw24 /dev/system_vg/redo2_3
/dev/raw/raw25 /dev/system_vg/redo2_4
/dev/raw/raw26 /dev/system_vg/users
/dev/raw/raw27 /dev/system_vg/spfile
/dev/raw/raw28 /dev/system_vg/pwdfile
/dev/raw/raw29 /dev/data_vg/undotbs3
/dev/raw/raw30 /dev/data_vg/redo3_1
/dev/raw/raw31 /dev/data_vg/redo3_2
/dev/raw/raw32 /dev/data_vg/redo3_3
/dev/raw/raw33 /dev/data_vg/redo3_4
[root@nmacct3 ~]# service rawdevices restart
Assigning devices:
/dev/raw/raw1 --> /dev/share_vg/ocr1
/dev/raw/raw1: bound to major 252, minor 2
/dev/raw/raw2 --> /dev/share_vg/ocr2
/dev/raw/raw2: bound to major 252, minor 3
/dev/raw/raw3 --> /dev/share_vg/voting1
/dev/raw/raw3: bound to major 252, minor 4
/dev/raw/raw4 --> /dev/share_vg/voting2
/dev/raw/raw4: bound to major 252, minor 5
/dev/raw/raw5 --> /dev/share_vg/voting3
/dev/raw/raw5: bound to major 252, minor 6
/dev/raw/raw6 --> /dev/data_vg/acct_tbs_01
/dev/raw/raw6: bound to major 252, minor 7
/dev/raw/raw7 --> /dev/data_vg/acct_tbs_02
/dev/raw/raw7: bound to major 252, minor 8
/dev/raw/raw8 --> /dev/data_vg/acct_index_01
/dev/raw/raw8: bound to major 252, minor 9
/dev/raw/raw9 --> /dev/data_vg/acct_index_02
/dev/raw/raw9: bound to major 252, minor 10
/dev/raw/raw10 --> /dev/system_vg/system
/dev/raw/raw10: bound to major 252, minor 11
/dev/raw/raw11 --> /dev/system_vg/sysaux
/dev/raw/raw11: bound to major 252, minor 12
/dev/raw/raw12 --> /dev/system_vg/undotbs1
/dev/raw/raw12: bound to major 252, minor 13
/dev/raw/raw13 --> /dev/system_vg/undotbs2
/dev/raw/raw13: bound to major 252, minor 14
/dev/raw/raw14 --> /dev/system_vg/temp
/dev/raw/raw14: bound to major 252, minor 15
/dev/raw/raw15 --> /dev/system_vg/control1
/dev/raw/raw15: bound to major 252, minor 16
/dev/raw/raw16 --> /dev/system_vg/control2
/dev/raw/raw16: bound to major 252, minor 17
/dev/raw/raw17 --> /dev/system_vg/control3
/dev/raw/raw17: bound to major 252, minor 18
/dev/raw/raw18 --> /dev/system_vg/redo1_1
/dev/raw/raw18: bound to major 252, minor 19
/dev/raw/raw19 --> /dev/system_vg/redo1_2
/dev/raw/raw19: bound to major 252, minor 20
/dev/raw/raw20 --> /dev/system_vg/redo1_3
/dev/raw/raw20: bound to major 252, minor 21
/dev/raw/raw21 --> /dev/system_vg/redo1_4
/dev/raw/raw21: bound to major 252, minor 22
/dev/raw/raw22 --> /dev/system_vg/redo2_1
/dev/raw/raw22: bound to major 252, minor 23
/dev/raw/raw23 --> /dev/system_vg/redo2_2
/dev/raw/raw23: bound to major 252, minor 24
/dev/raw/raw24 --> /dev/system_vg/redo2_3
/dev/raw/raw24: bound to major 252, minor 25
/dev/raw/raw25 --> /dev/system_vg/redo2_4
/dev/raw/raw25: bound to major 252, minor 26
/dev/raw/raw26 --> /dev/system_vg/users
/dev/raw/raw26: bound to major 252, minor 27
/dev/raw/raw27 --> /dev/system_vg/spfile
/dev/raw/raw27: bound to major 252, minor 28
/dev/raw/raw28 --> /dev/system_vg/pwdfile
/dev/raw/raw28: bound to major 252, minor 29
/dev/raw/raw29 --> /dev/data_vg/undotbs3
/dev/raw/raw29: bound to major 252, minor 30
/dev/raw/raw30 --> /dev/data_vg/redo3_1
/dev/raw/raw30: bound to major 252, minor 31
/dev/raw/raw31 --> /dev/data_vg/redo3_2
/dev/raw/raw31: bound to major 252, minor 32
/dev/raw/raw32 --> /dev/data_vg/redo3_3
/dev/raw/raw32: bound to major 252, minor 33
/dev/raw/raw33 --> /dev/data_vg/redo3_4
/dev/raw/raw33: bound to major 252, minor 34
done
3.1.8. 修改裸设备权限? 原有节点，修改新增的5个裸设备权限chown oracle:dba /dev/raw/raw29
chown oracle:dba /dev/raw/raw30
chown oracle:dba /dev/raw/raw31
chown oracle:dba /dev/raw/raw32
chown oracle:dba /dev/raw/raw33
chmod 660 /dev/raw/raw29
chmod 660 /dev/raw/raw30
chmod 660 /dev/raw/raw31
chmod 660 /dev/raw/raw32
chmod 660 /dev/raw/raw33
? 新节点，修改所有裸设备权限[root@nmacct3 raw]# chown oracle:dba /dev/raw/raw*
[root@nmacct3 raw]# chmod 660 /dev/raw/raw*
3.1.9. 修改启动文件使权限永久生效[root@nmacct3 raw]# vi /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
chown oracle:dba /dev/raw/*
chmod 660 /dev/raw/*
192.168.56.111 nmacct1
192.168.56.112 nmacct1-vip
192.168.56.121 nmacct2
192.168.56.122 nmacct2-vip
192.168.56.131 nmacct3
192.168.56.132 nmacct3-vip
10.10.10.11 nmacct1-priv
10.10.10.12 nmacct2-priv
10.10.10.13 nmacct3-priv
3.2.2. 新主机用户执行ssh-keygen[oracle@nmacct3 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
9c:a2:44:00:69:63:9f:e4:02:78:f5:0e:5f:d0:ce:02 oracle@nmacct3
[oracle@nmacct3 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
8a:30:63:3a:3e:5b:2f:26:e2:d9:12:7b:cf:6d:24:3f oracle@nmacct3
3.2.3. 在已有节点进行配置authorized_keys文件选择nmacct1主机进行操作：
[oracle@nmacct1 .ssh]$ ssh nmacct3 cat ~/.ssh/id_dsa.pub >> authorized_keys
The authenticity of host 'nmacct3 (192.168.56.131)' can't be established.
RSA key fingerprint is 65:8f:79:b4:fa:29:cb:6e:53:f4:90:dd:48:f9:81:5c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nmacct3,192.168.56.131' (RSA) to the list of known hosts.
oracle@nmacct3's password:
[oracle@nmacct1 .ssh]$ ssh nmacct3 cat ~/.ssh/id_rsa.pub >> authorized_keys
oracle@nmacct3's password:
3.2.4. 复制authorized_keys文件到远程节点[oracle@nmacct1 .ssh]$ scp authorized_keys nmacct2:~/.ssh
authorized_keys 100% 3000 2.9KB/s 00:00
[oracle@nmacct1 .ssh]$ scp authorized_keys nmacct3:~/.ssh
oracle@nmacct3's password:
authorized_keys 100% 3000 2.9KB/s 00:00
3.2.5. 修改新增节点目录、文件属性[oracle@nmacct3 ~]$ chmod 700 ~/.ssh
[oracle@nmacct3 ~]$ cd .ssh
[oracle@nmacct3 .ssh]$ chmod 600 authorized_keys
3.2.6. 在所有节点测试ssh认证[oracle@nmacct1 ~]$ vi time_validate.sh
date; ssh nmacct1 date;
date; ssh nmacct1-priv date;
date; ssh nmacct2 date;
date; ssh nmacct2-priv date;
date; ssh nmacct3 date;
date; ssh nmacct3-priv date;
[oracle@nmacct1 ~]$ chmod 700 time_validate.sh
[oracle@nmacct1 ~]$ ./time_validate.sh
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:24 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:24 CST 2013
Wed Jul 17 16:34:24 CST 2013
Wed Jul 17 16:34:24 CST 2013
Wed Jul 17 16:34:24 CST 2013
The authenticity of host 'nmacct3-priv (10.10.10.13)' can't be established.
RSA key fingerprint is 65:8f:79:b4:fa:29:cb:6e:53:f4:90:dd:48:f9:81:5c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nmacct3-priv,10.10.10.13' (RSA) to the list of known hosts.
Wed Jul 17 16:34:26 CST 2013
[oracle@nmacct1 ~]$ ./time_validate.sh
Wed Jul 17 16:34:29 CST 2013
Wed Jul 17 16:34:29 CST 2013
Wed Jul 17 16:34:29 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
备注：执行两次命令，第一次需要输入口令可以忽略。
[oracle@nmacct1 ~]$ scp time_validate.sh nmacct2:~
time_validate.sh 100% 159 0.2KB/s 00:00
[oracle@nmacct1 ~]$ scp time_validate.sh nmacct3:~
time_validate.sh
[oracle@nmacct2 ~]$ ./time_validate.sh
[oracle@nmacct3 ~]$ ./time_validate.sh
[root@nmacct3 ~]# chown -R oracle:oinstall /oracle
3.3.2. 配置新节点环境变量[oracle@nmacct1 ~]$ scp .bash_profile nmacct3:~
.bash_profile 100% 492 0.5KB/s 00:00
[oracle@nmacct3 ~]$ vi .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export ORACLE_BASE=/oracle
export CRS_HOME=/oracle/product/10.2.0/crs
export ORACLE_HOME=/oracle/product/10.2.0/db
export ORACLE_SID=nmjf3
export PATH=$ORACLE_HOME/bin:$CRS_HOME/bin:$PATH:$HOME/bin:/sbin:/usr/sbin:
alias oh='cd $ORACLE_HOME/bin'
alias crs='cd $CRS_HOME/bin'
alias sql='sqlplus / as sysdba'
alias tns='cd $ORACLE_HOME/network/admin'
tar: Removing leading `/' from member names
/oracle/product/10.2.0/db/
/oracle/product/10.2.0/db/wwg/
/oracle/product/10.2.0/db/wwg/admin/
/oracle/product/10.2.0/db/wwg/admin/images/
tar: Removing leading `/' from member names
/oracle/product/10.2.0/crs/
/oracle/product/10.2.0/crs/sysman/
/oracle/product/10.2.0/crs/sysman/lib32/
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 204576
Used space (kbytes) : 4608
Available space (kbytes) : 199968
ID : 96694627
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded
Device/File Name : /dev/raw/raw2
Device/File integrity check succeeded
Cluster registry integrity check succeeded
[oracle@nmacct1 ~]$ dd if=/dev/raw/raw1 of=/home/oracle/ocr.bak
409600+0 records in
409600+0 records out
209715200 bytes (210 MB) copied, 66.1798 seconds, 3.2 MB/s
0. 0 /dev/raw/raw3
1. 0 /dev/raw/raw4
2. 0 /dev/raw/raw5
located 3 votedisk(s).
[oracle@nmacct1 ~]$ dd if=/dev/raw/raw3 of=/home/oracle/votedisk.bak
409600+0 records in
409600+0 records out
209715200 bytes (210 MB) copied, 66.3791 seconds, 3.2 MB/s
/etc/init.d/init.cssd
/etc/init.d/init.crs
/etc/init.d/init.crsd
/etc/init.d/init.evmd
/var/opt/oracle
/etc/inittab
For Linux:
/etc/init.d/init.cssd
/etc/init.d/init.crs
/etc/init.d/init.crsd
/etc/init.d/init.evmd
/etc/oracle
/etc/inittab
For HP-UX:
/sbin/init.d/init.cssd
/sbin/init.d/init.crs
/sbin/init.d/init.crsd
/sbin/init.d/init.evmd
/var/opt/oracle
/etc/inittab
For HP Tru64:
/sbin/init.d/init.cssd
/sbin/init.d/init.crs
/sbin/init.d/init.crsd
/sbin/init.d/init.evmd
/var/opt/oracle
/etc/inittab
For IBM AIX:
/etc/init.cssd
/etc/init.crs
/etc/init.crsd
/etc/init.evmd
/etc/oracle
/etc/inittab
参考文档：10gR2/11gR1: What Files To Backup In Oracle Clusterware (CRS) Installation [ID 754369.1]
Performing post-checks for hardware and operating system setup
Checking node reachability...
Node reachability check passed from node "nmacct1".
Checking user equivalence...
User equivalence check passed for user "oracle".
Checking node connectivity...
Node connectivity check passed for subnet "192.168.56.0" with node(s) nmacct3,nmacct2,nmacct1.
Node connectivity check passed for subnet "10.10.10.0" with node(s) nmacct3,nmacct2,nmacct1.
Suitable interfaces for the private interconnect on subnet "192.168.56.0":
nmacct3 eth0:192.168.56.131 eth0:192.168.56.132
nmacct2 eth0:192.168.56.121 eth0:192.168.56.122
nmacct1 eth0:192.168.56.111 eth0:192.168.56.112
Suitable interfaces for the private interconnect on subnet "10.10.10.0":
nmacct3 eth1:10.10.10.13
nmacct2 eth1:10.10.10.12
nmacct1 eth1:10.10.10.11
ERROR:
Could not find a suitable set of interfaces for VIPs.
Node connectivity check failed.
Checking shared storage accessibility...
WARNING:
Package cvuqdisk not installed.
nmacct3,nmacct2,nmacct1
No shared storage found.
Shared storage check was successful on nodes "nmacct3,nmacct2,nmacct1".
Post-check for hardware and operating system setup was unsuccessful on all the nodes.
[oracle@nmacct1 bin]$ ./cluvfy stage -pre crsinst -n nmacct1,nmacct2,nmacct3
Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "nmacct1".
Checking user equivalence...
User equivalence check passed for user "oracle".
Checking administrative privileges...
User existence check passed for "oracle".
Group existence check passed for "oinstall".
Membership check for user "oracle" in group "oinstall" [as Primary] passed.
Administrative privileges check passed.
Checking node connectivity...
Node connectivity check passed for subnet "192.168.56.0" with node(s) nmacct3,nmacct2,nmacct1.
Node connectivity check passed for subnet "10.10.10.0" with node(s) nmacct3,nmacct2,nmacct1.
Suitable interfaces for the private interconnect on subnet "192.168.56.0":
nmacct3 eth0:192.168.56.131 eth0:192.168.56.132
nmacct2 eth0:192.168.56.121 eth0:192.168.56.122
nmacct1 eth0:192.168.56.111 eth0:192.168.56.112
Suitable interfaces for the private interconnect on subnet "10.10.10.0":
nmacct3 eth1:10.10.10.13
nmacct2 eth1:10.10.10.12
nmacct1 eth1:10.10.10.11
ERROR:
Could not find a suitable set of interfaces for VIPs.
Node connectivity check failed.
Checking system requirements for 'crs'...
Total memory check passed.
Free disk space check passed.
Swap space check passed.
System architecture check passed.
Kernel version check passed.
Package existence check passed for "binutils-2.17.50.0.6-2.el5".
Package existence check passed for "control-center-2.16.0-14.el5".
Package existence check passed for "gcc-4.1.1-52".
Package existence check passed for "glibc-2.5-12".
Package existence check passed for "glibc-common-2.5-12".
Package existence check passed for "libstdc++-4.1.1-52.el5".
Package existence check passed for "libstdc++-devel-4.1.1-52.el5".
Package existence check passed for "make-3.81-1.1".
Package existence check passed for "sysstat-7.0.0-3.el5".
Package existence check passed for "setarch-2.0-1.1".
Kernel parameter check passed for "semmsl".
Kernel parameter check passed for "semmns".
Kernel parameter check passed for "semopm".
Kernel parameter check passed for "semmni".
Kernel parameter check passed for "shmall".
Kernel parameter check passed for "shmmni".
Kernel parameter check passed for "file-max".
Kernel parameter check passed for "rmem_default".
Kernel parameter check passed for "rmem_max".
Kernel parameter check passed for "wmem_default".
Kernel parameter check passed for "wmem_max".
Group existence check passed for "dba".
User existence check passed for "nobody".
Hard resource limit check passed for "open file descriptors".
Soft resource limit check passed for "open file descriptors".
Hard resource limit check passed for "maximum user processes".
Soft resource limit check passed for "maximum user processes".
System requirement passed for 'crs'
Pre-check for cluster services setup was unsuccessful on all the nodes.
New 'nmacct1:1 (oracle)' desktop is nmacct1:1
Starting applications specified in /home/oracle/.vnc/xstartup
Log file is /home/oracle/.vnc/nmacct1:1.log
[oracle@nmacct1 bin]$ ./addNode.sh
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Oracle Universal Installer, Version 10.2.0.5.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
指定新节点后，点击Next，进行检测。
[root@nmacct3 ~]# /oracle/oraInventory/orainstRoot.sh
Changing permissions of /oracle/oraInventory to 770.
Changing groupname of /oracle/oraInventory to oinstall.
The execution of the script is complete
[root@nmacct1 ~]# /oracle/product/10.2.0/crs/install/rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 3: nmacct3 nmacct3-priv nmacct3
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/oracle/product/10.2.0/crs/bin/srvctl add nodeapps -n nmacct3 -A nmacct3-vip/255.255.255.0/eth0 -o /oracle/product/10.2.0/crs
[root@nmacct3 ~]# /oracle/oraInventory/orainstRoot.sh
Changing permissions of /oracle/oraInventory to 770.
Changing groupname of /oracle/oraInventory to oinstall.
The execution of the script is complete
[root@nmacct3 ~]# /oracle/product/10.2.0/crs/root.sh
WARNING: directory '/oracle/product/10.2.0' is not owned by root
WARNING: directory '/oracle/product' is not owned by root
WARNING: directory '/oracle' is not owned by root
No value set for the CRS parameter CRS_OCR_LOCATIONS. Using Values in paramfile.crs
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
OCR LOCATIONS = /dev/raw/raw1,/dev/raw/raw2
OCR backup directory '/oracle/product/10.2.0/crs/cdata/crs' does not exist. Creating now
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle/product/10.2.0' is not owned by root
WARNING: directory '/oracle/product' is not owned by root
WARNING: directory '/oracle' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: nmacct1 nmacct1-priv nmacct1
node 2: nmacct2 nmacct2-priv nmacct2
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
nmacct1
nmacct2
nmacct3
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Name Type Target State Host
------------------------------------------------------------
ora....T1.lsnr application ONLINE ONLINE nmacct1
ora....ct1.gsd application ONLINE ONLINE nmacct1
ora....ct1.ons application ONLINE ONLINE nmacct1
ora....ct1.vip application ONLINE ONLINE nmacct1
ora....T2.lsnr application ONLINE ONLINE nmacct2
ora....ct2.gsd application ONLINE ONLINE nmacct2
ora....ct2.ons application ONLINE ONLINE nmacct2
ora....ct2.vip application ONLINE ONLINE nmacct2
ora....ct3.gsd application ONLINE ONLINE nmacct3
ora....ct3.ons application ONLINE ONLINE nmacct3
ora....ct3.vip application ONLINE ONLINE nmacct3
ora.nmjf.db application ONLINE ONLINE nmacct1
ora....f1.inst application ONLINE ONLINE nmacct1
ora....f2.inst application ONLINE ONLINE nmacct2
[DATABASE.ONS_HOSTS]
[DATABASE.ONS_HOSTS.nmacct1]
[DATABASE.ONS_HOSTS.nmacct1.PORT]
[DATABASE.ONS_HOSTS.nmacct2]
[DATABASE.ONS_HOSTS.nmacct2.PORT]
6.8.2. 查看当前ONS_HOSTS远程端口配置[oracle@nmacct3 conf]$ ocrdump -stdout -keyname 'DATABASE.ONS_HOSTS.nmacct1'
07/18/2013 09:27:00
ocrdump -stdout -keyname DATABASE.ONS_HOSTS.nmacct1
[DATABASE.ONS_HOSTS.nmacct1]
ORATEXT : nmacct1
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : oinstall}
[DATABASE.ONS_HOSTS.nmacct1.PORT]
ORATEXT : 6200
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : oinstall}
[oracle@nmacct3 ~]$ cd $CRS_HOME/opmn/conf
[oracle@nmacct3 conf]$ more ons.config
localport=6113
remoteport=6200
loglevel=3
useocr=on
[oracle@nmacct3 conf]$
6.8.3. 添加ONS端口[oracle@nmacct3 ~]$ cd $CRS_HOME/bin
[oracle@nmacct3 bin]$ ./racgons add_config nmacct3:6200
[oracle@nmacct3 ~]$ ocrdump -stdout | grep ONS_HOSTS
[DATABASE.ONS_HOSTS]
[DATABASE.ONS_HOSTS.nmacct1]
[DATABASE.ONS_HOSTS.nmacct1.PORT]
[DATABASE.ONS_HOSTS.nmacct2]
[DATABASE.ONS_HOSTS.nmacct2.PORT]
[DATABASE.ONS_HOSTS.nmacct3]
[DATABASE.ONS_HOSTS.nmacct3.PORT]
[oracle@nmacct1 bin]$ ./addNode.sh
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Oracle Universal Installer, Version 10.2.0.5.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
[root@nmacct3 ~]# /oracle/product/10.2.0/db/root.sh
Running Oracle 10g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/10.2.0/db
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Oracle Net Services Configuration:
[oracle@nmacct1 bin]$ netca
Oracle Net Services Configuration:
Configuring Listener:LISTENER
ServiceAliasException: Could not save Service Alias: TNS-04406: Object already exists
Default local naming configuration complete.
nmacct3...
Listener configuration complete.
Oracle Net Services configuration successful. The exit code is 0
Name Type Target State Host
------------------------------------------------------------
ora....T1.lsnr application ONLINE ONLINE nmacct1
ora....ct1.gsd application ONLINE ONLINE nmacct1
ora....ct1.ons application ONLINE ONLINE nmacct1
ora....ct1.vip application ONLINE ONLINE nmacct1
ora....T2.lsnr application ONLINE ONLINE nmacct2
ora....ct2.gsd application ONLINE ONLINE nmacct2
ora....ct2.ons application ONLINE ONLINE nmacct2
ora....ct2.vip application ONLINE ONLINE nmacct2
ora....T3.lsnr application ONLINE ONLINE nmacct3
ora....ct3.gsd application ONLINE ONLINE nmacct3
ora....ct3.ons application ONLINE ONLINE nmacct3
ora....ct3.vip application ONLINE ONLINE nmacct3
ora.nmjf.db application ONLINE ONLINE nmacct1
ora....f1.inst application ONLINE ONLINE nmacct1
ora....f2.inst application ONLINE ONLINE nmacct2
/dev/raw/raw29 /dev/data_vg/undotbs3
/dev/raw/raw30 /dev/data_vg/redo3_1
/dev/raw/raw31 /dev/data_vg/redo3_2
/dev/raw/raw32 /dev/data_vg/redo3_3
/dev/raw/raw33 /dev/data_vg/redo3_4
Name Type Target State Host
------------------------------------------------------------
ora....T1.lsnr application ONLINE ONLINE nmacct1
ora....ct1.gsd application ONLINE ONLINE nmacct1
ora....ct1.ons application ONLINE ONLINE nmacct1
ora....ct1.vip application ONLINE ONLINE nmacct1
ora....T2.lsnr application ONLINE ONLINE nmacct2
ora....ct2.gsd application ONLINE ONLINE nmacct2
ora....ct2.ons application ONLINE ONLINE nmacct2
ora....ct2.vip application ONLINE ONLINE nmacct2
ora....T3.lsnr application ONLINE ONLINE nmacct3
ora....ct3.gsd application ONLINE ONLINE nmacct3
ora....ct3.ons application ONLINE ONLINE nmacct3
ora....ct3.vip application ONLINE ONLINE nmacct3
ora.nmjf.db application ONLINE ONLINE nmacct2
ora....f1.inst application ONLINE ONLINE nmacct1
ora....f2.inst application ONLINE ONLINE nmacct2
ora....f3.inst application ONLINE ONLINE nmacct3
在2个节点执行：
# crsctl stop crs
在节点1执行：
# ocrconfig -restore <filename> （其中filename是自动备份的文件)
在2个节点执行：
# crsctl start crs
1.1. 新节点IP地址规划 5
1.2. 新节点裸设备规划 5
2. 安装Linux 6
2.1. 安装操作系统 6
2.1.1. 选择语言 6
2.1.2. 格式化磁盘 7
2.1.3. 修改IP地址 (不要配置DNS) 9
2.1.4. 设置时区 11
2.1.5. 选择安装组件 12
2.1.6. 开始安装 13
2.1.7. 确认安装信息 14
2.1.8. 关闭安全设置 15
2.1.9. 确认时间 16
2.1.10. 不进行新建用户 17
2.1.11. 完成安装 18
2.2. 关闭不必要的服务 19
2.3. 修改启动模式为字符界面 19
2.4. Yum安装oracle-validated 20
2.5. 确认内核的调整 21
2.6. 检查用户、组，创建安装目录 (保持与其他节点一致) 21
2.7. 修改主机名 21
2.8. 图形界面使用vnc 22
2.8.1. 第一次启动配置vnc 22
2.8.2. 停止vnc 22
2.8.3. 修改vnc图形界面为gnome模式 22
2.9. 关闭虚拟机 23
3. 配置新主机环境 24
3.1. 准备共享存储 24
3.1.1. 磁盘信息 24
3.1.2. 扫描pv 24
3.1.3. 扫描vg 24
3.1.4. 为新节点添加undo和redo文件的裸设备 25
3.1.5. 激活vg (所有节点都需要操作) 26
3.1.6. 扫描lv (所有节点) 28
3.1.7. 绑定裸设备 (所有节点都需要操作) 30
3.1.8. 修改裸设备权限 33
3.1.9. 修改启动文件使权限永久生效 34
3.2. 准备ssh认证 34
3.2.1. 修改已有主机的hosts文件 (nmacct1, nmacct2) 34
3.2.2. 新主机用户执行ssh-keygen 35
3.2.3. 在已有节点进行配置authorized_keys文件 35
3.2.4. 复制authorized_keys文件到远程节点 36
3.2.5. 修改新增节点目录、文件属性 36
3.2.6. 在所有节点测试ssh认证 36
3.3. 配置安装目录及环境变量 38
3.3.1. 配置新节点安装目录 38
3.3.2. 配置新节点环境变量 38
4. 环境备份 40
4.1. $ORACLE_HOME备份 40
4.2. $CRS_HOME备份 40
4.3. OCR备份 40
4.4. VotingDisk备份 41
4.5. 主机脚本备份 41
5. 执行安装前检测 43
5.1. 操作系统及硬件检查 43
5.2. CRS安装检查 44
6. 向新节点安装CRS软件 47
6.1. 启动vnc服务 47
6.2. 在原有节点(nmacct1)执行$CRS_HOME/oui/bin/addNode.sh脚本 47
6.3. 选择需要添加的节点 49
6.4. 确认开始安装 51
6.5. 执行root脚本 (注意脚本执行节点) 53
6.6. 完成新节点CRS安装 55
6.7. 查看CRS状态 55
6.8. 向新节点添加ONS远程端口 56
6.8.1. 查看当前集群中ONS_HOSTS配置 56
6.8.2. 查看当前ONS_HOSTS远程端口配置 56
6.8.3. 添加ONS端口 57
7. 向新节点安装Database软件 58
7.1. 在原有节点(nmacct1)执行$ORACLE_HOME/oui/bin/addNode.sh脚本 58
7.2. 选择需要添加的节点 59
7.3. 确认开始安装 61
7.4. 新节点执行root脚本 62
7.5. 完成新节点Database软件安装 63
8. 在新节点使用netca配置监听 64
8.1. 启动netca 64
8.2. 只选择新增节点 (nmacct3) 65
8.3. 进行监听配置 65
8.4. 完成监听配置 68
8.5. 查看集群中监听状态 70
9. 向新节点添加数据库实例 71
9.1. 在原有节点(nmacct1)执行dbca命令 71
9.2. 选择实例管理 72
9.3. 选择添加实例 72
9.4. 选择需要添加实例的数据库，输入用户和口令 73
9.5. 选择实例名和节点 74
9.6. 选择数据文件 74
9.7. 确认安装信息 76
9.8. 查询集群信息 77
10. 回退方案 79
1. 虚拟机规划
1.1. 新节点IP地址规划
127.0.0.1 localhost192.168.56.111 nmacct1
192.168.56.112 nmacct1-vip
192.168.56.121 nmacct2
192.168.56.122 nmacct2-vip
192.168.56.131 nmacct3
192.168.56.132 nmacct3-vip
10.10.10.11 nmacct1-priv
10.10.10.12 nmacct2-priv
10.10.10.13 nmacct3-priv
1.2. 新节点裸设备规划
使用裸设备的情况下，需要为新节点新增数据文件，包括undo表空间和redo日志的文件。使用asm和共享存储的情况下，不需要考虑这个问题。表空间
占用空间
VG
裸设备
undotbs3
500M
data_vg
/dev/raw/raw29
redo3_1
100M
data_vg
/dev/raw/raw30
redo3_2
100M
data_vg
/dev/raw/raw31
redo3_3
100M
data_vg
/dev/raw/raw32
redo3_4
100M
data_vg
/dev/raw/raw33
2. 安装Linux
2.1. 安装操作系统
2.1.1. 选择语言
2.1.2. 格式化磁盘
2.1.3. 修改IP地址 (不要配置DNS)
2.1.4. 设置时区
2.1.5. 选择安装组件
2.1.6. 开始安装
2.1.7. 确认安装信息
2.1.8. 关闭安全设置
2.1.9. 确认时间
2.1.10. 不进行新建用户
2.1.11. 完成安装
2.2. 关闭不必要的服务
chkconfig --level 2345 bluetooth offchkconfig --level 2345 cups off
chkconfig --level 2345 ip6tables off
chkconfig --level 2345 iptables off
chkconfig --level 2345 irqbalance off
chkconfig --level 2345 pcscd off
chkconfig --level 2345 anacron off
chkconfig --level 2345 atd off
chkconfig --level 2345 auditd off
chkconfig --level 2345 avahi-daemon off
chkconfig --level 2345 avahi-dnsconfd off
chkconfig --level 2345 cpuspeed off
chkconfig --level 2345 gpm off
chkconfig --level 2345 hidd off
chkconfig --level 2345 mcstrans off
chkconfig --level 2345 microcode_ctl off
chkconfig --level 2345 netfs off
chkconfig --level 2345 nfslock off
chkconfig --level 2345 portmap off
chkconfig --level 2345 readahead_early off
chkconfig --level 2345 readahead_later off
chkconfig --level 2345 restorecond off
chkconfig --level 2345 rpcgssd off
chkconfig --level 2345 rhnsd off
chkconfig --level 2345 rpcidmapd off
chkconfig --level 2345 sendmail off
chkconfig --level 2345 setroubleshoot off
chkconfig --level 2345 smartd off
chkconfig --level 2345 xinetd off
2.3. 修改启动模式为字符界面
[root@nmacct3 ~]# vi /etc/inittab#
# inittab This file describes how the INIT process should set up
# the system in a certain run-level.
#
# Author: Miquel van Smoorenburg, <miquels@drinkel.nl.mugnet.org>
# Modified for RHS Linux by Marc Ewing and Donnie Barnes
#
# Default runlevel. The runlevels used by RHS are:
# 0 - halt (Do NOT set initdefault to this)
# 1 - Single user mode
# 2 - Multiuser, without NFS (The same as 3, if you do not have networking)
# 3 - Full multiuser mode
# 4 - unused
# 5 - X11
# 6 - reboot (Do NOT set initdefault to this)
#
id:3:initdefault:
修改系统为字符模式。
2.4. Yum安装oracle-validated
[root@nmacct3~]# mkdir /media/disk[root@nmacct3~]# mount /dev/cdrom /media/disk
mount: block device /dev/cdrom is write-protected, mounting read-only
[root@nmacct3~]# cp /etc/yum.repos.d/public-yum-el5.repo /etc/yum.repos.d/public-yum-el5.repo.bak
[root@nmacct3~]# vi /etc/yum.repos.d/public-yum-el5.repo
[oel5]
name = Enterprise Linux 5.9 DVD
baseurl=file:///media/disk/Server/
gpgcheck=0
enabled=1
[root@nmacct3~]# yum install oracle-validated
[root@nmacct3 ~]# umount /dev/cdrom
2.5. 确认内核的调整
[root@nmacct3~]# more /etc/sysctl.conf[root@nmacct3~]# more /etc/security/limits.conf
2.6. 检查用户、组，创建安装目录 (保持与其他节点一致)
[root@nmacct3 ~]# id oracleuid=54321(oracle) gid=54321(oinstall) groups=54321(oinstall),54322(dba)
[root@nmacct3 ~]# passwd oracle
Changing password for user oracle.
New UNIX password:
BAD PASSWORD: it is based on a dictionary word
Retype new UNIX password:
passwd: all authentication tokens updated successfully.
[root@nmacct3~]# mkdir /oracle
[root@nmacct3~]# chown -R oracle:oinstall /oracle
2.7. 修改主机名
[root@nmacct3 ~]# vi /etc/hosts127.0.0.1 localhost
192.168.56.111 nmacct1
192.168.56.112 nmacct1-vip
192.168.56.121 nmacct2
192.168.56.122 nmacct2-vip
192.168.56.131 nmacct3
192.168.56.132 nmacct3-vip
10.10.10.11 nmacct1-priv
10.10.10.12 nmacct2-priv
10.10.10.13 nmacct3-priv
2.8. 图形界面使用vnc
2.8.1. 第一次启动配置vnc[oracle@nmacct3~]$ vncserver :1You will require a password to access your desktops.
Password:
Verify:
New 'nmacct2:1 (oracle)' desktop is nmacct2:1
Creating default startup script /home/oracle/.vnc/xstartup
Starting applications specified in /home/oracle/.vnc/xstartup
Log file is /home/oracle/.vnc/nmacct2:1.log
2.8.2. 停止vnc[oracle@nmacct3~]$ vncserver -kill :1
Killing Xvnc process ID 16056
2.8.3. 修改vnc图形界面为gnome模式[oracle@nmacct3~]$ cd .vnc
[oracle@nmacct3.vnc]$ vi xstartup
#!/bin/sh
# Uncomment the following two lines for normal desktop:
# unset SESSION_MANAGER
# exec /etc/X11/xinit/xinitrc
[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
gnome-session &
最后一行，修改为gnome-session，重启vnc生效。
2.9. 关闭虚拟机
[root@nmacct3~]# init 03. 配置新主机环境
3.1. 准备共享存储
3.1.1. 磁盘信息[root@nmacct3 ~]# ls -la /dev/sd*brw-r----- 1 root disk 8, 0 Jul 17 16:10 /dev/sda
brw-r----- 1 root disk 8, 1 Jul 17 16:11 /dev/sda1
brw-r----- 1 root disk 8, 2 Jul 17 16:10 /dev/sda2
brw-r----- 1 root disk 8, 16 Jul 17 16:10 /dev/sdb
brw-r----- 1 root disk 8, 32 Jul 17 16:10 /dev/sdc
brw-r----- 1 root disk 8, 48 Jul 17 16:10 /dev/sdd
3.1.2. 扫描pv[root@nmacct3 ~]# pvscan
PV /dev/sdd VG share_vg lvm2 [2.00 GB / 1.02 GB free]
PV /dev/sdc VG data_vg lvm2 [5.00 GB / 1020.00 MB free]
PV /dev/sdb VG system_vg lvm2 [5.00 GB / 392.00 MB free]
PV /dev/sda2 VG VolGroup00 lvm2 [19.88 GB / 0 free]
Total: 4 [31.86 GB] / in use: 4 [31.86 GB] / in no VG: 0 [0 ]
[root@nmacct3 ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 VolGroup00 lvm2 a-- 19.88G 0
/dev/sdb system_vg lvm2 a-- 5.00G 392.00M
/dev/sdc data_vg lvm2 a-- 5.00G 1020.00M
/dev/sdd share_vg lvm2 a-- 2.00G 1.02G
3.1.3. 扫描vg[root@nmacct3 ~]# vgscan
Reading all physical volumes. This may take a while...
Found volume group "share_vg" using metadata type lvm2
Found volume group "data_vg" using metadata type lvm2
Found volume group "system_vg" using metadata type lvm2
Found volume group "VolGroup00" using metadata type lvm2
[root@nmacct3 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
VolGroup00 1 2 0 wz--n- 19.88G 0
data_vg 1 4 0 wz--n- 5.00G 1020.00M
share_vg 1 5 0 wz--n- 2.00G 1.02G
system_vg 1 19 0 wz--n- 5.00G 392.00M
3.1.4. 为新节点添加undo和redo文件的裸设备? 新节点空间要求表空间
占用空间
VG
裸设备
undotbs3
500M
data_vg
/dev/raw/raw29
redo3_1
100M
data_vg
/dev/raw/raw30
redo3_2
100M
data_vg
/dev/raw/raw31
redo3_3
100M
data_vg
/dev/raw/raw32
redo3_4
100M
data_vg
/dev/raw/raw33
? 存储空间检查[root@nmacct3 ~]# vgdisplay data_vg
--- Volume group ---
VG Name data_vg
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 4
Open LV 0
Max PV 0
Cur PV 1
Act PV 1
VG Size 5.00 GB
PE Size 4.00 MB
Total PE 1279
Alloc PE / Size 1024 / 4.00 GB
Free PE / Size 255 / 1020.00 MB
VG UUID Oo5yzk-v3BZ-OdeY-32Qj-kHd1-1xUK-6wfFe0
? 新增undolvcreate -L 500m -n undotbs3 data_vg
? 新增redolvcreate -L 100m -n redo3_1 data_vg
lvcreate -L 100m -n redo3_2 data_vg
lvcreate -L 100m -n redo3_3 data_vg
lvcreate -L 100m -n redo3_4 data_vg
3.1.5. 激活vg (所有节点都需要操作)[root@nmacct2 ~]# lvscan
ACTIVE '/dev/share_vg/ocr1' [200.00 MB] inherit
ACTIVE '/dev/share_vg/ocr2' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting1' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting2' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting3' [200.00 MB] inherit
ACTIVE '/dev/data_vg/acct_tbs_01' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_tbs_02' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_index_01' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_index_02' [1.00 GB] inherit
inactive '/dev/data_vg/undotbs3' [500.00 MB] inherit
inactive '/dev/data_vg/redo3_1' [100.00 MB] inherit
inactive '/dev/data_vg/redo3_2' [100.00 MB] inherit
inactive '/dev/data_vg/redo3_3' [100.00 MB] inherit
inactive '/dev/data_vg/redo3_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/system' [1000.00 MB] inherit
ACTIVE '/dev/system_vg/sysaux' [1000.00 MB] inherit
ACTIVE '/dev/system_vg/undotbs1' [500.00 MB] inherit
ACTIVE '/dev/system_vg/undotbs2' [500.00 MB] inherit
ACTIVE '/dev/system_vg/temp' [500.00 MB] inherit
ACTIVE '/dev/system_vg/control1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/control2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/control3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/users' [100.00 MB] inherit
ACTIVE '/dev/system_vg/spfile' [12.00 MB] inherit
ACTIVE '/dev/system_vg/pwdfile' [12.00 MB] inherit
ACTIVE '/dev/VolGroup00/LogVol00' [15.53 GB] inherit
ACTIVE '/dev/VolGroup00/LogVol01' [4.34 GB] inherit
[root@nmacct3 ~]# vgchange -ay
5 logical volume(s) in volume group "share_vg" now active
4 logical volume(s) in volume group "data_vg" now active
19 logical volume(s) in volume group "system_vg" now active
2 logical volume(s) in volume group "VolGroup00" now active
3.1.6. 扫描lv (所有节点)[root@nmacct2 ~]# lvscan
ACTIVE '/dev/share_vg/ocr1' [200.00 MB] inherit
ACTIVE '/dev/share_vg/ocr2' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting1' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting2' [200.00 MB] inherit
ACTIVE '/dev/share_vg/voting3' [200.00 MB] inherit
ACTIVE '/dev/data_vg/acct_tbs_01' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_tbs_02' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_index_01' [1.00 GB] inherit
ACTIVE '/dev/data_vg/acct_index_02' [1.00 GB] inherit
ACTIVE '/dev/data_vg/undotbs3' [500.00 MB] inherit
ACTIVE '/dev/data_vg/redo3_1' [100.00 MB] inherit
ACTIVE '/dev/data_vg/redo3_2' [100.00 MB] inherit
ACTIVE '/dev/data_vg/redo3_3' [100.00 MB] inherit
ACTIVE '/dev/data_vg/redo3_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/system' [1000.00 MB] inherit
ACTIVE '/dev/system_vg/sysaux' [1000.00 MB] inherit
ACTIVE '/dev/system_vg/undotbs1' [500.00 MB] inherit
ACTIVE '/dev/system_vg/undotbs2' [500.00 MB] inherit
ACTIVE '/dev/system_vg/temp' [500.00 MB] inherit
ACTIVE '/dev/system_vg/control1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/control2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/control3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo1_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_1' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_2' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_3' [100.00 MB] inherit
ACTIVE '/dev/system_vg/redo2_4' [100.00 MB] inherit
ACTIVE '/dev/system_vg/users' [100.00 MB] inherit
ACTIVE '/dev/system_vg/spfile' [12.00 MB] inherit
ACTIVE '/dev/system_vg/pwdfile' [12.00 MB] inherit
ACTIVE '/dev/VolGroup00/LogVol00' [15.53 GB] inherit
ACTIVE '/dev/VolGroup00/LogVol01' [4.34 GB] inherit
[root@nmacct2 ~]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
LogVol00 VolGroup00 -wi-ao 15.53G
LogVol01 VolGroup00 -wi-ao 4.34G
acct_index_01 data_vg -wi-ao 1.00G
acct_index_02 data_vg -wi-a- 1.00G
acct_tbs_01 data_vg -wi-ao 1.00G
acct_tbs_02 data_vg -wi-a- 1.00G
redo3_1 data_vg -wi-a- 100.00M
redo3_2 data_vg -wi-a- 100.00M
redo3_3 data_vg -wi-a- 100.00M
redo3_4 data_vg -wi-a- 100.00M
undotbs3 data_vg -wi-a- 500.00M
ocr1 share_vg -wi-ao 200.00M
ocr2 share_vg -wi-ao 200.00M
voting1 share_vg -wi-ao 200.00M
voting2 share_vg -wi-ao 200.00M
voting3 share_vg -wi-ao 200.00M
control1 system_vg -wi-ao 100.00M
control2 system_vg -wi-ao 100.00M
control3 system_vg -wi-ao 100.00M
pwdfile system_vg -wi-a- 12.00M
redo1_1 system_vg -wi-ao 100.00M
redo1_2 system_vg -wi-ao 100.00M
redo1_3 system_vg -wi-ao 100.00M
redo1_4 system_vg -wi-ao 100.00M
redo2_1 system_vg -wi-ao 100.00M
redo2_2 system_vg -wi-ao 100.00M
redo2_3 system_vg -wi-ao 100.00M
redo2_4 system_vg -wi-ao 100.00M
spfile system_vg -wi-a- 12.00M
sysaux system_vg -wi-ao 1000.00M
system system_vg -wi-ao 1000.00M
temp system_vg -wi-ao 500.00M
undotbs1 system_vg -wi-ao 500.00M
undotbs2 system_vg -wi-ao 500.00M
users system_vg -wi-ao 100.00M
3.1.7. 绑定裸设备 (所有节点都需要操作)[root@nmacct3 ~]# vi /etc/sysconfig/rawdevices
# raw device bindings
# format: <rawdev> <major> <minor>
# <rawdev> <blockdev>
# example: /dev/raw/raw1 /dev/sda1
# /dev/raw/raw2 8 5
/dev/raw/raw1 /dev/share_vg/ocr1
/dev/raw/raw2 /dev/share_vg/ocr2
/dev/raw/raw3 /dev/share_vg/voting1
/dev/raw/raw4 /dev/share_vg/voting2
/dev/raw/raw5 /dev/share_vg/voting3
/dev/raw/raw6 /dev/data_vg/acct_tbs_01
/dev/raw/raw7 /dev/data_vg/acct_tbs_02
/dev/raw/raw8 /dev/data_vg/acct_index_01
/dev/raw/raw9 /dev/data_vg/acct_index_02
/dev/raw/raw10 /dev/system_vg/system
/dev/raw/raw11 /dev/system_vg/sysaux
/dev/raw/raw12 /dev/system_vg/undotbs1
/dev/raw/raw13 /dev/system_vg/undotbs2
/dev/raw/raw14 /dev/system_vg/temp
/dev/raw/raw15 /dev/system_vg/control1
/dev/raw/raw16 /dev/system_vg/control2
/dev/raw/raw17 /dev/system_vg/control3
/dev/raw/raw18 /dev/system_vg/redo1_1
/dev/raw/raw19 /dev/system_vg/redo1_2
/dev/raw/raw20 /dev/system_vg/redo1_3
/dev/raw/raw21 /dev/system_vg/redo1_4
/dev/raw/raw22 /dev/system_vg/redo2_1
/dev/raw/raw23 /dev/system_vg/redo2_2
/dev/raw/raw24 /dev/system_vg/redo2_3
/dev/raw/raw25 /dev/system_vg/redo2_4
/dev/raw/raw26 /dev/system_vg/users
/dev/raw/raw27 /dev/system_vg/spfile
/dev/raw/raw28 /dev/system_vg/pwdfile
/dev/raw/raw29 /dev/data_vg/undotbs3
/dev/raw/raw30 /dev/data_vg/redo3_1
/dev/raw/raw31 /dev/data_vg/redo3_2
/dev/raw/raw32 /dev/data_vg/redo3_3
/dev/raw/raw33 /dev/data_vg/redo3_4
[root@nmacct3 ~]# service rawdevices restart
Assigning devices:
/dev/raw/raw1 --> /dev/share_vg/ocr1
/dev/raw/raw1: bound to major 252, minor 2
/dev/raw/raw2 --> /dev/share_vg/ocr2
/dev/raw/raw2: bound to major 252, minor 3
/dev/raw/raw3 --> /dev/share_vg/voting1
/dev/raw/raw3: bound to major 252, minor 4
/dev/raw/raw4 --> /dev/share_vg/voting2
/dev/raw/raw4: bound to major 252, minor 5
/dev/raw/raw5 --> /dev/share_vg/voting3
/dev/raw/raw5: bound to major 252, minor 6
/dev/raw/raw6 --> /dev/data_vg/acct_tbs_01
/dev/raw/raw6: bound to major 252, minor 7
/dev/raw/raw7 --> /dev/data_vg/acct_tbs_02
/dev/raw/raw7: bound to major 252, minor 8
/dev/raw/raw8 --> /dev/data_vg/acct_index_01
/dev/raw/raw8: bound to major 252, minor 9
/dev/raw/raw9 --> /dev/data_vg/acct_index_02
/dev/raw/raw9: bound to major 252, minor 10
/dev/raw/raw10 --> /dev/system_vg/system
/dev/raw/raw10: bound to major 252, minor 11
/dev/raw/raw11 --> /dev/system_vg/sysaux
/dev/raw/raw11: bound to major 252, minor 12
/dev/raw/raw12 --> /dev/system_vg/undotbs1
/dev/raw/raw12: bound to major 252, minor 13
/dev/raw/raw13 --> /dev/system_vg/undotbs2
/dev/raw/raw13: bound to major 252, minor 14
/dev/raw/raw14 --> /dev/system_vg/temp
/dev/raw/raw14: bound to major 252, minor 15
/dev/raw/raw15 --> /dev/system_vg/control1
/dev/raw/raw15: bound to major 252, minor 16
/dev/raw/raw16 --> /dev/system_vg/control2
/dev/raw/raw16: bound to major 252, minor 17
/dev/raw/raw17 --> /dev/system_vg/control3
/dev/raw/raw17: bound to major 252, minor 18
/dev/raw/raw18 --> /dev/system_vg/redo1_1
/dev/raw/raw18: bound to major 252, minor 19
/dev/raw/raw19 --> /dev/system_vg/redo1_2
/dev/raw/raw19: bound to major 252, minor 20
/dev/raw/raw20 --> /dev/system_vg/redo1_3
/dev/raw/raw20: bound to major 252, minor 21
/dev/raw/raw21 --> /dev/system_vg/redo1_4
/dev/raw/raw21: bound to major 252, minor 22
/dev/raw/raw22 --> /dev/system_vg/redo2_1
/dev/raw/raw22: bound to major 252, minor 23
/dev/raw/raw23 --> /dev/system_vg/redo2_2
/dev/raw/raw23: bound to major 252, minor 24
/dev/raw/raw24 --> /dev/system_vg/redo2_3
/dev/raw/raw24: bound to major 252, minor 25
/dev/raw/raw25 --> /dev/system_vg/redo2_4
/dev/raw/raw25: bound to major 252, minor 26
/dev/raw/raw26 --> /dev/system_vg/users
/dev/raw/raw26: bound to major 252, minor 27
/dev/raw/raw27 --> /dev/system_vg/spfile
/dev/raw/raw27: bound to major 252, minor 28
/dev/raw/raw28 --> /dev/system_vg/pwdfile
/dev/raw/raw28: bound to major 252, minor 29
/dev/raw/raw29 --> /dev/data_vg/undotbs3
/dev/raw/raw29: bound to major 252, minor 30
/dev/raw/raw30 --> /dev/data_vg/redo3_1
/dev/raw/raw30: bound to major 252, minor 31
/dev/raw/raw31 --> /dev/data_vg/redo3_2
/dev/raw/raw31: bound to major 252, minor 32
/dev/raw/raw32 --> /dev/data_vg/redo3_3
/dev/raw/raw32: bound to major 252, minor 33
/dev/raw/raw33 --> /dev/data_vg/redo3_4
/dev/raw/raw33: bound to major 252, minor 34
done
3.1.8. 修改裸设备权限? 原有节点，修改新增的5个裸设备权限chown oracle:dba /dev/raw/raw29
chown oracle:dba /dev/raw/raw30
chown oracle:dba /dev/raw/raw31
chown oracle:dba /dev/raw/raw32
chown oracle:dba /dev/raw/raw33
chmod 660 /dev/raw/raw29
chmod 660 /dev/raw/raw30
chmod 660 /dev/raw/raw31
chmod 660 /dev/raw/raw32
chmod 660 /dev/raw/raw33
? 新节点，修改所有裸设备权限[root@nmacct3 raw]# chown oracle:dba /dev/raw/raw*
[root@nmacct3 raw]# chmod 660 /dev/raw/raw*
3.1.9. 修改启动文件使权限永久生效[root@nmacct3 raw]# vi /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
chown oracle:dba /dev/raw/*
chmod 660 /dev/raw/*
3.2. 准备ssh认证
3.2.1. 修改已有主机的hosts文件 (nmacct1, nmacct2)127.0.0.1 localhost192.168.56.111 nmacct1
192.168.56.112 nmacct1-vip
192.168.56.121 nmacct2
192.168.56.122 nmacct2-vip
192.168.56.131 nmacct3
192.168.56.132 nmacct3-vip
10.10.10.11 nmacct1-priv
10.10.10.12 nmacct2-priv
10.10.10.13 nmacct3-priv
3.2.2. 新主机用户执行ssh-keygen[oracle@nmacct3 ~]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Created directory '/home/oracle/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
9c:a2:44:00:69:63:9f:e4:02:78:f5:0e:5f:d0:ce:02 oracle@nmacct3
[oracle@nmacct3 ~]$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_dsa.
Your public key has been saved in /home/oracle/.ssh/id_dsa.pub.
The key fingerprint is:
8a:30:63:3a:3e:5b:2f:26:e2:d9:12:7b:cf:6d:24:3f oracle@nmacct3
3.2.3. 在已有节点进行配置authorized_keys文件选择nmacct1主机进行操作：
[oracle@nmacct1 .ssh]$ ssh nmacct3 cat ~/.ssh/id_dsa.pub >> authorized_keys
The authenticity of host 'nmacct3 (192.168.56.131)' can't be established.
RSA key fingerprint is 65:8f:79:b4:fa:29:cb:6e:53:f4:90:dd:48:f9:81:5c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nmacct3,192.168.56.131' (RSA) to the list of known hosts.
oracle@nmacct3's password:
[oracle@nmacct1 .ssh]$ ssh nmacct3 cat ~/.ssh/id_rsa.pub >> authorized_keys
oracle@nmacct3's password:
3.2.4. 复制authorized_keys文件到远程节点[oracle@nmacct1 .ssh]$ scp authorized_keys nmacct2:~/.ssh
authorized_keys 100% 3000 2.9KB/s 00:00
[oracle@nmacct1 .ssh]$ scp authorized_keys nmacct3:~/.ssh
oracle@nmacct3's password:
authorized_keys 100% 3000 2.9KB/s 00:00
3.2.5. 修改新增节点目录、文件属性[oracle@nmacct3 ~]$ chmod 700 ~/.ssh
[oracle@nmacct3 ~]$ cd .ssh
[oracle@nmacct3 .ssh]$ chmod 600 authorized_keys
3.2.6. 在所有节点测试ssh认证[oracle@nmacct1 ~]$ vi time_validate.sh
date; ssh nmacct1 date;
date; ssh nmacct1-priv date;
date; ssh nmacct2 date;
date; ssh nmacct2-priv date;
date; ssh nmacct3 date;
date; ssh nmacct3-priv date;
[oracle@nmacct1 ~]$ chmod 700 time_validate.sh
[oracle@nmacct1 ~]$ ./time_validate.sh
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:24 CST 2013
Wed Jul 17 16:34:23 CST 2013
Wed Jul 17 16:34:24 CST 2013
Wed Jul 17 16:34:24 CST 2013
Wed Jul 17 16:34:24 CST 2013
Wed Jul 17 16:34:24 CST 2013
The authenticity of host 'nmacct3-priv (10.10.10.13)' can't be established.
RSA key fingerprint is 65:8f:79:b4:fa:29:cb:6e:53:f4:90:dd:48:f9:81:5c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'nmacct3-priv,10.10.10.13' (RSA) to the list of known hosts.
Wed Jul 17 16:34:26 CST 2013
[oracle@nmacct1 ~]$ ./time_validate.sh
Wed Jul 17 16:34:29 CST 2013
Wed Jul 17 16:34:29 CST 2013
Wed Jul 17 16:34:29 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
Wed Jul 17 16:34:30 CST 2013
备注：执行两次命令，第一次需要输入口令可以忽略。
[oracle@nmacct1 ~]$ scp time_validate.sh nmacct2:~
time_validate.sh 100% 159 0.2KB/s 00:00
[oracle@nmacct1 ~]$ scp time_validate.sh nmacct3:~
time_validate.sh
[oracle@nmacct2 ~]$ ./time_validate.sh
[oracle@nmacct3 ~]$ ./time_validate.sh
3.3. 配置安装目录及环境变量
3.3.1. 配置新节点安装目录[root@nmacct3 ~]# mkdir /oracle[root@nmacct3 ~]# chown -R oracle:oinstall /oracle
3.3.2. 配置新节点环境变量[oracle@nmacct1 ~]$ scp .bash_profile nmacct3:~
.bash_profile 100% 492 0.5KB/s 00:00
[oracle@nmacct3 ~]$ vi .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
export ORACLE_BASE=/oracle
export CRS_HOME=/oracle/product/10.2.0/crs
export ORACLE_HOME=/oracle/product/10.2.0/db
export ORACLE_SID=nmjf3
export PATH=$ORACLE_HOME/bin:$CRS_HOME/bin:$PATH:$HOME/bin:/sbin:/usr/sbin:
alias oh='cd $ORACLE_HOME/bin'
alias crs='cd $CRS_HOME/bin'
alias sql='sqlplus / as sysdba'
alias tns='cd $ORACLE_HOME/network/admin'
4. 环境备份
4.1. $ORACLE_HOME备份
[oracle@nmacct1 ~]$ tar cvf db.tar /oracle/product/10.2.0/db/tar: Removing leading `/' from member names
/oracle/product/10.2.0/db/
/oracle/product/10.2.0/db/wwg/
/oracle/product/10.2.0/db/wwg/admin/
/oracle/product/10.2.0/db/wwg/admin/images/
4.2. $CRS_HOME备份
[oracle@nmacct1 ~]$ tar cvf crs.tar /oracle/product/10.2.0/crs/tar: Removing leading `/' from member names
/oracle/product/10.2.0/crs/
/oracle/product/10.2.0/crs/sysman/
/oracle/product/10.2.0/crs/sysman/lib32/
4.3. OCR备份
[oracle@nmacct1 ~]$ ocrcheckStatus of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 204576
Used space (kbytes) : 4608
Available space (kbytes) : 199968
ID : 96694627
Device/File Name : /dev/raw/raw1
Device/File integrity check succeeded
Device/File Name : /dev/raw/raw2
Device/File integrity check succeeded
Cluster registry integrity check succeeded
[oracle@nmacct1 ~]$ dd if=/dev/raw/raw1 of=/home/oracle/ocr.bak
409600+0 records in
409600+0 records out
209715200 bytes (210 MB) copied, 66.1798 seconds, 3.2 MB/s
4.4. VotingDisk备份
[oracle@nmacct1 ~]$ crsctl query css votedisk0. 0 /dev/raw/raw3
1. 0 /dev/raw/raw4
2. 0 /dev/raw/raw5
located 3 votedisk(s).
[oracle@nmacct1 ~]$ dd if=/dev/raw/raw3 of=/home/oracle/votedisk.bak
409600+0 records in
409600+0 records out
209715200 bytes (210 MB) copied, 66.3791 seconds, 3.2 MB/s
4.5. 主机脚本备份
For Sun:/etc/init.d/init.cssd
/etc/init.d/init.crs
/etc/init.d/init.crsd
/etc/init.d/init.evmd
/var/opt/oracle
/etc/inittab
For Linux:
/etc/init.d/init.cssd
/etc/init.d/init.crs
/etc/init.d/init.crsd
/etc/init.d/init.evmd
/etc/oracle
/etc/inittab
For HP-UX:
/sbin/init.d/init.cssd
/sbin/init.d/init.crs
/sbin/init.d/init.crsd
/sbin/init.d/init.evmd
/var/opt/oracle
/etc/inittab
For HP Tru64:
/sbin/init.d/init.cssd
/sbin/init.d/init.crs
/sbin/init.d/init.crsd
/sbin/init.d/init.evmd
/var/opt/oracle
/etc/inittab
For IBM AIX:
/etc/init.cssd
/etc/init.crs
/etc/init.crsd
/etc/init.evmd
/etc/oracle
/etc/inittab
参考文档：10gR2/11gR1: What Files To Backup In Oracle Clusterware (CRS) Installation [ID 754369.1]
5. 执行安装前检测
5.1. 操作系统及硬件检查
[oracle@nmacct1 bin]$ ./cluvfy stage -post hwos -n nmacct1,nmacct2,nmacct3Performing post-checks for hardware and operating system setup
Checking node reachability...
Node reachability check passed from node "nmacct1".
Checking user equivalence...
User equivalence check passed for user "oracle".
Checking node connectivity...
Node connectivity check passed for subnet "192.168.56.0" with node(s) nmacct3,nmacct2,nmacct1.
Node connectivity check passed for subnet "10.10.10.0" with node(s) nmacct3,nmacct2,nmacct1.
Suitable interfaces for the private interconnect on subnet "192.168.56.0":
nmacct3 eth0:192.168.56.131 eth0:192.168.56.132
nmacct2 eth0:192.168.56.121 eth0:192.168.56.122
nmacct1 eth0:192.168.56.111 eth0:192.168.56.112
Suitable interfaces for the private interconnect on subnet "10.10.10.0":
nmacct3 eth1:10.10.10.13
nmacct2 eth1:10.10.10.12
nmacct1 eth1:10.10.10.11
ERROR:
Could not find a suitable set of interfaces for VIPs.
Node connectivity check failed.
Checking shared storage accessibility...
WARNING:
Package cvuqdisk not installed.
nmacct3,nmacct2,nmacct1
No shared storage found.
Shared storage check was successful on nodes "nmacct3,nmacct2,nmacct1".
Post-check for hardware and operating system setup was unsuccessful on all the nodes.
5.2. CRS安装检查
[oracle@nmacct1 ~]$ cd $CRS_HOME/bin[oracle@nmacct1 bin]$ ./cluvfy stage -pre crsinst -n nmacct1,nmacct2,nmacct3
Performing pre-checks for cluster services setup
Checking node reachability...
Node reachability check passed from node "nmacct1".
Checking user equivalence...
User equivalence check passed for user "oracle".
Checking administrative privileges...
User existence check passed for "oracle".
Group existence check passed for "oinstall".
Membership check for user "oracle" in group "oinstall" [as Primary] passed.
Administrative privileges check passed.
Checking node connectivity...
Node connectivity check passed for subnet "192.168.56.0" with node(s) nmacct3,nmacct2,nmacct1.
Node connectivity check passed for subnet "10.10.10.0" with node(s) nmacct3,nmacct2,nmacct1.
Suitable interfaces for the private interconnect on subnet "192.168.56.0":
nmacct3 eth0:192.168.56.131 eth0:192.168.56.132
nmacct2 eth0:192.168.56.121 eth0:192.168.56.122
nmacct1 eth0:192.168.56.111 eth0:192.168.56.112
Suitable interfaces for the private interconnect on subnet "10.10.10.0":
nmacct3 eth1:10.10.10.13
nmacct2 eth1:10.10.10.12
nmacct1 eth1:10.10.10.11
ERROR:
Could not find a suitable set of interfaces for VIPs.
Node connectivity check failed.
Checking system requirements for 'crs'...
Total memory check passed.
Free disk space check passed.
Swap space check passed.
System architecture check passed.
Kernel version check passed.
Package existence check passed for "binutils-2.17.50.0.6-2.el5".
Package existence check passed for "control-center-2.16.0-14.el5".
Package existence check passed for "gcc-4.1.1-52".
Package existence check passed for "glibc-2.5-12".
Package existence check passed for "glibc-common-2.5-12".
Package existence check passed for "libstdc++-4.1.1-52.el5".
Package existence check passed for "libstdc++-devel-4.1.1-52.el5".
Package existence check passed for "make-3.81-1.1".
Package existence check passed for "sysstat-7.0.0-3.el5".
Package existence check passed for "setarch-2.0-1.1".
Kernel parameter check passed for "semmsl".
Kernel parameter check passed for "semmns".
Kernel parameter check passed for "semopm".
Kernel parameter check passed for "semmni".
Kernel parameter check passed for "shmall".
Kernel parameter check passed for "shmmni".
Kernel parameter check passed for "file-max".
Kernel parameter check passed for "rmem_default".
Kernel parameter check passed for "rmem_max".
Kernel parameter check passed for "wmem_default".
Kernel parameter check passed for "wmem_max".
Group existence check passed for "dba".
User existence check passed for "nobody".
Hard resource limit check passed for "open file descriptors".
Soft resource limit check passed for "open file descriptors".
Hard resource limit check passed for "maximum user processes".
Soft resource limit check passed for "maximum user processes".
System requirement passed for 'crs'
Pre-check for cluster services setup was unsuccessful on all the nodes.
6. 向新节点安装CRS软件
6.1. 启动vnc服务
[oracle@nmacct1 ~]$ vncserver :1New 'nmacct1:1 (oracle)' desktop is nmacct1:1
Starting applications specified in /home/oracle/.vnc/xstartup
Log file is /home/oracle/.vnc/nmacct1:1.log
6.2. 在原有节点(nmacct1)执行$CRS_HOME/oui/bin/addNode.sh脚本
[oracle@nmacct1 ~]$ cd $CRS_HOME/oui/bin[oracle@nmacct1 bin]$ ./addNode.sh
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Oracle Universal Installer, Version 10.2.0.5.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
6.3. 选择需要添加的节点
指定新节点后，点击Next，进行检测。
6.4. 确认开始安装
6.5. 执行root脚本 (注意脚本执行节点)
[root@nmacct3 ~]# /oracle/oraInventory/orainstRoot.sh
Changing permissions of /oracle/oraInventory to 770.
Changing groupname of /oracle/oraInventory to oinstall.
The execution of the script is complete
[root@nmacct1 ~]# /oracle/product/10.2.0/crs/install/rootaddnode.sh
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Attempting to add 1 new nodes to the configuration
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 3: nmacct3 nmacct3-priv nmacct3
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
/oracle/product/10.2.0/crs/bin/srvctl add nodeapps -n nmacct3 -A nmacct3-vip/255.255.255.0/eth0 -o /oracle/product/10.2.0/crs
[root@nmacct3 ~]# /oracle/oraInventory/orainstRoot.sh
Changing permissions of /oracle/oraInventory to 770.
Changing groupname of /oracle/oraInventory to oinstall.
The execution of the script is complete
[root@nmacct3 ~]# /oracle/product/10.2.0/crs/root.sh
WARNING: directory '/oracle/product/10.2.0' is not owned by root
WARNING: directory '/oracle/product' is not owned by root
WARNING: directory '/oracle' is not owned by root
No value set for the CRS parameter CRS_OCR_LOCATIONS. Using Values in paramfile.crs
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
OCR LOCATIONS = /dev/raw/raw1,/dev/raw/raw2
OCR backup directory '/oracle/product/10.2.0/crs/cdata/crs' does not exist. Creating now
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/oracle/product/10.2.0' is not owned by root
WARNING: directory '/oracle/product' is not owned by root
WARNING: directory '/oracle' is not owned by root
clscfg: EXISTING configuration version 3 detected.
clscfg: version 3 is 10G Release 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: nmacct1 nmacct1-priv nmacct1
node 2: nmacct2 nmacct2-priv nmacct2
clscfg: Arguments check out successfully.
NO KEYS WERE WRITTEN. Supply -force parameter to override.
-force is destructive and will destroy any previous cluster
configuration.
Oracle Cluster Registry for cluster has already been initialized
Startup will be queued to init within 30 seconds.
Adding daemons to inittab
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
nmacct1
nmacct2
nmacct3
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
6.6. 完成新节点CRS安装
6.7. 查看CRS状态
[oracle@nmacct3 ~]$ crs_stat -tName Type Target State Host
------------------------------------------------------------
ora....T1.lsnr application ONLINE ONLINE nmacct1
ora....ct1.gsd application ONLINE ONLINE nmacct1
ora....ct1.ons application ONLINE ONLINE nmacct1
ora....ct1.vip application ONLINE ONLINE nmacct1
ora....T2.lsnr application ONLINE ONLINE nmacct2
ora....ct2.gsd application ONLINE ONLINE nmacct2
ora....ct2.ons application ONLINE ONLINE nmacct2
ora....ct2.vip application ONLINE ONLINE nmacct2
ora....ct3.gsd application ONLINE ONLINE nmacct3
ora....ct3.ons application ONLINE ONLINE nmacct3
ora....ct3.vip application ONLINE ONLINE nmacct3
ora.nmjf.db application ONLINE ONLINE nmacct1
ora....f1.inst application ONLINE ONLINE nmacct1
ora....f2.inst application ONLINE ONLINE nmacct2
6.8. 向新节点添加ONS远程端口
6.8.1. 查看当前集群中ONS_HOSTS配置[oracle@nmacct3 conf]$ ocrdump -stdout | grep ONS_HOSTS[DATABASE.ONS_HOSTS]
[DATABASE.ONS_HOSTS.nmacct1]
[DATABASE.ONS_HOSTS.nmacct1.PORT]
[DATABASE.ONS_HOSTS.nmacct2]
[DATABASE.ONS_HOSTS.nmacct2.PORT]
6.8.2. 查看当前ONS_HOSTS远程端口配置[oracle@nmacct3 conf]$ ocrdump -stdout -keyname 'DATABASE.ONS_HOSTS.nmacct1'
07/18/2013 09:27:00
ocrdump -stdout -keyname DATABASE.ONS_HOSTS.nmacct1
[DATABASE.ONS_HOSTS.nmacct1]
ORATEXT : nmacct1
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : oinstall}
[DATABASE.ONS_HOSTS.nmacct1.PORT]
ORATEXT : 6200
SECURITY : {USER_PERMISSION : PROCR_ALL_ACCESS, GROUP_PERMISSION : PROCR_READ, OTHER_PERMISSION : PROCR_READ, USER_NAME : oracle, GROUP_NAME : oinstall}
[oracle@nmacct3 ~]$ cd $CRS_HOME/opmn/conf
[oracle@nmacct3 conf]$ more ons.config
localport=6113
remoteport=6200
loglevel=3
useocr=on
[oracle@nmacct3 conf]$
6.8.3. 添加ONS端口[oracle@nmacct3 ~]$ cd $CRS_HOME/bin
[oracle@nmacct3 bin]$ ./racgons add_config nmacct3:6200
[oracle@nmacct3 ~]$ ocrdump -stdout | grep ONS_HOSTS
[DATABASE.ONS_HOSTS]
[DATABASE.ONS_HOSTS.nmacct1]
[DATABASE.ONS_HOSTS.nmacct1.PORT]
[DATABASE.ONS_HOSTS.nmacct2]
[DATABASE.ONS_HOSTS.nmacct2.PORT]
[DATABASE.ONS_HOSTS.nmacct3]
[DATABASE.ONS_HOSTS.nmacct3.PORT]
7. 向新节点安装Database软件
7.1. 在原有节点(nmacct1)执行$ORACLE_HOME/oui/bin/addNode.sh脚本
[oracle@nmacct1 ~]$ cd $ORACLE_HOME/oui/bin[oracle@nmacct1 bin]$ ./addNode.sh
Starting Oracle Universal Installer...
No pre-requisite checks found in oraparam.ini, no system pre-requisite checks will be executed.
Oracle Universal Installer, Version 10.2.0.5.0 Production
Copyright (C) 1999, 2010, Oracle. All rights reserved.
7.2. 选择需要添加的节点
7.3. 确认开始安装
7.4. 新节点执行root脚本
[root@nmacct3 ~]# /oracle/product/10.2.0/db/root.sh
Running Oracle 10g root.sh script...
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /oracle/product/10.2.0/db
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
7.5. 完成新节点Database软件安装
8. 在新节点使用netca配置监听
8.1. 启动netca
[oracle@nmacct1 bin]$ netcaOracle Net Services Configuration:
8.2. 只选择新增节点 (nmacct3)
8.3. 进行监听配置
8.4. 完成监听配置
[oracle@nmacct1 bin]$ netca
Oracle Net Services Configuration:
Configuring Listener:LISTENER
ServiceAliasException: Could not save Service Alias: TNS-04406: Object already exists
Default local naming configuration complete.
nmacct3...
Listener configuration complete.
Oracle Net Services configuration successful. The exit code is 0
8.5. 查看集群中监听状态
[oracle@nmacct3 ~]$ crs_stat -tName Type Target State Host
------------------------------------------------------------
ora....T1.lsnr application ONLINE ONLINE nmacct1
ora....ct1.gsd application ONLINE ONLINE nmacct1
ora....ct1.ons application ONLINE ONLINE nmacct1
ora....ct1.vip application ONLINE ONLINE nmacct1
ora....T2.lsnr application ONLINE ONLINE nmacct2
ora....ct2.gsd application ONLINE ONLINE nmacct2
ora....ct2.ons application ONLINE ONLINE nmacct2
ora....ct2.vip application ONLINE ONLINE nmacct2
ora....T3.lsnr application ONLINE ONLINE nmacct3
ora....ct3.gsd application ONLINE ONLINE nmacct3
ora....ct3.ons application ONLINE ONLINE nmacct3
ora....ct3.vip application ONLINE ONLINE nmacct3
ora.nmjf.db application ONLINE ONLINE nmacct1
ora....f1.inst application ONLINE ONLINE nmacct1
ora....f2.inst application ONLINE ONLINE nmacct2
9. 向新节点添加数据库实例
9.1. 在原有节点(nmacct1)执行dbca命令
[oracle@nmacct1 ~]$ dbca
9.2. 选择实例管理
9.3. 选择添加实例
9.4. 选择需要添加实例的数据库，输入用户和口令
9.5. 选择实例名和节点
9.6. 选择数据文件
新节点新增裸设备信息：/dev/raw/raw29 /dev/data_vg/undotbs3
/dev/raw/raw30 /dev/data_vg/redo3_1
/dev/raw/raw31 /dev/data_vg/redo3_2
/dev/raw/raw32 /dev/data_vg/redo3_3
/dev/raw/raw33 /dev/data_vg/redo3_4
9.7. 确认安装信息
9.8. 查询集群信息
[oracle@nmacct2 ~]$ crs_stat -tName Type Target State Host
------------------------------------------------------------
ora....T1.lsnr application ONLINE ONLINE nmacct1
ora....ct1.gsd application ONLINE ONLINE nmacct1
ora....ct1.ons application ONLINE ONLINE nmacct1
ora....ct1.vip application ONLINE ONLINE nmacct1
ora....T2.lsnr application ONLINE ONLINE nmacct2
ora....ct2.gsd application ONLINE ONLINE nmacct2
ora....ct2.ons application ONLINE ONLINE nmacct2
ora....ct2.vip application ONLINE ONLINE nmacct2
ora....T3.lsnr application ONLINE ONLINE nmacct3
ora....ct3.gsd application ONLINE ONLINE nmacct3
ora....ct3.ons application ONLINE ONLINE nmacct3
ora....ct3.vip application ONLINE ONLINE nmacct3
ora.nmjf.db application ONLINE ONLINE nmacct2
ora....f1.inst application ONLINE ONLINE nmacct1
ora....f2.inst application ONLINE ONLINE nmacct2
ora....f3.inst application ONLINE ONLINE nmacct3
10. 回退方案
在备份时已做好了充分的准备，考虑回退时只针对OCR进行恢复即可。操作步骤：在2个节点执行：
# crsctl stop crs
在节点1执行：
# ocrconfig -restore <filename> （其中filename是自动备份的文件)
在2个节点执行：
# crsctl start crs
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- ORACLE 10G RAC 10.2.0.5 删除节点
- oracle 10g rac一个节点offline处理
- AIX下Oracle 10g RAC 双节点时间不同步问题解决
- Redhat 5.4 Oracle 10g RAC 删除节点
- Oracle 10G RAC 删除已有节点
- Redhat 5.4 Oracle 10g RAC 删除节点
- Redhat 5.4 Oracle 10g RAC 删除节点
- (超详细)Oracle 10g RAC 添加节点实验操作总结
- AIX下Oracle 10g RAC 双节点时间不同步问题解决
- Redhat 5.4 Oracle 10g RAC 删除节点
- oracle 10g RAC添加节点遭遇ORA-02174错误解决
- (超详细)Oracle 10g RAC 删除节点实验操作总结
- RedHat 5.6_x86_64 + ASM + RAW+ Oracle 10g RAC （七） 添加节点—初始化环境
- oracle 10g rac一个节点监听状态不正常，但crs_stat -t 显示正常
- Redhat 5.4 Oracle 10g RAC 删除节点
- Oracle 10g RAC 添加节点
- Redhat 5.4 Oracle 10g RAC 添加节点
- RedHat 5.6_x86_64 + ASM + RAW+ Oracle 10g RAC （九） 添加节点—安装数据库软件
- Oracle 10g RAC 添加节点
- Redhat 5.4 Oracle 10g RAC 删除节点