rh436 - Facing "Volume group for uuid not found" error while run lvcreate on shared storage

Version:
[root@node2 ~]# cat /proc/version
Linux version 2.6.32-220.el6.x86_64 (mockbuild@x86-004.build.bos.redhat.com) (gcc version 4.4.5 20110214 (Red Hat 4.4.5-6) (GCC) ) #1 SMP Wed Nov 9 08:03:13 EST 2011
[root@node2 ~]#

Issue description:
I was doing a HA LVM on test in a three node cluster, but got "Volume group for uuid not found" error while run lvcreate on multipathed shared storage, please find below for detail steps:
[root@node2 ~]# fdisk -cu /dev/mapper/netstorage

Command (m for help): p

Disk /dev/mapper/netstorage: 4294 MB, 4294967296 bytes
171 heads, 40 sectors/track, 1226 cylinders, total 8388608 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xee9a74ab

Device Boot Start End Blocks Id System
/dev/mapper/netstoragep1 2048 2099199 1048576 8e Linux LVM

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4):
Value out of range.
Partition number (1-4): 2
First sector (2099200-8388607, default 2099200):
Using default value 2099200
Last sector, +sectors or +size{K,M,G} (2099200-8388607, default 8388607): +2G

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 22: Invalid argument.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root@node2 ~]# ll /dev/mapper/
total 0
lrwxrwxrwx. 1 root root 7 Jun 16 18:11 clusteredvg-webdata -> ../dm-5
crw-rw----. 1 root root 10, 58 Jun 16 18:10 control
lrwxrwxrwx. 1 root root 7 Jun 16 18:11 netstorage -> ../dm-3
lrwxrwxrwx. 1 root root 7 Jun 16 18:11 netstoragep1 -> ../dm-4
lrwxrwxrwx. 1 root root 7 Jun 16 18:10 vgsrv-home -> ../dm-2
lrwxrwxrwx. 1 root root 7 Jun 16 18:10 vgsrv-root -> ../dm-1
lrwxrwxrwx. 1 root root 7 Jun 16 18:10 vgsrv-swap -> ../dm-0
[root@node2 ~]# partprobe
device-mapper: remove ioctl failed: Device or resource busy
Warning: parted was unable to re-read the partition table on /dev/mapper/netstorage (Device or resource busy). This means Linux won't know anything about the modifications you made.
device-mapper: create ioctl failed: Device or resource busy
device-mapper: remove ioctl failed: Device or resource busy
Warning: WARNING: the kernel failed to re-read the partition table on /dev/vda (Device or resource busy). As a result, it may not reflect all of your changes until after reboot.
[root@node2 ~]# ll /dev/mapper/
total 0
lrwxrwxrwx. 1 root root 7 Jun 16 18:11 clusteredvg-webdata -> ../dm-5
crw-rw----. 1 root root 10, 58 Jun 16 18:10 control
lrwxrwxrwx. 1 root root 7 Jun 16 18:11 netstorage -> ../dm-3
lrwxrwxrwx. 1 root root 7 Jun 16 18:11 netstoragep1 -> ../dm-4
brw-rw----. 1 root disk 253, 6 Jun 16 18:24 netstoragep2
lrwxrwxrwx. 1 root root 7 Jun 16 18:10 vgsrv-home -> ../dm-2
lrwxrwxrwx. 1 root root 7 Jun 16 18:10 vgsrv-root -> ../dm-1
lrwxrwxrwx. 1 root root 7 Jun 16 18:10 vgsrv-swap -> ../dm-0
[root@node2 ~]# multipath -r
Jun 16 18:24:16 | /lib/udev/scsi_id exitted with 1
Jun 16 18:24:16 | /lib/udev/scsi_id exitted with 1
reload: netstorage (1IET_00010001) undef IET,VIRTUAL-DISK
size=4.0G features='0' hwhandler='0' wp=undef
|-+- policy='round-robin 0' prio=1 status=undef
| `- 2:0:0:1 sda 8:0 active ready running
`-+- policy='round-robin 0' prio=1 status=undef
`- 3:0:0:1 sdb 8:16 active ready running
[root@node2 ~]# ll /dev/mapper/
total 0
lrwxrwxrwx. 1 root root 7 Jun 16 18:11 clusteredvg-webdata -> ../dm-5
crw-rw----. 1 root root 10, 58 Jun 16 18:10 control
lrwxrwxrwx. 1 root root 7 Jun 16 18:24 netstorage -> ../dm-3
lrwxrwxrwx. 1 root root 7 Jun 16 18:24 netstoragep1 -> ../dm-4
lrwxrwxrwx. 1 root root 7 Jun 16 18:24 netstoragep2 -> ../dm-6
lrwxrwxrwx. 1 root root 7 Jun 16 18:10 vgsrv-home -> ../dm-2
lrwxrwxrwx. 1 root root 7 Jun 16 18:10 vgsrv-root -> ../dm-1
lrwxrwxrwx. 1 root root 7 Jun 16 18:10 vgsrv-swap -> ../dm-0
[root@node2 ~]# pvcreate /dev/mapper/netstoragep2
Writing physical volume data to disk "/dev/mapper/netstoragep2"
Physical volume "/dev/mapper/netstoragep2" successfully created
[root@node2 ~]# vgcreate -cy havg /dev/mapper/netstoragep2
Clustered volume group "havg" successfully created
[root@node2 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
clusteredvg 1 1 0 wz--nc 1020.00m 0
havg 1 0 0 wz--nc 2.00g 2.00g
vgsrv 1 3 0 wz--n- 8.78g 4.69g
[root@node2 ~]# pvs
p PV VG Fmt Attr PSize PFree
/dev/mapper/netstoragep1 clusteredvg lvm2 a-- 1020.00m 0
/dev/mapper/netstoragep2 havg lvm2 a-- 2.00g 2.00g
/dev/vda2 vgsrv lvm2 a-- 8.78g 4.69g
[root@node2 ~]# lvcreate -l 100%free -n nfsdata /dev/havg
Error locking on node node1.private.cluster20.example.com: Volume group for uuid not found: j6NzWCQCDIfr9oPaxFXYHYSNleMSGJNmEesWct32FM1W09X2qG15nq72RYfgm0Fe
Failed to activate new LV.
[root@node2 ~]# lvs
LV VG Attr LSize Origin Snap% Move Log Copy% Convert
webdata clusteredvg -wi-ao 1020.00m
home vgsrv -wi-ao 256.00m
root vgsrv -wi-ao 3.31g
swap vgsrv -wi-ao 544.00m
[root@node2 ~]# tail /var/log/messages
Jun 16 18:18:40 node2 modclusterd: startup succeeded
Jun 16 18:21:00 node2 ntpd[2299]: synchronized to 192.168.0.254, stratum 11
Jun 16 18:24:12 node2 kernel: sda: sda1 sda2
Jun 16 18:24:12 node2 kernel: sdb: sdb1 sdb2
Jun 16 18:24:12 node2 multipathd: dm-6: remove map (uevent)
Jun 16 18:24:12 node2 multipathd: dm-6: devmap not registered, can't remove
Jun 16 18:34:51 node2 multipathd: dm-7: remove map (uevent)
Jun 16 18:34:51 node2 multipathd: dm-7: devmap not registered, can't remove
Jun 16 18:34:51 node2 multipathd: dm-7: remove map (uevent)
Jun 16 18:34:51 node2 multipathd: dm-7: devmap not registered, can't remove
[root@node2 ~]#

Cause:
I forgot to run partprobe on node1 before vgcreate.

Solution:
Please run below commands before vgcreate on every node shares the same sharestorage in cluster.
[root@node1 ~]# partprobe ; multipath -r

Before creating lvm, we must fistly recreate vg, since the old vg created before partprobe did not be recognized on other node.
[root@node2 ~]# vgremove /dev/havg
Volume group "havg" successfully removed
[root@node2 ~]# vgcreate -cy havg /dev/mapper/netstoragep2
Clustered volume group "havg" successfully created
[root@node2 ~]# lvcreate -l 100%free -n nfsdata havg
Logical volume "nfsdata" created
[root@node2 ~]#

REF:
1. Re: [Linux-cluster] adding volume to cluster http://www.redhat.com/archives/linux-cluster/2008-October/msg00019.html