GlusterFS: {path} or a prefix of it is already part


Starting with GlusterFS 3.3, one change has been the check to see if a directory (or any of it's ancestors) is already part of a volume. This is causing many support questions in #gluster.

This was implemented because if you remove a brick from a volume and continue to use the volume, you can get file into a state where re-adding a former brick can cause all sort of problems, many of which can result in
data loss.

If you're going to reuse a brick, make sure you know what you're doing.

The Solution
For the directory (or any parent directories) that was formerly part of a volume, simply:

setfattr -x trusted.glusterfs.volume-id $brick_path
setfattr -x trusted.gfid $brick_path
rm -rf $brick_path/.glusterfs
Don't worry if it says that the attribute does not exist. As long as it doesn't exist, you're in good shape.

Finally, restart glusterd to ensure it's not "remembering" the old bricks.

See the bugzilla entry for more details and see Jeff Darcy's article for more information about how GlusterFS uses extended attributes.