Zookeeper管理多个HBase集群

zookeeper是hbase集群的"协调器"。由于zookeeper的轻量级特性，因此我们可以将多个hbase集群共用一个zookeeper集群，以节约大量的服务器。多个hbase集群共用zookeeper集群的方法是使用同一组ip，修改不同hbase集群的"zookeeper.znode.parent"属性，让它们使用不同的根目录。比如cluster1使用/hbase-c1,cluster2使用/hbase-c2，等等。

使用以上方法有一个现实的问题：如何避免各集群的相互干扰？因为client的配置权是在用户手上，并不能保证用户永远是配置正确的，那么会产生某个用户访问了不该他访问的hbase集群。此时数据安全性成了很大的问题，甚至可能出现误删除数据。我们需要在zookeeper层屏弊掉该问题。

zookeeper3.x版本起自带了简单的ACL功能(注意3.3.x版本起不再支持按hostname来分配权限)。见：http://zookeeper.apache.org/doc/r3.3.2/zookeeperProgrammers.html#sc_ZooKeeperAccessControl。进行权限配置主要使用digest和ip两种方法。其中digest是用户密码方式，对用户来说使用上并不透明。ip配置最简单，对用户也是透明的，用户并不知道的情况下就能限制它的访问权限。

zookeeper将访问权限分为了五类:READ/WRITE/DELETE/CREATE/ADMIN，其中admin为最高权限。zookeeper的权限是到znode级别的，限制了某一个node的权限并不能限制它的子节点权限。

不过使用IP做权限配置方案有一个缺陷：必须指定具体的ip，而不能使用通配符或者范围一类的。这样对于大规模的权限设置是非常不方便的一件事，因此作者略调整了一下zookeeper的代码：

IPAuthenticationProvider.java

Java代码


/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0 *
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.zookeeper.server.auth;

import org.apache.zookeeper.data.Id;
import org.apache.zookeeper.server.ServerCnxn;
import org.apache.zookeeper.KeeperException;

public class IPAuthenticationProvider implements AuthenticationProvider {

public String getScheme() {
return "ip";
}

public KeeperException.Code
handleAuthentication(ServerCnxn cnxn, byte[] authData)
{
String id = cnxn.getRemoteAddress().getAddress().getHostAddress();
cnxn.getAuthInfo().add(new Id(getScheme(), id));
return KeeperException.Code.OK;
}

// This is a bit weird but we need to return the address and the number of
// bytes (to distinguish between IPv4 and IPv6
private byte[] addr2Bytes(String addr) {
byte b[] = v4addr2Bytes1(addr);
// TODO Write the v6addr2Bytes
return b;
}

private byte v4addr2Bytes(String part) throws NumberFormatException{
try {
int v = Integer.parseInt(part);
if (v >= 0 && v <= 255) {
byte b = (byte) v;
return b;
} else {
throw new NumberFormatException("v < 0 or v > 255!");
}
} catch (NumberFormatException e) {
throw e;
}
}

private byte[] v4addr2Bytes1(String addr) {
String parts[] = addr.split("\\.", -1);
if (parts.length != 4) {
return null;
}
byte b[] = new byte[4];
for (int i = 0; i < 4; i++) {
try {
if(parts[i].split("/").length == 2){
v4addr2Bytes(parts[i].split("/")[0]);
v4addr2Bytes(parts[i].split("/")[1]);
continue;
}else{
b[i] = v4addr2Bytes(parts[i]);
}
} catch (NumberFormatException e) {
return null;
}
}
return b;
}

public boolean matches(String id, String aclExpr) {
String parts[] = aclExpr.split("/", 2);
byte aclAddr[] = addr2Bytes(parts[0]);
if (aclAddr == null) {
return false;
}
byte endAclAddr[] = new byte[aclAddr.length];
for(int i = 0; i < aclAddr.length; i ++){
endAclAddr[i] = aclAddr[i];
}
if (parts.length == 2) {
try {
int end = Integer.parseInt(parts[1]);
int e = endAclAddr[endAclAddr.length-1]<=0?endAclAddr[endAclAddr.length-1]+256:endAclAddr[endAclAddr.length-1];
if(end < e|| end < 0 || end > 255)
return false;
endAclAddr[endAclAddr.length-1] = (byte)end;
} catch (NumberFormatException e) {
return false;
}
}
byte remoteAddr[] = addr2Bytes(id);
if (remoteAddr == null) {
return false;
}
for (int i = 0; i < remoteAddr.length; i++) {
int r = remoteAddr[i]<=0?(int)remoteAddr[i]+256:remoteAddr[i];
int a = aclAddr[i]<=0?(int)aclAddr[i]+256:aclAddr[i];
int e = endAclAddr[i]<=0?(int)endAclAddr[i]+256:endAclAddr[i];
if (r < a || r > e) {
return false;
}
}
return true;
}

public boolean isAuthenticated() {
return false;
}

public boolean isValid(String id) {
return addr2Bytes(id) != null;
}
}

支持了使用/做为范围标识，比如进入hbase zkcli，执行：setAcl /test ip:192.168.0.3/10:cd，则将读写权限赋给了192.168.0.3-192.168.0.10这8台机器，其它机器将没有任何权限。

这样用同一个zookeeper管理多个集群、海量机器将不再有困扰。

最后写了一个帮助运维同学自动化管理zookeeper集群下多个hbase集群的ACL权限的工具，像以下这样：

Html代码


java -Djava.ext.dirs=libs/ -cp hbase-tools.jar dwbasis.hbase.tools.client.ZookeeperAcl aclFile.json
Usage: ZookeeperAcl acljsonfile [-plan]
/test/t ==> 'ip,'192.168.0.1
:cdrwa
/test ==> 'ip,'192.168.0.1/3
:cdrwa
/test ==> 'ip,'192.168.0.5
:cdrwa
do you really setAcl as above?(y/n)

补充：多集群共用zk后，每个集群的启动和停止不应该影响zk的稳定。因此请配置hbase-env.sh中export HBASE_MANAGES_ZK=false