关于HBase MVCC的设计原理以及MVCC所引起的一个scan问题

最近在使用HBase0.94版本的时，偶尔会出现，HRegionInfo was null or empty in Meta 的警告

java.io.IOException: HRegionInfo was null or empty in Meta for writetest, row=lot_let,9399239430349923234234,99999999999999

at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:170)

在客户端的MetaScanner.metaScan实现中

metaTable = new HTable(configuration, HConstants.META_TABLE_NAME);

Result startRowResult = metaTable.getRowOrBefore(searchRow,HConstants.CATALOG_FAMILY);

if (startRowResult == null) { throw new TableNotFoundException("Cannot find row in .META. for table: " + Bytes.toString(tableName) + ", row=" + Bytes.toStringBinary(searchRow)); }

byte[] value = startRowResult.getValue(HConstants.CATALOG_FAMILY,

HConstants.REGIONINFO_QUALIFIER);

if (value == null || value.length == 0) { throw new IOException("HRegionInfo was null or empty in Meta for " + Bytes.toString(tableName) + ", row=" + Bytes.toStringBinary(searchRow)); }

可以发现在扫描MetaScanner,rowkey所在的范围在Meta 表中不存在；通过RPC定位到服务端的实现

HRegion中：

public Result getClosestRowBefore(final byte [] row, final byte [] family)
throws IOException {
if (coprocessorHost != null) {
Result result = new Result();
if (coprocessorHost.preGetClosestRowBefore(row, family, result)) {
return result;
}
}
// look across all the HStores for this region and determine what the
// closest key is across all column families, since the data may be sparse
checkRow(row, "getClosestRowBefore");
startRegionOperation();
this.readRequestsCount.increment();
try {
Store store = getStore(family);
// get the closest key. (HStore.getRowKeyAtOrBefore can return null)
KeyValue key = store.getRowKeyAtOrBefore(row);
Result result = null;
if (key != null) {
Get get = new Get(key.getRow());
get.addFamily(family);
result = get(get, null);
}
if (coprocessorHost != null) {
coprocessorHost.postGetClosestRowBefore(row, family, result);
}
return result;
} finally {
closeRegionOperation();
}
}

在 KeyValue key = store.getRowKeyAtOrBefore(row);中获得了Meta表的rowkey，但是在后续的实现中

if (key != null) {
Get get = new Get(key.getRow());
get.addFamily(family);
result = get(get, null);
}

获得空的result导致了这个问题;

为什么会存在这个现象。

先讲一下HBase 的MVCC的原理，

MVCC是保证数据一致性的手段，HBase在写数据的过程中，需要经过好几个阶段，写HLog，写memstore，更新MVCC;

只有更新了MVCC，才算真正memstore写成功，其中事务的隔离需要有mvcc的来控制，比如读数据不可以获取别的线程还未提交的数据。

1、put、delete数据都会调用applyFamilyMapToMemstore

HRegion中

private long applyFamilyMapToMemstore(Map<byte[], List<KeyValue>> familyMap,
MultiVersionConsistencyControl.WriteEntry localizedWriteEntry) {
long size = 0;
boolean freemvcc = false;

try {
if (localizedWriteEntry == null) {
//开始一个写memstore，mvcc中的memstoreWrite++，并add待write pending队列中
localizedWriteEntry = mvcc.beginMemstoreInsert();
freemvcc = true;
}

for (Map.Entry<byte[], List<KeyValue>> e : familyMap.entrySet()) {
byte[] family = e.getKey();
List<KeyValue> edits = e.getValue();

Store store = getStore(family);
for (KeyValue kv: edits) {
kv.setMemstoreTS(localizedWriteEntry.getWriteNumber());
size += store.add(kv);
}
}
} finally {
if (freemvcc) {
mvcc.completeMemstoreInsert(localizedWriteEntry);
}
}

return size;
}

mvcc.completeMemstoreInsert，更新mvcc 的memstoreRead，也就是可以读的位置， 并通知readWaiters.notifyAll()，释放因flushcache调用waitForRead引起的阻塞;
waitForRead参见以下代码：
public void waitForRead(WriteEntry e) {
boolean interrupted = false;
synchronized (readWaiters) {
//小于，表示还有写未提交
while (memstoreRead < e.getWriteNumber()) {
try {
readWaiters.wait(0);
} catch (InterruptedException ie) {
// We were interrupted... finish the loop -- i.e. cleanup --and then
// on our way out, reset the interrupt flag.
interrupted = true;
}
}
}
if (interrupted) Thread.currentThread().interrupt();
}

2、 在flushcache的过程中，获取到memstore中的keyvalues后,会调用mvcc.waitForRead(w)(因memstore所有的keyvalue,包括还未真正提交的，所以要等待其他事务提交后，才可以进行后续的flush操作，保证事务的一致性。

w = mvcc.beginMemstoreInsert();
mvcc.advanceMemstore(w);

mvcc.waitForRead(w);

3、scan数据

在RegionScannerImpl.next方法实现中：

public synchronized boolean next(List<KeyValue> outResults, int limit)
throws IOException {
if (this.filterClosed) {
throw new UnknownScannerException("Scanner was closed (timed out?) " +
"after we renewed it. Could be caused by a very slow scanner " +
"or a lengthy garbage collection");
}
startRegionOperation();
readRequestsCount.increment();
try {

// This could be a new thread from the last time we called next().
//this.readPoint在构造的时，初始化（readpoint为当前hregion的mvcc中的memstoreRead，为当前可读的点）和当前线程绑定
MultiVersionConsistencyControl.setThreadReadPoint(this.readPt);

在MemStore中过滤掉还未提交的事务（新的keyvalue中有最新的point）

protected KeyValue getNext(Iterator<KeyValue> it) {
long readPoint = MultiVersionConsistencyControl.getThreadReadPoint();

while (it.hasNext()) {
KeyValue v = it.next();
//过滤掉大于当前线程readPoint的keyvalue
if (v.getMemstoreTS() <= readPoint) {
return v;
}
}

return null;
}

纵观MVCC的整个过程，再分析HRegion中的getClosestRowBefore方法实现，
KeyValue key = store.getRowKeyAtOrBefore(row);
这个调用不会进行MVCC的控制，可以读到memstore中所有的数据
而get方法是会进行MVCC进行控制的，所以一种可能情况是在get调用的时， store.getRowKeyAtOrBefore(row)读到的key值还未提交，
所有都过滤掉了，查询范围为null。