Solr4.7.2启动时的Index locked for write for core问题分析

Solr在启动时，通过多线程的方式加载core，在加载完每个core的配置文件后，实例化了一个SolrCore，在SolrCore的构造函数中，会初始化index，Index locked for write for core的Exception就是在这个时候报出来的，Solr启动过程和SolrCore的构造函数可看之前的文章：
http://blog.csdn.net/wenchanter/article/details/37603621
此处主要看下引发Index locked for write for core错误的代码：

首先看下SolrCore构造函数中的initIndex方法：

void initIndex(boolean reload) throws IOException {
// 着里面用dataDir生成了一个Directory，并设置了锁工厂
String indexDir = getNewIndexDir();
boolean indexExists = getDirectoryFactory().exists(indexDir);
boolean firstTime;
synchronized (SolrCore.class) {
firstTime = dirs.add(getDirectoryFactory().normalize(indexDir));
}
boolean removeLocks = solrConfig.unlockOnStartup;

// solrconfig.xml中得indexReaderFactory,若不配置，也会默认StandardIndexReaderFactory
initIndexReaderFactory();

if (indexExists && firstTime && !reload) {
// 除了上面那个getNewIndexDir中的用dataDir创建的Directory设置了lockFactory，这里又用其返回的index路径创建了一个Directory，并设置了一遍lockFactory，真正判断是否加锁的时用这个Directory
Directory dir = directoryFactory.get(indexDir, DirContext.DEFAULT,
getSolrConfig().indexConfig.lockType);
try {
if (IndexWriter.isLocked(dir)) {
if (removeLocks) {
log.warn(
logid
+ "WARNING: Solr index directory '{}' is locked.  Unlocking...",
indexDir);
IndexWriter.unlock(dir);
} else {
log.error(logid
+ "Solr index directory '{}' is locked.  Throwing exception",
indexDir);
throw new LockObtainFailedException(
"Index locked for write for core " + name);
}

}
} finally {
directoryFactory.release(dir);
}
}

// Create the index if it doesn't exist.
if(!indexExists) {
log.warn(logid+"Solr index directory '" + new File(indexDir) + "' doesn't exist."
+ " Creating new index...");

SolrIndexWriter writer = SolrIndexWriter.create("SolrCore.initIndex", indexDir, getDirectoryFactory(), true,
getLatestSchema(), solrConfig.indexConfig, solrDelPolicy, codec);
writer.close();
}
}

可见是用IndexWriter.isLocked(dir)判断是否加锁，如果已经加了锁，则分为两种情况，一种是在solrconfig.xml中配置了unlockOnStartup，会尝试unlock，如果没有配置unlockStartup，则会抛出Index locked for write for core异常。

这个被校验的dir是一个Directory，这个Directory是通过

Directory dir
= directoryFactory.get(indexDir, DirContext.DEFAULT,

getSolrConfig().indexConfig.lockType);

得到的，其中indexDir是上面取到的$solrhome/collection1/data/index索引存放的文件夹，directoryFactory此处是NRTCachingDirectoryFactory实例，看下其中的get方法：

@Override
public final Directory get(String path,  DirContext dirContext, String rawLockType)
throws IOException {
String fullPath = normalize(path);
synchronized (this) {
if (closed) {
throw new AlreadyClosedException("Already closed");
}

final CacheValue cacheValue = byPathCache.get(fullPath);
Directory directory = null;
if (cacheValue != null) {
directory = cacheValue.directory;
}

if (directory == null) {
// 由于是NRTCachingDirectoryFactory，这里面直接new了一个NTRCachingDirectory，里面的FSDirectory.open(File)的方法根据不同的系统new不同的FSDirectory.
directory = create(fullPath, dirContext);

directory = rateLimit(directory);

CacheValue newCacheValue = new CacheValue(fullPath, directory);
// 这里设置锁工厂
injectLockFactory(directory, fullPath, rawLockType);

byDirectoryCache.put(directory, newCacheValue);
byPathCache.put(fullPath, newCacheValue);
log.info("return new directory for " + fullPath);
} else {
cacheValue.refCnt++;
log.debug("Reusing cached directory: {}", cacheValue);
}

return directory;
}
}

get方法中的injectLockFactory设置了锁工厂，根据不同的属性设置不同的锁工厂，此处配置的是native，因此设置了一个NativeFSLockFactory：

private static Directory injectLockFactory(Directory dir, String lockPath,
String rawLockType) throws IOException {
if (null == rawLockType) {
// we default to "simple" for backwards compatibility
log.warn("No lockType configured for " + dir + " assuming 'simple'");
rawLockType = "simple";
}
final String lockType = rawLockType.toLowerCase(Locale.ROOT).trim();

if ("simple".equals(lockType)) {
// multiple SimpleFSLockFactory instances should be OK
dir.setLockFactory(new SimpleFSLockFactory(lockPath));
} else if ("native".equals(lockType)) {
dir.setLockFactory(new NativeFSLockFactory(lockPath));
} else if ("single".equals(lockType)) {
if (!(dir.getLockFactory() instanceof SingleInstanceLockFactory)) dir
.setLockFactory(new SingleInstanceLockFactory());
} else if ("hdfs".equals(lockType)) {
Directory del = dir;

if (dir instanceof NRTCachingDirectory) {
del = ((NRTCachingDirectory) del).getDelegate();
}

if (del instanceof BlockDirectory) {
del = ((BlockDirectory) del).getDirectory();
}

if (!(del instanceof HdfsDirectory)) {
throw new SolrException(ErrorCode.FORBIDDEN, "Directory: "
+ del.getClass().getName()
+ ", but hdfs lock factory can only be used with HdfsDirectory");
}

dir.setLockFactory(new HdfsLockFactory(((HdfsDirectory)del).getHdfsDirPath(), ((HdfsDirectory)del).getConfiguration()));
} else if ("none".equals(lockType)) {
// Recipe for disaster
log.error("CONFIGURATION WARNING: locks are disabled on " + dir);
dir.setLockFactory(NoLockFactory.getNoLockFactory());
} else {
throw new SolrException(SolrException.ErrorCode.SERVER_ERROR,
"Unrecognized lockType: " + rawLockType);
}
return dir;
}

上面再new NativeFSLockFactory时，构造函数中调用父类FSLockFactory中的setLockDir(File lockDir)方法，这个方法只能调用一次，另外，在dir.setLockFactory时，也做了一些工作，看下一连串的方法调用：

NTRCachingDirectory中的setLockFactory:
@Override
public void setLockFactory(LockFactory lf) throws IOException {
// 这里是了NIOFSDirectory
delegate.setLockFactory(lf);
}
这里调用的是FSDirectory的setLockFactory:
@Override
public void setLockFactory(LockFactory lockFactory) throws IOException {
super.setLockFactory(lockFactory);

// for filesystem based LockFactory, delete the lockPrefix, if the locks are placed
// in index dir. If no index dir is given, set ourselves
if (lockFactory instanceof FSLockFactory) {
final FSLockFactory lf = (FSLockFactory) lockFactory;
final File dir = lf.getLockDir();
// if the lock factory has no lockDir set, use the this directory as lockDir
if (dir == null) {
lf.setLockDir(directory);
lf.setLockPrefix(null);
} else if (dir.getCanonicalPath().equals(directory.getCanonicalPath())) {
lf.setLockPrefix(null);
}
}

}
super.setLockFactory是指父类BaseDirectory：
@Override
public void setLockFactory(LockFactory lockFactory) throws IOException {
assert lockFactory != null;
this.lockFactory = lockFactory;
lockFactory.setLockPrefix(this.getLockID());
}
然后着里面的getLockID方法又使调用的FSDirectory中的,看看是不是可以发现LockID计算方法和String的hash算法是一样的：
@Override
public String getLockID() {
ensureOpen();
String dirName;                               // name to be hashed
try {
dirName = directory.getCanonicalPath();
} catch (IOException e) {
throw new RuntimeException(e.toString(), e);
}

int digest = 0;
for(int charIDX=0;charIDX<dirName.length();charIDX++) {
final char ch = dirName.charAt(charIDX);
digest = 31 * digest + ch;
}
return "lucene-" + Integer.toHexString(digest);
}

看完了设置锁工厂，再来看下校验锁的方法：

IndexWriter.isLocked(dir);

/**
* Returns <code>true</code> iff the index in the named directory is
* currently locked.
* @param directory the directory to check for a lock
* @throws IOException if there is a low-level IO error
*/
public static boolean isLocked(Directory directory) throws IOException {
return directory.makeLock(WRITE_LOCK_NAME).isLocked();
}

其中makeLock只是返回了一个NativeFSLock实例，WRITE_LOCK_NAME是默认的“write.lock”，看下NativeFSLock的isLocked()方法：

@Override
public synchronized boolean isLocked() {
// The test for is isLocked is not directly possible with native file locks:

// First a shortcut, if a lock reference in this instance is available
if (lockExists()) return true;

// Look if lock file is present; if not, there can definitely be no lock!
if (!path.exists()) return false;

// Try to obtain and release (if was locked) the lock
try {
boolean obtained = obtain();
if (obtained) close();
return !obtained;
} catch (IOException ioe) {
return false;
}
}

看下obtain()方法，这时的lockDir是目录$solrhome/collection1/data/index，这个属性是当时在创建Directory时setLockFactory时，新建NativeFSLock实例的时候设置的，就是那个只能设置一次的方法，而obtain后面的close方法，会释放锁：

@Override
public synchronized boolean obtain() throws IOException {

if (lockExists()) {
// Our instance is already locked:
return false;
}

// Ensure that lockDir exists and is a directory.
if (!lockDir.exists()) {
if (!lockDir.mkdirs())
throw new IOException("Cannot create directory: " +
lockDir.getAbsolutePath());
} else if (!lockDir.isDirectory()) {
// TODO: NoSuchDirectoryException instead?
throw new IOException("Found regular file where directory expected: " +
lockDir.getAbsolutePath());
}
// path是个File，/Users/wanghui/Documents/workspace/lucene_solr_4_7_2/solr/solrweb/solrhome/collection1/data/index/write.lock
String canonicalPath = path.getCanonicalPath();

boolean markedHeld = false;

try {

// Make sure nobody else in-process has this lock held
// already, and, mark it held if not:

synchronized(LOCK_HELD) {
if (LOCK_HELD.contains(canonicalPath)) {
// Someone else in this JVM already has the lock:
return false;
} else {
// This "reserves" the fact that we are the one
// thread trying to obtain this lock, so we own
// the only instance of a channel against this
// file:
LOCK_HELD.add(canonicalPath);
markedHeld = true;
}
}

try {
f = new RandomAccessFile(path, "rw");
} catch (IOException e) {
// On Windows, we can get intermittent "Access
// Denied" here.  So, we treat this as failure to
// acquire the lock, but, store the reason in case
// there is in fact a real error case.
failureReason = e;
f = null;
}

if (f != null) {
try {
channel = f.getChannel();
try {
lock = channel.tryLock();
} catch (IOException e) {
// At least on OS X, we will sometimes get an
// intermittent "Permission Denied" IOException,
// which seems to simply mean "you failed to get
// the lock".  But other IOExceptions could be
// "permanent" (eg, locking is not supported via
// the filesystem).  So, we record the failure
// reason here; the timeout obtain (usually the
// one calling us) will use this as "root cause"
// if it fails to get the lock.
failureReason = e;
} finally {
if (lock == null) {
try {
channel.close();
} finally {
channel = null;
}
}
}
} finally {
if (channel == null) {
try {
f.close();
} finally {
f = null;
}
}
}
}

} finally {
if (markedHeld && !lockExists()) {
synchronized(LOCK_HELD) {
if (LOCK_HELD.contains(canonicalPath)) {
LOCK_HELD.remove(canonicalPath);
}
}
}
}
return lockExists();
}

可以看到其中LOCK_HELD这个set如果里面包含这个路径也会直接返回false，如果LOCK_HELD不包含此路径，则会尝试获得锁。

从上面的分析可以看出，此处只是校验是否可以加锁，而真正加锁的地方，其实是在后面实例化SolrIndexWriter的时候，SolrIndeWriter调用父类IndexWriter的构造函数的时候加的锁，具体过程可见solr启动分析的这篇文章：
http://blog.csdn.net/wenchanter/article/details/37743977