hive的shims相关类分析 推荐

在hive的源码中经常可以看到shims相关的类，shims相关类是用来兼容不同的hadoop和hive版本的，以HadoopShims为例org.apache.hadoop.hive.shims.HadoopShims是一个接口，具体的实现类为

org.apache.hadoop.hive.shims.Hadoop20Shims
org.apache.hadoop.hive.shims.Hadoop20SShims
org.apache.hadoop.hive.shims.Hadoop23Shims

具体对应的那个shims类在ShimLoader类（org.apache.hadoop.hive.shims.ShimLoader）中有所定义，以HADOOP_SHIM_CLASSES 为例：

/**
* The names of the classes for shimming Hadoop for each major version.
*/
private static final HashMap<String, String> HADOOP_SHIM_CLASSES =
new HashMap<String, String>();  // 首先定义了一个hashmap
static {
HADOOP_SHIM_CLASSES.put( "0.20", "org.apache.hadoop.hive.shims.Hadoop20Shims" );  // 0.20
HADOOP_SHIM_CLASSES.put( "0.20S", "org.apache.hadoop.hive.shims.Hadoop20SShims" );  // 1.x和0.20 security版本的hadoop
HADOOP_SHIM_CLASSES.put( "0.23", "org.apache.hadoop.hive.shims.Hadoop23Shims" ); // 2.x版本的hadoop
} // 向hashmap中插入对应的项，key为version信息，value为实际对应的类

ShimLoader类通过getHadoopShims方法返回具体的HadoopShims

private static HadoopShims hadoopShims;
......
public static synchronized HadoopShims getHadoopShims() {
if (hadoopShims == null) {
hadoopShims = loadShims(HADOOP_SHIM_CLASSES, HadoopShims.class); // 调用loadShims方法
}
return hadoopShims;
}

而loadShims方法通过判断hadoop的版本，来获取hashmap中对应的设置类

private static <T> T loadShims(Map<String, String> classMap, Class<T> xface) {
String vers = getMajorVersion();  // 获取hadoop的版本
String className = classMap.get(vers);
return createShim(className, xface);
}

具体哪个hadoop版本是由getMajorVersion获取，比如我们线上使用的是2.0.0-cdh4.6.0，那么getMajorVersion返回的值为0.23，对应的HadoopShims 实现类为org.apache.hadoop.hive.shims.Hadoop23Shims

public static String getMajorVersion() {
String vers = VersionInfo.getVersion();  // org.apache.hadoop.util.VersionInfo类的getVersion方法   比如2.0.0-cdh4.6.0
String[] parts = vers.split( "\\.");
if (parts. length < 2) {
throw new RuntimeException("Illegal Hadoop Version: " + vers +
" (expected A.B.* format)");
}
// Special handling for Hadoop 1.x and 2.x
switch (Integer.parseInt(parts[0])) {
case 0:
break;
case 1:
return "0.20S";
case 2:
return "0.23";
default:
throw new IllegalArgumentException("Unrecognized Hadoop major version number: " + vers);
}
String majorVersion = parts[0] + "." + parts[1];
// If we are running a security release, we won't have UnixUserGroupInformation
// (removed by HADOOP-6299 when switching to JAAS for Login)
try {
Class.forName("org.apache.hadoop.security.UnixUserGroupInformation" );
} catch (ClassNotFoundException cnf) {
if ("0.20".equals(majorVersion)) {
majorVersion += "S";
}
}
return majorVersion;
}

而不同的shims中规定了不同版本下的属性和方法等，比如判断是否localmode：
1). Hadoop20SShims 类(Hadoop 0.20 with Security和1.x版本 )

@Override
public boolean isLocalMode(Configuration conf) {
return "local".equals(getJobLauncherRpcAddress(conf));
}
@Override
public String getJobLauncherRpcAddress( Configuration conf) { //通过判断mapred.job.tracker的设置
return conf.get( "mapred.job.tracker");
}

2).Hadoop20Shims类 (Hadoop 0.20版本) 同上
3).Hadoop23Shims (Hadoop 2.x版本)

@Override
public boolean isLocalMode( Configuration conf) {
return "local".equals(conf.get("mapreduce.framework.name" ));   //根据mapreduce.framework.name设置
}