Hadoop URL读取数据

1. 原理

通过URL的setURLStreamHandlerFactory方法设置Hadoop文件系统的URLStreamHandlerFactory的实现类FsUrlStreamHandlerFactory，这个方法在一个java虚拟机中只能被调用一次，因此一般都是放在一个static块中执行。

2. 代码

import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;

import java.io.InputStream;
import java.net.URL;

public class HDFSURLReader {
static {
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
}

public static void main(String[] args) {
InputStream inputStream = null;
try {
inputStream = new URL(args[0]).openStream();
IOUtils.copyBytes(inputStream, System.out, 1024, false);
} catch (Exception e) {
IOUtils.closeStream(inputStream);
}
}
}

3. 运行结果

[hadoop@master hdfs]$ hdfs dfs -cat hdfs://master:9000/user/hadoop/study/mr/WordCount/input/file1.txt
Hello, i love coding
are you ok?
Hello, i love hadoop
areyou ok?
[hadoop@master hdfs]$ hadoop jar HDFSURLReader.jar hdfs://master:9000/user/hadoop/study/mr/WordCount/input/file1.txt
Hello, i love coding
are you ok?
Hello, i love hadoop
areyou ok?