hadoop第二周作业

一、在Hadoop集群中编译并运行《权威指南》中的例3.2

在hdfs系统中创建测试文件test1.txt，并随便编写一些内容



在hadoop目录下创建myclass文件夹



编写class文件FileSystemCat.java

import java.io.IOException;
import java.io.InputStream;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class FileSystemCat {
public static void main(String[] args) throws IOException {
String uri  =args[0];
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(uri),conf);
InputStream in = null;
try{
in = fs.open(new Path(uri));
IOUtils.copyBytes(in, System.out, 4096,false);
}finally{
IOUtils.closeStream(in);
}
}
}

编译java文件

// 由于环境是hadoop2.x所以，没有hadoop1.x中的hadoop-core.jar文件，需要用到hadoop-common和hadoop-annotations
javac -classpath $HADOOP_HOME/share/hadoop/common/hadoop-common-2.4.1.jar:$HADOOP_HOME/share/hadoop/common/lib/hadoop-annotations-2.4.1.jar FileSystemCat.java

执行命令



生成的.class文件



利用编译得到的class文件执行查看功能



可以看到，test1.txt中的内容是“hello hadoop”

二、上传

自行在本地文件系统生成一个大约一百多字节的文本文件，写一段程序（可以利用Java API或C API），读入这个文件，并将其第101-120字节的内容写入HDFS成为一个新文件，提供代码和执行结果演示抓图

创建本地测试文件test2.txt

There are moments in life when you miss someone so much that you just want to pick them from your dreams and hug them for real! Dream what you want to dream;go where you want to go;be what you want to be,because you have only one life and one chance to do all the things you want to doThere are moments in life when you miss someone so much that you just want to pick them from your dreams and hug them for real! Dream what you want to dream;go where you want to go;

编写上传到hdfs的代码

import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class UploadFileBetwen101To102Byte {

public static void main(String[] args) throws IOException {
String src = args[0];
String tar = args[1];
FileInputStream in = null;
OutputStream out = null;
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(tar),conf);
try{
in = new FileInputStream(new File(src));
out = fs.create(new Path(tar));
in.skip(100);
byte[] buffer = new byte[20];
int read = in.read(buffer);
out.write(buffer,0,read);
}finally{
IOUtils.closeStream(in);
IOUtils.closeStream(out);
}

}
}

编译运行，并查看文件

三、下载

2的反向操作，在HDFS中生成一个大约一百多字节的文本文件，写一段程序（可以利用Java API或C API），读入这个文件，并将其第101-120字节的内容写入本地文件系统成为一个新文件，提供代码和执行结果演示抓图

在hdfs上创建测试文件test3.txt

There are moments in life when you miss someone so much that you just want to pick them from your dreams and hug them for real! Dream what you want to dream;go where you want to go;be what you want to be,because you have only one life and one chance to do all the things you want to doThere are moments in life when you miss someone so much that you just want to pick them from your dreams and hug them for real! Dream what you want to dream;go where you want to go;

编写下载到本地的代码

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class DownloadFileBetwen101To102Byte {

public static void main(String[] args) throws IOException {
String src = args[0];
String tar = args[1];
InputStream in = null;
OutputStream out = null;
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(URI.create(tar),conf);
try{
in = fs.open(new Path(src));
out = new FileOutputStream(new File(tar));
in.skip(100);
byte[] buffer = new byte[20];
int read = in.read(buffer);
out.write(buffer,0,read);
}finally{
IOUtils.closeStream(in);
IOUtils.closeStream(out);
}

}
}

编译，运行，查看文件