hive 存储格式和压缩方式 一：Snappy + SequenceFile

为什么要用Sequence File:

a).压缩 b).这种格式可分割，可供多个mapper 并发读取

贴一段《Programming Hive》的：

Compressing files results in space savings but one of the downsides of storing raw
compressed files in Hadoop is that often these files are not splittable. Splittable files
can be broken up and processed in parts by multiple mappers in parallel. Most com-
pressed files are not splittable because you can only start reading from the beginning.
The sequence file format supported by Hadoop breaks a file into blocks and then op-
tionally compresses the blocks in a splittable way.

下面就用它一用

1、设置三个参数：

hive.exec.compress.output 声明对 hive 查询的输出结果进行压缩，并指定压缩方式为 Snappy。
对SequenceFile 有 mapred.output.compression.type，在CDH4中默认就是 BLOCK。

SET hive.exec.compress.output=true;

SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;

SET mapred.output.compression.type=BLOCK;

> SET hive.exec.compress.output;
hive.exec.compress.output=false
hive (sequence)> SET hive.exec.compress.output=true;
hive (sequence)> SET mapred.output.compression.codec;
mapred.output.compression.codec=org.apache.hadoop.io.compress.DefaultCodec
hive (sequence)> SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;
hive (sequence)> SET mapred.output.compression.type;
mapred.output.compression.type=BLOCK
hive (sequence)> set io.seqfile.compress.blocksize;
io.seqfile.compress.blocksize=1000000

2、创建TEXTFILE表，用来提供数据。

CREATE TABLE info_text(
......)
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;' LINES TERMINATED BY '\n' STORED AS TEXTFILE;

3、建目标SequenceFile 表

CREATE TABLE info_sequ(
...... )
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\;' LINES TERMINATED BY '\n' STORED AS SEQUENCEFILE;

4、为text表导入数据,一个400+M的数据文件

load data local inpath 'file.txt' OVERWRITE into table info_text;

5、为目标 SequenceFile 表导入数据

insert into table info_sequ select * from info_text;

6、测试结果是否相同

select * from info_sequ limit 20;
select * from info_text limit 20;

7、查看HDFS文件大小

drwxrwxrwx   - hive hadoop          0 2013-08-30 01:33 /user/hive/warehouse/sequence.db/info_sequ
-rw-r--r--   3 root hadoop  124330943 2013-08-30 01:33 /user/hive/warehouse/sequence.db/info_sequ/000000_0
-rw-r--r--   3 root hadoop   77730350 2013-08-30 01:33 /user/hive/warehouse/sequence.db/info_sequ/000001_0
drwxr-xr-x   - root hadoop          0 2013-08-30 01:33 /user/hive/warehouse/sequence.db/info_text
-rw-r--r--   3 root hadoop  438474539 2013-08-30 01:33 /user/hive/warehouse/sequence.db/info_text/file.txt

000000_0 文件开头：

SEQ^F"org.apache.hadoop.io.BytesWritable^Yorg.apache.hadoop.io.Text^A^A)org.apache.hadoop.io.compress.SnappyCodec^@^@^@^@<84>çé<9f>³ÅZ<97>/.¹*¿I²6ÿÿÿÿ<84>çé<9f>³ÅZ<97>/.¹*¿I²6<8e>2^M<8e>^Bg^@^@2

000001_0 文件开头：

SEQ^F"org.apache.hadoop.io.BytesWritable^Yorg.apache.hadoop.io.Text^A^A)org.apache.hadoop.io.compress.SnappyCodec^@^@^@^@[Ü^^Ü<9c>rx1µå'^HçÕcöÿÿÿÿ[Ü^^Ü<9c>rx1µå'^HçÕcö<8e>2Y<8e>^Bj^@^@2Y^@^@^BbÙ

Hive 中有两个虚拟列：

INPUT__FILE__NAME  map 任务的输入文件名

BLOCK__OFFSET__INSIDE__FILE   当前文件的 position。据我理解，就如NIO的Buffer中的position。

hive> describe formatted locallzo;
# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: com.hadoop.mapred.DeprecatedLzoTextInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

hive> describe formatted raw;
# Storage Information
SerDe Library: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe
InputFormat: org.apache.hadoop.mapred.TextInputFormat
OutputFormat: org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat

可以比较两个表 BLOCK__OFFSET__INSIDE__FILE 的区别：

存储文件为文本文件:
hive> select INPUT__FILE__NAME, unum, BLOCK__OFFSET__INSIDE__FILE from raw limit 10；
hdfs://indigo:8020/user/hive/warehouse/raw/state=26/rawdata.txt 3246065 0
hdfs://indigo:8020/user/hive/warehouse/raw/state=26/rawdata.txt 2037265 73
hdfs://indigo:8020/user/hive/warehouse/raw/state=26/rawdata.txt 2287465 149
hdfs://indigo:8020/user/hive/warehouse/raw/state=26/rawdata.txt 6581865 225
hdfs://indigo:8020/user/hive/warehouse/raw/state=26/rawdata.txt 6581865 298
hdfs://indigo:8020/user/hive/warehouse/raw/state=26/rawdata.txt 6581865 371
hdfs://indigo:8020/user/hive/warehouse/raw/state=26/rawdata.txt 1629568 447
hdfs://indigo:8020/user/hive/warehouse/raw/state=26/rawdata.txt 2185765 523

存储文件为lzo压缩文件:
hive> select INPUT__FILE__NAME, office_no, BLOCK__OFFSET__INSIDE__FILE from locallzo limit 20;
hdfs://indigo:8020/tmp/table/tb/rawdata.txt.lzo 3246065 0
hdfs://indigo:8020/tmp/table/tb/rawdata.txt.lzo 2037265 28778
hdfs://indigo:8020/tmp/table/tb/rawdata.txt.lzo 2287465 28778
hdfs://indigo:8020/tmp/table/tb/rawdata.txt.lzo 6581865 28778
hdfs://indigo:8020/tmp/table/tb/rawdata.txt.lzo 6581865 28778
hdfs://indigo:8020/tmp/table/tb/rawdata.txt.lzo 6581865 28778
hdfs://indigo:8020/tmp/table/tb/rawdata.txt.lzo 1629568 28778