Hadoop2.6.0 Snappy 压缩安装与配置

先说点题外话，我们集群主要使用hive,Impala,spark sql等来处理数据，介于hdfs里面的数据没有压缩占用空间太大所以我们要对其进行压缩

snappy比较适中

snappy 原生被hive等支持

首先看一下你的集群是否支持/安装了snappy

hadoop checknative -a

6/07/08 14:20:48 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native

16/07/08 14:20:48 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library

Native library checking:

hadoop:  true /home/docker/hadoop/apache/hadoop-2.6.0-cdh5.7.0/lib/native/libhadoop.so

zlib:    true /lib64/libz.so.1

snappy:  true /home/docker/hadoop/apache/hadoop-2.6.0-cdh5.7.0/lib/native/libsnappy.so.1

lz4:     true revision:99

bzip2:   true /lib64/libbz2.so.1

openssl: true /lib64/libcrypto.so

我们集群已经安装过了所以都是true 如果有false 就说明你对应的native库没有安装好

hadoop native lib的定义

Native Hadoop Library

Hadoop has native implementations of certain components for performance reasons and for non-availability of Java implementations. These components are available in a single, dynamically-linked native library called the native hadoop library. On the *nix platforms the library is named libhadoop.so.

你需要安装重新编译hadoop源码with native

编程完成后目录应该类似是这样的

[root@dev03 ~]# ll $HADOOP_HOME/lib/native

总用量 19716

-rw-r--r--. 1 hadoop hadoop 1309890 6月  28 12:11 libhadoop.a

-rw-r--r--. 1 hadoop hadoop 1485708 6月  28 12:11 libhadooppipes.a

-rwxr-xr-x. 1 hadoop hadoop  769368 6月  28 12:11 libhadoop.so

-rwxr-xr-x. 1 hadoop hadoop  769368 6月  28 12:11 libhadoop.so.1.0.0

-rw-r--r--. 1 hadoop hadoop  582056 6月  28 12:11 libhadooputils.a

-rw-r--r--. 1 hadoop hadoop  369310 6月  28 12:11 libhdfs.a

-rwxr-xr-x. 1 hadoop hadoop  235700 6月  28 12:11 libhdfs.so

-rwxr-xr-x. 1 hadoop hadoop  235700 6月  28 12:11 libhdfs.so.0.0.0

-rw-r--r--. 1 hadoop hadoop 7705772 6月  28 12:11 libnativetask.a

-rwxr-xr-x. 1 hadoop hadoop 3082245 6月  28 12:11 libnativetask.so

-rwxr-xr-x. 1 hadoop hadoop 3082245 6月  28 12:11 libnativetask.so.1.0.0

-rw-r--r--. 1 hadoop hadoop  233386 6月  28 12:11 libsnappy.a

-rwxr-xr-x. 1 hadoop hadoop     960 6月  28 12:11 libsnappy.la

lrwxrwxrwx. 1 hadoop hadoop      23 6月  28 11:14 libsnappy.so -> /usr/lib64/libsnappy.so

-rwxr-xr-x. 1 hadoop hadoop  147670 6月  28 12:11 libsnappy.so.1

-rwxr-xr-x. 1 hadoop hadoop  147670 6月  28 12:11 libsnappy.so.1.2.0

[root@dev03 ~]#

同步全部节点 重启服务