shell命令之uniq

uniq，顾名思义是去重，但需注意，它统计的是相邻两行是否重复，因此很多时候会配合sort。

常用参数如下：

-u   只显示不重复的行
-c   每行开头显示重复行出现的次数，重复行显示一次
-d   只显示重复行
-i   对大小写不敏感
-f   跳过n个域
-s   跳过n个字符

举例开始，文件file.txt如下：

this is a test
this is a test
this is a test
i am tank
i love tank
i love tank
this is a test
whom have a try
WhoM have a try
you  have a try
i want to abroad
those are good men
we are good men

1、不加参数的uniq

uniq file.txt

this is a test
i am tank
i love tank
this is a test
whom have a try
WhoM have a try
you  have a try
i want to abroad
those are good men
we are good men

2、显示不重复的行

uniq -u Desktop/test

i am tank
this is a test
whom have a try
WhoM have a try
you  have a try
i want to abroad
those are good men
we are good men

3、统计行的次数

uniq -c file.txt
3 this is a test
1 i am tank
2 i love tank
1 this is a test
1 whom have a try
1 WhoM have a try
1 you  have a try
1 i want to abroad
1 those are good men
1 we are good men

4、忽略大小写，并统计重复行次数

uniq -i -c file.txt
3 this is a test
1 i am tank
2 i love tank
1 this is a test
2 whom have a try    ／／注意这里和上面例子的区别
1 you  have a try
1 i want to abroad
1 those are good men
1 we are good men

5、显示重复行

uniq -d file.txt

this is a test
i love tank

6、跳过n个域

uniq -f 1 -c file.txt

3 this is a test
1 i am tank
2 i love tank
1 this is a test
2 whom have a try
1 you  have a try
1 i want to abroad
2 those are good men

这里跳过第一个域，所以1至3行跳过this，4至6行跳过i，当whom和whoM被跳过时，这两行被当作一样的，至于接下来的一行，跳过you后，还剩2个空格，和上面两行并不一样，这里需要注意。

7、跳过n个字符

uniq -s 4 -c file.txt

3 this is a test
1 i am tank
2 i love tank
1 this is a test
3 whom have a try
1 i want to abroad
1 those are good men
1 we are good men

这里跳过前4个字符，所以和上面的例子类似，但是此时whoM have a try 和 you have a try就是一样的了。