LC_ALL=en_US.UTF-8 让 awk 慢了 40 倍!
2010-02-05 15:44
295 查看
无意中发现,在一台服务器上,非常简单的awk程序,比C的等价物要慢40倍,感觉有点不太正常,还以为的确是awk太慢。不得其解,到另一台服务上试了一下,相同的 awk 程序,相同的测试数据,这台服务器的速度与C相当,也就是说,同样是awk,两台机器速度差了 40倍,而两台机器配置基本相当。非常困惑,找了两小时的原因,终于发现gawk手册里面有一段话:
For other single-character record separators, using ‘LC_ALL=C’will give you much better performance when reading records. Otherwise,gawk has to make
several function calls, per inputcharacter to find the record terminator.
再看两台机器的 locale,结果发现,慢的机器上:
[root@slow-server]# locale
LANG=en_US.UTF-8
LC_XXXX=en_US.UTF-8
...
LC_ALL=en_US.UTF-8
快的机器上:
[root@fast-server]# locale
LANG=en_US
LC_XXXX=en_US
...
LC_ALL= <空>
马上试验,将slow-server的locale改掉:
export LC_ALL=C
速度马上快了40倍,与fast-server相当。
这应该是awk实现上的一个缺陷,即便是对utf8,也不应该慢这么多,如果缓冲合适,最多慢2~3倍就可以了,为什么非要gawk has to make several function calls,
per inputcharacter ?
For other single-character record separators, using ‘LC_ALL=C’will give you much better performance when reading records. Otherwise,gawk has to make
several function calls, per inputcharacter to find the record terminator.
再看两台机器的 locale,结果发现,慢的机器上:
[root@slow-server]# locale
LANG=en_US.UTF-8
LC_XXXX=en_US.UTF-8
...
LC_ALL=en_US.UTF-8
快的机器上:
[root@fast-server]# locale
LANG=en_US
LC_XXXX=en_US
...
LC_ALL= <空>
马上试验,将slow-server的locale改掉:
export LC_ALL=C
速度马上快了40倍,与fast-server相当。
这应该是awk实现上的一个缺陷,即便是对utf8,也不应该慢这么多,如果缓冲合适,最多慢2~3倍就可以了,为什么非要gawk has to make several function calls,
per inputcharacter ?
相关文章推荐
- LC_ALL=en_US.UTF-8 让 awk 慢了 40 倍! 【转】
- LC_ALL=en_US.UTF-8 让 awk 慢了 40 倍!
- LC_ALL=en_US.UTF-8 让 awk 慢了 40 倍! 【转】
- bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
- warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8): No such file or directory
- -bash: warning: setlocale: LC_CTYPE: cannot change locale (EN_US.UTF-8)
- -bash: warning: setlocale: LC_CTYPE: cannot change locale (EN_US.UTF-8)
- bug宝典linux篇 LC_CTYPE: cannot change locale (en_US.UTF-8): No such file or directory
- -bash: warning: setlocale: LC_CTYPE: cannot change locale (EN_US.UTF-8)
- 【云计算】Docker 镜像如何设置语言环境?bash: warning: setlocale: LC_ALL: cannot change locale (en_US)
- -bash: warning: setlocale: LC_CTYPE: cannot change locale (EN_US.UTF-8): No such file or directory
- Linux之——bash: warning: setlocale: LC_CTYPE: cannot change locale (EN_US.UTF-8)
- CentOS -bash: warning: setlocale: LC_MESSAGES: cannot change locale (en_US.UTF-8)
- -bash: warning: setlocale: LC_CTYPE: cannot change locale (EN_US.UTF-8)
- Linux操作系统下终端乱码的终极解决方案 export LANG=zh_CN.UTF-8 export LANG=en_US
- UTF-8、en_US.UTF-8和zh_CN.UTF-8的区别
- CentOS7添加en_US.UTF-8字符集
- 将Ubuntu系统语言环境改为英文的en_US.UTF-8
- Linux操作系统下终端乱码的终极解决方案 export LANG=zh_CN.UTF-8 export LANG=en_US
- 将ubuntu 8.04 (server x86_64)的系统编码由英文改成中文,即由en_US.UTF-8改为zh_CN.GBK