Python学习笔记:Convert UTF-8 with BOM to UTF-8 without BOM in Python
2017-03-01 20:58
531 查看
前言
windows对于utf-8编码的文件自带BOM,但是其他系统utf-8编码默认不带BOM。这就造成在某些情况下字符解码会出现问题,比如Python自带的json在读取在window下编码得来的utf-8文件时,会报如下错误:
ValueError: No JSON object could be decoded
BOM
From Wikipedia, the free encyclopediaBOM:The byte order mark (BOM) is a Unicode character, U+FEFF BYTE ORDER MARK (BOM), whose appearance as a magic number at the start of a text stream can signal several things to a program consuming
the text:
What byte order, or endianness, the text stream is stored in;
The fact that the text stream is Unicode, to a high level of confidence;
Which of several Unicode encodings that text stream is encoded as.
BOM use is optional, and, if used, should appear at the start of the text stream.
Unicode can be encoded as 8-bit, 16-bit, or 32-bit integers. For the 16- and 32-bit representations, a computer receiving text from arbitrary sources needs to know which byte order the integers
are encoded in. Because the BOM itself is encoded in the same scheme as the rest of the document, but has a known value, the consumer of the text can examine these first few bytes to determine the encoding. The BOM thus gives the producer of the text a way
to describe the text stream’s endianness to the consumer of the text without requiring some contract or metadata outside of the text stream itself.
Once the receiving computer has consumed the text stream, it is free to process the characters in its own native byte order and no longer needs the BOM. Hence the need for a BOM arises in the context
of text interchange, rather than in text processing within a closed environment.
一句话总结:BOM对于utf-16和utf-32有用,对于utf-8没啥大用。。所以能去掉就去掉好了。。
去掉BOM
方法1:
f = open("data","r") s = f.read() u = s.decode("utf-8-sig") # 得到一个不含BOM的unicode string s = u.encode("utf-8") # 将unicode转换为utf-8 f.close()1
2
3
4
5
1
2
3
4
5
方法2:
import codecs f = open("data","r") s = f.read() if s.startswith(codecs.BOM_UTF8): s = s[len(codecs.BOM_UTF8):] f.close()
相关文章推荐
- Python学习笔记:Convert UTF-8 with BOM to UTF-8 without BOM in Python
- python学习笔记 --- 实现将文件转换编码为utf-8-sig(带BOM)
- 【coursera 学习笔记】An Introduction to Interactive Programming in Python--week1
- 【coursera 学习笔记】An Introduction to Interactive Programming in Python--week0
- Convert HTML Table to an Array in Python
- Dive in Python学习笔记四:自省的威力
- java socket client to python server with utf-8
- Dive in Python学习笔记二:第一个Python程序
- string - How to get the filename without the extension from a path in Python? - Stack Overflow
- Python学习笔记三(Try-cacth&with方法)
- How to use UTF-8_with_BOM, XML and Java together
- How to convert ArrayList to Set in Java with Example
- Python学习笔记十二:列表(4)--len 、in、For的使用
- Python学习笔记之疑问 6:#coding=utf-8 与#-*- coding: utf-8 -*-的区别
- (pbc lua 加入)c++_lua_Python with/without extension性能测试 (10万次SerializeToString & ParseFromString)
- Convert SVG to PNG in Python - Stack Overflow
- python built-in function 学习笔记
- 《Beginning Python From Novice to Professional》学习笔记三:List
- javascript学习笔记(4)--with和for(in)对象语句
- How to convert a QString to unicode object in python 2?