您的位置：首页 > 其它

[Erlang]用erlang实现binary_to_term

2015-06-04 14:06 429 查看

erlang提供了两个函数用于erlang数据与二进制的转换，term_to_binary 把erlang数据转成一个二进制数据，binary_to_term 则是把二进制数据转为原始的erlang数据。这两个函数都是c实现的，效率很高，可用于序列化和反序列化数据，可以被直接用来当作网络封包协议。文章主要探讨这种二进制协议数据的组织形式，以及用erlang语言实现binary_to_term

前言

接触erlang可能有这些疑问，或者听别人这样说起：1、erlang数字可以无限长，远远超过了INT64所能表示的范围，怎么表示的？2、原子不能太长，会影响封包大小？3、还有，旧版本erlang浮点数占用太多空间？别再疑惑了，以上这些都可以在文章中找到答案。
下面演示下这两个函数：[plain] view plain copy1> term_to_binary(a).
<<131,100,0,1,97>>
2> term_to_binary(1).
<<131,97,1>>
3> term_to_binary({a,b,c}).
<<131,104,3,100,0,1,97,100,0,1,98,100,0,1,99>>
4> binary_to_term(v(1)).
a
5> binary_to_term(v(2)).
1
6> binary_to_term(v(3)).
{a,b,c}
可以看出 term_to_binary/1 生成的二进制数据是比较有规律的，第1个字节固定是131，遵循着一定的封包格式，数据才能被解析出来。知道这个二进制数据的组织形式，就可以用任意语言解包Erlang数据。

初识binary数据

现在重点介绍长整形、原子、浮点数、列表的协议规则，其他数据协议参考文档：External Term Format

erlang长整形

现在重点看下erlang长整形（erlang叫法是Bignums）:

10.18 SMALL_BIG_EXT

1	1	1	n
110	n	Sign	d(0) ... d(n-1)

Table 10.24: Bignums are stored in unary form with a Sign byte that is 0 if the binum is positive and 1 if is negative. The digits are stored with the LSB byte stored first. To calculate the integer the following formula can be used:
B = 256
(d0*B^0 + d1*B^1 + d2*B^2 + ... d(N-1)*B^(n-1))

实际上，erlang一个长整形会被“切分”成多个段，每个段数据用8位表示。公式为：(d0*B^0 + d1*B^1 + d2*B^2 + ... d(N-1)*B^(n-1))，其中，B = 256所以，理论上erlang数字可以无限长。

erlang原子

10.7 ATOM_EXT

1	2	Len
100	Len	AtomName

Table 10.13: An atom is stored with a 2 byte unsigned length in big-endian order, followed by Len numbers of 8 bit Latin1 characters that forms theAtomName. Note: The maximum allowed value for Len is 255.

erlang原子会转成字符串，所以，太长会影响二进制封包大小。

erlang浮点数

10.6 FLOAT_EXT

1	31
99	Float String

Table 10.12: A float is stored in string format. the format used in sprintf to format the float is "%.20e" (there are more bytes allocated than necessary). To unpack the float use sscanf with format "%lf".

以上是旧版本erlang浮点数的表示方式，一个erlang浮点数就使用了31个字节来表示，相当浪费。

10.26 NEW_FLOAT_EXT

1	8
70	IEEE float

Table 10.32: A float is stored as 8 bytes in big-endian IEEE format.This term is used in minor version 1 of the external format.

以上的新版的erlang浮点数表示，使用了8个字节来表示一个浮点数，精简了很多。[plain] view plain copy

Eshell V5.9.1 (abort with ^G)
1> term_to_binary(19.2).
<<131,99,49,46,57,49,57,57,57,57,57,57,57,57,57,57,57,57,
  57,57,48,48,48,48,101,43,48,48,49,...>>
[plain] view plain copy

Eshell V6.2 (abort with ^G)
1> term_to_binary(19.2).
<<131,70,64,51,51,51,51,51,51,51>>

erlang列表

10.16 LIST_EXT

1	4
108	Length	Elements	Tail

Table 10.22: Length is the number of elements that follows in the Elements section. Tail is the final tail of the list; it is NIL_EXT for a proper list, but may be anything type if the list is improper (for instance [a|b]).

列表由三部分组成：长度、元素、nil标志，这里需要注意的是这个nil标志，如果是空列表，就只有nil标志，记NIL_EXT；元祖跟列表类似，少了nil标志。
好了，理论到这里。下面贴个例子，简单识别binary数据：[plain] view plain copy

7> c(test).
{ok,test}
8> test:term_to_data({a,1,"abc"}).
atom a
int 1
string <<97,98,99>>
ok
9> term_to_binary(11111111111111).
<<131,110,6,0,199,177,212,1,27,10>>
10> test:binary_to_data(term_to_binary(11111111111111)).
big 11111111111111
ok

实现binary_to_term

下面改写上面的例子，用erlang语言实现binary_to_term，这里除了要识别二进制数据，还要将这些数据转成原始的 erlang 数据。[plain] view plain copy

-module(test).
-compile(export_all).

term_to_data(Term) ->
    Bin = term_to_binary(Term),
    binary_to_data(Bin).

binary_to_data(<<131, Bin/binary>>) ->
    binary_to_data1(Bin, [], []);
binary_to_data(_) ->
    error.

-define(NEW_FLOAT_EXT, 70).
-define(SMALL_INTEGER_EXT, 97).
-define(INTEGER_EXT, 98).
-define(FLOAT_EXT, 99).
-define(ATOM_EXT, 100).
-define(SMALL_TUPLE_EXT, 104).
-define(LARGE_TUPLE_EXT, 105).
-define(NIL_EXT, 106).
-define(STRING_EXT, 107).
-define(LIST_EXT, 108).
-define(BINARY_EXT, 109).
-define(SMALL_BIG_EXT, 110).
-define(LARGE_BIG_EXT, 111).
-define(SMALL_ATOM_EXT, 115).

binary_to_data2( DataList, SizeList, Data) ->
    DataList1 = case Data of
        undefined -> DataList;
        _ -> [Data|DataList]
    end,
    case SizeList of
        [{Type, Size, Index}|R] ->
            Index1 = Index +1,
            case Size=:=Index1 of
                true ->
                    {List, DataList2} = split_list(Type, DataList1, Size, []),
                    DataList3 = gen_data_block(Type, List, DataList2),
                    case R of
                        [_|_] ->
                            binary_to_data2( DataList3, R, undefined);
                        _ ->
                            {DataList3, R}
                    end;
                _ ->
                    {DataList1, [{Type, Size, Index1}|R]}
            end;
        _ ->
            {DataList1, SizeList}
    end.

split_list(list, [[]|TailList], Size, List) ->
    split_list(list, TailList, Size-1, List);
split_list(_Type, [], _Size, List) ->
    {List,[]};
split_list(_Type, TailList, 0, List) ->
    {List, TailList};
split_list(Type, [Data|TailList], Size, List) ->
    split_list(Type, TailList, Size-1, [Data|List]).


gen_data_block(tuple, List, DataList) ->
    [list_to_tuple(List)|DataList];
gen_data_block(list, List, DataList) ->
    [List|DataList].

binary_to_data1(<<?LARGE_TUPLE_EXT, ElemtSize:32, Bin/binary>>, DataList, SizeList) ->
    binary_to_data1(Bin, DataList, [{tuple, ElemtSize, 0}|SizeList]);
binary_to_data1(<<?SMALL_TUPLE_EXT, ElemtSize:8, Bin/binary>>, DataList, SizeList) ->
    binary_to_data1(Bin, DataList, [{tuple, ElemtSize, 0}|SizeList]);
binary_to_data1(<<?SMALL_INTEGER_EXT, Int:8, Bin/binary>>, DataList, SizeList) ->
    %%msg(int, Int),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, Int),
    binary_to_data1(Bin, DataList1, SizeList1);
binary_to_data1(<<?INTEGER_EXT, Int:32, Bin/binary>>, DataList, SizeList) ->
    %%msg(int2, Int),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, Int),
    binary_to_data1(Bin, DataList1, SizeList1);
binary_to_data1(<<?FLOAT_EXT, F:31/binary, Bin/binary>>, DataList, SizeList) ->
    Float = erlang:binary_to_float(F),
    %%msg(float, Float),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, Float),
    binary_to_data1(Bin, DataList1, SizeList1);
binary_to_data1(<<?NEW_FLOAT_EXT, Float:64/unsigned-big-float, Bin/binary>>, DataList, SizeList) ->
    %%msg(float2, Float),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, Float),
    binary_to_data1(Bin, DataList1, SizeList1);
binary_to_data1(<<?SMALL_BIG_EXT, N:8, _Sign:8, Bin:N/binary, Rest/binary>>, DataList, SizeList) ->
    {N, Big} = gen_small_big(Bin),
    %%msg(big, Big),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, Big),
    binary_to_data1(Rest, DataList1, SizeList1);
binary_to_data1(<<?LARGE_BIG_EXT, N:32, _Sign:8, Bin:N/binary, Rest/binary>>, DataList, SizeList) ->
    {N, Big} = gen_small_big(Bin),
    %%msg(big2, Big),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, Big),
    binary_to_data1(Rest, DataList1, SizeList1);
binary_to_data1(<<?ATOM_EXT, Len:16, Bin:Len/binary, Rest/binary>>, DataList, SizeList) ->
    Atom = erlang:binary_to_atom(Bin, latin1),
    %%msg(atom, Atom),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, Atom),
    binary_to_data1(Rest, DataList1, SizeList1);
binary_to_data1(<<?SMALL_ATOM_EXT, Len:8, Bin:Len/binary, Rest/binary>>, DataList, SizeList) ->
    Atom = erlang:binary_to_atom(Bin, latin1),
    %%msg(atom2, Atom),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, Atom),
    binary_to_data1(Rest, DataList1, SizeList1);
binary_to_data1(<<?STRING_EXT, Len:16, Bin:Len/binary, Rest/binary>>, DataList, SizeList) ->
    String = binary_to_list(Bin),
    %%msg(string, String),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, String),
    binary_to_data1(Rest, DataList1, SizeList1);
binary_to_data1(<<?BINARY_EXT, Len:32, Bin:Len/binary, Rest/binary>>, DataList, SizeList) ->
    %%msg(binary, Bin),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, Bin),
    binary_to_data1(Rest, DataList1, SizeList1);
binary_to_data1(<<?LIST_EXT, ElemtSize:32, Bin/binary>>, DataList, SizeList) ->
    %%msg(list, Bin),
    binary_to_data1(Bin, DataList, [{list, ElemtSize+1, 0}|SizeList]);
binary_to_data1(<<?NIL_EXT, Rest/binary>>, DataList, SizeList) ->
    %%msg(list_nil, []),
    {DataList1, SizeList1} = binary_to_data2( DataList, SizeList, []),
    binary_to_data1(Rest, DataList1, SizeList1);
binary_to_data1(<<>>, DataList, _SizeList) ->
    %%msg(final, DataList),
    case lists:reverse(DataList) of
        [Data] -> next;
        [] -> Data=error;
        Data -> next
    end,
    Data;
binary_to_data1(Bin, _DataList, _SizeList) ->
    msg(unknown, Bin),
    error.

gen_small_big(<<Bin/binary>>) ->
    gen_small_big(Bin, 0, 0).
gen_small_big(<<>>, Number, Index) ->
    {Index, Number};
gen_small_big(<<Num:8, Rest/binary>>, Number, Index) ->
    gen_small_big(Rest, Number + Num * (1 bsl (Index * 8)), Index+1).

msg(Type, Data) ->
    io:format("~w ~w~n", [Type,Data]),
    ok.
保存为test.erl，运行结果如下：[plain] view plain copy

16> c(test).
{ok,test}
17> test:term_to_data({a,b,c}).
{a,b,c}
18> test:term_to_data([]).
[]
19> test:term_to_data({a,b,{c,{d,{e,"TTT"}},f,[g,h,i],[j]}}).
{a,b,{c,{d,{e,"TTT"}},f,[g,h,i],[j]}}
以上的例子有一个小问题，erlang R15以下版本不支持 binary_to_float 函数，所以在这些erlang版本测试float使用会有问题。有兴趣的小伙伴还可以自己动手试试实现 binary_to_float
代码比较长，这里也贴了例子下载地址：http://download.csdn.net/detail/cwqcwk1/8342895，欢迎下载

内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理

标签： erlang

相关文章推荐

新的分享

章节导航