序列化战争:主流序列化框架Benchmark
2015-07-03 21:26
176 查看
序列化战争:主流序列化框架Benchmark
GitHub上有这样一个关于序列化的Benchmark,被好多文章引用。但这个项目考虑到完整性,代码有些复杂。为了个人学习,自己实现了个简单的Benchmark测试类,也算是总结一下当今主流序列化框架的用法。1.序列化的战争
按照序列化后的数据格式,主流的序列化框架主要可以分为四大类:JSON、二进制、XML、RPC。从更高层次来说,JSON和XML都可以算作是文本类的,而RPC类因为不只是序列化,框架往往还提供了底层RPC以及跨语言代码生成等基础设施,所以单列作一类。具体说来,本次测试涵盖了以下这些:JSON类
非常流行的Jackson
Google的Gson
类JSON的MessagePack
阿里的FastJSON
二进制类
老牌劲旅Hessian(以前很喜欢用的)
功能全面而强大的FST
后起之秀Kryo
XML类
StAX(Streaming API for XML)
Thoughwork的XStream
RPC类
Protobuf:这里“偷了点懒”,因为Protobuf和Thrift都要安装、编译,所以这里使用了Protostuff,可以在运行时自动获取对象的Schema信息,省去了额外安装和手动编写协议格式文件的过程(Protostuff真是太好了!)。
Thrift、Apache Avro:同上,都需要预编译。
Why does Jackson-JSON call BSON the “smile format” of JSON?
BSON and Smile are two distinct binary formats. They are related in that they are both based on the logical format of JSON (i.e., key-value objects) but they are distinct in that they write incompatible binary formats (you can neither directly read Smile as BSON nor vice-versa). They also have different incompatible features (e.g., BSON defines a date type, while Smile does not as far as I can tell.) BSON is the binary serialization used by MongoDB for network transfer and disk serialization. Smile is the binary JSON format used by the Jackson project.
<!-- JSON BEGIN --> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.5.4</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.module</groupId> <artifactId>jackson-module-afterburner</artifactId> <version>2.5.4</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.module</groupId> <artifactId>jackson-module-scala_2.10</artifactId> <version>2.5.3</version> </dependency> <dependency> <groupId>com.google.code.gson</groupId> <artifactId>gson</artifactId> <version>2.3.1</version> </dependency> <dependency> <groupId>com.alibaba</groupId> <artifactId>fastjson</artifactId> <version>1.2.6</version> </dependency> <dependency> <groupId>io.fastjson</groupId> <artifactId>boon</artifactId> <version>0.33</version> </dependency> <!-- JSON END --> <!-- JSON-like BEGIN --> <dependency> <groupId>com.fasterxml.jackson.dataformat</groupId> <artifactId>jackson-dataformat-smile</artifactId> <version>2.5.4</version> </dependency> <dependency> <groupId>org.msgpack</groupId> <artifactId>msgpack</artifactId> <version>0.6.12</version> </dependency> <dependency> <groupId>org.mongodb</groupId> <artifactId>bson</artifactId> <version>3.0.2</version> </dependency> <dependency> <groupId>com.fasterxml.jackson.dataformat</groupId> <artifactId>jackson-dataformat-yaml</artifactId> <version>2.5.4</version> </dependency> <!-- JSON-like END --> <!-- Binary BEGIN --> <dependency> <groupId>com.caucho</groupId> <artifactId>hessian</artifactId> <version>4.0.38</version> </dependency> <dependency> <groupId>de.ruedigermoeller</groupId> <artifactId>fst</artifactId> <version>2.31</version> </dependency> <dependency> <groupId>com.esotericsoftware</groupId> <artifactId>kryo</artifactId> <version>3.0.2</version> </dependency> <!-- Binary END --> <!-- XML BEGIN --> <dependency> <groupId>com.thoughtworks.xstream</groupId> <artifactId>xstream</artifactId> <version>1.4.8</version> </dependency> <dependency> <groupId>com.fasterxml</groupId> <artifactId>aalto-xml</artifactId> <version>0.9.11</version> </dependency> <!-- XML END --> <!-- RPC BEGIN --> <dependency> <groupId>io.protostuff</groupId> <artifactId>protostuff-core</artifactId> <version>1.3.5</version> </dependency> <dependency> <groupId>io.protostuff</groupId> <artifactId>protostuff-runtime</artifactId> <version>1.3.5</version> </dependency> <dependency> <groupId>org.apache.avro</groupId> <artifactId>avro</artifactId> <version>1.7.7</version> </dependency> <!-- RPC END -->
2.Benchmark代码
2.1 测试对象
用Serializer接口实现表示不同的序列化框架,作为测试对象集合。测试主要关注序列化数据大小、序列化时间消耗、反序列化时间消耗三个指标。public class SerializerBenchmark { private static final int WARMUP_COUNT = 100; private static final int TEST_COUNT = 1000 * 1000; /** Column index */ private static final int COL_SER_SIZE = 0; private static final int COL_SER_COST = 1; private static final int COL_DER_COST = 2; /** Dictionary for random generation */ private static final char[] ALPHA = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ".toCharArray(); public static void main(String[] args) throws Exception { Serializer[] serializers = { // ============= JSON ============== new Serializer<Person>() { private ObjectMapper mapper = new ObjectMapper(); @Override public String name() { return "Jackson"; } @Override public byte[] serialize(Person obj) throws Exception { return mapper.writeValueAsBytes(obj); } @Override public Person deserialize(byte[] data, Class<Person> type) throws Exception { return mapper.readValue(data, type); } }, new Serializer<Person>() { private Gson gson = new GsonBuilder().create(); @Override public String name() { return "Gson"; } @Override public byte[] serialize(Person obj) { return gson.toJson(obj).getBytes(); } @Override public Person deserialize(byte[] data, Class<Person> type) { return gson.fromJson(new String(data), type); } }, new Serializer<Person>() { @Override public String name() { return "FastJSON"; } @Override public byte[] serialize(Person obj) { return JSON.toJSONBytes(obj); } @Override public Person deserialize(byte[] data, Class<Person> type) { return JSON.parseObject(data, type); } }, // ============= JSON-like ============== new Serializer<Person>() { private ObjectMapper mapper = new ObjectMapper(new SmileFactory()); @Override public String name() { return "Jackson-smile"; } @Override public byte[] serialize(Person obj) throws Exception { return mapper.writeValueAsBytes(obj); } @Override public Person deserialize(byte[] data, Class<Person> type) throws Exception { return mapper.readValue(data, type); } }, new Serializer<Person>() { private ObjectMapper mapper = new ObjectMapper(new SmileFactory()); { mapper.registerModule(new AfterburnerModule()); } @Override public String name() { return "Jackson-smile-afterburner"; } @Override public byte[] serialize(Person obj) throws Exception { return mapper.writeValueAsBytes(obj); } @Override public Person deserialize(byte[] data, Class<Person> type) throws Exception { return mapper.readValue(data, type); } }, new Serializer<Person>() { private ObjectMapper mapper = new ObjectMapper(new SmileFactory()); { mapper.registerModule(new DefaultScalaModule()); } @Override public String name() { return "Jackson-smile-scala"; } @Override public byte[] serialize(Person obj) throws Exception { return mapper.writeValueAsBytes(obj); } @Override public Person deserialize(byte[] data, Class<Person> type) throws Exception { return mapper.readValue(data, type); } }, new Serializer<Person>() { private ObjectMapper mapper = new ObjectMapper(new YAMLFactory()); @Override public String name() { return "Jackson-yaml"; } @Override public byte[] serialize(Person obj) throws Exception { return mapper.writeValueAsBytes(obj); } @Override public Person deserialize(byte[] data, Class<Person> type) throws Exception { return mapper.readValue(data, type); } }, new Serializer<Person>() { private MessagePack msgpack = new MessagePack(); { msgpack.register(Person.class); } @Override public String name() { return "MessagePack"; } @Override public byte[] serialize(Person obj) throws Exception { return msgpack.write(obj); } @Override public Person deserialize(byte[] data, Class type) throws Exception { return msgpack.read(data, Person.class); } }, // ============= Binary ============== new Serializer<Person>() { private Schema<Person> schema = RuntimeSchema.getSchema(Person.class); private LinkedBuffer buffer = LinkedBuffer.allocate(); @Override public String name() { return "Protostuff"; } @Override public byte[] serialize(Person obj) { byte[] data = ProtobufIOUtil.toByteArray(obj, schema, buffer); buffer.clear(); return data; } @Override public Person deserialize(byte[] data, Class<Person> type) { Person obj = new Person(); ProtobufIOUtil.mergeFrom(data, obj, schema); return obj; } }, new Serializer<Person>() { @Override public String name() { return "Hessian"; } @Override public byte[] serialize(Person obj) throws Exception { ByteArrayOutputStream bytes = new ByteArrayOutputStream(); Hessian2Output output = new Hessian2Output(bytes); output.writeObject(obj); output.close(); // flush to avoid EOF error return bytes.toByteArray(); } @Override public Person deserialize(byte[] data, Class<Person> type) throws Exception { Hessian2Input input = new Hessian2Input(new ByteArrayInputStream(data)); return (Person) input.readObject(); } }, new Serializer<Person>() { private FSTObjectInput input = new FSTObjectInput(); private FSTObjectOutput output = new FSTObjectOutput(); @Override public String name() { return "FST"; } @Override public byte[] serialize(Person obj) throws Exception { output.resetForReUse(); output.writeObject(obj); return output.getCopyOfWrittenBuffer(); } @Override public Person deserialize(byte[] data, Class<Person> type) throws Exception { input.resetForReuseUseArray(data); return (Person) input.readObject(); } }, new Serializer<Person>() { private Kryo kryo = new Kryo(); { kryo.setReferences(false); kryo.setRegistrationRequired(true); kryo.register(Person.class); } private byte[] buffer = new byte[512]; private Output output = new Output(buffer, -1); private Input input = new Input(buffer); @Override public String name() { return "Kryo"; } @Override public byte[] serialize(Person obj) { output.setBuffer(buffer, -1); // reset kryo.writeObject(output, obj); return output.toBytes(); } @Override public Person deserialize(byte[] data, Class<Person> type) { input.setBuffer(data); return kryo.readObject(input, type); } }, new Serializer<Person>() { @Override public String name() { return "JDK Built-in"; } @Override public byte[] serialize(Person obj) throws Exception { ByteArrayOutputStream out = new ByteArrayOutputStream(); new ObjectOutputStream(out).writeObject(obj); return out.toByteArray(); } @Override public Person deserialize(byte[] data, Class<Person> type) throws Exception { return (Person) new ObjectInputStream(new ByteArrayInputStream(data)).readObject(); } }, // ============= XML ============== new Serializer<Person>() { private XStream xstream = new XStream(); @Override public String name() { return "XStream"; } @Override public byte[] serialize(Person obj) throws Exception { ByteArrayOutputStream out = new ByteArrayOutputStream(); xstream.toXML(obj, out); return out.toByteArray(); } @Override public Person deserialize(byte[] data, Class<Person> type) throws Exception { return (Person) xstream.fromXML(new ByteArrayInputStream(data)); } }, }; // Sheet int[] testCase = { 10, 100, 1000 }; String[] sheetNames = new String[testCase.length]; for (int i = 0; i < sheetNames.length; i++) { sheetNames[i] = "Size=" + testCase[i]; } // Row String[] rowNames = new String[serializers.length]; for (int i = 0; i < rowNames.length; i++) { rowNames[i] = serializers[i].name(); } // Column String[] colNames = new String[3]; colNames[0] = "Size"; colNames[1] = "Ser"; colNames[2] = "Der"; Reporter reporter = new Reporter(sheetNames, rowNames, colNames); for (int i = 0; i < testCase.length; i++) { int length = testCase[i]; System.out.printf("===== Round [%d]: %d =====\n", i, length); for (int j = 0; j < serializers.length; j++) { testSerializer(reporter, length, i, j, serializers[j]); } } System.out.println(reporter.generateFinalReport()); } ... }
2.2 测试Runner
每轮测试前都先Warmup并GC,避免JIT和GC对测试的影响。同时,Warmup时检测序列化和反序列化的正确性。private static void testSerializer(Reporter reporter, int length, int sheet, int row, Serializer<Person> serializer) throws Exception { System.out.println("===== " + serializer.name() + " ====="); // 1.Warm-up and validate System.out.println("Pre-warmup & Check correctness..."); Person p1 = newPerson(length); for (int i = 0; i < WARMUP_COUNT; i++) { byte[] bytes = serializer.serialize(p1); Person p2 = serializer.deserialize(bytes, Person.class); if (!p1.equals(p2)) { throw new IllegalStateException(p1 + " not equals to " + p2); } } int serSize = serializer.serialize(p1).length; System.out.printf("%s serialization size[%d]\n", serializer.name(), serSize); reporter.report(sheet, row, COL_SER_SIZE, serSize); doGc(); // 2.Serialization long startTime = System.currentTimeMillis(); for (int i = 0; i < TEST_COUNT; i++) { serializer.serialize(p1); } long serCostTime = System.currentTimeMillis() - startTime; System.out.printf("%s serialization benchmark[%d]\n", serializer.name(), serCostTime); reporter.report(sheet, row, COL_SER_COST, serCostTime); // Warm up again for (int i = 0; i < WARMUP_COUNT; i++) { byte[] bytes = serializer.serialize(p1); serializer.deserialize(bytes, Person.class); } doGc(); // 3.De-Serialization byte[] bytes = serializer.serialize(p1); startTime = System.currentTimeMillis(); for (int i = 0; i < TEST_COUNT; i++) { serializer.deserialize(bytes, Person.class); } long derCostTime = System.currentTimeMillis() - startTime; System.out.printf("%s de-serialization benchmark[%d]\n", serializer.name(), derCostTime); reporter.report(sheet, row, COL_DER_COST, derCostTime); System.out.println(); }
3.测试报告
3.1 报告生成
这里“偷了点小懒”,用Apache Common Lang提供的StringUtils中的pad()方法排版。static class Reporter { private final String[] sheetNames; private final String[] rowNames; private final String[] colNames; private final long[][][] table; Reporter(String[] sheetNames, String[] rowNames, String[] colNames) { this.sheetNames = sheetNames; this.rowNames = rowNames; this.colNames = colNames; this.table = new long[sheetNames.length] [rowNames.length] [colNames.length]; } public void report(int sheet, int row, int col, long val) { table[sheet][row][col] = val; } public String generateFinalReport() { StringBuilder report = new StringBuilder(); for (int i = 0; i < table.length; i++) { report.append(center(sheetNames[i], 50, '*')) .append("\n"); // 1.Header final int width0 = 30; final int width1 = 10; report.append(rightPad("", width0)); for (String colName : colNames) { report.append(rightPad(colName, width1)); } report.append("\n"); // 2.Row for (int j = 0; j < table[i].length; j++) { report.append(rightPad(rowNames[j], width0)); for (int k = 0; k < table[i][j].length; k++) { report.append(rightPad( String.valueOf(table[i][j][k]), width1)); } report.append("\n"); } report.append("\n"); } return report.toString(); } }
3.2 测试结果
测试结果可以简单总结如下:Kryo占用空间最小,其次是MessagePack和Protostuff(Protobuf)。
Protostuff在不同数据长度下表现都非常出色!
JSON以及类JSON框架中,Jackson+Smile格式+Afterburner模块的组合表现最好。
XStream出奇地慢,印象中XStream挺快的吧,难道有优化参数没配?
*********************Size=10********************** Size Ser Der Jackson 39 602 758 Gson 38 1204 1181 FastJSON 38 573 608 Jackson-smile 35 415 465 Jackson-smile-afterburner 35 305 377 Jackson-smile-scala 34 522 590 Jackson-yaml 39 4233 5638 MessagePack 15 891 1075 Protostuff 17 148 130 Hessian 84 2459 1233 FST 73 334 481 Kryo 13 98 117 JDK Built-in 138 1462 4526 XStream 169 6088 13007 *********************Size=100********************* Size Ser Der Jackson 129 403 565 Gson 128 1056 1248 FastJSON 129 522 571 Jackson-smile 126 426 472 Jackson-smile-afterburner 126 454 371 Jackson-smile-scala 126 452 639 Jackson-yaml 129 5250 5330 MessagePack 108 948 976 Protostuff 107 172 192 Hessian 176 2528 1513 FST 163 288 470 Kryo 105 440 134 JDK Built-in 228 1332 4559 XStream 259 5913 12797 ********************Size=1000********************* Size Ser Der Jackson 1029 1412 1411 Gson 1029 4614 3855 FastJSON 1029 2476 2011 Jackson-smile 1026 1052 1343 Jackson-smile-afterburner 1025 1105 1232 Jackson-smile-scala 1025 1058 1452 Jackson-yaml 1029 18983 13065 MessagePack 1008 2101 2010 Protostuff 1008 1172 838 Hessian 1075 4358 6587 FST 1063 1083 1567 Kryo 1005 2675 921 JDK Built-in 1128 2502 8537 XStream 1158 10633 16981
相关文章推荐
- NYOJ 37 回文字符串
- PostgreSQL游标使用举例
- 【动手写排序】堆排序
- stm32学习笔记
- 【Leetcode Algorithm】Min Stack
- 【动手写排序】快速排序
- linux 虚拟机安装VMware Tools
- ubuntu 14.04开机出现错误“Error found when loading /root/.profile”解决
- C# .NETWEB开发6大内置对象
- NYOJ 972 核桃的数量
- javascript 创建私有成员和静态私有成员
- Java程序员的Golang入门指南(下)
- Java程序员的Golang入门指南(下)
- [USACO Open10]数三角形Triangle Counting解题报告
- 红野猪动画
- NYOJ 822 画图
- HTML01
- 电子书makefile分析
- NYOJ 88 汉诺塔(一)
- hdu4336压缩率大方的状态DP