MongoDB的nodejs 键值压缩driver

这是E叔同事tzlastom 所写的nodejs的一个mongodb driver，征得他同意后E叔转到咱中文站了。

原文地址：https://www.npmjs.com/package/mongodbx

A wrapper of Node.js mongoDB driver with document key compression

examples

Code:

var mongodb = require(‘mongodbx’);
mongodb.mongodbx.initialize({
    ‘collections’: {
        ‘mongodbxTest’: {
            columns: {‘name’: ‘n’, ‘expireTime’: ‘e’, ‘deleteFlag’: ‘d’},
            enableCompact: true,
        }
    }
});
 
var record = {‘name’: ‘name’, ‘expireTime’: new Date(), ‘deleteFlag’: true, ‘notConfiguredKey’: ‘ext’};
var db = new mongodb.Db(‘test’, new mongodb.Server(‘localhost’, 27017));
db.open(function(err, db){
    var collection = db.collection(“mongodbxTest”);
    collection.insert(record, {w:1}, function(){
        collection.find().toArray(console.log);
    });
});

Will print:

[tong@localhost mongodbx]$ node test.js
null [{_id: 551b6c63ee8ff65407e59847, name: 'name', expireTime: Wed Apr 01 2015 11:56:19 GMT+0800 (SGT), deleteFlag: true, notConfiguredKey: 'ext'}]
 

Within mongo shell:

rs0:PRIMARY> db.mongodbxTest.find()
{“_id”: ObjectId(“551b6c63ee8ff65407e59847″), “n”: “name”, “e”: ISODate(“2015-04-01T03:56:19.625Z”), “d”: true, “notConfiguredKey”: “ext”}

Background

As a key-value database, one fault of mongodb is to store the key string for each document. This will cost a lot of IO resources. For example, a collection with column named ‘deleteFlag’, mongodb need 11 bytes for key string storage –The value itself(true/false)
only need 1 byte!

There do have a ‘best pratice’ for this: shorten the key string. For ‘deleteFlag’, design/code with ‘fdel’, even ‘d’. However, this solution will cause some software nightmare. The code’s maintainability && readability will be compromised.

To optimise mongodb’s storage && be friendly to code/design, we can put a middleware between our application and mongodb driver. The middleware will translate document bi-direction. For example, {deleteFlag: true}(application model) <=> {d: true}(mongodb document).

Test status

Pass all test cases in mongodb driver(driver version: 2.0.25,
server version: 3.0.1).
Performance: about 250ms to compress/decompress 10k documents(Core i7 3.40GHZ).

Api spec

mongodbx.initialize(params)

params.collections (Object)
params.collections[].enableCompact (Boolean, default: params.debug). collection level compress config.
params.collections[].columns (Array/Object).
Object: translate map;
Array: using base62(a-z, A-Z, 0-9) to compress, the compressed value is key’s index in columns array. Make sure this array is append only.

params.enableCompact (Boolean, default:false). Global compress config.
params.debug (Boolean, default:false).
params.mongodb mongodb driver object(require(‘mongodb’)). For those don’t want embedded mongodb driver.
mongodb.mongodbx.initialize({
    ‘collections’: {
        ‘mongodbxTest’: {
            columns: {‘name’: ‘n’, ‘expireTime’: ‘e’, ‘deleteFlag’: ‘d’},
            enableCompact: true,
        },
        ‘mongodbxTestBase62Map’: {
            columns: ['name', 'expireTime', 'deleteFlag'],
            enableCompact: true,
        },
        ‘noCompactCol’: {
            columns: {‘name’: ‘n’, ‘expireTime’: ‘e’, ‘deleteFlag’: ‘d’},
            enableCompact: false,
        },
        ‘dependOnBlockConfig’: {
            columns: {‘name’: ‘n’, ‘expireTime’: ‘e’, ‘deleteFlag’: ‘d’},
        },
    },
    enableCompact: true
});
 
#Specify you mongodb driver
var mongodb = require(‘mongodb’);
require(‘mongodbx’).initialize({
    collections: …
    mongodb: mongodb,
});
mongodb.mongodbx.[Api]
});

mongodbx.addCollection(collectionName, params)

params. see mongodbx.initialize params.collections[]

mongodbx.getColumnNameMap = function(collectionName);

console.log(mongodb.mongodbx.getColumnNameMap(‘mongodbxTest’));

Will print:
{ name: ‘n’, expireTime: ‘e’, deleteFlag: ‘d’ }

mongodbx.translateToCompact(inputStr, collectionName)

console.log(‘Map translate normal:’, mongodb.mongodbx.translateToCompact(‘name’, ‘mongodbxTest’));
console.log(‘Map translate notConfiguredKey:’, mongodb.mongodbx.translateToCompact(‘notConfiguredKey’, ‘mongodbxTest’));
 
console.log(‘Base62 translate 1st Key:’, mongodb.mongodbx.translateToCompact(‘name’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate 2nd Key:’, mongodb.mongodbx.translateToCompact(‘expireTime’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate 3rd Key:’, mongodb.mongodbx.translateToCompact(‘deleteFlag’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate notConfiguredKey:’, mongodb.mongodbx.translateToCompact(‘notConfiguredKey’, ‘mongodbxTestBase62Map’));

Will print:
Map translate normal: n
Map translate notConfiguredKey: notConfiguredKey
Base62 translate 1st Key: a
Base62 translate 2nd Key: b
Base62 translate 3rd Key: c
Base62 translate notConfiguredKey: notConfiguredKey

mongodbx.translateToOriginal(encodeStr, collectionName)

console.log(‘Map translate normal:’, mongodb.mongodbx.translateToOriginal(‘n’, ‘mongodbxTest’));
console.log(‘Map translate notConfiguredKey:’, mongodb.mongodbx.translateToOriginal(‘notConfiguredKey’, ‘mongodbxTest’));
 
console.log(‘Base62 translate 1st Key:’, mongodb.mongodbx.translateToOriginal(‘a’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate 2nd Key:’, mongodb.mongodbx.translateToOriginal(‘b’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate 3rd Key:’, mongodb.mongodbx.translateToOriginal(‘c’, ‘mongodbxTestBase62Map’));
console.log(‘Base62 translate notConfiguredKey:’, mongodb.mongodbx.translateToOriginal(‘notConfiguredKey’, ‘mongodbxTestBase62Map’));

Will print
Map translate normal: name
Map translate notConfiguredKey: notConfiguredKey
Base62 translate 1st Key: name
Base62 translate 2nd Key: expireTime
Base62 translate 3rd Key: deleteFlag
Base62 translate notConfiguredKey: notConfiguredKey

Known issues

mongodb.Collection.mapReduce not supported
mongodb.Collection.group not supported
mongodb.Collection.find $where not supported
embeded-document’s key compress not supported

Todo

With mongodbx, application can get tranlated record. But if we use mongodb shell, the output is still less-readability. Try to solve this…