「Deep Learning」Dropout - Drop out the Units in a Neural Network

Sina Weibo：东莞小锋子Sexyphone

Tencent E-mail：403568338@qq.com http://blog.csdn.net/dgyuanshaofeng/article/details/52134243
这个Dropout技术，念成「draw泡」，主要解决神经网络的过拟合问题，提升泛化能力。
我并不打算对其进行解说，因为相关资料很多，包括论文、书籍和博客。

参考资料：
1、2012论文，Improving_Neural_Networks_by_Preventing_Co-Adaptation_of_Feature_Detectors
2、2014论文，很详细，Dropout_A_Simple_Way_to_Prevent_Neural_Networks_From_Overfitting
3、书籍，《deep learning》，2015年10月版本7.11，2016年4月版本7.12
4、博客，tornadomeet

实验：代码来自tornadomeet和DeepLearnToolbox

% load mnist data
load mnist_uint8 ;

% use some of data to train and test
% for example, 4000 examples for training, 2000 examples for testing
train_x = double(train_x(1:4000, :)) / 255 ;
test_x  = double(test_x(1:2000, :))  / 255 ;
train_y = double(train_y(1:4000, :)) ;
test_y  = double(test_y(1:2000, :)) ;

% normalize: subtract and divide
[train_x, mu, sigma] = zscore(train_x) ; % toolbox function
% note that mu and sigma of training data is useful in testing phase,
% testing data is performed normalization by using the mu and sigma
test_x = normalize(test_x, mu, sigma) ; % toolbox function

diary('log.txt')
diary on
%% without dropout
rng(0) ; % restore the seed
%nn = nnsetup([784 100 10]) ; % neural network
nn = nnsetup([784 500 10]) ;
opts.numepochs =  40 ;
opts.batchsize = 100 ;
[nn, L] = nntrain(nn, train_x, train_y, opts) ;
[er, bad] = nntest(nn, test_x, test_y) ;
str = sprintf('testing error rate is: %f', er) ;
disp(str)

%% with dropout
rng(0) ; % restore the seed
%nn = nnsetup([784 100 10]) ;
nn = nnsetup([784 500 10]) ;
nn.dropoutFraction = 0.5 ;
opts.numepochs =  40 ;
opts.batchsize = 100 ;
nn = nntrain(nn, train_x, train_y, opts) ;
[er, bad] = nntest(nn, test_x, test_y) ;
str = sprintf('testing error rate is: %f', er) ;
disp(str)

diary off

结论：测试误差率的确降低。
epoch 1/40. Took 1.0921 seconds. Mini-batch mean squared error on training set is 0.88014; Full-batch train err = 0.237818
...

epoch 40/40. Took 1.2243 seconds. Mini-batch mean squared error on training set is 0.027368; Full-batch train err = 0.027243

testing error rate is: 0.107000

epoch 1/40. Took 1.3454 seconds. Mini-batch mean squared error on training set is 0.8309; Full-batch train err = 0.179294
...

epoch 40/40. Took 1.3908 seconds. Mini-batch mean squared error on training set is 0.030229; Full-batch train err = 0.011828

testing error rate is: 0.084000