神经网络压缩(6):Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
2017-06-29 16:32
701 查看
Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
方法介绍
目的:探索稀疏性和预测精度之间的关系
能不能在稀疏性的规则中找到一个平衡点,这个点是如何提升硬件实现效率的
为了探索上面的问题,文章设置了几种不同的剪枝粒度等级,在前人的基础上进行对比实验,探索剪枝粒度和预测精度之间的关系,分析不同粒度等级下的存储量和硬件效率
Range of granularity粒度范围也是文章的一个点,按照dim的增加划分为不同的粒度等级,四个,第一个是细粒度,也就是逐元素的,剩下三个都归为粗粒度
因为文章更关注粒度尺寸在实验中的影响而不是prune的方法,所以文章里用了一个简单直观的方法->
pruning criterion in “Learning both weights and connections for efficient neural network.”
Gi代表不同的粒度等级,包括按不同粒度等级区分的一组weights;Si就代表在粒度等级的划分下,不同组间进行比较的依据;
剪枝时确定稀疏性参数,就是通过敏感性分析,逐层的剪枝,在训练集上衡量精度损失来确定最终的稀疏性参数
实验探索和结论
相同稀疏度情况下在不同剪枝粒度的情况下的accuracyFigure 4,相同稀疏度情况下在不同剪枝粒度的情况下的accuracy
和表哥里的第一大行是对应的,按横坐标1-0.248的位置竖向比较,
粗粒度比如减filter确实会造成很大的精度损失,另一方面,比较小的粒度尺寸下,粗粒度也能得到和细粒度比较相似的精度曲线
剪枝就像是一个正则化a regularizer. Coarse-grained sparsity not only reduces the number of parameters but also constrains the positions of parameters限制了这些参数的位置, which is an even stronger regularizer. That’s why at low sparsity rate we observed the accuracy improvement in Figure 4.图四,可以看到accuracy有提升
存储空间上的比较
存储空间上的比较,稀疏网络一般都按CRS压缩行存储,存储值和索引,粗粒度由于有极强的规律性,能够节省一定的索引的存储空间,比细粒度占的存储空间要少,如图5;
Figure 7,compare the accuracy of different pruning granularities at the same level of storage相同存储率的情况下不同剪枝粒度对精度的影响,可以看到在80baseline上更紧凑,也说明粗粒度 index saving 也是比较有效的。
最后的存储,use 4-bit indices to store the distances between adjacent non-zeros, following the method in Deep Compression同样沿用了韩松在deep compression中介绍的方法
粗粒度对于硬件实现方面的帮助
这个图是说,我把weights和input activations都用稀疏格式存储,输出是用正常的dense存储,在计算输出的时候,先用weights和activations的索引进行一一对应,也就是这两条虚线,索引的比较也就是相当于一个控制信号,满足这个控制信号了再做weights和activations的对应位置相乘和累加的操作在一层计算结束后,输出的activations再压缩成稀疏存储重复刚才的过程
总结
总的来说,文章还是比较支持粗粒度稀疏的,节约索引存储空间,能得到更大的压缩率,对硬件效率的提升也比较有帮助Due to the advantage of index saving, coarse-grained pruning is able to achieve a higher model compression ratio.
coarse-grained sparsity saves 2* output memory access compared with fine-grained sparsity, and 3* compared with dense implementation.
coarse-grained sparsity enables more efficient hardware architecture design of deep neural networks.
相关文章推荐
- 循环神经网络The Unreasonable Effectiveness of Recurrent Neural Networks
- [神经网络]1.3-Using neural nets to recognize handwritten digits-The architecture of neural networks(翻译)
- Reducing the Dimensionality of Data with Neural Networks:神经网络用于降维
- 【神经网络】Reducing the Dimensionality of Data with Neural Networks
- 神经网络压缩(4) Learning Structured Sparsity in Deep Neural Networks
- 神经网络不同激活函数比较--读《Understanding the difficulty of training deep feedforward neural networks》
- 用平常语言介绍神经网络(Neural Networks in Plain English)
- CHAPTER 3 改进神经网络的学习(Improving the way neural networks learn)
- CNN网络结构 - Refining Architectures of Deep Convolutional Neural Networks
- 神经网络压缩(9) WAE-Learning a Wavelet-like Auto-Encoder to Accelerate Deep Neural Networks
- 【论文笔记2】图像压缩神经网络在Kodak数据集上首次超越JPEG——Full Resolution Image Compression with Recurrent Neural Networks
- 论文笔记 Ensemble of Deep Convolutional Neural Networks for Learning to Detect Retinal Vessels in Fundus
- 机器学习中的神经网络Neural Networks for Machine Learning:Programming Assignment 1: The perceptron learning alg
- 有效感受野--Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
- How to calculate the number of parameters of Convolutional Neural Networks(CNNs) correctly?
- 蒸馏神经网络(Distill the Knowledge in a Neural Network) 论文笔记
- 论文阅读:Reading Text in the Wild with Convolutional Neural Networks
- Neural Networks and Deep Learning(神经网络与深度学习)_On the exercises and problems
- (笔记)神经网络压缩,Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks(一)
- VGG Convolutional Neural Networks Practical(5)the theory of back-propagation