Source-linked AI summary
Fast ConvNets Using Group-wise Brain Damage
Vadim Lebedev, Victor Lempitsky
TL;DR
The paper addresses the limited computational benefit of unstructured neural-network pruning for speeding convolutional layers. It introduces group-wise brain damage aligned with lowering-based matrix multiplication and learns sparsity through group regularization. On AlexNet, the method achieves 8.5x simultaneous acceleration of conv2 and conv3 with less than 1% prediction-accuracy loss.
Problem
Unstructured brain damage sparsifies parameters but usually provides insufficient structured sparsity for substantial convolutional speedup.
Method
The method applies group-wise pruning to convolutional kernels, using group-sparsity regularization to learn aligned sparsity patterns for lowered matrix multiplication.
Results
8.5x simultaneous acceleration of AlexNet conv2 and conv3 incurred less than 1% prediction-accuracy loss.
Takeaways & Limitations
Group-wise brain damage can provide state-of-the-art ConvNet speedup while learning receptive-field structure from data.
Takeaways & Limitations
For small target density τ, selecting regularization strength λ can cause excessive shrinkage and significant accuracy drops during learning.
Abstract
from arXiv · showhide
We revisit the idea of brain damage, i.e. the pruning of the coefficients of a neural network, and suggest how brain damage can be modified and used to speedup convolutional layers. The approach uses the fact that many efficient implementations reduce generalized convolutions to matrix multiplications. The suggested brain damage process prunes the convolutional kernel tensor in a group-wise fashion by adding group-sparsity regularization to the standard training process. After such group-wise pruning, convolutions can be reduced to multiplications of thinned dense matrices, which leads to speedup. In the comparison on AlexNet, the method achieves very competitive performance.
1. Introduction
The paper targets convolutional speed bottlenecks by replacing unstructured pruning with group-wise brain damage aligned to matrix-multiplication implementations. Group sparsity makes convolutions faster while preserving competitive prediction accuracy.
- Motivation: Unstructured brain damage can remove many parameters but usually provides insufficient structured sparsity for substantial computational speedup.The limitation arises because conventional pruning does not align with efficient convolution implementations.
- Motivation: Generalized convolution is a core ConvNet operation and a speed bottleneck, motivating structured brain damage of convolutional kernel tensors.The approach modifies standard generalized convolution to obtain speedup for a particular structure.
- Approach: Efficient convolution implementations reduce generalized convolutions to matrix multiplications, so pruning must account for both filter and patch-matrix structure.Unstructured kernel pruning sparsifies the filter matrix but does not necessarily thin the other factor.
- Approach: Group sparsity embedded in stochastic-gradient training learns coordinated zero groups and optimizes receptive-field connectivity.The method uses a conventional group-sparsity regularizer to induce group-wise brain damage.
- Results: 8.5x acceleration of AlexNet conv2 and conv3 simultaneously incurred only modest (<1%) prediction-accuracy loss.This is the paper’s representative reported comparison for bottleneck layers.
2. Related work
Related work addresses convolutional acceleration through Fourier transforms, tensor factorization, and structured sparsity, while also distinguishing memory reduction and receptive-field learning objectives.
- Convolution acceleration: Fourier-transform approaches offer theoretical appeal but face memory-usage limitations, and most existing packages use lowering-based convolution.The comparison concerns implementation choices for accelerating generalized convolution.
- Convolution acceleration: Tensor-factorization methods replace full convolution tensors with sequences of smaller convolutions, often trading approximation error against speed through decomposition rank.The proposed approach also reduces parameters but does not factorize the kernel tensor.
- Structured sparsity: Structured-sparsity methods have been used to discover optimal architectures, including graphical-model structures and receptive fields.The paper relates its receptive-field learning to these architecture-discovery approaches.
- Distinctions: Some sparsification methods target memory savings rather than acceleration, while another sparsifies output locations and fills gaps by interpolation instead of pruning kernels.These objectives and mechanisms differ from group-wise brain damage.
- Scope: The paper focuses on speeding convolutional layers, complementing methods that reduce fully connected layers’ size or memory footprint.The distinction is between convolutional computation and fully connected-layer compression.
3. Group-Sparse Convolutions
The method lowers generalized convolution to matrix multiplication, then imposes aligned group-wise sparsity so both filter and patch matrices become thinner. This reduces scalar operations and produces near-theoretical speedups.
- Convolution representation: Generalized convolution transforms S input maps into T output maps using a four-dimensional kernel tensor K of size d×d×S×T.The kernel dimensions represent spatial positions, input maps, and output maps.
- Dense lowering: Standard lowering reshapes the kernel into filter matrix F and input patches into matrix P, whose product contains all output-map values.The resulting matrix is reshaped into the output tensor.
- Computational effect: Figure 1 contrasts dense filters and patches with group-sparse filters and patch sampling, showing why the resulting multiplication is faster.Both cases reshape the filter-matrix product into the output map stack.
- Group-sparse convolution: Group-wise brain damage assigns each input map s a spatial sparsity pattern Qs, a subset of the d-by-d kernel grid.Aligned patterns ensure consistent pruning across convolutional filters and sampled patches.
- Empirical speedup: Figure 2 reports relative CPU forward-propagation speedup in LeNet’s second convolutional layer versus density τ, with speedup almost linear in sparsity level.Error bars represent 100 runs.
- Computational effect: After pruning, dense matrices of sizes T × d^2S and d^2S × W′′H′′ are replaced by thinner dense matrices, reducing scalar operations according to the retained pattern sizes.The implementation keeps dense matrix multiplication while reducing its dimensions.
- Empirical speedup: The reduction’s measured wall-clock convolution-time improvement nearly matches its theoretical speedup factor.This observation comes from experiments with the reference implementation.
4. Fast ConvNets with Group-sparse convolutions
The paper develops group-wise brain damage for learning or modifying fast ConvNets, using structured sparsity to remove receptive-field groups while preserving accuracy. It presents fixed-pattern, fine-tuning, and gradual sparsification procedures, with gradual sparsification addressing key tuning and accuracy-drop difficulties.
- Training scenarios: Group-wise sparsity can be imposed either from scratch or by modifying pretrained ConvNet architectures.The paper considers both predefined patterns and learned patterns, with pretrained-network acceleration as its main focus.
- Predefined group-sparsity pattern: Predefined sparsity patterns reduce filter shapes but offer no clear design principles for choosing shapes across multiple convolution layers.The approach permits non-rectangular filters, which the experiments found useful.
- Training with group-sparsity regularizer: The l2,1 regularizer shrinks groups Γijs of kernel entries toward zero, enabling zeroed groups to define sparse receptive-field patterns Ωs.Its gradient can be added to the learning-loss gradient during stochastic gradient updates, with λ controlling regularization strength.
- Group-wise sparsification with fine-tuning: Fine-tuning-based sparsification makes λ difficult to select, and aggressive sparsity can cause substantial accuracy loss during learning.These complications motivate gradual group-wise sparsification.
- Group-wise sparsification with fine-tuning: Fine-tuning fixes sparsity patterns after pruning and retrains without group-sparse regularization so the network adapts and recovers most of the accuracy drop.For target density τ, the procedure zeros d^2S(1−τ) groups.
- Gradual group-wise sparsification: Gradual sparsification combines brain damage with fine-tuning, varies θ according to validation accuracy, and greedily fixes groups below ϵ to zero.The process increases sparsity monotonically while keeping the tolerated validation-accuracy drop below δ.
- Gradual group-wise sparsification: The procedure is reported as relatively insensitive to λ and ϵ and more practical, with higher group-sparsity and speed-ups than sparsification followed by fine-tuning.Shared λ, ϵ, and θ values can be used when sparsifying multiple layers simultaneously.
5. Experiments
Experiments evaluate group-wise brain damage on MNIST, pretrained AlexNet, an external retrieval task, and VGGNet. The method achieves substantial acceleration with modest accuracy loss and compares favorably with baselines and tensor-factorization methods.
- Experimental settings: Experiments cover MNIST training from scratch, pretrained AlexNet acceleration, an external image-retrieval task, and preliminary VGGNet results.AlexNet evaluation targets its slowest convolutional layers individually, jointly, and across all five layers.
- ILSVRC experiments: AlexNet gradual sparsification outperforms tensor-factorization methods and fine-tuning-based sparsification for similar prediction-accuracy drops.The procedure uses validation data to adjust θ while holding out another test half for reporting.
- External computer vision task: At 3.2x speedup, group-sparsifying all AlexNet convolutional layers changes INRIA Holidays mAP from 0.783 to 0.780 and Oxford Buildings performance from 0.45 to 0.41.The external task uses sum-pooled pretrained convolutional features with PCA whitening and normalization.
- VGGNet results: VGGNet reaches density τ = 0.13 with a 0.2% top-1 accuracy drop, while 29 of 64 feature maps become all-zero.The result suggests the tested layer contains excessive feature maps and supports sparsification with 3 × 3 filters.
6. Discussion
The discussion frames group-wise brain damage as both a practical acceleration method and a way to learn network connectivity from data. Learned receptive fields tend to become smaller and roughly circular, while VGGNet can lose entire feature maps.
- Practical acceleration: Group-wise brain damage prunes convolution kernels after group-sparsity learning and fine-tuning, achieving state-of-the-art performance for speeding up ConvNets.The approach accounts for reducing generalized convolutions to matrix multiplications and prunes kernel entries groupwise.
- Learned receptive fields: In AlexNet’s second convolutional layer, increasing sparsity generally shrinks receptive fields toward the center and makes them roughly circular.Figure 4 displays nonzero weights in white across different sparsity levels.
- Gradual sparsification: Figure 5 tracks monotonic sparsity growth across all five AlexNet layers, validation and hold-out accuracy drops, and adaptive θ adjustment.θ changes according to whether validation accuracy loss exceeds or falls below 1.2%.
- Architecture discovery: VGGNet sparsification can eliminate entire feature maps, creating additional speedup because the preceding layer no longer needs to compute those maps.Hierarchical group-sparsity regularizers can explicitly encourage this elimination.