Source-linked AI summary
Memory Bounded Deep Convolutional Networks
Maxwell D. Collins, Pushmeet Kohli
TL;DR
Large CNNs create deployment memory costs, motivating methods that simplify their connectivity without sacrificing much accuracy. This paper trains CNNs with sparsity-inducing regularizers using stochastic gradient descent, and reports substantial memory reductions across MNIST, CIFAR, and ImageNet, including a factor-of-four AlexNet reduction with minimal accuracy loss.
Problem
Deep CNNs have high deployment memory costs, particularly in large models, while conventional regularization does not directly address this issue.
Method
The paper applies sparsity-inducing regularizers to convolutional and fully connected layers and optimizes the resulting objectives with stochastic gradient descent.
Results
Across MNIST, CIFAR, and ImageNet, the method reduces memory consumption by a factor of 3 or more with minimal loss in accuracy.
Takeaways & Limitations
Sparse models can reduce deployment resources and can also be combined into ensembles within fixed memory budgets.
Takeaways & Limitations
The ℓ1 update produces many near-zero weights but almost never exact zeros, so thresholding requires attention to its schedule.
Abstract
from arXiv · showhide
In this work, we investigate the use of sparsity-inducing regularizers during training of Convolution Neural Networks (CNNs). These regularizers encourage that fewer connections in the convolution and fully connected layers take non-zero values and in effect result in sparse connectivity between hidden units in the deep network. This in turn reduces the memory and runtime cost involved in deploying the learned CNNs. We show that training with such regularization can still be performed using stochastic gradient descent implying that it can be used easily in existing codebases. Experimental evaluation of our approach on MNIST, CIFAR, and ImageNet datasets shows that our regularizers can result in dramatic reductions in memory requirements. For instance, when applied on AlexNet, our method can reduce the memory consumption by a factor of four with minimal loss in accuracy.
1. Introduction
Deep CNNs achieve strong vision performance but impose substantial deployment memory costs. The paper proposes sparsity-inducing regularization that reduces connectivity and deployment cost while retaining accuracy and remaining compatible with standard training.
- Motivation: Deep CNNs require substantial memory at deployment, especially when fully connected layers dominate model size.This is particularly problematic on computationally constrained architectures such as mobile devices.
- Approach: Sparsity-inducing regularizers reduce the number of non-zero connections in convolutional and fully connected layers.The resulting sparse connectivity restricts model capacity while targeting simpler deployed networks.
- Results: The method reduces memory consumption by a factor of 3 or more with minimal loss in accuracy across MNIST, CIFAR, and ImageNet.The authors also report that sparse models can support ensembles without incurring the usual memory cost of storing multiple dense models.
- Optimization: The regularized objectives can be optimized using stochastic gradient descent and implemented within standard existing training algorithms.This preserves the compatibility of the approach with common deep-learning training procedures.
- Related approaches: Unlike dropout, which removes entire units, this approach retains a constant number of simpler units by reducing their parameters.Model compression and low-rank methods likewise target smaller models but use different mechanisms and connectivity structures.
2. Convolutional Neural Nets
CNNs combine convolutional feature extraction with fully connected prediction layers, whose parameters are learned by minimizing a regularized loss. Because this optimization is expensive for modern networks, stochastic gradient descent estimates updates from training subsets.
- Architecture: CNNs use learned convolutions in early layers to process visual primitives and may use fully connected layers to map features to output labels.The fully connected layers often account for most of a network’s parameters.
- Optimization formulation: The network is represented as a function f of inputs and learned parameters W, with training minimizing prediction loss plus a weighted regularization term.For image classification, the loss is a soft-max loss.
- Optimization formulation: Stochastic gradient descent replaces the full objective gradient with an estimate computed from a sampled subset of training examples.The stochastic gradient is equal in expectation to the full objective gradient.
3. Regularization Updates
The paper develops sparsity-inducing updates for neural-network weights, contrasting ℓ1, shrinkage, and direct ℓ0 projection approaches. These methods aim to produce genuinely sparse models while retaining stochastic-gradient training, although ℓ0 optimization lacks the guarantees available for convex problems.
- ℓ1 regularization: ℓ1 updates often create many near-zero weights without producing exact zeros, so the resulting network may remain unnecessarily complex.Thresholding these weights requires choosing a suitable threshold and schedule relative to gradients and regularization.
- Shrinkage operator: The shrinkage operator produces exact zeros without sign overshoot and can eliminate the need for separate thresholding.It is a proximal mapping that can be alternated with loss-gradient descent steps, including approximate or stochastic gradients.
- ℓ0 projection: ℓ0 regularization directly targets sparsity by constraining the number of nonzero parameters rather than merely shrinking their magnitudes.The proposed operator periodically keeps only the t largest-magnitude weights, enforcing ∥W∥0 ≤ t.
- ℓ0 projection: Every n updates, the ℓ0 operator sets all but the t largest-magnitude parameters to zero, acting as a projection onto an ℓ0 ball.The projection removes the smallest-magnitude elements and imposes a hard nonzero-count constraint.
- ℓ0 projection: The ℓ0 update is inspired by projected-gradient methods, but neither its objective nor its feasible set is convex.Consequently, the theoretical guarantees available for convex proximal stochastic-gradient methods do not directly apply.
4. Implementation
The implementation combines sparsity updates with existing CNN training workflows and evaluates optimization, progressive sparsification, and layer-wise sparsity selection. Experiments show that stochastic optimization remains effective and that sparsity should be allocated unevenly across layers.
- Implementation: The ℓ1 update was extended in Caffe to support mixing different regularization types across layers.This enabled retaining ordinary weight decay in selected portions while analyzing sparsity elsewhere.
- Implementation: The ImageNet network contained five convolutional layers, three fully-connected layers, and 60 million parameters.For this larger-scale experiment, the network was trained using a computationally cheaper fine-tuning procedure.
- Optimization: Stochastic gradient optimization consistently converged across 10 repetitions for each sparsity update method.The convergence experiment used a baseline CIFAR-10 model with sparsity penalties on the fully-connected layers.
- Optimization: ℓ0-norm projection was robust across layer choices, structured updates, and other hyperparameters, without observed failures relative to baseline training.The authors also report high test accuracies for very sparse models in other experiments.
- Layer-wise distribution of sparsity: A greedy procedure reduced each layer’s nonzeros by 20% and selected the layer producing the best validation accuracy for the next iteration.The procedure began by separating a validation set and reused each selected network as the next iteration’s starting point.
- Layer-wise distribution of sparsity: For CIFAR-10, the strongest sparsity constraints targeted the final convolutional and fully-connected layers, while the first two convolutional layers were relatively untouched.The greatest reductions in weights occurred in the final convolutional and first fully-connected layers, and these networks outperformed uniformly constrained baselines.
5. Experiments
Experiments on MNIST, CIFAR-10, and ImageNet evaluate sparsity-inducing regularization, layer-specific sparsity, thresholding baselines, and sparse ensembles. The results show that learned sparsity can preserve accuracy better than simple thresholding while substantially reducing parameters and memory.
- Experiments use MNIST, CIFAR-10, and ImageNet, including AlexNet fine-tuning with random crops, horizontal mirroring, and mean subtraction.
- 5.2. Accuracy and Regularization Updates: Sparse models achieve high accuracy with few nonzero parameters, while layer choice strongly affects performance, especially when sparsifying the first convolutional layer.The layer-wise distribution of sparsity is reported as the hyperparameter with the greatest effect on sparse-model performance.
- 5.1. Thresholding: ℓ0-constrained optimization produces higher end test accuracy than simply thresholding an ℓ2-regularized model at matched sparsity.The comparison duplicates the thresholded model’s per-layer nonzero distribution in an ℓ0-regularized model.
- 5.2. Accuracy and Regularization Updates: ℓ0 projection is used in later experiments because it produces good models across nearly any imposed nonzero count, unlike ℓ1 regularization’s narrow useful multiplier range.The ℓ0 projection tends to outperform ℓ1 updates as the number of nonzeros approaches that of the dense model.
- 5.3. Ensembles: Sparse ensembles improve accuracy under a fixed parameter budget, but gains diminish as increasing ensemble size makes individual models too sparse to approximate the task.The ensemble procedure combines bagged resampling with layer-wise ℓ0 constraints whose total nonzeros are bounded by 1/n.
- 5.4. Training Set Size: With less CIFAR-10 training data, differences between simpler and denser models narrow, and simpler models can outperform dense models in some cases.
6. Discussion
The discussion presents sparsity-inducing regularization as a way to construct simpler vision models that remain effective at image classification. It also uses very sparse models to build ensembles that outperform dense baselines within fixed resources.
- Sparsity-inducing regularization is presented as a technique for regularizing and constructing simpler deep vision models.
- The method reduces model parameters more effectively than reducing network units or applying simple thresholding.
- Very sparse models perform surprisingly well on benchmark image classification tasks and can form ensembles that outperform dense baselines within fixed resources.
A. Memory Usage
The paper estimates sparse-network memory using dense, bitmask, and indexed storage formats, selecting the cheapest format for each sparsity pattern. Memory accounting assumes single-precision weights and includes sparse-structure overhead.
- Memory estimates compare dense storage with bitmask and indexed sparse formats for storing CNN weights.Bitmask storage records parameter occupancy separately from nonzero values, while indexed storage stores each nonzero value with its original-array index.
- Compressed Sparse Row formats may also be suitable for some layers, with memory cost approximately equal to indexed storage.
- The memory calculations assume 32-bit, 4-byte floating-point weights; double precision has relatively smaller sparse-structure overhead, while half precision and fixed-point formats are unsupported by standard hardware or CNN codes.
- Figure 1 plots memory for candidate sparse networks from a greedy search over per-layer nonzero distributions and reports indexed-format memory for an ImageNet sparse CNN.Reported units are Kibibytes and Mebibytes, abbreviated KB and MB.