Source-linked AI summary
Compressing Convolutional Neural Networks
Wenlin Chen, James T. Wilson, Stephen Tyree, Kilian Q. Weinberger, Yixin Chen
TL;DR
Large CNNs create memory and storage problems, especially on mobile devices. FreshNets combines DCT-based frequency representation with hashed parameter sharing and allocates compression according to frequency importance. Evaluations on eight datasets report better compressed performance than competing baselines, while the reported scheme can be further improved with lower-precision parameters.
Problem
CNNs use increasingly large models, but mobile devices have tight memory and storage limits that constrain deployable parameter sizes.
Method
FreshNets transforms convolutional filters with DCT, shares frequency parameters through hashing, and assigns more hash capacity to important low-frequency components.
Results
Across eight benchmark datasets, FreshNets preserves prediction accuracy significantly better than competing baselines at high compression rates.
Takeaways & Limitations
Frequency-sensitive parameter sharing provides a compressed CNN model that performs better than relevant baselines on almost all evaluated classification tasks.
Takeaways & Limitations
The reported compression rates could be further improved by learning and storing parameters in lower precision.
Abstract
from arXiv · showhide
Convolutional neural networks (CNN) are increasingly used in many areas of computer vision. They are particularly attractive because of their ability to "absorb" great quantities of labeled data through millions of parameters. However, as model sizes increase, so do the storage and memory requirements of the classifiers. We present a novel network architecture, Frequency-Sensitive Hashed Nets (FreshNets), which exploits inherent redundancy in both convolutional layers and fully-connected layers of a deep learning model, leading to dramatic savings in memory and storage consumption. Based on the key observation that the weights of learned convolutional filters are typically smooth and low-frequency, we first convert filter weights to the frequency domain with a discrete cosine transform (DCT) and use a low-cost hash function to randomly group frequency parameters into hash buckets. All parameters assigned the same hash bucket share a single value learned with standard back-propagation. To further reduce model size we allocate fewer hash buckets to high-frequency components, which are generally less important. We evaluate FreshNets on eight data sets, and show that it leads to drastically better compressed performance than several relevant baselines.
1 Introduction
CNNs achieve strong results but their growing parameter counts conflict with mobile devices’ tight memory and storage limits. FreshNets addresses this challenge through frequency-aware hashing and offers favorable compressed performance across eight benchmarks.
- 1 Introduction: FreshNets targets CNN compression because mobile devices impose tight memory and storage limits, sometimes restricting models to only a few megabytes.The paper motivates compression by the shift of computing toward mobile platforms.
- 1 Introduction: The method transforms smooth convolutional filters into the frequency domain and applies hashing to share parameters, extending compression beyond fully connected layers.The approach exploits local pixel correlation and low-frequency structure in CNN filters.
- 1 Introduction: FreshNets makes parameter count independent of the number of convolutional filters and supports reconstruction and training with low-cost hashing and standard back-propagation.The gradient for a hash bucket is the sum of gradients from its assigned frequency components.
- 1 Introduction: Across eight image benchmarks and four competitive baselines, FreshNets is the most effective compression method and has the lowest generalization error on almost all classification tasks.The paper reports that increasing compression lowers test accuracy for all schemes, but FreshNets degrades least overall.
2 Background
Feature hashing reduces model storage by mapping high-dimensional inputs into smaller spaces, while DCT represents smooth signals compactly through low-frequency components. Together, these ideas motivate frequency-domain compression for CNN filters.
- 2 Background: Feature hashing maps x ∈ R^d into a much smaller R^k space using hash functions, reducing stored parameters from O(d) to O(k).The mapping uses an index hash and a sign hash, with inner products preserved in expectation.
- 2 Background: The sign factor in feature hashing corrects bias, while larger hash tables or multiple hashing can reduce information loss from collisions.The passage notes that collision loss is less severe for sparse feature vectors.
- 2 Background: DCT expresses a function as weighted sinusoids whose coefficients indicate frequency magnitudes, and its inverse reconstructs the spatial representation without loss at sufficient precision.The losslessness statement excludes quantization and other compression operations.
- 2 Background: DCT is useful for compression because smooth images concentrate weight in a few low-frequency components, and it produces real-valued coefficients unlike DFT.This spectral compaction property motivates preserving low-frequency components during CNN compression.
3 Frequency-Sensitive Hashed Nets
FreshNets compress convolutional filters by sharing frequency-domain weights, reconstructing spatial filters through the inverse DCT. Frequency-sensitive hashing allocates separate hash spaces and fewer buckets to less important high-frequency components.
- Frequency-Sensitive Hashed Nets: FreshNets stores shared convolutional parameters in the frequency domain and reconstructs spatial filters with the inverse DCT.Filters are converted with the DCT, while the inverse transform recovers their spatial representation.
- Random Weight Sharing by Hashing: Hashing ties each filter frequency weight to a shared value using a hash function and sign factor without auxiliary assignment storage.The sign factor is intended to preserve inner products in expectation, while hashing avoids additional storage for assignments.
- Gradients over Shared Frequency Weights: Shared frequency weights remain trainable with standard back-propagation through gradients derived for the frequency-domain parameters.The gradient derivation connects spatial-domain filter gradients to the shared frequency weights.
- Frequency-Sensitive Hashing: Because CNN filters concentrate magnitude in low frequencies, FreshNets separates frequency groups to avoid collisions between components of different importance.Low-frequency components generally have larger magnitudes, whereas high-frequency components are near zero.
- Frequency-Sensitive Hashing: A smaller compression rate r_j creates more hash collisions, so the method typically assigns larger r_j values to low-frequency regions and squeezes high-frequency regions.This reserves more hash-space capacity for lower-frequency parameters under a fixed layer-size budget.
4 Related Work
Prior compression work reduces neural-network storage through low-rank decomposition, distillation, reduced precision, or hashing. Related methods also target convolutional evaluation or fully connected layers, whereas FreshNets focuses on compressed convolutional filters through frequency-domain sharing.
- 4 Related Work: Low-rank decomposition can replace O(mn) fully connected parameters with O(k(m + n)) storage when k is much smaller than both dimensions.Denil et al. factorize W as AB, reducing the stored parameter count.
- 4 Related Work: Other compression approaches distill complex models into one-layer networks, reduce numerical precision, or hash fully connected parameters before learning.HashedNets reports notable compression with less accuracy loss than low-rank and randomly sparse alternatives.
- 4 Related Work: As architectures use fewer fully connected and more convolutional layers, compressing convolutional filters becomes increasingly important.Frequency-domain convolution methods primarily target evaluation speed and may introduce test-time memory overhead through frequency oversampling.
5 Experimental Results
Experiments evaluate FreshNets across eight benchmark datasets and multiple frequency-sensitive compression schemes. FreshNets consistently outperforms the baselines, especially at higher compression, while preserving smoother filters than spatial-domain HashedNets.
- Comprehensive evaluation: The evaluation covers CIFAR10, CIFAR100, SVHN, and five MNIST variants using a five-convolutional-layer architecture with one fully connected layer.The datasets include rotated, background-superimposed, and reduced-training-size MNIST variants; the common architecture uses 5 × 5 filters and max-pooling in layers 2, 4, and 5.
- Comprehensive evaluation: FreshNets consistently outperforms DropFilt, DropFreq, low-rank decomposition, and HashedNets, particularly at the 1/64 compression rate.Table 2 reports test-error comparisons across the benchmark evaluation; DropFilt and DropFreq are excluded at 1/64 because they do not support that compression in all layers.
- Varying compression by frequency: Fewer shared weights for high-frequency components outperform alternative frequency-compression schemes, while the inverse allocation performs worst.The comparison uses normalized CIFAR10 test error averaged over compression rates from 1 to 1/256.
- Filter visualization: FreshNets preserves smooth convolutional filters at 1/16 compression, whereas spatial-domain HashedNets produces more chaotic filters.The visualization uses first-layer filters from four-layer networks trained on MNIST with 11 × 11 filters.
6 Conclusion
FreshNets compresses CNN storage by combining hashed parameter sharing with frequency-sensitive preservation of convolutional filters’ important low-frequency components, maintaining prediction accuracy better than competing baselines at high compression rates.
- FreshNets preserves prediction accuracy significantly better than competing baselines at high compression rates.The method uses frequency-sensitive compression to better preserve significant parameters, such as low-frequency components.