Source-linked AI summary
Training CNNs with Low-Rank Filters for Efficient Image Classification
Yani Ioannou, Duncan Robertson, Jamie Shotton, Roberto Cipolla, Antonio Criminisi
TL;DR
CNN accuracy gains have increased computational and parameter costs, motivating more efficient convolutional architectures. The paper trains low-rank filters from scratch by combining learned basis filters and introduces initialization for mixed-shape composite layers. Across CIFAR, ILSVRC, and MIT Places, the method achieves similar or higher accuracy with substantially less computation and fewer parameters.
Problem
CNN accuracy improvements have increased model size and computational complexity, limiting deployment on low-power devices and rapid analysis.
Method
The method learns small basis filters from scratch, combines them into larger convolutional filters, and initializes mixed-shape composite layers using a generalized variance rule.
Results
Across three image-classification datasets, the method achieves similar or higher accuracy with much greater computational efficiency; one variant reaches 89.7% top-5 center-crop accuracy with 16% lower computation than the baseline.
Takeaways & Limitations
Learning computationally efficient models with fewer, more relevant parameters can preserve accuracy and may improve generalization.
Takeaways & Limitations
The study addresses spatial filter extents, while redundancy in channel extents remains for future extension.
Abstract
from arXiv · showhide
We propose a new method for creating computationally efficient convolutional neural networks (CNNs) by using low-rank representations of convolutional filters. Rather than approximating filters in previously-trained networks with more efficient versions, we learn a set of small basis filters from scratch; during training, the network learns to combine these basis filters into more complex filters that are discriminative for image classification. To train such networks, a novel weight initialization scheme is used. This allows effective initialization of connection weights in convolutional layers composed of groups of differently-shaped filters. We validate our approach by applying it to several existing CNN architectures and training these networks from scratch using the CIFAR, ILSVRC and MIT Places datasets. Our results show similar or higher accuracy than conventional CNNs with much less compute. Applying our method to an improved version of VGG-11 network using global max-pooling, we achieve comparable validation accuracy using 41% less compute and only 24% of the original VGG-11 model parameters; another variant of our method gives a 1 percentage point increase in accuracy over our improved VGG-11 model, giving a top-5 center-crop validation accuracy of 89.7% while reducing computation by 16% relative to the original VGG-11 model. Applying our method to the GoogLeNet architecture for ILSVRC, we achieved comparable accuracy with 26% less compute and 41% fewer model parameters. Applying our method to a near state-of-the-art network for CIFAR, we achieved comparable accuracy with 46% less compute and 55% fewer parameters.
1 INTRODUCTION
CNN accuracy improvements have increased model size and computational cost, limiting deployment on low-power devices and rapid analysis. This paper learns low-rank filters from scratch to reduce computation while preserving or improving classification accuracy.
- CNN accuracy gains have increased model size and computational complexity, creating deployment challenges for low-power devices and fast analysis.
- The paper further sparsifies convolutional connections by representing filters with groups of differently shaped basis filters, reducing computation without compromising classification accuracy.
- The method learns small basis filters and combines them into larger filters instead of approximating filters in previously trained networks.
- Networks are trained from scratch, with a specialized initialization method for composite layers containing differently shaped filters.
- Prior approaches reduce efficiency costs through quantization, pruning, low-dimensional embeddings, or approximations of pretrained filters, but differ in training strategy and accuracy guarantees.
2 USING LOW-RANK FILTERS IN CNNS
The paper replaces full-rank convolutional filters with combinations of lower-cost basis filters. Composite layers combine differently oriented filters and can produce cross-shaped responses while preserving expressive spatial structure.
- Full-rank convolutional filters are represented as linear combinations of basis vectors to reduce convolutional-layer complexity.
- Unlike retrospective approximation methods, this approach trains separable basis filters and their combination weights simultaneously from scratch.
- Composite layers use horizontal and vertical basis filters, concatenate their responses, and combine them with 1 × 1 filters into the output feature map.
- Equal numbers of horizontal and vertical filters give computational complexity O(m(wc/2 + hc/2 + d)) and yield cross-shaped linear combinations.
- Composite layers may combine more than two filter sizes or omit the 1 × 1 layer so a subsequent layer learns the combination.
3 TRAINING CNNS WITH MIXED-SHAPE LOW-RANK FILTERS
The paper extends variance-based initialization to composite convolutional layers containing groups of differently shaped filters. The resulting rule accounts for the total outgoing connections across all filter groups.
- The initialization method generalizes He-style variance initialization to composite layers containing multiple groups of differently shaped filters.
- A composite layer is initialized as a single layer whose effective connection count is ˆn = P w[i]h[i]d[i].
- For ReLU layers, the initialization uses a zero-mean Gaussian distribution with a standard deviation derived from the summed connection count.
4 RESULTS AND COMPARISONS
Across ILSVRC, MIT Places, GoogLeNet, and CIFAR-10, low-rank convolutional representations reduce computation while preserving or improving classification accuracy. The strongest variants combine differently shaped basis filters, sometimes with full-rank filters, and are trained from scratch with specialized initialization.
- VGG-11 architectures: The simple horizontal/vertical basis made VGG approximately 49% faster than the original while keeping ILSVRC and MIT Places accuracy within 1 percentage point.Each original 3 × 3 convolution was replaced by 1 × 3 and 3 × 1 filters followed by 1 × 1 filters.
- VGG-11 architectures: 89.7% top-5 center-crop accuracy with approximately 16% less computation than the baseline was achieved by mixing 25% 3 × 3 with 75% 1 × 3 and 3 × 1 filters.This variant was approximately 1 percentage point more accurate than GoogLeNet.
- VGG-11 architectures: Removing the intermediate 1 × 1 combination layer reduced the VGG variant to one-third of baseline compute without reducing accuracy.The subsequent convolutional layer could learn effective combinations after the intermediate ReLU.
- VGG-11 architectures: Doubling the number of low-rank convolutional filters increased ILSVRC top-5 accuracy to 88.9% while remaining approximately 58% faster than baseline.The low-dimensional-embedding variant reduced computation by approximately 86% but lost 1.2 top-5 percentage points, while remaining 2.5 points above original VGG-11.
- GoogLeNet for ILSVRC: Comparable GoogLeNet accuracy was achieved with 26% less compute using the low-rank network.Test-time model size is smaller than training-time size because intermediate losses contain the only fully connected layers.
- Network-in-Network for CIFAR-10: The CIFAR-10 low-rank NiN variant reached similar accuracy, 91.8% versus 91.9%, at approximately 54% of the original computation and 45% of its parameters.The preceding 3 × 3 redesign was already 26% faster than standard NiN with 54% of its parameters.
5 DISCUSSION
The paper argues that low-rank filters can match or exceed full filters because image-classification structure is often low-rank, while basis combinations retain useful complexity and may reduce over-fitting.
- Low-rank filters can perform as well as or better than full k × k filters because edges and gradients are well characterized by low-rank structure.
- Basis-filter architectures combine complex k×k and simple 1 × k or k × 1 filters to learn effective spatial representations.
- Restricting the number of complex spatial filters reduces representational freedom and helps improve generalization by preventing over-fitting.
6 CONCLUSION
The method trains CNNs from scratch with low-rank filters and specialized initialization, achieving similar or higher accuracy with greater computational efficiency across three image-classification datasets. The paper focuses on spatial filter redundancy, while leaving channel redundancy for future work.
- Similar or higher accuracy than state-of-the-art methods was achieved across three image-classification datasets with much greater computational efficiency.
- Learning computationally efficient models with fewer, more relevant parameters may prevent over-fitting, increase generalization, and increase accuracy.
- The method addresses spatial filter extents, but channel extents also contain redundancy and remain an avenue for future extension.
A INITIALIZING CNNS WITH MIXED-SHAPE LOW-RANK FILTERS
The paper derives a weight-initialization scheme for composite convolutional layers that combine heterogeneous filter shapes, preserving gradient scale during training. The scheme extends standard initialization to mixed-shape low-rank architectures by accounting for each filter group's outgoing connections.
- Weight Initialization: Unsuitable random weight initialization can cause training failure through exploding or vanishing gradients, making layer-specific variance settings critical.The authors found published settings unsuitable for their new architectures.
- Weight Initialization in Composite Layers: Composite-layer initialization generalizes standard initialization to groups of filters with different spatial dimensions, a main contribution of the work.The layer response concatenates outputs from the heterogeneous filter groups.
- Forward Propagation: Standard convolutional forward propagation maps each local input region through a filter-weight matrix and bias to produce output feature responses.The input vector represents a spatial subregion across input channels, while the weight matrix contains one row per filter.
- Backward Propagation: During back-propagation, gradients are propagated through reshaped filter weights from output pixels to input pixels, with activation derivatives mediating the signal.The gradient representation accounts for the spatial output pixels that depend on each input pixel.
- Weight Initialization in Composite Layers: A heterogeneous filter group should be initialized as one layer whose effective connection count sums w[i]h[i]d[i] across groups.This rule follows the variance analysis for composite layers and is intended to keep gradient magnitudes stable.
C COMPARING WITH STATE OF THE ART NETWORKS FOR ILSVRC
The comparison is complicated by differing training and test-time augmentation protocols, yet the proposed models achieve comparable accuracy with substantially lower computational complexity.
- Published error rates use varied scale, photometric, ensemble, and multiview or dense-oversampling procedures, complicating architectural comparisons.
- The proposed models achieve comparable accuracy to state-of-the-art networks without training or test-time augmentation, while using far less computation.The comparison includes networks such as VGG-19, oversampled GoogLeNet, and MSRA-C.
- State-of-the-art networks can be orders of magnitude more computationally complex than the proposed networks.
- Table 5 reports single-view and augmented top-5 validation accuracy, multiply-accumulate counts, and parameter counts for state-of-the-art models.
D PLOTS OF RESULTS
The supplementary plots extend the evaluation across VGG-derived, GoogLeNet-derived, and Network-in-Network-derived models on three image-classification benchmarks.
- Supplementary plots report VGG-derived results on MIT Places, GoogLeNet-derived results on ILSVRC, and Network-in-Network-derived results on CIFAR-10.
E VGG-DERIVED MODEL TABLE
The VGG-derived model appendix documents the architectures and plots their efficiency results alongside related benchmark figures.
- Figure 7 compares test-time multiply-accumulate operations with top-5 error for single state-of-the-art ILSVRC models.
- Figure 8 compares test-time parameter counts with top-5 error and notes that global pooling or removing fully connected layers reduces model size.
- Figure 9 plots multiply-accumulate operations against top-5 error for VGG-derived models on MIT Places.
- Figures 10 and 11 plot multiply-accumulate operations against error for GoogLeNet-derived ILSVRC models and Network-in-Network-derived CIFAR-10 models, respectively.
- The VGG-derived model architectures are documented in Table 6, including filter shapes, channel counts, strides, fully connected layers, and composite-layer concatenation.