Source-linked AI summary
Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
Yani Ioannou, Duncan Robertson, Roberto Cipolla, Antonio Criminisi
TL;DR
Efficient CNN design must reduce computation and parameters without sacrificing recognition accuracy. The paper introduces tree-root-like hierarchical filter groups that impose sparse inter-layer dependencies and trains modified CNN architectures from scratch. Across CIFAR10 and ILSVRC variants, the method reports similar or higher accuracy with fewer parameters and less computation, while homogeneous grouping remains a stated limitation.
Problem
State-of-the-art CNNs are largely monolithic, motivating a way to reduce channel inter-connectivity and computational cost without compromising accuracy.
Method
The method uses hierarchical filter groups arranged in a tree-root topology, with root modules combining grouped spatial filters and 1×1 embeddings to impose sparse filter dependencies.
Results
Across CIFAR10 and ILSVRC, efficient variants achieved similar accuracy with fewer parameters and less computation; ResNet 50 used 27% fewer parameters and was 24% (11%) faster on CPU (GPU).
Takeaways & Limitations
Structured channel sparsity can produce more efficient variants of Network-in-Network, GoogLeNet, and ResNet while retaining comparable accuracy.
Takeaways & Limitations
The paper focuses on homogeneous filter groups, which may be suboptimal compared with heterogeneous groups that better reflect filter co-dependencies.
Abstract
from arXiv · showhide
We propose a new method for creating computationally efficient and compact convolutional neural networks (CNNs) using a novel sparse connection structure that resembles a tree root. This allows a significant reduction in computational cost and number of parameters compared to state-of-the-art deep CNNs, without compromising accuracy, by exploiting the sparsity of inter-layer filter dependencies. We validate our approach by using it to train more efficient variants of state-of-the-art CNN architectures, evaluated on the CIFAR10 and ILSVRC datasets. Our results show similar or higher accuracy than the baseline architectures with much less computation, as measured by CPU and GPU timings. For example, for ResNet 50, our model has 40% fewer parameters, 45% fewer floating point operations, and is 31% (12%) faster on a CPU (GPU). For the deeper ResNet 200 our model has 25% fewer floating point operations and 44% fewer parameters, while maintaining state-of-the-art accuracy. For GoogLeNet, our model has 7% fewer parameters and is 21% (16%) faster on a CPU (GPU).
1. Introduction
The paper targets the inefficiency of monolithic CNNs by applying structured sparsity to filter inter-connectivity. Its filter-group approach reduces computation and model size while preserving accuracy.
- Contribution: The proposed sparse connection structure resembles a tree root and reduces computational cost and parameters without compromising accuracy.The method is presented as a way to create computationally efficient and compact CNNs.
- Motivation: 13% of learned weights can be redundant, motivating methods that make neural networks smaller and more computationally efficient.The passage frames redundancy as a property already exploited by network-compression approaches.
- Motivation: Deep CNNs often use monolithic layers in which each filter operates on all previous-layer feature maps.This contrasts with the specialized, specifically interconnected arrangements observed in biological neural networks.
- Contribution: Filter grouping applies sparsity to channel extents, complementing prior low-rank methods that addressed only spatial filter extents.The proposed alteration targets filter inter-connectivity rather than the spatial dimensions h and w.
2. Related Work
Prior CNN-efficiency work reduced spatial complexity, precision, dimensionality, or pretrained-model size, while filter groups exposed a channel-sparsity opportunity. This paper instead trains a structurally sparse architecture from scratch, extending efficiency gains to channel inter-connectivity.
- Prior efficiency methods: Earlier approaches reduced CNN cost through spatial low-rank or Fourier methods, reduced-precision representations, or compression of previously trained models.These methods primarily addressed spatial convolution, numerical representation, or post-training model size.
- Filter groups: Filter grouping reduces filter dimensions from h×w×c1 to h×w×c1/g without changing input or output feature-map dimensions.The grouped operation uses g independent groups, each processing a fraction of the input channels.
- Filter groups: Filter groups reduce incoming channel connections; AlexNet used approximately 57% fewer connection weights while achieving comparable ILSVRC accuracy.The grouped network reportedly achieved approximately 1% lower top-5 validation error than its ungrouped counterpart.
- Related architectures: Low-dimensional embeddings map feature maps from c channels to d channels where d < c, reducing computation for subsequent spatial filters.This approach is used in many state-of-the-art image-classification networks.
- Related architectures: GoogLeNet combines low-dimensional embeddings with heterogeneously sized spatial filters in inception modules to optimize computational efficiency.Its design limits expensive 5×5 convolutions while using more 3×3 and 1×1 convolutions.
- This work: Unlike low-rank approximations of trained weights, this work creates an explicitly sparse architecture and trains it from random initialization.The distinction allows the method to alter network structure rather than approximate an existing model.
3. Root Architectures
The paper introduces root architectures: sparsely connected filter groups that limit inter-layer dependence while reducing CNN computation and model size. Root modules combine grouped spatial convolutions with low-dimensional embeddings to represent full-depth filters with limited dependence.
- Root architectures use sparsely connected structures resembling tree roots to reduce computational complexity and model size in deep image-recognition networks.
- Filter groups force filters to depend on only subsets of channels from previous layers, directly limiting feature co-adaptation.
- Reduced channel connectivity makes grouped filters smaller, lowering both computational complexity and model size.
- Each root module combines grouped spatial convolutional layers with following 1×1 low-dimensional embeddings.
- Increasing the number of filter groups decreases connections to outputs of the previous layer.
4. Results
Root modules replace spatial convolutions in established CNNs and substantially reduce computation and model size while preserving comparable accuracy. Experiments on NiN, ResNet, and deeper architectures identify root topologies as effective configurations.
- Improving Network in Network on CIFAR-10: Root-8 NiN achieves equivalent accuracy with 46% of baseline FLOPS, 33% of its parameters, and approximately 37% faster CPU and 23% faster GPU timings.
- Improving Network in Network on CIFAR-10: Root modules reveal block-diagonal inter-layer filter correlations, with stronger grouping producing denser block-diagonal organization.
- Improving Network in Network on CIFAR-10: The root topology gives the smallest accuracy reduction for a given reduction in model size and computational complexity.
- ResNet 50: Root-16 ResNet-50 exceeds baseline accuracy by 0.2% while reducing model size by 27%, FLOPS by 37%, CPU time by 23%, and GPU time by 13%.
- ResNet 200: Root-32 ResNet-200 maintains comparable error while using 25% fewer FLOPS and 44% fewer parameters.
- Improving GoogLeNet on ILSVRC: GoogLeNet configurations reduce computational complexity and model size, with many top-5 accuracies within 0.5% of baseline.
5. GPU Implementation
Measured runtime reductions broadly accompany the method’s lower FLOPS and parameter counts, but GPU gains are smaller than theoretical complexity estimates because existing software is optimized for less-grouped architectures.
- CPU and GPU runtimes decrease alongside FLOPS and floating-point parameter counts without compromising accuracy.
- GPU runtime reductions are smaller than expected from theoretical FLOPS reductions.
- Caffe’s optimization for architectures with limited filter grouping contributes to the GPU timing disparity.
- CuBLAS overhead grows with many smaller matrix multiplications created by high filter-group counts.
- Modified Caffe using CuBLAS batched calls achieved a 25% speedup on the root-16 modified GoogLeNet architecture, but remained slower than Caffe with CuDNN.
6. Future Work
The study focuses on homogeneous filter groups, while heterogeneous groups and combined spatial-channel bases remain possible directions for improving efficiency.
- Homogeneous filter groups may not be optimal because heterogeneous groups could better reflect filter co-dependencies in deep networks.
- Learning a combined spatial and channel basis may further improve efficiency.
7. Conclusion
The paper introduces a root topology that reduces CNN computation and model size while maintaining accuracy, and validates it across several state-of-the-art architectures and datasets.
- A structured decrease in filter grouping with depth forms a root topology for efficient variants of state-of-the-art CNNs without compromising accuracy.The approach appears complementary to low-dimensional embeddings and trains deep networks more efficiently than methods approximating pretrained weights.
- The method was evaluated in efficient Network-in-Network, GoogLeNet, and ResNet variants on CIFAR10 and ILSVRC.
- 33% of the original Network-in-Network parameters and approximately 37% (23%) faster CPU (GPU) timings were achieved on CIFAR10.
- 27% fewer parameters and 24% (11%) faster CPU (GPU) timings were achieved for the ResNet 50 variant.
- GoogLeNet used 7% fewer parameters and was 21% (16%) faster on a CPU (GPU).
A.1. Full Network-in-Network Results
The supplied passage identifies the Network-in-Network CIFAR10 results table but does not report its contents.
- Table 8 presents Network-in-Network results on CIFAR10.
A.2. Inter-Layer Covariance
The paper measures covariance between adjacent featuremaps to examine inter-layer filter relationships, using whitening to separate learned relationships from within-featuremap covariance. The resulting correlations reveal structured grouping and block-diagonal sparsity in root networks.
- Covariance between adjacent featuremaps is calculated from responses of convolutional layers with c1 and c2 filters.
- Each featuremap is represented as N pixel responses, with nearest-neighbor up-sampling used to equalize sample counts after pooling.
- Whitening the featuremap responses is used to clarify inter-layer covariance by removing covariance inherent within the individual responses.
- Root networks show more structured intra-layer filter correlation, with filters grouped alongside strongly correlated filters.
- ResNet 50 filter groups placed after conv1 provide direct visual intuition about organization in root networks.
- Block-diagonal sparsity is visible between conv3a and conv2c and across grouped layers in the Network-in-Network variants.
A.3. The Affect on Image-level Filters of Root Modules
Root modules organize image-level filters into groups with strong mutual information while preserving similarity to filters learned by the original ResNet. They reduce computation mainly in spatial convolutional layers.
- Root-model image-level filters are similar to those in the original ResNet, although they may be inverted or differently ordered.
- Filter groups produce consistent ordering across random initializations by placing filters with strong mutual information adjacent to one another.
- In the root-8 network, each filter row corresponds to the input of an independent conv2 filter group.Rows can contain related color-gradient filters that combine into color edges, although shortcut layers make some groupings less obvious.
- Spatial 3×3 convolutional layers have the highest computational complexity in standard ResNet-50 models.
- Using a root module dramatically reduces the computational complexity of the spatial convolutional layers, while leaving 1×1 low-dimensional embedding layers unchanged.