Source-linked AI summary
Dynamic Convolution: Attention over Convolution Kernels
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, Zicheng Liu
TL;DR
Light-weight CNNs lose representation capability when tight computational budgets constrain depth and width. The paper introduces dynamic convolution, which aggregates multiple input-dependent convolution kernels without increasing network depth or width. It reports improvements across ImageNet classification and COCO keypoint detection with small computational overhead.
Problem
Severe computational constraints reduce efficient CNN depth and width, limiting their representation capability and performance.
Method
Dynamic convolution replaces a single convolution kernel with multiple parallel kernels aggregated dynamically according to input-dependent attentions.
Results
Dynamic convolution improves ImageNet classification and COCO keypoint detection; MobileNetV3-Small gains 2.9% top-1 accuracy with 4% additional computational cost.
Takeaways & Limitations
Dynamic convolution can be integrated into existing efficient CNN architectures to improve representation capability with negligible extra computation.
Takeaways & Limitations
DY-CNNs are more difficult to train because convolution kernels and attention across multiple layers require joint optimization.
Abstract
from arXiv · showhide
Light-weight convolutional neural networks (CNNs) suffer performance degradation as their low computational budgets constrain both the depth (number of convolution layers) and the width (number of channels) of CNNs, resulting in limited representation capability. To address this issue, we present Dynamic Convolution, a new design that increases model complexity without increasing the network depth or width. Instead of using a single convolution kernel per layer, dynamic convolution aggregates multiple parallel convolution kernels dynamically based upon their attentions, which are input dependent. Assembling multiple kernels is not only computationally efficient due to the small kernel size, but also has more representation power since these kernels are aggregated in a non-linear way via attention. By simply using dynamic convolution for the state-of-the-art architecture MobileNetV3-Small, the top-1 accuracy of ImageNet classification is boosted by 2.9% with only 4% additional FLOPs and 2.9 AP gain is achieved on COCO keypoint detection.
1. Introduction
Efficient CNNs lose accuracy under severe computational constraints because reduced depth and width limit representation capability. Dynamic convolution addresses this by aggregating input-dependent parallel kernels without increasing network depth or width, yielding accuracy gains with small computational overhead.
- Motivation: When MobileNetV3’s computational cost falls from 219M to 66M Multi-Adds, ImageNet top-1 accuracy drops from 75.2% to 67.4%.The low budget constrains network depth and width, both important for performance.
- Method: Dynamic convolution replaces each layer’s single kernel with K parallel convolution kernels aggregated using input-dependent attention.The aggregated kernel and bias are formed from attention-weighted kernel components.
- Method: Dynamic convolution increases representation capability without increasing network depth or width.Parallel kernels share output channels through aggregation, while extra computation is limited to attention calculation and kernel aggregation.
- Training: DY-CNNs are harder to train because all convolution kernels and attention mechanisms must be jointly optimized across multiple layers.The paper uses attention normalization and early-training flattening, implemented with softmax at a large temperature, to facilitate optimization.
- Results: At a 100M Multi-Adds budget, dynamic convolution improves ImageNet top-1 accuracy by 4.5% for MobileNetV2 and 2.9% for MobileNetV3, with 4% additional computational cost.The method is evaluated by replacing static convolution in MobileNet V2 and V3.
2. Related Work
Prior efficient-CNN research combines efficient operators and architecture design, while related dynamic-network methods adapt execution structure. Dynamic convolution instead keeps the network structure static and adapts convolution kernels.
- Efficient CNNs: Efficient CNNs reduce computation through operators such as depthwise convolution, inverted residuals, linear bottlenecks, and squeeze-and-excitation.Architecture search is also used to design efficient convolutional networks.
- Dynamic Neural Networks: Dynamic neural-network methods adapt computation by skipping blocks, exiting early, or supporting multiple network widths.Examples include D2NN, SkipNet, BlockDrop, MSDNet, Slimmable Nets, and Once-for-all.
- Distinction: Dynamic convolution differs from these methods by using dynamic convolution kernels with a static network structure.The related methods generally use static kernels with dynamic network structure.
3. Dynamic Convolutional Neural Networks
Dynamic convolution increases model capability without widening or deepening efficient CNNs by aggregating multiple convolution kernels through input-dependent attention. The design targets better performance–compute trade-offs while keeping aggregation overhead small.
- Dynamic Convolutional Neural Networks: Dynamic convolution aggregates multiple convolution kernels through attention without increasing network depth or width.Different input images produce different kernel assemblies.
- Dynamic Perceptron: A dynamic perceptron aggregates K linear functions using attention weights that vary with the input.The aggregated weight and bias are input-dependent functions.
- Dynamic Perceptron: The aggregated dynamic model is nonlinear and therefore has more representation power than its static counterpart.Its attention weights represent an input-specific aggregation of linear models.
- Dynamic Convolution: Dynamic convolution meets its computational constraint because attention computation and kernel aggregation cost much less than convolution.Convolution is identified as a suitable operator, unlike a fully connected layer.
- Dynamic Convolution: 4%: dynamic convolution with K = 4 kernels increases MobileNetV2 (×1.0) computation cost by 4%.The method increases model size but does not increase each layer’s output dimension.
4. Two Insights of Training Deep DY-CNNs
Training deep DY-CNNs is difficult because attention and multiple convolution kernels must be jointly optimized. The paper addresses this with normalized, initially near-uniform attention, using high-temperature softmax and annealing to improve training and accuracy.
- Insight 1: Sum the Attention to One: Constraining attention weights to sum to one keeps the aggregated kernel within the convex hull of the individual kernels.This normalization compresses the kernel space and simplifies learning the attention model jointly with the convolution kernels.
- Insight 2: Near-uniform Attention in Early Training Epochs: Near-uniform attention during early training helps optimize all convolution kernels simultaneously.The authors identify this as a second key insight for efficient joint optimization.
- Insight 2: Near-uniform Attention in Early Training Epochs: Softmax with temperature τ = 30 raises final top-1 accuracy from 65.4% for the static counterpart to 69.4% in the described DY-MobileNetV2 experiment.Higher temperature makes attention less sparse and improves training efficiency.
- Insight 2: Near-uniform Attention in Early Training Epochs: Temperature annealing from 30 to 1 over the first 10 epochs further improves top-1 accuracy from 69.4% to 69.9%.These results support the importance of near-uniform attention in early training epochs.
- Relation to Concurrent Work: Compared with CondConv, the method uses a smaller kernel space and achieves higher accuracy with fewer kernels, a smaller model, and fewer computations.The paper attributes the difference to normalized attention and near-uniform early training behavior.
5. Experiments: ImageNet Classification
ImageNet experiments show that dynamic convolution improves efficient CNNs across architectures and configurations with small computational overhead. Ablations indicate that attention placement, kernel count, and temperature affect performance and optimization.
- Inspecting DY-CNN: Attention is more effective at higher layers with lower resolutions: enabling it only there reaches 67.0% top-1 accuracy versus 42.5% when enabled only at lower levels.
- Ablation Studies: Dynamic convolution outperforms its static counterpart for all tested depth and width multipliers, even with K = 2 kernels.Accuracy stops increasing once K exceeds 4, as optimization becomes harder and over-fitting becomes more likely.
- Ablation Studies: Using dynamic convolution in all three MobileNetV2 bottleneck layers yields the best accuracy, while the final 1 × 1 convolution performs best when only one layer is dynamic.
- Ablation Studies: 69.9% top-1 accuracy is obtained with temperature annealing, improving on 69.4% with τ = 30.Large softmax temperature early in training is important for effective optimization.
- Ablation Studies: Dynamic convolution without SE exceeds MobileNetV3-Small with SE by 2.2% top-1 accuracy, while combining both mechanisms adds another 0.7%.
- Main Results: Dynamic convolution outperforms static convolution across MobileNetV2, MobileNetV3, and ResNet with approximately 4% extra computational cost.DY-ResNet and DY-MobileNetV2 gain more than 2.3% and 3.2% top-1 accuracy, respectively.
- Main Results: 2.9% higher top-1 accuracy is achieved by DY-MobileNetV3-Small over MobileNetV3-Small.
6. DY-CNNs for Human Pose Estimation
Dynamic convolution is evaluated for single-person COCO keypoint detection using lightweight network designs. Improvements arise mainly from the backbone, while dynamic convolution in the head also helps.
- Experimental Setup: Dynamic convolution is evaluated on COCO 2017 single-person keypoint detection using mean average precision over 10 OKS thresholds.The validation set contains 5000 images, and the training set contains 57K images with 150K person instances.
- Results: Dynamic convolution improves keypoint-detection AP when used in both the backbone and the head.The evaluated dynamic-convolution layers use K = 4 kernels.
- Ablation: Most of the improvement comes from dynamic convolution in the backbone, although applying it in the head is also helpful.The backbone contains more convolution layers than the head.
7. Conclusion
The paper introduces dynamic convolution as an input-dependent aggregation of multiple convolution kernels. It reports improved classification and human pose estimation with negligible additional computation and easy integration into existing CNNs.
- Dynamic convolution aggregates multiple convolution kernels according to input-dependent attentions for each input.
- Replacing convolution kernels with dynamic convolution improves MobileNet V2 and V3 for image classification and human pose estimation.
- Dynamic convolution increases representation capability with negligible extra computation and can be integrated into existing CNN architectures.
A. Appendix
The appendix reports runtime measurements and additional analyses of dynamic convolution.
- The appendix reports running time and additional analysis for dynamic convolution.
A.1. Inference Running Time
DY-MobileNetV2 adds modest inference overhead relative to MobileNetV2 while improving model performance.
- Running time was measured on a single-threaded Intel Xeon CPU using 5,000 batch-size-one images.Both models were implemented in PyTorch, and inference times were averaged.
- 10% more running time and 4% more Multi-Adds are reported for DY-MobileNetV2 versus its static counterpart.The running-time overhead exceeds the Multi-Adds increase.
- The authors attribute the larger running-time overhead to less efficient optimization of global average pooling and small inner-product operations than convolution.
A.2. Dynamic Convolution in Shallower and Thinner Networks
Dynamic convolution is especially useful in smaller networks, where it improves capability and can provide a better accuracy–cost trade-off than simply increasing depth.
- Shallower DY-MobileNetV2 achieves a better accuracy–computational-cost trade-off than deeper MobileNetV2.The comparison concerns depth ×0.5 DY-MobileNetV2 versus depth ×1.0 MobileNetV2.
- Dynamic convolution provides greater improvement for thinner and shallower networks with small width and depth multipliers.
- DY-MobileNetV2 uses K = 4 dynamic kernels in every convolution layer except the first layer.
- Shallower networks suffer performance degradation for both dynamic and static MobileNetV2, but dynamic convolution improves the capability of underfitted models.The passage attributes underfitting to limited model size.
A.3. Example: Learning XOR
The XOR example illustrates that dynamic perceptron can solve the task with one layer by using input-dependent attentions, unlike the referenced two-layer static solution.
- The XOR task uses four two-dimensional input points: [0, 0]T, [0, 1]T, [1, 0]T, and [1]T.
- Dynamic perceptron requires only a single layer for the XOR example, compared with the referenced solution using two static perceptron layers.
- The example sets the attentions to π1(x) = x2 and π2(x) = 1 − x2.
- The authors attribute the dynamic perceptron's greater representation power to its non-linearity.