Source-linked AI summary
Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference
Thomas Verelst, Tinne Tuytelaars
TL;DR
Conventional convolutions apply identical computation at every image location despite unequal regional importance. The paper learns pixel-wise gating masks end-to-end with Gumbel-Softmax and executes dynamic convolutions only at selected locations. It reports state-of-the-art classification results, strong pose-estimation results, and practical speedups from an efficient CUDA implementation.
Problem
Convolutional networks apply the same operations at every image location, although many images contain spatially sparse regions that need fewer computations.
Method
The method uses small gating networks to learn pixel-wise execution masks for dynamic convolutions in residual blocks, trained end-to-end with Gumbel-Softmax.
Results
The method achieves state-of-the-art image-classification results, strong human-pose-estimation results, and practical inference speedups with CUDA dynamic convolutions.
Takeaways & Limitations
Dynamic convolutions reduce operations by evaluating only predicted important locations and are especially suitable for high-resolution tasks such as pose estimation and object detection.
Abstract
from arXiv · showhide
Modern convolutional neural networks apply the same operations on every pixel in an image. However, not all image regions are equally important. To address this inefficiency, we propose a method to dynamically apply convolutions conditioned on the input image. We introduce a residual block where a small gating branch learns which spatial positions should be evaluated. These discrete gating decisions are trained end-to-end using the Gumbel-Softmax trick, in combination with a sparsity criterion. Our experiments on CIFAR, ImageNet and MPII show that our method has better focus on the region of interest and better accuracy than existing methods, at a lower computational complexity. Moreover, we provide an efficient CUDA implementation of our dynamic convolutions using a gather-scatter approach, achieving a significant improvement in inference speed with MobileNetV2 residual blocks. On human pose estimation, a task that is inherently spatially sparse, the processing speed is increased by 60% with no loss in accuracy.
1. Introduction
The paper targets the inefficiency of applying identical convolutional operations across all image locations, especially when images contain spatially sparse regions. It proposes input-conditioned, pixel-wise gating with efficient dynamic convolutions to improve the performance–accuracy trade-off.
- Motivation: Real-time neural networks need a better trade-off between computational complexity, frames per second, and accuracy.The motivation is particularly strong for consumer devices and increasingly complex tasks such as detection and pose estimation.
- Motivation: Conditional execution adapts network operations to image content instead of applying the same computation everywhere.The paper focuses on spatial conditioning, where different image locations can receive different amounts of computation.
- Motivation: Spatially sparse images contain background regions whose features can be extracted with few operations, unlike important regions requiring more computation.Flat regions such as blue sky are given as an example of locations that can be processed cheaply.
- Method: A small gating network predicts pixel-wise masks that select locations for dynamic convolutions in each residual block.The gating decisions are learned end-to-end with the Gumbel-Softmax trick and progress through the network from complex regions toward higher-level regions of interest.
- Implementation: The method is designed for practical inference efficiency through CUDA dynamic convolutions using tensor rearrangement and gather-scatter operations.The implementation aims to reduce wall-clock time rather than only theoretical floating-point operations, with minimal changes to existing libraries.
- Results: The paper reports state-of-the-art classification results, strong pose-estimation results, and practical speedups with MobileNetV2 and ShuffleNetV2.Its contributions include end-to-end mask training, improved performance–accuracy trade-offs, and a CUDA implementation for residual blocks.
2. Related work
Prior conditional-execution methods vary computation by layers, channels, or spatial regions, but they differ in granularity, applicability, and practical efficiency. The paper positions its approach as a finer-grained spatial method with integrated mask generation and sparse execution.
- Taxonomy: Conditional execution methods are commonly distinguished as layer-based, channel-based, or spatial methods.These approaches adapt computation according to the input image rather than relying only on static compression.
- Layer-based methods: Layer-based methods skip network layers or blocks, allowing easy images to use shallower computation than complex examples.Examples include halting residual refinement and early-exit branches.
- Channel-based methods: Channel-based methods dynamically prune channels during inference because advanced features are needed only for subsets of images.Reported approaches rank channels, use reinforcement learning, or apply Gumbel-Softmax gating.
- Spatial methods: Spatial methods exploit unequal regional importance, but glimpse-based approaches can lose information outside the crop.Cascades and finer-grained methods address some of these constraints for pixel-wise tasks.
- Spatial methods: SACT varies residual-block execution per spatial location but requires many consecutive residual layers and does not report inference speedup.The paper describes its own method as more flexible because it makes skipping decisions per residual block and spatial location.
- Practical efficiency: SBNet demonstrates practical spatial speedup using tiled dense convolutions, but tile overlap creates overhead and limits typical use to large images.The paper contrasts this with integrated mask generation, pixel-wise control, and efficient inference.
3. Method
The method learns spatial execution masks for each residual block, converts soft gating decisions into hard masks, and evaluates convolutions only at selected positions. Efficient gather-scatter execution and a computational-budget loss reduce the work performed during inference.
- Block architecture: Each residual block uses a small mask unit to generate pixel-wise execution masks from its input.The masks are learned independently for spatial locations and residual blocks.
- Gumbel-Softmax gating: The Gumbel-Softmax module converts soft gating decisions into binary masks while preserving end-to-end backpropagation.Hard samples are used during inference, while training uses differentiable soft decisions and a straight-through estimator.
- Sparse execution: During inference, the residual function is evaluated only at active spatial positions indicated by the execution mask.Training applies mask multiplication, whereas inference conditionally evaluates the residual function.
- Efficient inference implementation: Gather-scatter execution packs selected positions into a dense intermediate tensor, applies standard non-spatial operations, and restores outputs to their original locations.A modified CUDA 3×3 depthwise convolution retrieves spatial neighbors through index mappings.
- Sparsity loss: A computational budget θ controls the desired relative number of operations, while the sparsity loss minimizes deviation from that budget across residual blocks.The budget is expressed as the relative number of FLOPS executed in conditional layers.
4. Experiments and results
Experiments across classification, image recognition, and human pose estimation show that DynConv improves the accuracy–complexity trade-off and provides practical inference speedups through conditional spatial execution.
- Evaluation scope: The experiments evaluate conditional execution on CIFAR and ImageNet, practical inference speed on MobileNetV2 and ShuffleNetV2, and pose estimation with a sparsity-criterion study.The evaluation spans accuracy-complexity comparisons, throughput measurements, and ablations.
- CIFAR-10: DynConv improves the accuracy-complexity trade-off on CIFAR-10 while showing a smaller accuracy drop as computational budgets decrease.The comparison uses ResNet-32 and evaluates multiple budget targets.
- ImageNet analysis: ImageNet images with typically sparse content, such as birds, receive fewer computations than images of consumer goods.Downsampling blocks remain evaluated at all spatial positions, while computation varies across images and residual blocks.
- Qualitative evaluation: DynConv focuses computation more strongly on regions of interest than SACT at the same average complexity of 5.7 GMACs.Computational-cost heatmaps compare simple and cluttered examples.
- Human pose estimation: On MPII pose estimation, DynConv reduces operations by more than 45% with almost no accuracy loss and outperforms non-conditional models with similar operation counts.The performance gap over comparable baselines increases at larger FLOPS reductions.
- Human pose estimation: DynConv achieves a 60% inference speedup without accuracy loss and a 125% speedup with 0.6% accuracy loss in pose estimation.Mask generation, bookkeeping, and gather/scatter operations are small relative to the residual function cost.
5. Conclusion and future work
The paper proposes dynamic convolutions that learn pixel-wise execution masks end-to-end to speed inference by operating only at predicted locations. It reports state-of-the-art image-classification results and identifies advantages for high-resolution tasks such as pose estimation and object detection.
- Conclusion and future work: Dynamic convolutions speed inference by operating only on locations selected by learned pixel-wise execution masks.The masks are learned end-to-end.
- Conclusion and future work: For pose estimation, annealed lower and upper bounds per residual block provide better initialization and more stable training than a simple network-wise sparsity criterion.
- Conclusion and future work: The method achieves state-of-the-art results on image classification.
- Conclusion and future work: The method is especially suitable for high-resolution tasks such as pose estimation and object detection.