PaperlayerView exact revision on arXiv

Source-linked AI summary

Omni-Dimensional Dynamic Convolution

Chao Li, Aojun Zhou, Anbang Yao

arXiv:2209.07947v1cs.CVcs.AIcs.LG

TL;DR

Existing dynamic convolutions adapt kernel combinations but overlook three dimensions of the kernel space, motivating a more comprehensive design. ODConv applies parallel, complementary attentions across all four dimensions and, in experiments, improves CNN performance across ImageNet and MS-COCO while retaining efficiency advantages.

  • Problem

    Existing dynamic convolutions adapt convolutional kernels through kernel-number attention while overlooking spatial, input-channel, and output-channel dimensions.

  • Method

    ODConv is a drop-in dynamic convolution that uses a parallel multi-dimensional attention mechanism to learn complementary attentions along all four kernel-space dimensions.

  • Results

    ODConv improves recognition across ImageNet and MS-COCO CNN experiments, and one-kernel ODConv can compete with or outperform multi-kernel dynamic convolutions with fewer extra parameters.

  • Takeaways & Limitations

    ODConv can replace regular convolutions across CNN architectures while offering a stronger accuracy–efficiency tradeoff than existing dynamic convolution designs.

  • Takeaways & Limitations

    Existing dynamic convolution increases convolutional parameters by n times relative to regular convolution, although its extra Multiply-Adds are marginal.

Abstract

from arXiv · show

Learning a single static convolutional kernel in each convolutional layer is the common training paradigm of modern Convolutional Neural Networks (CNNs). Instead, recent research in dynamic convolution shows that learning a linear combination of $n$ convolutional kernels weighted with their input-dependent attentions can significantly improve the accuracy of light-weight CNNs, while maintaining efficient inference. However, we observe that existing works endow convolutional kernels with the dynamic property through one dimension (regarding the convolutional kernel number) of the kernel space, but the other three dimensions (regarding the spatial size, the input channel number and the output channel number for each convolutional kernel) are overlooked. Inspired by this, we present Omni-dimensional Dynamic Convolution (ODConv), a more generalized yet elegant dynamic convolution design, to advance this line of research. ODConv leverages a novel multi-dimensional attention mechanism with a parallel strategy to learn complementary attentions for convolutional kernels along all four dimensions of the kernel space at any convolutional layer. As a drop-in replacement of regular convolutions, ODConv can be plugged into many CNN architectures. Extensive experiments on the ImageNet and MS-COCO datasets show that ODConv brings solid accuracy boosts for various prevailing CNN backbones including both light-weight and large ones, e.g., 3.77%~5.71%|1.86%~3.72% absolute top-1 improvements to MobivleNetV2|ResNet family on the ImageNet dataset. Intriguingly, thanks to its improved feature learning ability, ODConv with even one single kernel can compete with or outperform existing dynamic convolution counterparts with multiple kernels, substantially reducing extra parameters. Furthermore, ODConv is also superior to other attention modules for modulating the output features or the convolutional weights.

1 INTRODUCTION

ODConv addresses limitations in dynamic convolution by applying complementary attentions across all four dimensions of the convolutional kernel space. As a drop-in replacement, it improves CNN accuracy and generalizes across architectures and tasks.

  • Existing dynamic convolution methods primarily improve lightweight CNNs by combining input-weighted convolutional kernels.Their attention mechanisms focus on the kernel-number dimension while overlooking other kernel-space dimensions.
  • ODConv learns four complementary attentions over the spatial, input-channel, output-channel, and kernel dimensions.These attentions are applied in parallel within the kernel space.
  • ODConv with one kernel can compete with or outperform dynamic convolution methods using multiple kernels while reducing extra parameters.This result is attributed to its improved feature learning ability.
  • ODConv can replace regular convolutions in many CNN architectures and transfers performance improvements from ImageNet classification to MS-COCO object detection.The experiments also compare ODConv with attention methods that recalibrate output features or modify convolutional weights.

2 RELATED WORK

Related work spans manually designed and automatically searched CNN architectures, attentive feature recalibration, and dynamic weight mechanisms. These approaches improve representations or adapt weights through attention and dynamic parameter generation.

  • Deep CNN Architectures: Deep CNN research has produced increasingly deep architectures with sophisticated connection topologies, alongside efforts to automate network design.Examples include VGGNet, InceptionNet, ResNet, DenseNet, ResNeXt, MobileNets, ShuffleNet, and neural architecture search methods.
  • Attentive Feature Recalibration: Attentive feature recalibration modules improve CNNs by modeling feature-channel dependencies, spatial attention, global context, or feature statistics.Representative methods include SE, BAM, CBAM, GE, SRM, and SKNets.
  • Dynamic Weight Networks: Dynamic weight networks make neural-network weights sample-adaptive or task-adaptive through mechanisms such as hypernetworks and meta learning.These methods target increased model capacity and generalization.

3 METHOD

Dynamic convolution makes convolution input-dependent by weighting multiple kernels, but existing methods assign one scalar per kernel and overlook the other kernel-space dimensions. ODConv addresses this limitation with parallel attentions over spatial locations, input channels, output filters, and kernels.

  • 3.1 REVIEW OF DYNAMIC CONVOLUTION: Dynamic convolution combines n convolutional kernels with input-dependent attention weights, unlike regular convolution’s single static kernel.The output is produced by convolving the input with an attention-weighted linear combination of kernels.
  • 3.1 REVIEW OF DYNAMIC CONVOLUTION: CondConv and DyConv assign one scalar αwi to each kernel, giving all its filters the same value and ignoring spatial, input-channel, and output-channel dimensions.This provides only coarse exploitation of the four-dimensional kernel space.
  • 3.2 OMNI-DIMENSIONAL DYNAMIC CONVOLUTION: ODConv learns four complementary attentions in parallel: spatial αsi, input-channel αci, output-filter αfi, and kernel-level αwi.The attentions are progressively multiplied with each kernel along the corresponding dimensions before convolution.
  • 3.2 OMNI-DIMENSIONAL DYNAMIC CONVOLUTION: ODConv’s progressively ordered multiplications modulate locations, input channels, filters, and kernels, enabling convolution operations to vary across all four kernel-space dimensions.The paper describes these attentions as complementary and links their combination to richer context capture.
  • 3.2 OMNI-DIMENSIONAL DYNAMIC CONVOLUTION: With n = 1, ODConv can compete with or outperform CondConv and DyConv while introducing substantially fewer extra parameters.ODConv is presented as a drop-in replacement for regular convolutions and as a more generalized dynamic convolution.
  • 3.2 OMNI-DIMENSIONAL DYNAMIC CONVOLUTION: ODConv uses a SE-typed multi-head module with GAP, an FC layer, and four heads producing attentions of sizes k × k, cin, cout, and n.The implementation uses a reduction ratio r = 1/16 in the main experiments and shares three attentions across kernels for easier implementation.

4 EXPERIMENTS

Experiments evaluate ODConv across ImageNet classification, MS-COCO detection, ablations, and inference speed using lightweight and larger CNN backbones. ODConv generally improves accuracy-efficiency tradeoffs over existing dynamic convolution and attention methods.

  • Experimental setup: Experiments cover ImageNet classification and MS-COCO detection with MobileNetV2 and ResNet backbones, comparing ODConv against dynamic convolution and attention methods.The evaluated backbones include MobileNetV2 variants, ResNet18, ResNet50, and ResNet101.
  • Image classification on ImageNet: ODConv (1×) outperforms CondConv (8×) on MobileNetV2 while matching DyConv (4×) with substantially fewer extra parameters.ODConv (4×) achieves the best results across the MobileNetV2 backbones.
  • Image classification on ImageNet: 2.85% top-1 gain is achieved by ODConv (1×) on ResNet18, while ODConv (4×) reaches a 3.72% top-1 gain.On larger ResNet50 models, existing dynamic convolution methods perform worse than most comparison methods, whereas ODConv remains effective.
  • Object detection on MS-COCO: ODConv improves MS-COCO detection accuracy over the baseline, with ODConv (1×) gaining 1.8% AP for Faster R-CNN and 1.9% AP for Mask R-CNN using ResNet50.ODConv (1×) also outperforms CondConv (8×) and nearly matches DyConv (4×) with MobileNetV2; ODConv (4×) achieves the best AP scores.
  • Ablation studies: ODConv maintains large gains across reduction ratios, with 2.85∼3.16% top-1 improvements on ResNet18 and negligible extra MAdds.The experiments select r = 1/16 as the best accuracy-efficiency tradeoff.
  • Ablation studies: ODConv with one convolutional kernel brings near 3.0% top-1 gain to the ResNet18 baseline, supporting its parameter-efficiency advantage.Inference tests also find ODConv (1×) faster than DyConv and DCD on CPU, while CondConv has the fastest runtime on GPU and CPU.

5 CONCLUSION

ODConv is a dynamic convolution design that applies parallel, complementary attentions across all four dimensions of the convolutional kernel space. Experiments on ImageNet and MS-COCO validate its superior performance across CNN architectures.

  • 5 CONCLUSION: ODConv learns four complementary attention types along all dimensions of the convolutional kernel space in parallel.Progressively applying these attentions strengthens the feature extraction ability of basic convolution operations.
  • 5 CONCLUSION: Experiments on ImageNet and MS-COCO validate ODConv's superior performance across prevailing CNN architectures.
  • 5 CONCLUSION: ODConv is presented to promote the representation power of deep CNNs.

A APPENDIX

The appendix provides supplementary analyses of ODConv's computational cost, implementation, ablations, training stability, and additional potential applications.

  • A APPENDIX: Supplementary materials analyze ODConv's computational cost and provide ImageNet and MS-COCO implementation details.
  • A APPENDIX: Additional materials include ImageNet ablations, training and validation curve comparisons, and experiments studying other potentials of ODConv.

A.1 COMPUTATIONAL COST OF ODCONV

The appendix expresses ODConv's extra multiplication-addition cost relative to regular convolution and notes that the additional cost is small compared with regular convolution's total cost.

  • A.1 COMPUTATIONAL COST OF ODCONV: For ODConv (1×), the extra MAdds over regular convolution are calculated as r + k^2c_in(1 + 2c_out).Regular convolution has hwk^2c_inc_out MAdds, excluding bias.
  • A.1 COMPUTATIONAL COST OF ODCONV: For ODConv (n×), the extra MAdds over regular convolution are calculated as r + k^2c_in(1 + c_out + 2nc_out).
  • A.1 COMPUTATIONAL COST OF ODCONV: ODConv adds relatively little computational cost compared with the hwk^2c_inc_out MAdds of regular convolution.

A.2 EXPERIMENTAL DETAILS

The ImageNet experiments evaluate ODConv with MobileNetV2 and ResNet backbones, while MS-COCO experiments use Faster R-CNN and Mask R-CNN with FPNs. Evaluation reports standard detection AP across IoU thresholds and object sizes.

  • ImageNet experiments cover both lightweight MobileNetV2 and larger ResNet18, ResNet50, and ResNet101 backbones.
  • MS-COCO experiments use pre-trained ResNet50 and MobileNetV2 backbones with Faster R-CNN and Mask R-CNN detectors using FPN necks.
  • MS-COCO validation reports Average Precision from IoU 0.5 to 0.95 and retains AP scores for small, medium, and large objects.

A.3 MORE ABLATIVE EXPERIMENTS ON IMAGENET

Additional ImageNet ablations examine layer placement, temperature annealing, activation functions, attention sharing, and longer MobileNetV2 training. They identify important training and design choices while preserving ODConv’s accuracy–size tradeoff.

  • Layer locations: Applying ODConv to fewer layers can improve runtime speed, motivating experiments on different layer locations in MobileNetV2.
  • Temperature annealing: 1.06% and 1.04% top-1 improvements result from temperature annealing for ODConv (1×) and ODConv (4×), respectively.
  • Activation functions: Activation-function choices are compared for computing ODConv’s four attention types, following prior discussions for channel and kernel dimensions.
  • Attention sharing: Removing attention sharing improves accuracy when separate αsi, αci, and αfi attentions are trained for each additive convolutional kernel.
  • Longer training: 300-epoch MobileNetV2 experiments preserve the performance-gain trend and ODConv’s better accuracy–size tradeoff versus CondConv and DyConv.

A.4 ILLUSTRATION OF MODEL TRAINING AND VALIDATION CURVES

Training and validation curves show that ODConv maintains higher top-1 gains during ResNet18 training and offers a favorable accuracy–parameter tradeoff against multi-kernel dynamic convolutions.

  • Training and validation curves: ODConv shows consistent high top-1 gains throughout training compared with CondConv, DyConv, ODConv (1×), and ODConv (4×).
  • Model accuracy and size: ODConv (1×) provides a better accuracy–size tradeoff than CondConv and DyConv on lightweight MobileNetV2 backbones.
  • Training and validation curves: ODConv (1×) outperforms CondConv and DyConv on ResNet18 while using 14.68% and 26.26% of their parameters, respectively.
  • Training and validation curves: ODConv (4×) reaches the best validation accuracy, exceeding CondConv and DyConv by 1.98% and 1.21% top-1 gain, respectively.

A.5 MORE EXPERIMENTS FOR STUDYING OTHER POTENTIALS OF ODCONV

Further experiments test ODConv’s compatibility with SE, aggressive training, alternative feature pooling, training cost, attention behavior, and larger deployment settings. Results support broad applicability while identifying computational and tuning boundaries.

  • SE combination: ODConv significantly improves a ResNet18 backbone already containing SE, indicating combination potential between kernel attention and output-feature recalibration.
  • Training enhancements: ODConv works with label smoothing, mixup, their combination, and a longer training schedule, producing further improved performance.
  • Feature pooling: Spatial kernel attention αsi brings the largest gain among the first three tested attentions, motivating analysis of feature pooling spatial size.
  • Training cost: Training cost comparisons measure seconds per batch, minutes per epoch, and total training hours across dynamic convolution methods.
  • Learnt attentions: ODConv learns αsi, αci, αfi, and αwi in parallel and progressively applies them along the kernel-space dimensions.
  • Limitations: ODConv introduces slightly higher FLOPs, inference latency, and training cost than reference methods at similar model sizes.
  • Visualization and attention behavior: Grad-CAM++ examples show that individual attentions can fail on some inputs, whereas the full ODConv (4×) attention combination makes correct predictions in the reported cases.
  • Limitations: Hyperparameters tuned with ResNet18 on ImageNet are not necessarily optimal for every backbone, and models beyond ResNet101 remain unexplored.
Loading 2209.07947v1…