PaperlayerView exact revision on arXiv

Source-linked AI summary

HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions

Yongming Rao, Wenliang Zhao, Yansong Tang, Jie Zhou, Ser-Nam Lim, Jiwen Lu

arXiv:2207.14284v3cs.CV

TL;DR

Vision Transformers are effective partly because self-attention provides input-adaptive, long-range, and high-order spatial interactions, but its quadratic complexity limits efficient use at high resolutions. This paper introduces recursive gated convolutions and builds HorNet to reproduce these interactions with convolutions; HorNet outperforms Swin Transformers and ConvNeXt across classification, detection, and segmentation, while also improving dense prediction decoders.

  • Problem

    Vision Transformers provide powerful spatial interactions, but self-attention has quadratic complexity and higher-order spatial interactions have not been efficiently implemented in convolution-based models.

  • Method

    The paper proposes Recursive Gated Convolution (gnConv), which uses gated convolutions and recursive designs to implement input-adaptive, long-range, and arbitrary-order spatial interactions, then constructs HorNet from gnConv.

  • Results

    HorNet outperforms Swin Transformers and ConvNeXt by a large margin across ImageNet classification, COCO detection, and ADE20K segmentation, with ImageNet-22K results of 87.7% top-1 accuracy, 57.9% mIoU, and 59.2% bounding box AP.

  • Takeaways & Limitations

    gnConv serves as a drop-in spatial-mixing module for vision Transformers and convolutional models, and its decoder application consistently improves dense prediction with lower computational costs.

  • Takeaways & Limitations

    HorNet is slower than ConvNeXt with similar FLOPs on GPU, likely because its high-order interaction design is more complex.

Abstract

from arXiv · show

Recent progress in vision Transformers exhibits great success in various tasks driven by the new spatial modeling mechanism based on dot-product self-attention. In this paper, we show that the key ingredients behind the vision Transformers, namely input-adaptive, long-range and high-order spatial interactions, can also be efficiently implemented with a convolution-based framework. We present the Recursive Gated Convolution ($\textit{g}^\textit{n}$Conv) that performs high-order spatial interactions with gated convolutions and recursive designs. The new operation is highly flexible and customizable, which is compatible with various variants of convolution and extends the two-order interactions in self-attention to arbitrary orders without introducing significant extra computation. $\textit{g}^\textit{n}$Conv can serve as a plug-and-play module to improve various vision Transformers and convolution-based models. Based on the operation, we construct a new family of generic vision backbones named HorNet. Extensive experiments on ImageNet classification, COCO object detection and ADE20K semantic segmentation show HorNet outperform Swin Transformers and ConvNeXt by a significant margin with similar overall architecture and training configurations. HorNet also shows favorable scalability to more training data and larger model sizes. Apart from the effectiveness in visual encoders, we also show $\textit{g}^\textit{n}$Conv can be applied to task-specific decoders and consistently improve dense prediction performance with less computation. Our results demonstrate that $\textit{g}^\textit{n}$Conv can be a new basic module for visual modeling that effectively combines the merits of both vision Transformers and CNNs. Code is available at https://github.com/raoyongming/HorNet

1 Introduction

The paper argues that vision Transformers derive power from input-adaptive, long-range, and high-order spatial interactions, while efficient convolution-based implementations of all three remain underexplored. It introduces gnConv and HorNet to address this gap and reports stronger performance than Swin Transformers and ConvNeXt across major vision tasks.

  • Vision Transformers achieve leading performance across image classification, object detection, semantic segmentation, and video understanding.
  • Explicit input-adaptive, long-range, and high-order spatial interactions are identified as key ingredients behind vision Transformers.Prior work had migrated Transformer meta-architecture, input-adaptive weights, and large-range modeling to CNNs, but not higher-order interactions.
  • gnConv implements arbitrary-order spatial interactions using gated convolutions and recursive designs without self-attention’s quadratic complexity.Progressively increasing channel width enables higher-order interactions with bounded complexity.
  • HorNet uses gnConv as a generic convolution-based backbone and outperforms Swin Transformers and ConvNeXt by a large margin across tasks and complexity levels.The comparisons use similar overall architectures, 7×7 kernels or windows, and training configurations.
  • With ImageNet-22K pre-training, HorNet reaches 87.7% top-1 accuracy on ImageNet, 57.9% mIoU on ADE20K val, and 59.2% bounding box AP on COCO val.
  • Applying gConv to feature-fusion decoders produces HorFPN, which consistently improves dense prediction models with lower computational costs.

2 Related Work

Related work has adapted Transformer architecture, large-kernel and dynamic convolutions, and hybrid CNN-Transformer designs for visual modeling. HorNet instead seeks a self-attention-free model that combines favorable properties of vision Transformers and CNNs.

  • Vision Transformers: Vision Transformer research adapts the original Transformer architecture for visual recognition by modifying its blocks and patch embedding.
  • Convolution-based models: Convolution-based models use Transformer-style architectures, large or global kernels, dynamic weights, and gated convolutions to improve spatial modeling.Examples include ConvNeXt, RepLKNet, GFNet, VAN, and FocalNet.
  • Hybrid models: Hybrid models combine vision Transformer and CNN blocks sequentially or in parallel, often using lightweight convolutions for local patterns or reduced complexity.
  • Hybrid models: HorNet differs from these hybrid models by targeting a self-attention-free architecture that combines the favorable properties of vision Transformers and CNNs.

3 Method

gnConv replaces conventional spatial mixing with recursive gated convolutions that combine input-adaptive, long-range, and arbitrary-order spatial interactions. HorNet uses gnConv in Transformer-like blocks and can also enhance convolution-based feature-pyramid decoders.

  • gnConv: Recursive Gated Convolutions: gnConv combines standard convolutions, linear projections, and element-wise multiplication to provide input-adaptive spatial mixing.Its gated-convolution formulation explicitly interacts neighboring features through element-wise multiplication.
  • gnConv: Recursive Gated Convolutions: Recursive gating increases the interaction order by one at each step, enabling gnConv to achieve n-order spatial interactions.The operation projects input features into p0 and qk components, then recursively applies gated convolutions before the final output projection.
  • gnConv: Recursive Gated Convolutions: Lower interaction orders use fewer channels in a coarse-to-fine design, while the total FLOPs remain strictly bounded as n increases.The resulting gnConv achieves high-order interactions with computational cost similar to a convolutional layer.
  • Long-term interactions with large kernel convolutions: Long-range interactions are supported through depth-wise convolutions implemented with either the default 7×7 kernel or a global filter.The 7×7 configuration follows the default window or kernel size used by Swin Transformers and ConvNeXt for comparison.
  • Relation to dot-product self-attention: gnConv produces input-adaptive spatial mixing weights from recursively processed features, extending self-attention's spatial mixing to higher orders.The mixing weights are computed from features containing n−1 order interactions, allowing more complex spatial interactions than the self-attention formulation described here.
  • Model Architectures: HorNet inserts gnConv as the spatial-mixing layer in a Transformer-style meta-architecture, with variants based on model size and depth-wise convolution implementation.The same operation is also placed after multi-level feature fusion in FPNs to enhance spatial interactions for downstream tasks.

4 Experiments

Across ImageNet, ADE20K, and COCO, HorNet consistently outperforms comparable Swin and ConvNeXt models, while showing favorable scaling and accuracy-complexity trade-offs. Ablations and decoder experiments support gnConv’s generality for high-order spatial modeling, although GPU latency remains a limitation at similar FLOPs.

  • ImageNet Classification: HorNet surpasses comparable Swin Transformers and ConvNeXt models on ImageNet classification across model sizes, resolutions, and training-data settings.The experiments use comparable architectures, FLOPs, parameters, and similar training configurations.
  • Dense Prediction Tasks: HorNet7×7 and HorNetGF outperform similarly sized Swin and ConvNeXt backbones on ADE20K semantic segmentation.HorNet-L variants also outperform ConvNeXt-XL with approximately 25% fewer FLOPs, while global-filter models achieve larger single-scale mIoU gains.
  • Dense Prediction Tasks: +1.2∼2.0 box AP and +1.0∼1.9 mask AP distinguish HorNetGF from ConvNeXt on COCO object detection and instance segmentation.HorNet models consistently improve both box AP and mask AP over comparable Swin and ConvNeXt counterparts.
  • Dense Prediction Tasks: HorFPN outperforms standard FPN across multiple backbones and both dense prediction tasks with about 30G fewer FLOPs.HorFPNGF is consistently better than HorFPN7×7, indicating a benefit from global interactions during hierarchical feature fusion.
  • Analysis: Progressively increasing gnConv order from stages 1 to 4 improves accuracy beyond shared-order designs, while gnConv also improves isotropic models, depth-wise convolution, and pooling.The shared-order ablation saturates at 82.5 for n = 4; the final design uses g{2,3,4,5}Conv.
  • Analysis: HorNet achieves better model-size, FLOPs, and GPU-latency trade-offs than Swin and ConvNeXt, but is slower than ConvNeXt at similar FLOPs on GPU.The authors attribute the latency gap to the more complex designs used for high-order interactions.

5 Conclusion

The paper presents gnConv as an efficient, extendable operation for translation-equivariant high-order spatial interactions and uses it to build HorNet.

  • gnConv combines gated convolutions and recursive designs to perform efficient, extendable, translation-equivariant high-order spatial interactions.
  • gnConv can replace spatial-mixing layers in vision Transformers and convolution-based models as a drop-in module.
  • HorNet is a family of generic vision backbones constructed from gnConv.

A FLOPs of gnConv

The FLOPs analysis divides gnConv computation into projection layers, depth-wise convolution, and recursive gating, then sums their costs.

  • The FLOPs calculation separates gnConv computation into three parts.These parts are projection layers, depth-wise convolution, and recursive gating.
  • Projection layers: The two projection layers, φin and φout, contribute separately to the total FLOPs.
  • Depth-wise convolution: Depth-wise convolution is evaluated across recursive feature groups qk with channel dimensions Ck.
  • Recursive Gating: Recursive gating accounts for both projection by gk and element-wise multiplication.
  • The total computational cost is obtained by summing the FLOPs from all three components.

B Spatial Interactions in Vision Models.

The analysis characterizes spatial interactions by their order: convolution has zero-order interaction, gated operations have first-order interaction, self-attention has second-order interaction, and gnConv supports higher orders.

  • The interaction analysis studies how an operation directly combines a feature xi with neighboring features xj.
  • Convolution: Standard convolution is classified as a 0-order interaction because its output weights do not depend on xi.
  • SE Block/Gated Convolution: SE blocks and gated convolutions introduce 1-order interactions through dependence on the feature at location i.
  • Self-attention: Self-attention is classified as a 2-order interaction because it first computes query-key attention and then aggregates value features.
  • gnConv: gnConv can achieve n-order interactions with bounded computational cost and supports arbitrary orders up to 1 + log2 C.
  • gnConv: Unlike earlier gated methods limited to 2-order interactions, gnConv extends higher-order interactions under a controllable computational budget.

C.1 Architecture Details.

The architecture details describe two minimal changes to Swin Transformer designs for fair comparison and training stability.

  • HorNet adds one block in stage 2 to keep computation and parameter counts close to previous models.
  • HorNet applies LayerScale to improve training stability, following ConvNeXt practice.

C.2 Experimental Settings for Image Classification.

The ImageNet experiments use standard 1K and 22K pretraining protocols, with stabilization and regularization adjusted for HorNet’s convolutional variants. Fine-tuning transfers 22K- or 224×224-pretrained models to ImageNet-1K at 224×224 or 384×384.

  • ImageNet-1K: ImageNet-1K evaluation uses approximately 1.2 million training images across 1,000 categories and reports single-crop top-1 accuracy on 50,000 validation images.The setup follows common practice and largely matches ConvNeXt training configurations.
  • ImageNet-1K: Gradient clipping with maximal norm 5 stabilizes HorNet training with 7×7 convolutions, while global-filter models use norm 1 and stronger RandAug regularization.RandAug magnitudes are 9, 12, and 15 for tiny, small, and base models, respectively.
  • ImageNet-22K: ImageNet-22K pretraining uses roughly 14 million images and more than 21,000 classes, with 90 epochs, stochastic depth 0.2, and adjusted regularization.The accessible subset removes categories with few images, leaving roughly half as many categories and 13% fewer images than the original.
  • Fine-tuning: ImageNet-1K fine-tuning runs for 30 epochs at 224×224 or 384×384 using batch size 512, cosine decay, initial learning rate 5e−5, and weight decay 1e−6.MixUp and CutMix are disabled, and classifier weights are initialized from corresponding ImageNet-22K classes when applicable.

C.3 Experimental Settings for Downstream Tasks.

Downstream evaluation uses established detection and segmentation frameworks with fixed optimization schedules, batch sizes, and weight decay across HorNet variants. COCO uses Cascade Mask R-CNN, while ADE20K uses UperNet 160K.

  • COCO: COCO detection and instance segmentation use Cascade Mask R-CNN with a 3× schedule of 36 epochs, AdamW, and global batch size 16.HorNet-T/S/B backbones use ImageNet-1K pretraining, while HorNet-L uses ImageNet-22K pretraining.
  • COCO: COCO experiments use learning rates of 2e−4 for HorNet-T/S/B and 1e−4 for HorNet-L, with weight decay 0.05 for all models.
  • ADE20K: ADE20K semantic segmentation uses UperNet 160K, global batch size 16, 160 iterations, AdamW, learning rate 1e−4, and weight decay 0.05.ImageNet-1K-pretrained models use 512×512 images, whereas the ImageNet-22K-pretrained large model uses 640×640 images.

D More Analysis

Additional analyses examine ImageNet state-of-the-art comparisons, throughput, recursive-output scaling, and activation choices in gated convolutions. They also identify remaining opportunities for improving HorNet’s ImageNet-1K performance and speed.

  • Comparisons with state-of-the-art methods on ImageNet: HorNet’s ImageNet-1K comparison follows Swin and ConvNeXt architectures and training configurations, while recent hybrid or more carefully designed models achieve better performance.The authors identify architecture, patch embedding, adaptive-weight computation, training, and hybrid-design improvements as future opportunities.
  • Throughput analysis: 7% 15% slower than ConvNeXt with similar FLOPs, HorNet nevertheless achieves similar or slightly faster GPU speed than typical vision Transformers.Multiple small matrix multiplications from gnConv affect GPU throughput, while efficient depth-wise convolutions help preserve competitiveness.
  • Effects of α: α = 3 gives the best ImageNet performance for HorNet-B7×7 and is used in all models.Rescaling gated-convolution outputs avoids large recursive values and stabilizes training.
  • Effects of activation functions in gated convolutions: Gated convolutions can be viewed as spatially generated channel attention with location-dependent attention weights.The analysis compares activation functions for producing these weights, motivated by bounded attention in methods such as SE-Net.
Loading 2207.14284v3…