PaperlayerView exact revision on arXiv

Source-linked AI summary

InceptionNeXt: When Inception Meets ConvNeXt

Weihao Yu, Pan Zhou, Shuicheng Yan, Xinchao Wang

arXiv:2303.16900v3cs.CVcs.AIcs.LG

TL;DR

Large-kernel depthwise convolutions improve receptive fields but incur memory-access costs, while smaller kernels can degrade accuracy. InceptionNeXt addresses this trade-off by decomposing the operation into efficient parallel branches and reports a better speed-accuracy trade-off than ConvNeXt.

  • Problem

    The paper addresses how to speed up large-kernel CNNs while preserving performance, given depthwise convolution’s memory-access costs and smaller kernels’ accuracy degradation.

  • Method

    InceptionNeXt decomposes large-kernel depthwise convolution along the channel dimension into identity mapping, a small square kernel, and two orthogonal band-kernel branches.

  • Results

    Experiments on image classification and semantic segmentation show that InceptionNeXt achieves a better speed-accuracy trade-off than ConvNeXt.

  • Takeaways & Limitations

    InceptionNeXt is proposed as a new CNN baseline for speeding up neural architecture design research.

  • Takeaways & Limitations

    The simpler MetaNeXt-style design limits the token mixer from being complicated.

Abstract

from arXiv · show

Inspired by the long-range modeling ability of ViTs, large-kernel convolutions are widely studied and adopted recently to enlarge the receptive field and improve model performance, like the remarkable work ConvNeXt which employs 7x7 depthwise convolution. Although such depthwise operator only consumes a few FLOPs, it largely harms the model efficiency on powerful computing devices due to the high memory access costs. For example, ConvNeXt-T has similar FLOPs with ResNet-50 but only achieves ~60% throughputs when trained on A100 GPUs with full precision. Although reducing the kernel size of ConvNeXt can improve speed, it results in significant performance degradation, which poses a challenging problem: How to speed up large-kernel-based CNN models while preserving their performance. To tackle this issue, inspired by Inceptions, we propose to decompose large-kernel depthwise convolution into four parallel branches along channel dimension, i.e., small square kernel, two orthogonal band kernels, and an identity mapping. With this new Inception depthwise convolution, we build a series of networks, namely IncepitonNeXt, which not only enjoy high throughputs but also maintain competitive performance. For instance, InceptionNeXt-T achieves 1.6x higher training throughputs than ConvNeX-T, as well as attains 0.2% top-1 accuracy improvement on ImageNet-1K. We anticipate InceptionNeXt can serve as an economical baseline for future architecture design to reduce carbon footprint. Code is available at https://github.com/sail-sg/inceptionnext.

1. Introduction

Modern CNNs use large-kernel depthwise convolutions to enlarge receptive fields and improve performance, but their memory access costs reduce practical speed. InceptionNeXt addresses this trade-off by replacing the expensive operator with efficient parallel branches while preserving accuracy.

  • Large-kernel depthwise convolution provides the large receptive fields shared by modern CNN models such as ConvNeXt.
  • Reducing ConvNeXt-T’s kernel from 7 × 7 to 3 × 3 causes a 0.6% top-1 accuracy drop on ImageNet-1K.
  • Inception depthwise convolution decomposes depthwise convolution into three small-kernel branches and an identity branch, leaving some channels unaltered.The convolutional branches use 3×3, 1×k, and k×1 kernels over separate channel groups.
  • InceptionNeXt achieves a better speed–accuracy trade-off than ConvNeXt, with InceptionNeXt-T reaching 1.6× higher training throughput and higher accuracy.The reported throughput is similar to ResNet-50 while accuracy exceeds ConvNeXt-T’s.

2. Related work

Related work improves vision models through both Transformer-based designs and efficient large-kernel convolutional constructions. These studies motivate decomposing large kernels while retaining broad spatial context.

  • Vision Transformers such as DeiT and Swin achieve strong performance across vision tasks, motivating extensive Transformer adoption.
  • Prior efficiency strategies include stacking 3×3 convolutions, sequentially factorizing k × k kernels, and splitting depthwise kernels into groups.Examples include VGG, Inception v3, and MixConv.

3. Formulation and Method

InceptionNeXt reformulates ConvNeXt’s spatial token mixer with an efficient Inception depthwise convolution, targeting the speed bottleneck of large-kernel depthwise operators while retaining receptive-field benefits. The resulting models use parallel channel-wise branches and a four-stage architecture with improved convolutional complexity.

  • 3.1. MetaNeXt: ConvNeXt abstracts its depthwise convolution as a token mixer, followed by normalization, an MLP, and a shortcut connection.The input is processed spatially before normalization and two fully connected layers or equivalent 1 × 1 convolutions.
  • 3.1. MetaNeXt: MetaNeXt simplifies MetaFormer by using one shortcut connection instead of two, increasing speed but restricting the token mixer from using complex operators such as attention.The limitation is reported in experiments comparing the two block designs.
  • 3.2. Inception depthwise convolution: 1.4× higher training throughput accompanies reducing ConvNeXt-T’s depthwise kernel from 7 × 7 to 3 × 3, but top-1 accuracy falls from 82.1% to 81.5%.Reducing the kernel improves speed while shrinking the receptive field and degrading ImageNet-1K performance.
  • 3.2. Inception depthwise convolution: Inception depthwise convolution has parameters and FLOPs linear in kernel size, unlike vanilla and depthwise convolution, whose complexity is quadratic in k.The corresponding FLOP comparison shows increasing efficiency gains as kernel size grows.
  • 3.2. Inception depthwise convolution: Inception depthwise convolution splits channels across a 3 × 3 square branch, horizontal and vertical band-kernel branches, and an identity branch.The large kernel kh × kw is decomposed into 1 × kw and kh × 1 band kernels, and branch outputs are concatenated.
  • 3.2. Inception depthwise convolution: The branch channel count is controlled by g = r_gC, after which each split is processed in parallel or left unchanged through identity mapping.The outputs are concatenated to form the convolution result.
  • 3.3. InceptionNeXt: InceptionNeXt models are built from four stages with size-specific depths, using Batch Normalization and an MLP ratio of 3 in stage 4.The stage depths are [2] for Atto, [9] for Small, and [27] for Base; configurations are compared with ConvNeXt.
  • 3.3. InceptionNeXt: The method is designed to improve the practical speed–performance trade-off over previous architectures while supporting competitive image-classification and semantic-segmentation results.The paper presents InceptionNeXt as a CNN baseline built around the proposed convolution.

4. Experiment

Experiments evaluate InceptionNeXt on image classification and semantic segmentation, including speed, accuracy, and ablations against established baselines. Results show improved speed-accuracy trade-offs, with design choices affecting receptive field, throughput, and performance.

  • Image classification: ImageNet-1K experiments follow DeiT training practices, using AdamW and standard augmentation and regularization techniques.The setup omits distillation and uses LayerScale alongside ConvNeXt-style training choices.
  • Image classification: InceptionNeXt-T delivers 1.6×/1.2× training/inference throughputs on A100 than ConvNeXt-T while surpassing it by 0.2%.Its speed is described as similar to ResNet-50 while retaining ConvNeXt-T-level accuracy.
  • Image classification: Speed gains are larger for lightweight models and diminish as model size increases because MLP computation increasingly dominates depthwise-convolution computation.Depthwise and Inception depthwise convolutions scale as O(C), whereas MLPs scale as O(C2).
  • Semantic segmentation: On ADE20K semantic segmentation, InceptionNeXt consistently outperforms Swin and ConvNeXt across model sizes and surpasses PVT and PoolFormer with Semantic FPN.Evaluations use UperNet and Semantic FPN frameworks with reported MACs and FPS under the table-specific benchmarking setups.
  • Ablation studies: Removing either orthogonal band-kernel branch lowers accuracy from 82.3% to 81.9%, while removing the 3×3 branch reaches 82.0% with higher throughput.The band kernels enlarge receptive field; the simplified variant is suggested when speed is prioritized.
  • Ablation studies: Kernel size 11 improves performance over size 7, but size 13 reduces it; the default band-kernel size is therefore 11 except for atto models.The authors associate the decline at size 13 with optimization and mention structural re-parameterization as a possible remedy.
  • Ablation studies: Reducing the convolution branch ratio to 1/16 causes a serious performance drop, while increasing it from 1/8 to 1/4 yields no observed improvement.The default ratio is 1/8 except for atto models, because smaller ratios limit token mixing.
  • Ablation studies: Replacing Batch Normalization with Layer Normalization improves performance by 0.1% but reduces training and inference throughput.Batch Normalization is retained because the paper emphasizes efficiency.

5. Conclusion

InceptionNeXt is presented as an efficient CNN architecture that improves the practical speed-performance trade-off through a four-branch decomposition of large-kernel depthwise convolution.

  • Conclusion: InceptionNeXt decomposes large-kernel depthwise convolution along channels into identity, small-square-kernel, and two orthogonal-band-kernel branches.The branches are intended to be computationally efficient while jointly providing a large spatial receptive field.
  • Conclusion: Extensive experiments demonstrate InceptionNeXt’s competitive performance and practical efficiency.

A.1. ImageNet-1K image classification

The ImageNet-1K classification appendix describes training and fine-tuning configurations for InceptionNeXt using ConvNeXt-aligned hyperparameters.

  • A.1. ImageNet-1K image classification: InceptionNeXt is trained at input resolution 2242 and fine-tuned at 3842 using hyperparameters shown in Table 9.The implementation uses PyTorch based on the timm library.

A.2. Semantic segmentation

The semantic-segmentation appendix describes ConvNeXt-backed InceptionNeXt evaluation on ADE20K using UperNet and FPN configurations.

  • A.2. Semantic segmentation: ADE20K models use ImageNet-1K-pretrained backbones and are evaluated with UperNet and FPN under configurations following Swin, PVT, and PoolFormer.The implementation uses PyTorch and mmsegmentation.

B. Qualitative results

Grad-CAM visualizations compare activation maps for several ImageNet-1K models. InceptionNeXt-T locates key parts more accurately with smaller activation areas than the other compared models.

  • B. Qualitative results: Grad-CAM visualizes activation maps for RSB-ResNet-50, Swin-T, ConvNeXt-T, and InceptionNeXt-T trained on ImageNet-1K.The visualized images come from the ImageNet-1K validation set.
  • B. Qualitative results: InceptionNeXt-T locates key parts more accurately with smaller activation areas than the other compared models.
  • B. Qualitative results: The paper separately lists ImageNet-1K classification hyper-parameters and stochastic depth rates for ADE20K segmentation backbones with UperNet and FPN.
Loading 2303.16900v3…