PaperlayerView exact revision on arXiv

Source-linked AI summary

A ConvNet for the 2020s

Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie

arXiv:2201.03545v2cs.CV

TL;DR

ConvNets’ standing relative to hierarchical Transformers as generic vision backbones remains unclear. This paper modernizes ResNets into pure ConvNeXts, which compete favorably with Transformers across classification, detection, and segmentation while retaining standard ConvNet simplicity and efficiency.

  • Problem

    It remains unclear whether hierarchical Transformers’ performance advantage reflects Transformer superiority or convolutional inductive biases, limiting evidence for pure ConvNets as generic vision backbones.

  • Method

    The authors gradually modernize a standard ResNet toward hierarchical Transformer design, studying macro, block, kernel, and layer-wise decisions to construct pure ConvNeXt models.

  • Results

    ConvNeXts compete favorably with similarly sized Swin Transformers across ImageNet, COCO, and ADE20K, reaching 87.8% ImageNet top-1 accuracy.

  • Takeaways & Limitations

    Pure ConvNets can match or surpass hierarchical Transformers across multiple vision benchmarks while retaining the simplicity and efficiency of standard ConvNets.

  • Takeaways & Limitations

    The initial exploration was limited to a small scale, leaving ConvNet scaling and downstream-task competitiveness to be evaluated in subsequent experiments.

Abstract

from arXiv · show

The "Roaring 20s" of visual recognition began with the introduction of Vision Transformers (ViTs), which quickly superseded ConvNets as the state-of-the-art image classification model. A vanilla ViT, on the other hand, faces difficulties when applied to general computer vision tasks such as object detection and semantic segmentation. It is the hierarchical Transformers (e.g., Swin Transformers) that reintroduced several ConvNet priors, making Transformers practically viable as a generic vision backbone and demonstrating remarkable performance on a wide variety of vision tasks. However, the effectiveness of such hybrid approaches is still largely credited to the intrinsic superiority of Transformers, rather than the inherent inductive biases of convolutions. In this work, we reexamine the design spaces and test the limits of what a pure ConvNet can achieve. We gradually "modernize" a standard ResNet toward the design of a vision Transformer, and discover several key components that contribute to the performance difference along the way. The outcome of this exploration is a family of pure ConvNet models dubbed ConvNeXt. Constructed entirely from standard ConvNet modules, ConvNeXts compete favorably with Transformers in terms of accuracy and scalability, achieving 87.8% ImageNet top-1 accuracy and outperforming Swin Transformers on COCO detection and ADE20K segmentation, while maintaining the simplicity and efficiency of standard ConvNets.

1. Introduction

ConvNets dominated visual recognition because their inductive biases suited broad vision tasks, but Vision Transformers and hierarchical variants reshaped the field. The authors therefore modernize ResNets toward hierarchical Transformers and propose pure ConvNeXt models to reassess convolution’s contribution.

  • ConvNet dominance: ConvNets became foundational in visual recognition because sliding-window processing and translation equivariance suited high-resolution computer vision tasks.Region-based detectors further elevated ConvNets as fundamental building blocks in visual recognition systems.
  • Transformer transition: Vision Transformers introduced a step change in network architecture design in 2020, challenging ConvNets’ dominance in vision.The passage contrasts this shift with Transformers replacing recurrent networks as the dominant backbone architecture in natural language processing.
  • Hierarchical Transformers: Hierarchical Transformers bridged the gap to general vision by reintroducing local-window processing and achieving state-of-the-art performance beyond image classification.Swin Transformer is identified as a milestone example of this approach.
  • Motivation: The paper questions whether Transformers’ advantage reflects intrinsic superiority, since hierarchical designs share ConvNet inductive biases while differing in training and architecture.Recent comparisons often evaluate complete systems such as Swin Transformer versus ResNet, while performance differences are commonly attributed to Transformer scaling and multi-head self-attention.
  • Contributions: The authors gradually modernize a trained ResNet toward a hierarchical Transformer design, identify components affecting ConvNet performance, and propose the pure ConvNet family ConvNeXt.The exploration is guided by how Transformer design decisions impact ConvNet performance.

2. Modernizing a ConvNet: a Roadmap

The authors modernize ResNet toward a hierarchical Transformer-like design while preserving a simple, attention-free ConvNet architecture. They progressively evaluate training, macro-architecture, bottlenecks, kernels, and layer-wise choices, culminating in ConvNeXt and motivating larger-scale downstream evaluation.

  • Roadmap: The exploration starts from ResNet-50/200 models trained with vision Transformer techniques, using ResNet-50/Swin-T complexity models around 4.5×10^9 FLOPs as the primary setting.The higher-capacity ResNet-200/Swin-B regime has around 15.0 × 10^9 FLOPs, with consistent conclusions reported in the appendix.
  • Architecture modernization: The roadmap studies macro design, ResNeXt, inverted bottlenecks, large kernels, and layer-wise micro-designs while avoiding attention-based modules.The macro design adopts a multi-stage hierarchy, a 1:1:3:1 stage compute ratio, and a 4×4 non-overlapping patchify stem.
  • Inverted bottleneck: 80.6% performance follows the inverted-bottleneck change from 80.5%, while whole-network FLOPs decrease to 4.6G through shortcut 1×1 convolution savings.In the ResNet-200/Swin-B regime, performance increases from 81.9% to 82.6% with reduced FLOPs.
  • Large kernels: 80.6% performance is reached with a 7×7 depthwise kernel versus 79.9% with a 3×3 kernel at roughly unchanged FLOPs, with gains saturating beyond 7×7.The larger-capacity ResNet-200 regime also shows no further gain beyond 7×7.
  • ConvNeXt: The final architecture uses separate downsampling layers and is named ConvNeXt, a pure ConvNet that can outperform Swin Transformer on ImageNet-1K in this compute regime.The remaining question is whether ConvNeXt scales and competes with Swin Transformers on object detection and semantic segmentation.

3. Empirical Evaluations on ImageNet

ConvNeXt variants are evaluated on ImageNet-1K, including models pre-trained on ImageNet-22K, and show strong accuracy, scaling, throughput, and resolution-flexibility results against Transformer and ConvNet baselines. ConvNeXt-B at 3842 surpasses Swin-B while preserving higher inference throughput, and ConvNeXt remains competitive after large-scale pre-training.

  • Model variants: ConvNeXt-T/S/B/L match Swin-T/S/B/L in complexity, while ConvNeXt-XL extends the family to test scalability.Variants differ in stage channels and block counts, with channels doubling at each new stage.
  • Evaluation setup: ConvNeXts are simpler to fine-tune at different resolutions because they are fully convolutional and require neither input patch-size adjustment nor position-bias interpolation.The ImageNet evaluation reports top-1 accuracy on ImageNet-1K and also fine-tunes ImageNet-22K-pre-trained models on ImageNet-1K.
  • ImageNet-1K: ConvNeXt competes favorably with RegNet and EfficientNet in accuracy-computation trade-offs and inference throughput, while outperforming similarly complex Swin Transformers.The comparison includes DeiT, Swin Transformers, RegNets, EfficientNets, and EfficientNetsV2.
  • ImageNet-1K: 85.1% vs. 84.5%: ConvNeXt-B at 3842 outperforms Swin-B by 0.6%, with 12.5% higher inference throughput, 95.7 vs. 85.1 image/s.The FLOPs/throughput advantage becomes larger as resolution increases from 2242 to 3842.
  • ImageNet-1K: 85.5%: ConvNeXt-L improves further when scaled beyond ConvNeXt-B.This result is reported after the 3842-resolution ConvNeXt-B comparison.
  • ImageNet-22K: ConvNeXts remain on par or better than similarly sized Swin Transformers after ImageNet-22K pre-training, showing that properly designed ConvNets are not inferior at larger training scale.With ImageNet-22K pre-training, ConvNeXt also outperforms EfficientNetV2-L, whose ImageNet-1K result uses advanced modules and progressive training.

4. Empirical Evaluation on Downstream Tasks

ConvNeXt backbones match or outperform Swin Transformers on COCO detection and instance segmentation, while achieving competitive ADE20K segmentation performance. Despite using depthwise convolutions, ConvNeXts retain comparable or higher inference throughput and require less training memory than Swin Transformers.

  • Experimental settings: COCO experiments fine-tune Mask R-CNN and Cascade Mask R-CNN with ConvNeXt backbones using multi-scale training, AdamW, and a 3× schedule.ADE20K experiments train UperNet models for 160K iterations with batch size 16.
  • Object detection and segmentation on COCO: ConvNeXt achieves on-par or better COCO box and mask AP than Swin Transformer across model complexities.For larger ImageNet-22K-pretrained ConvNeXt-B/L/XL models, it is significantly better in many cases.
  • Object detection and segmentation on COCO: +1.0 AP: Larger ImageNet-22K-pretrained ConvNeXt models exceed Swin Transformers in many COCO box and mask AP comparisons.The comparison includes ConvNeXt-B, ConvNeXt-L, and ConvNeXt-XL.
  • Semantic segmentation on ADE20K: ConvNeXt models achieve competitive validation mIoU across different model capacities on ADE20K semantic segmentation.The evaluation uses UperNet with multi-scale testing.
  • Remarks on model efficiency: ConvNeXt inference throughputs are comparable to or exceed those of Swin Transformers for classification and higher-resolution tasks.Training ConvNeXts also requires less memory than training Swin Transformers.

5. Related Work

Related work explores hybrid convolution-attention models and recent convolution-based alternatives to Transformer token mixing. These approaches use convolutions or FFTs to capture dependencies or mix tokens, with some achieving performance comparable to Swin.

  • Hybrid models: Hybrid models augmented ConvNets with self-attention or non-local modules for long-range dependencies, while later works reintroduced convolutional priors into ViTs explicitly or implicitly.The original ViT also studied a hybrid configuration.
  • Recent convolution-based approaches: Han et al. showed local Transformer attention is equivalent to inhomogeneous dynamic depthwise convolution, and replacing Swin’s MSA with dynamic or regular depthwise convolution achieved comparable performance.This connects local attention and convolutional formulations while retaining comparable results to Swin.
  • Recent convolution-based approaches: ConvMixer demonstrated depthwise convolution as a promising mixing strategy in small-scale settings, but its smaller patch size produced much lower throughput than other baselines.The throughput limitation arose from using a smaller patch size to achieve the best results.
  • Recent convolution-based approaches: GFNet used the Fast Fourier Transform for token mixing, which is also a form of convolution with a global kernel.FFT provides a global-kernel alternative for token mixing.

6. Conclusions · Appendix · A. Experimental Settings

The conclusion presents ConvNeXt as a pure ConvNet that competes with hierarchical vision Transformers across computer vision benchmarks while retaining ConvNet simplicity and efficiency. The appendix documents additional evaluations, experimental details, specifications, throughput benchmarks, limitations, and societal impact.

  • 6. Conclusions: ConvNeXt is presented as a pure ConvNet competing favorably with state-of-the-art hierarchical vision Transformers across multiple computer vision benchmarks.The model retains the simplicity and efficiency of standard ConvNets.
  • 6. Conclusions: The work challenges the prevailing belief that vision Transformers are more accurate, efficient, and scalable than ConvNets.The conclusion frames ConvNeXt as evidence that pure ConvNets can remain competitive with hierarchical Transformers.
  • 6. Conclusions: The authors acknowledge Kaiming He, Eric Mintun, Xingyi Zhou, Ross Girshick, and Yann LeCun for valuable discussions and feedback.
  • Appendix: The appendix provides further experimental details in Section A and robustness evaluation results in Section B.
  • Appendix: The appendix reports more modernization experiment results in Section C and a detailed network specification in Section D.
  • Appendix: The appendix benchmarks model throughput on A100 GPUs and discusses the work’s limitations and societal impact.
  • A. Experimental Settings: Section A is identified as the location of the paper’s further experimental details.

A.1. ImageNet (Pre-)training · A.2. ImageNet Fine-tuning · A.3. Downstream Tasks

The appendix specifies shared ConvNeXt training protocols, distinct fine-tuning choices, and task-specific downstream evaluation procedures. It also documents the hyperparameter sweeps and testing conventions used for COCO and ADE20K.

  • A.1. ImageNet (Pre-)training: All ConvNeXt variants use the same ImageNet-1K/22K training settings, except for customized stochastic depth rates.These settings support the main results, with model-specific stochastic depth rates.
  • A.1. ImageNet (Pre-)training: For modernization experiments, ImageNet-1K training follows Table 5 except EMA is disabled because it severely hurts BatchNorm models.This modification applies specifically to the experiments in Section 2.
  • A.1. ImageNet (Pre-)training: Isotropic ConvNeXts extend ImageNet-1K warmup to 50 epochs and disable layer scale for isotropic ConvNeXt-S/B.Their stochastic depth rates are 0.1/0.2/0.5 for isotropic ConvNeXt-S/B/L.
  • A.2. ImageNet Fine-tuning: ImageNet-1K fine-tuning starts from final pre-trained model weights without EMA weights, and Table 6 provides the fine-tuning settings.The authors report no improvement from fine-tuning with EMA weights.
  • A.2. ImageNet Fine-tuning: Fine-tuning uses layer-wise learning-rate decay with every 3 consecutive blocks forming a group.At 3842 resolution, testing uses a 1.0 crop ratio instead of 0.875 at 2242.
  • A.3. Downstream Tasks: ADE20K and COCO follow BEiT and Swin training settings, use the MMDetection and MMSegmentation toolboxes, and initialize from final ImageNet pre-training weights.Final weights are used instead of EMA weights.
  • A.3. Downstream Tasks: COCO experiments sweep learning rate {1e-4, 2e-4}, layer-wise decay {0.7, 0.8, 0.9, 0.95}, and stochastic depth {0.3, 0.4, 0.5, 0.6, 0.7, 0.8}, also applying the sweep to ImageNet-22K pre-trained Swin-B/L.The Swin-B/L models use official code and pre-trained weights.
  • A.3. Downstream Tasks: ADE20K sweeps learning rate {8e-5, 1e-4}, layer-wise decay {0.8, 0.9}, and stochastic depth {0.3, 0.4, 0.5}, reporting validation mIoU with multi-scale testing.Additional single-scale testing results appear in Table 7.

B. Robustness Evaluation

ConvNeXt models are evaluated on several ImageNet robustness and domain-generalization benchmarks using standard classification models. Large-scale ConvNeXts show strong robustness, with ConvNeXt-XL achieving high accuracy across ImageNet-A, ImageNet-R, and ImageNet-Sketch without specialized modules or additional fine-tuning.

  • Benchmark setup: ConvNeXt classification models are tested on ImageNet-A, ImageNet-R, ImageNet-Sketch, ImageNet-C, and ImageNet-¯C robustness benchmarks.The evaluation reports mCE for ImageNet-C, corruption error for ImageNet-¯C, and top-1 Accuracy for the other datasets.
  • Results: 69.3%/68.2%/55.0% accuracy is achieved by ConvNeXt-XL on ImageNet-A/R/Sketch, respectively, with extra ImageNet-22K data.These results indicate strong domain generalization for the large-scale model variant.
  • Results: ConvNeXt, particularly its large-scale variants, outperforms state-of-the-art robust transformer models on several robustness benchmarks.The reported robustness results use no specialized modules or additional fine-tuning procedures.

C. Modernizing ResNets: detailed results

The modernization experiments report ImageNet-1K top-1 accuracy and FLOPs for ResNet-50/Swin-T and ResNet-200/Swin-B regimes. Results are broadly consistent across regimes, with larger gains from dimension inversion and fewer normalization layers in ResNet-200.

  • Experimental setup: ImageNet-1K top-1 accuracies and FLOPs are tabulated for modernization steps in both ResNet-50/Swin-T and ResNet-200/Swin-B regimes.ResNet-50 experiments use three random seeds, with mean and standard deviation reported in Table 10.
  • ResNet-200 results: +0.79% vs. +0.14%: inverting dimensions improves ResNet-200 more than ResNet-50.The observations in the ResNet-200 regime are otherwise mostly consistent with those in the ResNet-50 regime.
  • ResNet-200 results: +0.46% vs. +0.14%: using fewer normalization layers provides a larger gain in ResNet-200 than in ResNet-50.This difference is reported alongside the broader observation that modernization behavior is mostly consistent across the two regimes.
  • ResNet-200 results: Kernel-size gains appear to saturate at kernel size 5 rather than 7 in the ResNet-200 regime.This is identified as an interesting difference in the detailed ResNet-200 observations.

D. Detailed Architectures

This section compares the detailed architectures of ResNet-50, ConvNeXt-T, and Swin-T, highlighting ConvNeXt’s standard-ConvNet simplicity and favorable competition with Swin Transformers. Across ConvNeXt sizes, only stage block counts and channel counts differ from ConvNeXt-T.

  • D. Detailed Architectures: The architecture comparison covers ResNet-50, ConvNeXt-T, and Swin-T.The comparison is presented in Table 9.
  • D. Detailed Architectures: For differently sized ConvNeXts, only the number of blocks and channels at each stage differ from ConvNeXt-T.This defines the architecture changes across ConvNeXt model sizes.
  • D. Detailed Architectures: ConvNeXts retain the simplicity of standard ConvNets while competing favorably with Swin Transformers in visual recognition.The passage characterizes both their construction and recognition performance relative to Swin Transformers.

E. Benchmarking on A100 GPUs

On A100 GPUs using TF32 and channel-last memory layout, ConvNeXt achieves substantially higher inference throughput than Swin Transformer, reaching up to approximately 49% higher throughput at similar FLOPs.

  • E. Benchmarking on A100 GPUs: The benchmark uses PyTorch 1.10 with TF32 tensor cores and the latest “Channel Last” memory layout for further speedup.A100 GPUs support TF32 tensor cores, and the channel-last layout is used to improve speed.
  • E. Benchmarking on A100 GPUs: ConvNeXt’s A100 inference-throughput advantage is significantly greater than on V100 GPUs, sometimes reaching 49% faster.Both model families achieve faster inference throughput on A100 than on V100.
  • E. Benchmarking on A100 GPUs: Up to ∼49% higher throughput: ConvNeXt outperforms Swin Transformer on A100 GPUs with similar FLOPs.The comparison uses TF32 data format and a “channel last” memory layout.

F. Limitations · G. Societal Impact

ConvNeXt matches hierarchical vision Transformers across several evaluated vision tasks, but architecture choice should depend on task requirements and simplicity. The paper also highlights the computing demands and carbon emissions associated with increasingly large vision models.

  • F. Limitations: ConvNeXt performs as well as hierarchical vision Transformers on image classification, object detection, and instance and semantic segmentation.The evaluation spans multiple computer vision tasks, while the authors acknowledge that applications are even more diverse.
  • F. Limitations: Computer vision applications extend beyond the evaluated tasks, so ConvNeXt may be better suited to some tasks than Transformers.The authors explicitly limit their conclusion to the evaluated range of tasks.
  • F. Limitations: In multimodal learning, cross-attention may be preferable for modeling feature interactions across many modalities.This is presented as a case where Transformers may be more suitable than ConvNeXt.
  • F. Limitations: Transformers may be more flexible for tasks requiring discretized, sparse, or structured outputs.The paper recommends choosing the architecture according to the needs of the task while striving for simplicity.
  • F. Limitations: The authors argue that architecture choice should meet task needs while preserving simplicity.This principle follows their comparison of ConvNeXt and Transformers across different task requirements.
  • G. Societal Impact: ViT, Swin, and ConvNeXt perform best with huge model variants, increasing the computing demands of visual representation learning.The paper notes that larger models and datasets improve performance but introduce additional challenges.
  • G. Societal Impact: Investigating increasingly large model designs inevitably results in increased carbon emissions.The authors identify simplicity as an important direction and motivation for the work.
Loading 2201.03545v2…