PaperlayerView exact revision on arXiv

Source-linked AI summary

ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders

Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, Saining Xie

arXiv:2301.00808v1cs.CV

TL;DR

ConvNets were not well served by simply applying transformer-oriented masked autoencoders, motivating a co-designed approach. The paper introduces FCMAE and GRN within ConvNeXt V2, which improves pure ConvNets across major recognition tasks and model sizes. The resulting family reaches 88.9% top-1 ImageNet accuracy for its Huge model using public training data.

  • Problem

    Transformer-oriented masked autoencoders and standard ConvNet architectures are difficult to combine effectively, limiting masked self-supervised pre-training for ConvNets.

  • Method

    The paper co-designs ConvNeXt V2 with a fully convolutional masked autoencoder using sparse convolutions, a lightweight decoder, and Global Response Normalization.

  • Results

    ConvNeXt V2 significantly improves pure ConvNets across ImageNet classification, COCO detection, and ADE20K segmentation; its 650M-parameter Huge model reaches 88.9% top-1 ImageNet accuracy.

  • Takeaways & Limitations

    Co-designing architecture and masked self-supervised pre-training yields a ConvNet family spanning efficient and large compute regimes with improved downstream performance.

Abstract

from arXiv · show

Driven by improved architectures and better representation learning frameworks, the field of visual recognition has enjoyed rapid modernization and performance boost in the early 2020s. For example, modern ConvNets, represented by ConvNeXt, have demonstrated strong performance in various scenarios. While these models were originally designed for supervised learning with ImageNet labels, they can also potentially benefit from self-supervised learning techniques such as masked autoencoders (MAE). However, we found that simply combining these two approaches leads to subpar performance. In this paper, we propose a fully convolutional masked autoencoder framework and a new Global Response Normalization (GRN) layer that can be added to the ConvNeXt architecture to enhance inter-channel feature competition. This co-design of self-supervised learning techniques and architectural improvement results in a new model family called ConvNeXt V2, which significantly improves the performance of pure ConvNets on various recognition benchmarks, including ImageNet classification, COCO detection, and ADE20K segmentation. We also provide pre-trained ConvNeXt V2 models of various sizes, ranging from an efficient 3.7M-parameter Atto model with 76.7% top-1 accuracy on ImageNet, to a 650M Huge model that achieves a state-of-the-art 88.9% accuracy using only public training data.

1. Introduction

ConvNeXt models and masked autoencoders are individually effective, but simply combining a supervised-learning architecture with MAE can be suboptimal for ConvNets. ConvNeXt V2 co-designs the architecture and pre-training framework, adding FCMAE and GRN to improve ConvNet representation learning across scales and tasks.

  • Motivation: Masked autoencoders have become popular for visual representation learning, but standard MAE encoder-decoder designs are not directly compatible with dense-sliding-window ConvNets.The transformer-oriented design processes only visible patches efficiently, whereas standard ConvNets preserve dense 2D structure.
  • Contribution: ConvNeXt V2 co-designs the network architecture and masked autoencoder to make mask-based self-supervised learning effective for ConvNeXt models.The goal is to approach results obtained with transformer-based models while retaining a convolutional architecture.
  • Contribution: FCMAE uses sparse convolutions for visible regions and a lightweight ConvNeXt decoder, while converting weights back to dense layers for fine-tuning.This fully convolutional design reduces the mismatch between masked pre-training and ConvNet processing.
  • Contribution: Feature-space analysis identifies feature collapse at ConvNeXt MLP layers when training directly on masked input, motivating Global Response Normalization.The issue involves dead or saturated features and redundant activations across channels.
  • Results: ConvNeXt V2 improves pure ConvNets across ImageNet classification, COCO object detection, and ADE20K segmentation, spanning models from 3.7M to 650M parameters.The Atto model reaches 76.7% top-1 ImageNet accuracy, while the Huge model reaches 88.9% using IN-22K labels.

2. Related Work

Related work established ConvNets as influential visual representation architectures and introduced masked autoencoders as a major self-supervised pre-training strategy. Prior attempts to adapt masked image modeling to ConvNets produced mixed results, leaving architecture-framework compatibility unresolved.

  • ConvNets: ConvNets enabled generic feature learning for visual recognition instead of relying on manual feature engineering.Their development was driven largely by supervised training on ImageNet.
  • Masked Autoencoders: Masked autoencoders transferred masked language modeling ideas to vision and became a prominent self-supervised pre-training framework.The related-work discussion presents MAE as broadly influential in visual recognition.
  • Masked Autoencoders: Original MAE designs are not directly applicable to ConvNets, while alternative convolutional adaptations have achieved mixed results.The central compatibility issue is the asymmetric encoder-decoder design used by transformer-based MAE.

3. Fully Convolutional Masked Autoencoder

FCMAE adapts masked image modeling to ConvNeXt through sparse convolution, an asymmetric lightweight decoder, and masked-region reconstruction loss. Experiments show that preventing information leakage is essential, while FCMAE improves initialization but does not match the strongest supervised baseline in the reported comparison.

  • FCMAE Framework: The framework randomly removes 60% of the 32 × 32 input patches and uses minimal augmentation consisting of random resized cropping.The hierarchical convolutional mask is generated at the last stage and recursively upsampled.
  • FCMAE Framework: FCMAE masks raw visual input and reconstructs missing regions using a fully convolutional, asymmetric encoder-decoder.The encoder processes visible pixels, while the decoder combines encoded pixels with mask tokens and predicts masked content.
  • Decoder Design: A single lightweight ConvNeXt block serves as the decoder, reducing pre-training time while maintaining fine-tuning accuracy relative to more complex alternatives.The decoder is asymmetric because the encoder is heavier and hierarchical.
  • Reconstruction Target: The reconstruction objective is mean squared error between patch-wise normalized target and reconstructed images, applied only to masked patches.The loss excludes visible patches.
  • Encoder Design: Sparse convolution prevents information leakage from masked regions, which empirical findings identify as essential for good representations.The sparse-data perspective treats the masked image as a two-dimensional sparse pixel array.

4. Global Response Normalization

Global Response Normalization (GRN) addresses feature collapse in ConvNeXt masked-autoencoder pre-training by aggregating and normalizing channel responses, then calibrating the original features. Integrated into ConvNeXt V2, GRN preserves feature diversity and improves fine-tuning performance without added parameter or FLOP overhead.

  • Motivation: Feature collapse in FCMAE-pre-trained ConvNeXt appears as dead or saturated feature maps and redundant activations, especially in dimension-expansion MLP layers.Cosine distance analysis uses higher distances to indicate greater feature diversity and lower distances to indicate redundancy.
  • GRN design: GRN aggregates each channel’s spatial feature map with an L2 norm, normalizes each response relative to all channels, and recalibrates the input responses.The design adds zero-initialized affine parameters and a residual connection so the layer initially behaves like an identity function.
  • GRN design: GRN creates inter-channel feature competition through mutual inhibition, while divisive normalization outperforms alternative normalization functions on the same L2-norm aggregates.Standardization produces similar results, whereas global average pooling performs worse in the reported experiments.
  • Impact: GRN mitigates feature collapse in ConvNeXt V2, maintaining consistently high cosine distances across layers and producing behavior similar to MAE-pre-trained ViT.The analysis evaluates features from 1,000 ImageNet-1K validation images across model layers.
  • Impact: 84.6 fine-tuning accuracy is reported for V2 + FCMAE, compared with 83.8 for V1 + supervised 300ep and 83.7 for V1 + FCMAE.The GRN-equipped FCMAE model significantly outperforms the supervised counterpart without additional parameter overhead or increased FLOPs.
  • Design choices: GRN should be retained during both pre-training and fine-tuning, and its inclusion makes LayerScale redundant in the ConvNeXt V2 block.The block places GRN after the dimension-expansion MLP layer; removing or newly adding GRN only at fine-tuning causes performance degradation.

5. ImageNet Experiments

ImageNet experiments show that FCMAE and ConvNeXt V2 are most effective when co-designed, yielding consistent gains across model sizes and a 88.9% state-of-the-art Huge-model result using public data.

  • Co-design matters: FCMAE alone has limited impact, while GRN alone has a small supervised effect; combining both significantly improves fine-tuning performance.The result supports jointly designing the model architecture and self-supervised learning framework.
  • Model scaling: Eight ConvNeXt V2 models spanning 3.7M Atto to roughly 660M Huge consistently improve over supervised baselines.The models were pre-trained with FCMAE and compared with fully supervised counterparts.
  • Comparisons with previous methods: ConvNeXt V2 with FCMAE outperforms Swin with SimMIM across all tested model sizes.Against plain ViT with MAE, it performs similarly through Large while using 198M versus 307M parameters, but slightly lags in the Huge regime.
  • ImageNet-22K intermediate fine-tuning: 88.9% top-1 accuracy on ImageNet-1K is achieved by ConvNeXt V2 Huge after ImageNet-22K fine-tuning using only public data.This is reported as a new state-of-the-art result among methods using public data only.

6. Transfer Learning Experiments

Transfer-learning experiments show that GRN and FCMAE provide complementary gains for detection and segmentation, with ConvNeXt V2 outperforming or matching Swin depending on task and model regime.

  • Object detection and segmentation on COCO: COCO evaluation uses Mask R-CNN on COCO val2017 and reports detection mAPbox and segmentation mAPmask.All COCO fine-tuning experiments rely on ImageNet-1K pre-trained models.
  • Object detection and segmentation on COCO: COCO performance improves progressively from ConvNeXt V1 to V2 and from supervised initialization to FCMAE, with the best results using both.ConvNeXt V2 with FCMAE outperforms Swin across all model sizes, with the largest gap in the Huge regime.
  • Semantic segmentation on ADE20K: ConvNeXt V2 with FCMAE significantly improves over supervised ConvNeXt V1 on ADE20K semantic segmentation.It is on par with Swin in Base and Large regimes and outperforms Swin in the Huge regime.
  • Semantic segmentation on ADE20K: ADE20K evaluation uses UPerNet, with most experiments relying on ImageNet-1K pre-training.The FCMAE, 22K fine-tuning condition additionally applies ImageNet-22K supervised fine-tuning.

7. Conclusion

The paper introduces ConvNeXt V2, a broader ConvNet family designed for self-supervised learning through fully convolutional masked-autoencoder pre-training. It reports improved pure-ConvNet performance across major recognition tasks.

  • 7. Conclusion: ConvNeXt V2 covers a broader range of model complexity while making minimal architectural changes for self-supervised learning.The family is introduced as a new ConvNet model family.
  • 7. Conclusion: Fully convolutional masked-autoencoder pre-training significantly improves pure ConvNets on ImageNet classification, COCO detection, and ADE20K segmentation.These are the downstream tasks named in the conclusion.
  • 7. Conclusion: The appendix documents model configurations, training recipes, sparse and dense FCMAE encoding, feature analysis, and additional ablations.These implementation and analysis details are organized across appendix sections A through D.

A. Implementation Details

The implementation scales ConvNeXt V2 across a broad complexity spectrum by retaining standard Tiny, Base, and Large configurations and adding efficient and high-capacity variants.

  • Basic models: Tiny, Base, and Large retain ConvNeXt V1 stage, block, and channel configurations, with 28M, 89M, and 198M parameters respectively.These are the basic model configurations.
  • Model scaling: The model family is scaled across sizes to target versatile scenarios and provide a broad model-size spectrum.The appendix describes both downscaling for efficient models and upscaling for large capacity.
  • Efficient models: Atto, Femto, Pico, and Nano contain 3.7M, 5.2M, 9.1M, and 15.6M parameters, respectively.These efficient variants were originally designed in prior work.
  • Large-capacity model: Huge is a newly presented 659M-parameter large-capacity model.It is the upscaled variant in this configuration family.

A.2. ImageNet Experiments

The ImageNet experiments use model-size-specific fine-tuning recipes and report the configurations for pre-training, fine-tuning, and intermediate ImageNet-22K adaptation. Sparse encoding is supported through either sparse convolutions or masked dense convolutions, while segmentation models use supervised ImageNet-1K weights for initialization.

  • Fine-tuning setup: Fine-tuning recipes vary by model size, with longer schedules helping smaller models and different layer-wise decay strategies used across model groups.The default is layer-wise decay; group-wise decay is applied to Base and Large models.
  • Intermediate fine-tuning: ImageNet-22K intermediate fine-tuning is conducted for Nano, Tiny, Base, Large, and Huge FCMAE-pretrained models.Larger layer-wise learning-rate decay values are helpful for small models.
  • Pre-training setup: FCMAE pre-training supports sparse convolution encoding or masked dense convolution simulation, which produce numerically identical outputs.The choice depends on the use case and implementation availability.
  • Downstream setup: COCO experiments train ImageNet-1K-initialized models with MMDetection, a 3x schedule, batch size 32, and AdamW optimization.The experiments sweep layer-wise decay and stochastic-depth settings.
  • Downstream setup: ADE20K experiments use MMSegmentation, AdamW, batch size 16, and 160K training iterations.The models are initialized with supervised ImageNet-1K weights because they performed better than direct self-supervised initialization.

B. Complete comparisons with V1

ConvNeXt V2 consistently improves over V1 across model sizes in ImageNet comparisons, with larger V2 models surpassing larger V1 counterparts. The Huge model reaches 88.9%, while sparse encoding also improves pre-training efficiency.

  • ImageNet comparisons: V2 shows consistent improvement over V1 across the reported Atto, Femto, Nano, Pico, Tiny, Base, Large, and Huge model sizes.Table 14 reports single-crop ImageNet-1K fine-tuning results and Table 15 reports ImageNet-22K intermediate fine-tuning results.
  • Pre-training efficiency: Sparse encoding improves pre-training efficiency relative to dense masked convolution in the reported controlled setup.The accompanying figure measures throughput and maximum GPU memory usage with per-GPU batch size 64 and 20 forward-backward steps.
  • ImageNet comparisons: V2 Base and Large outperform the next-level V1 model sizes on ImageNet-1K and ImageNet-22K comparisons.V2 Base reports 86.8%/87.7% versus V1 Large at 86.6%/87.5%, while V2 Large reports 87.3%/88.2% versus V1 XLarge at 87.0%/87.8%.
  • ImageNet comparisons: 88.9% is achieved by ConvNeXt V2 Huge, establishing a new state-of-the-art result in the reported comparison.The result supports the paper’s claim that pure convolutional models can scale with mask-based pre-training.

C. Further Analyses

Further analyses examine sparse pre-training efficiency and feature selectivity, showing moderate resource gains and more class-generic features in deeper V2 layers. The GRN analysis also compares its component choices.

  • Sparse encoding efficiency: Sparse convolution blocks information from masked regions while applying kernels only to visible pixels during pre-training.Its practical efficiency depends on the framework and available sparse-convolution library optimization.
  • Sparse encoding efficiency: Sparse encoding yields an average 1.3× throughput increase and a 2× reduction in maximum memory usage across the evaluated models.The measurements use image size 224×224, masking ratio 0.6, mask size 32×32, and a specified PyTorch/CUDA/A6000 environment.
  • Feature analysis: V2 develops more class-generic features than V1 in deeper layers, while their class-selectivity distributions remain closely matched early in the network.The analysis uses the ImageNet-1K validation set and outputs from every residual block.
  • Feature analysis: The class-selectivity analysis is exploratory, with further investigation left for future work.The paper links class-agnostic features to transferability and downstream fine-tuning performance.

D. Additional Experiments

Additional experiments analyze GRN components, masking ratios, and self-supervised learning alternatives. They find that the proposed normalization components work jointly, a 0.6 masking ratio performs best, and FCMAE outperforms the reported baselines.

  • GRN analysis: Removing either global aggregation or feature normalization significantly decreases performance in the GRN component analysis.The analysis uses ConvNeXt V2-Base after 800-epoch FCMAE pre-training.
  • Masking-ratio analysis: A masking ratio of 0.6 provides the highest ImageNet-1K accuracy, while ratios from 0.5 to 0.7 produce the best results.Performance declines when nearly all or nearly none of the input is masked.
  • SSL comparison: FCMAE achieves better representation quality than MoCo V3 and the 300-epoch supervised baseline with the same ConvNeXt V2-Base encoder.The comparison evaluates end-to-end ImageNet-1K fine-tuning.
Loading 2301.00808v1…