PaperlayerView exact revision on arXiv

Source-linked AI summary

InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions

Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, Xiaogang Wang, Yu Qiao

arXiv:2211.05778v4cs.CV

TL;DR

Large-scale CNNs have not matched the progress of ViTs, motivating a CNN foundation model that scales with parameters and massive data. InternImage centers on deformable convolution and transformer-like architecture designs, and is reported to achieve comparable or better performance than leading ViTs across vision tasks.

  • Problem

    Large-scale CNN models remain less developed than ViTs, despite the need to determine whether CNNs can scale effectively with parameters and massive training data.

  • Method

    InternImage uses an improved 3×3 deformable convolution with flexible receptive fields and adaptive spatial aggregation, combined with tailored blocks, stacking rules, and scaling strategies.

  • Results

    InternImage achieves comparable or better performance than state-of-the-art ViTs across a wide range of vision tasks, including classification, detection, and segmentation.

  • Takeaways & Limitations

    The results support CNNs as a viable direction for large-scale vision foundation model research.

Abstract

from arXiv · show

Compared to the great progress of large-scale vision transformers (ViTs) in recent years, large-scale models based on convolutional neural networks (CNNs) are still in an early state. This work presents a new large-scale CNN-based foundation model, termed InternImage, which can obtain the gain from increasing parameters and training data like ViTs. Different from the recent CNNs that focus on large dense kernels, InternImage takes deformable convolution as the core operator, so that our model not only has the large effective receptive field required for downstream tasks such as detection and segmentation, but also has the adaptive spatial aggregation conditioned by input and task information. As a result, the proposed InternImage reduces the strict inductive bias of traditional CNNs and makes it possible to learn stronger and more robust patterns with large-scale parameters from massive data like ViTs. The effectiveness of our model is proven on challenging benchmarks including ImageNet, COCO, and ADE20K. It is worth mentioning that InternImage-H achieved a new record 65.4 mAP on COCO test-dev and 62.9 mIoU on ADE20K, outperforming current leading CNNs and ViTs. The code will be released at https://github.com/OpenGVLab/InternImage.

1. Introduction

InternImage is proposed as a large-scale CNN foundation model built around deformable convolution, targeting the long-range, adaptive aggregation, and scaling capabilities associated with ViTs. Across vision tasks, it is reported to match or surpass large-scale ViTs and CNNs.

  • Large-scale CNN research has lagged behind ViTs, which have advanced rapidly through large parameters, data, and global or adaptive representations.
  • InternImage uses an improved 3×3 deformable convolution whose offsets learn flexible receptive fields and whose modulation adapts spatial aggregation to input data.The common 3×3 window avoids the optimization and cost issues of large dense kernels.
  • The model combines the deformable operator with transformer-like block and architecture designs, stacking rules, and scaling strategies for large parameters and data.
  • 65.4 box AP is achieved by InternImage-H on COCO test-dev, establishing a reported new record and outperforming state-of-the-art CNNs and large-scale ViTs.
  • InternImage is evaluated on classification, detection, instance segmentation, and semantic segmentation while scaling from 30 million to 1 billion parameters and 1 million to 400 million images.

2. Related Work

Related work spans the evolution of CNNs, transformer-based vision foundation models, and large-scale model scaling. The paper positions InternImage against ViTs’ strong performance and prior CNN efforts using large or recursive kernels.

  • CNNs progressed from classic architectures to more sophisticated operations such as depth-wise and deformable convolution.
  • ViTs achieved strong vision performance through global receptive fields and dynamic spatial aggregation, but global attention is costly on large feature maps.
  • Large-scale vision models expanded ViTs to billions of parameters and raised performance on basic vision tasks, while CNN-based scaling remained comparatively limited.
  • Recent CNNs introduced long-range dependencies with very large or recursive gated kernels, yet the paper describes a remaining gap from state-of-the-art ViTs.

3. Proposed Method

InternImage develops a large-scale CNN foundation model around deformable convolution, extending DCNv2 into DCNv3 and combining it with transformer-inspired blocks and scaling rules.

  • InternImage starts from DCNv2 and explores stacking and scaling principles to build a CNN model trainable on massive data.
  • Deformable convolution: DCNv2 provides flexible sampling offsets and input-conditioned offsets and modulation scalars, supporting long-range dependencies and adaptive spatial aggregation.
  • Deformable convolution: DCNv3 extends DCNv2 with shared convolutional weights, multi-group aggregation, and softmax-normalized modulation scalars.
  • Deformable convolution: DCNv3 uses sparse sampling and retains convolutional inductive bias, making it more computationally and memory efficient than attention-based alternatives.
  • InternImage architecture: InternImage’s basic block combines DCNv3 with layer normalization, feed-forward networks, and GELU, while separable convolution predicts offsets and modulation scales.
  • InternImage architecture: The architecture uses convolutional stem and downsampling layers for hierarchical feature maps, alongside transformer-inspired blocks centered on DCNv3.

4. Experiment

InternImage is evaluated across ImageNet, COCO detection and instance segmentation, and ADE20K semantic segmentation, with results showing strong performance against CNN and transformer backbones. Ablations further examine parameter sharing, multi-group aggregation, and receptive-field behavior.

  • Object detection and instance segmentation: 47.2 APb is reported for InternImage-T with Mask R-CNN under the 1× schedule, outperforming Swin-T by 4.5 points.InternImage also reports higher mask AP across representative COCO comparisons.
  • Object detection and instance segmentation: 65.4 APb is achieved on COCO test-dev, surpassing FD-SwinV2-G by 1.2 points with 27% fewer parameters and without distillation.
  • Semantic segmentation: 62.9 mIoU is achieved by InternImage-H on ADE20K with Mask2Former and multi-scale testing, exceeding the current best BEiT-3 result.With UperNet, InternImage-B reaches 50.8 mIoU and InternImage-H reaches 60.3 MS mIoU.
  • Ablation study: Sharing DCNv3 weights saves 42.0% of parameters and 84.2% of GPU memory at the -H scale while preserving similar task accuracy at the -T scale.Removing multi-group aggregation reduces ImageNet accuracy by 1.2 points and COCO performance by 3.4 points.

5. Conclusion & Limitations

InternImage is presented as a scalable CNN foundation model built around deformable convolution and evaluated across diverse vision tasks. The paper reports competitive or superior performance to large-scale ViTs, while identifying latency as a remaining issue for DCN-based operators.

  • InternImage provides strong representations for image classification, object detection, and semantic segmentation.
  • InternImage combines a tuned DCNv2 operator with tailored blocks, stacking rules, and scaling rules for large-scale CNNs.
  • Extensive experiments report comparable or better performance than well-designed large-scale vision transformers trained with massive data.
  • Limitations: Latency remains an issue for DCN-based operators in downstream applications with high-speed requirements.

A. Detailed Training Settings

The paper specifies training procedures for classification, detection, and segmentation, then studies stacking and scaling choices used to construct InternImage variants. These experiments identify a default stacking configuration and scaling factors associated with the best observed performance.

  • Training settings: Image classification uses ImageNet-1K for smaller models, ImageNet-22K pre-training for larger models, and additional M3I pre-training for InternImage-H.
  • Training settings: COCO detection uses Mask R-CNN or Cascade Mask R-CNN with pre-trained classification weights and 1× or 3× training schedules.
  • Training settings: ADE20K segmentation uses pre-trained backbones with UperNet, while InternImage-H is additionally paired with Mask2Former under advanced training settings.
  • Stacking choices: The stacking search varies C1, L1, and C′ across 30 approximately 30M-parameter models, with the selected configuration set to (64, 16, 4, 18).
  • Model scaling: The scaling search finds (α, β) = (1.09, 1.36) as the best-performing setting, with other model sizes also confirming the scaling rules.C1 must be divisible by C′, causing small parameter-count fluctuations during specific scaling choices.

B.3. Kernel Size

The paper examines kernel size, receptive fields, and transfer performance to characterize InternImage’s operator and large-scale representation. Larger dense kernels increase cost without reliable accuracy gains, while trained InternImage develops increasingly global receptive fields and strong results on specialized datasets.

  • Effective receptive field: After sufficient training, InternImage can perceive information across the entire image in its third and fourth stages.The effective receptive field is measured by aggregating gradient maps from an activated point at the dog’s eye.
  • Classification transfer: InternImage-H achieves 92.6 accuracy on iNaturalist 2018, 3.9 points above the previous best MetaFormer model.
  • Classification transfer: InternImage achieves 71.7 accuracy on Places205, outperforming MixMIM-L by 2.4 points.
  • Classification transfer: InternImage achieves 61.2 accuracy on Places365 and is reported as state of the art against the previous best SWAG model.

C.2. Object Detection

InternImage-H is evaluated across diverse object-detection benchmarks, achieving strong results on large-vocabulary, general, crowded, and real-world driving scenarios.

  • Additional benchmarks: InternImage-H is also evaluated on Pascal VOC and CrowdHuman using Objects365-pre-trained weights and the same general fine-tuning setup.The supplied passages describe the datasets and setup but do not provide complete corresponding results.
  • OpenImages v6: 74.1 mAP on OpenImages v6 improves the previous best result by 1.9 mAP.OpenImages v6 covers 600 object classes across a highly diverse image collection.
  • BDD100K: 38.8 mAP on BDD100K validation achieves state-of-the-art performance, surpassing the previous best model by 3.2 mAP.BDD100K represents diverse weather and lighting conditions in moving-vehicle imagery.

C.3. Semantic Segmentation

InternImage-H is paired with Mask2Former and evaluated across semantic-segmentation benchmarks spanning broad-category, street-scene, and RGB-D settings.

  • COCO-Stuff: 59.6 MS mIoU on COCO-Stuff-10K test surpasses the previous best ViT-Adapter by 5.4 mIoU.The model is pre-trained on COCO-Stuff-164K and fine-tuned on COCO-Stuff-10K with 512×512 crops.
  • Cityscapes: 87.0 MS mIoU on Cityscapes validation and 86.1 MS mIoU on its test set are reported for high-resolution street-scene segmentation.The setup uses Mask2Former, Mapillary Vistas pre-training, and 1024×1024 crops.
  • NYU Depth V2: 68.1 MS mIoU on NYU Depth V2 validation is 11.2 points better than CMX-B5.NYU Depth V2 contains 40 semantic categories across 1449 RGB-D images.

D. Throughput Analysis

InternImage is analyzed for inference throughput and robustness under geometric transformations, with the supplied results indicating efficiency and stronger invariance in several settings.

  • Throughput: InternImage has better parameter efficiency and significantly faster inference than a DCNv2 variant at both 224×224 and 800×800 resolutions.The throughput comparison includes ConvNeXt, RepLKNet, and deformable-attention DAT counterparts.
  • Translation invariance: InternImage shows the best translation robustness among InternImage-T, ConvNeXt-T, PVTv2-B2, and Swin-T.Translation invariance is measured by label consistency while dithering images from 0 to 64 pixels.
  • Rotation invariance: At rotations above 10°, InternImage is clearly superior to the other compared models, despite comparable small-angle consistency.Images are rotated from 0° to 45° in 5° steps.
  • Scaling invariance: When scaling images up, InternImage achieves better box consistency and bounding-box mAP than the other methods.Scaling factors range from 0.25 to 3.0; all methods are sensitive to downscaling.

E.4. How Hungry the Model is for Data Scale?

InternImage-T remains robust across training-data scales, combining competitive low-data performance with stronger full-data accuracy than the compared variant and ConvNeXt.

  • Data-scale robustness: 83.5% accuracy on full ImageNet-1K data, 56.0% on 10%, and 5.9% on 1% demonstrate robustness across data scales.The experiments train models for 300 rounds on uniformly sampled 1%, 10%, and 100% ImageNet-1K data.
  • Data-scale robustness: ResNet performs best on 1% and 10% data but reaches only 80.4% on full data, while Swin-T performs well only with the full dataset.The passage attributes ResNet’s low-data advantage to inductive biases.
Loading 2211.05778v4…