PaperlayerView exact revision on arXiv

Source-linked AI summary

Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks

Micah Goldblum, Hossein Souri, Renkun Ni, Manli Shu, Viraj Prabhu, Gowthami Somepalli, Prithvijit Chattopadhyay, Mark Ibrahim, Adrien Bardes, Judy Hoffman, Rama Chellappa, Andrew Gordon Wilson, Tom Goldstein

arXiv:2310.19909v2cs.CVcs.LG

TL;DR

Practitioners face many pretrained-backbone choices across architectures, algorithms, and datasets, without a single comparison resource. BoB benchmarks public checkpoints across diverse vision tasks and finds supervised CNNs strongest on most tasks, while same-scale comparisons favor SSL models.

  • Problem

    Practitioners face many pretrained-backbone choices across architectures, algorithms, and datasets, but lack a resource for comparing them systematically.

    Source: §1 · p. 2

  • Method

    BoB compares publicly available pretrained checkpoints and random-initialized baselines across classification, detection, segmentation, OOD generalization, and retrieval.

    Source: §1 · p. 2

  • Results

    Supervised convolutional networks outperform transformers on most considered tasks, while same-scale comparisons find SSL models outperform supervised counterparts.

    Source: §1.1 · p. 3

  • Takeaways & Limitations

    BoB aims to assist practitioners’ backbone choices and guide research toward strengths, weaknesses, and promising directions for improvement.

    Source: §1 · p. 2

    The benchmark excludes backbones larger than ConvNeXt-Base except Stable Diffusion, so rankings may change at larger scales.

    Source: §6 · p. 10

Abstract

from arXiv · show

Neural network based computer vision systems are typically built on a backbone, a pretrained or randomly initialized feature extractor. Several years ago, the default option was an ImageNet-trained convolutional neural network. However, the recent past has seen the emergence of countless backbones pretrained using various algorithms and datasets. While this abundance of choice has led to performance increases for a range of systems, it is difficult for practitioners to make informed decisions about which backbone to choose. Battle of the Backbones (BoB) makes this choice easier by benchmarking a diverse suite of pretrained models, including vision-language models, those trained via self-supervised learning, and the Stable Diffusion backbone, across a diverse set of computer vision tasks ranging from classification to object detection to OOD generalization and more. Furthermore, BoB sheds light on promising directions for the research community to advance computer vision by illuminating strengths and weakness of existing approaches through a comprehensive analysis conducted on more than 1500 training runs. While vision transformers (ViTs) and self-supervised learning (SSL) are increasingly popular, we find that convolutional neural networks pretrained in a supervised fashion on large training sets still perform best on most tasks among the models we consider. Moreover, in apples-to-apples comparisons on the same architectures and similarly sized pretraining datasets, we find that SSL backbones are highly competitive, indicating that future works should perform SSL pretraining with advanced architectures and larger pretraining datasets. We release the raw results of our experiments along with code that allows researchers to put their own backbones through the gauntlet here: https://github.com/hsouri/Battle-of-the-Backbones

1 Introduction

BoB addresses the lack of a single resource for choosing among pretrained backbones by comparing them across diverse tasks. Its findings show supervised CNN strength overall, while same-scale comparisons favor SSL models.

1 Introduction

  • Practitioners face many choices across architecture, pretraining algorithm, and dataset, but lack a single resource for comparing backbones.Source: §1 · p. 2
  • BoB compares public pretrained checkpoints and random-initialized baselines across natural, medical, and satellite classification, detection, segmentation, OOD generalization, and retrieval.Source: §1 · p. 2

1.1 Battle of the Backbones: The TLDR

  • Supervised ConvNeXt-Base, ImageNet-21k SwinV2-Base, and CLIP ViT-Base come out on top across BoB’s evaluations; ConvNeXt-Tiny and SwinV2-Tiny lead smaller scales, followed by DINO ViT-Small.Source: §1.1 · p. 2
  • Supervised convolutional networks outperform transformers on most considered tasks, while same-scale comparisons find SSL models outperform supervised counterparts.Source: §1.1 · p. 3
  • ViTs are more sensitive than CNNs to pretraining-data amount and parameter count.Source: §1.1 · p. 3
  • Performance is strongly correlated across tasks, with BoB’s top-performing backbones tending to do well across tasks and settings.Source: Figure 1 · p. 2, §1.1 · p. 3

1.2 Previous Benchmarks

  • Earlier benchmark studies examined more limited model sets or settings, while pretraining algorithms, datasets, and architectures have since advanced.Source: §1.2 · p. 3

2 A Guide to BoB

BoB catalogs backbone architectures, pretraining routines, and datasets, then compares publicly available checkpoints under practitioner-oriented constraints. Its task suite probes classification, dense prediction, OOD generalization, and retrieval.

2 A Guide to BoB

  • BoB catalogs algorithms, datasets, and architectures, then compares publicly available checkpoints using equally sized downstream hyperparameter sweeps under practitioner-like compute limits.Source: §2 · p. 3, Table 1 · p. 4

2.1 The Tasks

  • The evaluation probes backbones both fine-tuned and frozen across four downstream task categories.Source: §2.1 · p. 4
  • Classification covers natural, medical, and satellite image datasets, with both fine-tuning and linear-probe evaluation.Source: §2.1 · p. 4
  • Detection and segmentation test whether features preserve object locations at bounding-box and pixel level.Source: §2.1 · p. 4
  • OOD evaluation examines generalization to new downstream domains.Source: §2.1 · p. 4
  • Retrieval matches images by semantic content or visual similarity in feature space.Source: §2.1 · p. 4

3 Experimental Setup

The experimental setup specifies learning protocols, datasets, and metrics for each task. It evaluates classification, detection and segmentation, OOD generalization, and image retrieval.

3.1 Classification

  • Classification uses end-to-end fine-tuning, including 1% and 10% training subsets, and linear probes across six datasets, measured by accuracy or AUC.Source: §3.1 · pp. 4–5

3.2 Object Detection and Segmentation

  • Detection and instance segmentation use Cascade Mask R-CNN with random-init training, backbone fine-tuning, or frozen backbones, evaluated on COCO AP metrics.Source: §3.2 · p. 5

3.3 Out-of-Distribution Generalization

  • OOD tests ImageNet structure-and-style shifts and synthetic-to-real classification and detection, using accuracy and mAP@50, respectively.Source: §3.3 · pp. 5–6

3.4 Image Retrieval

  • Retrieval spans landmark, instance, fine-grained, natural-image, and real-world datasets, with mAP, Recall@k, and MRR as metrics.Source: §3.4 · p. 6

4 I’m a Practitioner. Which Backbone Should I Choose?

BoB ranks pretrained backbones for individual tasks and across tasks using setting-specific z-scores aggregated into a coarse overall comparison. Its rankings identify different task leaders, while supervised ConvNeXt and SwinV2 models and CLIP ViT-Base place highly overall; among small backbones, EfficientNet-B0 leads overall.

4 I’m a Practitioner. Which Backbone Should I Choose?

  • BoB computes z-scores within each task and setting, then aggregates them for a coarse cross-setting ranking of backbones.The scores measure performance relative to the mean and standard deviation of backbones in that setting.Source: §4 · p. 6

4.2 Best Backbones Overall

  • For practitioners without a specific task, Supervised ConvNeXt-Base trained on IN-21k ranks first overall, followed by Supervised SwinV2-Base trained on IN-21k (finetuned on IN-1k) and CLIP ViT-Base trained on LAION-2B.The table lists the top three backbones for various tasks and settings.Source: Table 2 · p. 7, §4.2 · p. 7

4.1 Task-Specific Backbones

  • Across classification settings, Supervised SwinV2-Base trained on IN-21k (finetuned on IN-1k) leads, while Supervised ConvNeXt-Base trained on IN-21k leads retrieval.CLIP ViT-Base and Supervised ConvNeXt-Base trained on IN-21k follow in classification; SwinV2-Base and CLIP ViT-B trained on LAION-2B rank second and third in retrieval.Source: §4.1 · p. 7
  • For object detection and instance segmentation, Supervised SwinV2-Base trained on IN-21k (finetuned on IN-1k) ranks above Supervised ConvNeXt-Base trained on IN-1k.Source: §4.1 · p. 7
  • For OOD classification, Supervised ConvNeXt-Base trained on IN-21k ranks first; for Syn→Real object detection, Supervised ConvNeXt-Base trained on IN-1k ranks first.The OOD classification ranking continues with SwinV2-Base and CLIP ViT-Base; Syn→Real detection continues with ConvNeXt-Tiny trained on IN-1k and ConvNeXt-Base trained on IN-21k.Source: §4.1 · p. 7

4.3 Backbones on a Tight Budget

  • Among the small supervised ImageNet-1k backbones, EfficientNet-B0 performs best overall and across classification, retrieval, and OOD classification, while ResNets outperform newer efficient architectures for detection and segmentation.The table ranks lightweight backbones across tasks and settings.Source: §4.3 · p. 7, Table 3 · p. 8

5 Observations and Trends

Across the evaluated models, supervised convolutional backbones often lead, while scale, dataset size, pretraining style, and fine-tuning strategy shape the comparisons. Performance is correlated across tasks, with notable qualifications for retrieval and generative backbones.

A performance comparison of ViTs and CNNs

  • ConvNeXt-Base leads, SwinV2-Base is close, and CLIP is strongest among vanilla ViTs except on OOD detection; small-scale winners are ConvNeXt-Tiny and SwinV2-Tiny, followed by DINO ViT-Small.ConvNeXt and SwinV2 pretrained with supervision on ImageNet-21k outperform SSL backbones on all tasks.Source: §5 · pp. 8–9

ViTs benefit more from scale than CNNs

  • Relative performance correlates more strongly with parameter count for ViTs than CNNs (ρ = 0.58 vs. ρ = 0.35) and pretraining-data size (ρ = 0.72 vs. ρ = 0.33).The authors suggest that much larger benchmarks might have different winners, potentially transformer-based.Source: §5 · p. 8

Supervised or not?

  • On similarly sized pretraining datasets, SSL or vision-language methods outperform supervised models on classification and retrieval, while supervised models retain an edge on detection and segmentation.For the same ViT-Base architecture, SSL beats ImageNet-1k supervised backbones but not ImageNet-21k trained backbones.Source: §5 · p. 8

Performance across tasks is highly correlated

  • Task-pair performance correlations are typically ρ > 0.8, but retrieval correlates less with classification and ranking (ρ = 0.49), rising to ρ = 0.8 when MiDaS and MAE are removed.The authors attribute the lower retrieval correlation to those models’ retrieval limitations.Source: §5 · p. 8

Transformers excel under end-to-end fine-tuning while convolutional networks excel under linear probing

  • ViTs benefit significantly more than CNNs from end-to-end fine-tuning, including for supervised and self-supervised pretraining.Figure 2 plots the difference between fine-tuning and fixed-backbone z-scores; dense prediction tasks show the larger transformer benefit.Source: Figure 2 · p. 9, §5 · p. 9

Generative backbones and performance versus speed

  • Generative-objective backbones such as MAE and Stable Diffusion perform comparatively worse, while throughput and average performance are negatively correlated (ρ = −0.41).The authors caution that Stable Diffusion was evaluated on select tasks; throughput was measured on an NVIDIA RTX A5000.Source: §5 · p. 9

Monocular depth-estimation as a general purpose pretraining strategy

  • MiDaS performs competitively with top supervised and SSL backbones in classification, detection, and segmentation, including satellite imagery.The authors suggest depth estimation could be a generalizable primary or auxiliary pretraining task for foundation models, supporting Lao et al. [49].Source: §5 · p. 9

Calibration, likelihood, and adversarial robustness

  • Test likelihood correlates more strongly with ImageNet accuracy than ECE (r = −0.8278 vs. r = −0.4876), while supervised pretraining yields inferior adversarial robustness to SSL for matched architectures.ViTs are more vulnerable to adversarial examples than CNNs; supervised ConvNeXt is more robust than other evaluated backbones.Source: §5 · pp. 9–10

6 Where Are Things Going From Here?

BoB offers practitioners a broad comparison of backbones and identifies research directions in generalization, scale, and self-supervision. Its conclusions are bounded by the models, tasks, and evaluation dimensions included.

6 Where Are Things Going From Here?

  • Supervised ConvNeXt, SwinV2, and CLIP perform well across the evaluated tasks; for smaller backbones, ConvNeXt-Tiny and SwinV2-Tiny lead, followed by DINO with a small ViT.BoB compares more than 1,500 training runs and presents its results as a guide for backbone selection.Source: §6 · p. 10
  • Strong cross-task correlations suggest a shift from specialized toward universal backbones, while the inverse throughput-performance relationship makes scaling a promising avenue.The authors also report that self-supervised learning holds promise in apples-to-apples comparisons with standard supervised training.Source: §6 · p. 10
  • BoB’s takeaways depend on the tasks, backbones, and settings considered, and its performance-focused analysis leaves other axes, including model biases, to explore.The authors expect practical considerations to evolve as more backbones, tasks, and settings are included.Source: §6 · p. 10
  • The benchmark includes no backbones larger than ConvNeXt-Base other than Stable Diffusion, so rankings may change at larger scales, where transformers could pull ahead.The authors find that transformers benefit more than convolutional backbones from greater model size and pretraining-data scale.Source: §6 · p. 10

7 Computation Cost and Carbon Footprint

The paper reports the computational and carbon cost of its experiments and reviews prior backbone benchmarking across classification, detection and segmentation, OOD generalization, and retrieval.

7 Computation Cost and Carbon Footprint

  • 127k GPU hours on NVIDIA RTX A100 cards produced estimated emissions of 11792.36 kgCO2eq.The estimate assumes average carbon efficiency of 0.37 kgCO2eq/kWh.Source: §7 · p. 10

A Additional Related Work

  • The authors describe their classification evaluation as a systematic comparison of supervised and self-supervised backbones across natural, satellite, and medical image datasets.They use the same learning setting across backbones and datasets.Source: A Additional Related Work · p. 17
  • Prior detection and instance-segmentation comparisons either cover few backbones or vary detectors, while this work presents a comprehensive study across architectures and pretraining methods.The authors note that modern vision-language and self-supervised backbones have not been extensively studied for these tasks.Source: A Additional Related Work · p. 17
  • The paper studies OOD generalization at larger scale than a prior broad study of pretraining strategies, while including backbones designed for varied tasks.Prior benchmarks include classification shifts and detection shifts such as sim-to-real, cross-weather, and cross-city settings.Source: A Additional Related Work · p. 17
  • The authors describe their study as the first comprehensive quantitative examination of multiple pretrained deep-learning backbones for image retrieval.They report that earlier surveys explained methods but did not quantitatively analyze these backbones [11] [108].Source: A Additional Related Work · p. 17

B.1 The Architectures

The benchmark varies backbone architectures and pretraining algorithms to examine their distinct properties. It includes convolutional and transformer architectures, a Stable Diffusion backbone, supervised and self-supervised methods, and vision-language models.

B.1 The Architectures

  • The architecture set spans ResNet, ConvNeXt, ViT, SwinV2, and a Stable Diffusion encoder, with multiple versions included where available.The authors vary architectures to examine potential benefits of architectural differences and include multiple sizes or versions when checkpoints allow.Source: §B.1 · p. 17
  • ResNet-50 and ResNet-101 represent staple CNNs, while ConvNeXt-Tiny and ConvNeXt-Base add modern convolutional designs.ConvNeXt uses depthwise convolutions, inverted bottlenecks, large kernels, and a larger share of layers in its third stage.Source: §B.1 · pp. 17–18
  • The transformer choices are ViT-Small and ViT-Base, plus hierarchical SwinV2-Tiny and SwinV2-Base models.SwinV2 uses spatial-window attention and iterative window merging; Base uses window size 24 unless otherwise stated.Source: §B.1 · p. 18
  • The Stable Diffusion backbone combines its learned image encoder and U-Net, with the text encoder frozen and its embedding used.The encoder maps images into the latent space used for diffusion and combines convolution with attention.Source: §B.1 · p. 18

B.2 The Pretraining Algorithms

  • Pretraining diversity includes supervised learning, self-supervised learning, and vision-language methods, selected as prototypical categories with hypothesized distinct properties.The paper notes beliefs that SSL features transfer more readily and vision-language models resist domain shifts.Source: §B.2 · p. 18
  • The benchmark includes ImageNet-1k/-21k classifiers and MiDaS, which is supervised for monocular depth estimation and outputs dense depth values.ImageNet pretraining uses labels for supervised models and can also be used without labels for self-supervised models.Source: §B.2 · p. 18
  • MoCo v3 uses contrastive learning with a momentum encoder and multi-crop training, whereas VICReg regularizes feature variance and covariance instead of using negative pairs.VICRegL also applies its criterion to local features to promote localization information for dense prediction.Source: §B.2 · p. 18
  • DINO replaces contrastive learning with self-distillation, while MAE reconstructs masked input patches rather than matching augmented-image features.The benchmark also includes CLIP image features trained contrastively on image-caption pairs, a Stable Diffusion generative backbone, and randomly initialized downstream baselines.Source: §B.2 · pp. 18–19

B.3 The Pretraining Datasets

The benchmark draws on pretraining datasets spanning image classification, image-text pairs, and depth annotations, and evaluates across several computer-vision task datasets. The appendix also notes dataset-use restrictions.

B.3 The Pretraining Datasets

  • Pretraining data range from labeled or unlabeled classification images to image-text pairs and images with depth annotations.The paper includes ImageNet, LAION, OpenAI's CLIP training dataset, and MiDaS's multi-dataset depth training set.Source: §B.3 · p. 19
  • ImageNet-21k contains over 14 million images in 21,841 classes; ImageNet-1k has almost 1.3 million images in 1,000 classes.LAION-2B contains 2.32 billion English-captioned image-text pairs and is noted to contain many duplicates [97].Source: §B.3 · p. 19
  • Evaluation datasets cover classification, detection, segmentation, OOD generalization, and retrieval, with tables summarizing class counts and test-sample counts.EuroSAT and CheXpert use 10% of their labeled data; the passage also describes non-commercial-use restrictions for several datasets.Source: §B.3 · p. 19
  • Most retrieval datasets are restricted to non-commercial research and educational use; Objectnet is available for research and commercial applications.Source: §B.3 · p. 20

C.1 Implementation Details

The implementation appendix specifies classification and dense-prediction training protocols, plus settings for domain generalization and retrieval. It notes that benchmark cost and training choices constrain conclusions about ViT performance.

C.1 Implementation Details

  • Regular classification fine-tuning uses AdamW for 100 epochs with batch size 1024 and cosine annealing; linear probing uses LARS, batch size 4096, for 90 epochs.For low-shot ImageNet, transformer-based backbones use AdamW and convolution-only backbones use SGD.Source: §C.1 · p. 20
  • Detection and segmentation use Cascade Mask R-CNN with AdamW, a 36-epoch schedule, and grid-searched learning rates; ViTs and Stable Diffusion receive distinct training settings.ViTs use large-scale jitter at 1024×1024, while Stable Diffusion uses a single 512×512 resolution to limit its substantially higher computational cost.Source: §C.1 · p. 21
  • Larger models are omitted because they would be prohibitively expensive, although ViTs may overtake other models at larger scales.The paper says ViTs benefit more from scale and that architectural modifications and very long training schedules can benefit them in particular.Source: §C.1 · p. 21
  • For VisDA synthetic-to-real classification, models train on the synthetic source domain and are evaluated on the real target after 10 epochs.For Sim10k detection, models train on all of Sim10k and are evaluated on Cityscapes validation images downsampled to 1024×512.Source: §C.1 · p. 21
  • Image retrieval evaluation uses pretrained models only.The appendix points to the main body for datasets and metrics and to Table 7 for a dataset summary.Source: §C.1 · p. 21

D Results

D Results reports backbone performance across classification, detection and segmentation, OOD generalization, retrieval, and adversarial robustness. The results also show scale benefits and a slight advantage for modern convolutional architectures on LVIS, while E Assets and Licenses describes the experiment code and dependencies.

D Results

  • ImageNet classification results cover Top-1 and Top-5 accuracy, while fine-tuning results cover Top-1 accuracy on various datasets.The tables also include ImageNet calibration and uncertainty estimation; ImageNet results for ImageNet-pretrained backbones are omitted because they coincide with random-initialization results.Source: D Results · pp. 21–22, Table 8 · p. 22, Table 9 · p. 22
  • COCO results cover bounding-box detection and instance segmentation with frozen backbones, scratch training, and end-to-end fine-tuning.The experiments train on COCO train2017 and evaluate on val2017.Source: D Results · p. 22, Table 11 · p. 23, Table 12 · p. 24
  • LVIS results show a benefit of scale and slightly superior performance from modern convolutional architectures compared with transformer-based architectures.The comparison uses the best SwinV2 and ConvNeXt architectures at several sizes.Source: D Results · p. 23, Table 13 · p. 24
  • OOD results cover image classification and object detection, while retrieval results report mAP, MRR, and Recall@5 across datasets.The OOD detection comparison evaluates models trained on Sim10k on Cityscapes for car detection, including frozen-backbone settings.Source: D Results · pp. 24–25, Table 14 · p. 24, Table 15 · p. 25
  • Adversarial robustness is evaluated on ImageNet under untargeted PGD attacks with ℓ∞ radii of 1/255 and 2/255.The reported measure is Top-1 classification accuracy.Source: D Results · p. 25

E Assets and Licenses

  • The released experiment code builds on timm, MAE, a medical-dataset pretraining repository, and MMDetection, with PyTorch, HuggingFace utilities, and MMCV among key dependencies.The backbones are publicly available pretrained models, and the full list is provided in Appendix B.Source: E Assets and Licenses · p. 26

F Observations about Hyperparameters

Fine-tuning needs depend on both pretraining method and downstream task: learning-rate choices vary, and classification can use fewer epochs than scratch training. The authors speculate that parameter-magnitude differences contribute to learning-rate sensitivity, while detection and segmentation save fewer epochs because they add randomly initialized parameters.

F Observations about Hyperparameters

  • The best tried ImageNet learning rate was 1e-4 for CLIP and 1e-3 for MAE, illustrating method- and dataset-dependent learning-rate sensitivity.The authors speculate this sensitivity occurs because different pretraining algorithms produce parameter vectors with different magnitudes.Source: F Observations about Hyperparameters · p. 26
  • Image classification fine-tuning uses 100 epochs, one-third the scratch-training schedule, and yields much higher accuracy on smaller datasets such as Flowers-102 and aircraft.The authors contrast this with detection and segmentation, where fine-tuning saves fewer epochs.Source: F Observations about Hyperparameters · p. 27
  • Detection and segmentation save fewer fine-tuning epochs than classification because their systems contain many new parameters initialized randomly for downstream training.Source: F Observations about Hyperparameters · p. 27
Loading 2310.19909v2…