PaperlayerView exact revision on arXiv

Source-linked AI summary

Big Transfer (BiT): General Visual Representation Learning

Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, Neil Houlsby

arXiv:1912.11370v3cs.CVcs.LG

TL;DR

Deep vision models often require substantial task-specific data, compute, and tuning, motivating more efficient transfer from generic pre-training. The paper scales supervised pre-training into BiT, combines selected components with a simple transfer heuristic, and reports strong performance across more than 20 datasets and data regimes. BiT also finds that standard benchmarks approach saturation while more diverse tasks retain substantial room for progress.

  • Problem

    Task-specific deep-learning training can require large amounts of data, compute, and tuning, making new vision tasks expensive.

  • Method

    BiT scales supervised pre-training and combines selected components with BiT-HyperRule, which sets key fine-tuning hyperparameters from task resolution and dataset size.

  • Results

    BiT attains strong performance across over 20 diverse tasks and data regimes ranging from 1 example per class to over 1M total examples.

  • Takeaways & Limitations

    BiT provides a clean transfer setup with cheap fine-tuning and limited per-task hyperparameter search across diverse downstream tasks.

  • Takeaways & Limitations

    On more diverse tasks such as localization, simulated-environment, medical, and satellite-imaging tasks, BiT-L performs well but substantial room for progress remains.

Abstract

from arXiv · show

Transfer of pre-trained representations improves sample efficiency and simplifies hyperparameter tuning when training deep neural networks for vision. We revisit the paradigm of pre-training on large supervised datasets and fine-tuning the model on a target task. We scale up pre-training, and propose a simple recipe that we call Big Transfer (BiT). By combining a few carefully selected components, and transferring using a simple heuristic, we achieve strong performance on over 20 datasets. BiT performs well across a surprisingly wide range of data regimes -- from 1 example per class to 1M total examples. BiT achieves 87.5% top-1 accuracy on ILSVRC-2012, 99.4% on CIFAR-10, and 76.3% on the 19 task Visual Task Adaptation Benchmark (VTAB). On small datasets, BiT attains 76.8% on ILSVRC-2012 with 10 examples per class, and 97.0% on CIFAR-10 with 10 examples per class. We conduct detailed analysis of the main components that lead to high transfer performance.

1 Introduction

BiT revisits pre-training on a large supervised dataset followed by fine-tuning, aiming for a minimal recipe that transfers effectively across diverse tasks and data regimes. Its design reduces downstream training and hyperparameter costs while retaining strong transfer performance.

  • Transfer learning replaces repeated task-specific data and compute requirements with one large-scale pre-training phase followed by downstream fine-tuning.
  • BiT combines a few carefully selected components rather than introducing new complexity, using large supervised pre-training and a simple transfer recipe.
  • BiT-L performs well with only 1 to 100 images per class when transferred to downstream tasks.
  • BiT-L transfers across tasks ranging from 1 example per class to 1M total examples, including 19 diverse VTAB datasets.
  • BiT requires one pre-training phase, then uses cheap fine-tuning and a heuristic instead of extensive hyperparameter tuning for each new task.
  • The paper analyzes how scale, architecture, and training hyperparameters interact, and plans to release BiT-M trained on ImageNet-21k.

2 Big Transfer

Big Transfer uses large-scale pre-training and selected normalization, optimization, and transfer components to support efficient adaptation across downstream tasks. Its fine-tuning procedure emphasizes task-dependent heuristics while accounting for resolution, dataset size, and task-specific augmentation needs.

  • BiT organizes its transfer recipe into upstream pre-training components and downstream fine-tuning components.
  • The study examines how pre-training dataset scale, architecture size, and computational budget interact in transfer learning.
  • Group Normalization and Weight Standardization are selected because Batch Normalization performs poorly with small per-device batches and complicates transfer through running statistics.
  • BiT-HyperRule selects schedule length, resolution, and MixUp usage from the task’s intrinsic image resolution and number of datapoints.
  • Augmentation is modified when flipping or cropping would destroy label semantics, such as orientation or pixel-coordinate labels.
  • Resolution changes are included during fine-tuning to address inconsistency between training and test resolutions.
  • MixUp is not useful for BiT pre-training but can help transfer on mid-sized datasets, unlike few-shot settings.
  • Downstream tuning omits weight decay and dropout, while schedule length is adjusted according to dataset size.

3 Experiments

BiT is evaluated across standard, few-shot, VTAB, ObjectNet, and detection benchmarks using models trained at multiple dataset scales and a lightweight transfer protocol. The results show strong performance across data regimes, with benefits from larger pre-training datasets and architectures.

  • 3 Experiments: BiT models trained on ILSVRC-2012, ImageNet-21k, and JFT-300M achieve strong performance across high- and low-data downstream regimes.BiT-S, BiT-M, and BiT-L are evaluated on diverse downstream tasks ranging from standard benchmarks to few-shot settings.
  • Standard Benchmarks: BiT-L outperforms previously reported generalist state-of-the-art models and, in many cases, specialist models that condition pre-training on the target task.Table 1 reports results using one model and one BiT-HyperRule setting per task, while specialist methods incur large per-task training costs.
  • Few-Shot Evaluation: 84.1% top-1 accuracy is achieved on ILSVRC-2012 with 100 labeled samples per class, while 72.0% is achieved with 5 samples per class.On CIFAR-100, BiT-L reaches 82.6% with 10 samples per class, demonstrating strong few-shot transfer.
  • VTAB-1k: BiT-L with BiT-HyperRule substantially outperforms the previously reported state-of-the-art across the 19 VTAB-1k tasks.BiT is reported as best on natural, specialized, and structured task subsets.
  • ObjectNet: 80.0% top-5 accuracy on ObjectNet represents an almost 25% absolute improvement over the previous state-of-the-art.The results attribute higher accuracies to scaling both architecture size and pre-training data; cropped single-object images further improve performance.
  • Object Detection: ImageNet-21k pre-training improves Average Precision by 1.5 points over ILSVRC-2012, while JFT-300M adds a further 0.6 points.The comparison uses BiT models and shows benefits from pre-training on datasets larger than ILSVRC-2012.

4 Analysis

BiT analysis shows that transfer performance depends on jointly scaling upstream data, model capacity, and computational budget, while GN/WS improves large-batch training and downstream transfer.

  • Scaling Models and Datasets: Larger models provide greater benefits on larger upstream datasets, whereas scaling either model size or data alone can yield limited or negative returns.A ResNet-50x1 trained on JFT-300M can underperform the same architecture trained on ImageNet-21k, while larger architectures benefit substantially from JFT-300M.
  • Scaling Models and Datasets: With one example per class, larger architectures outperform smaller ones when pre-trained on large upstream datasets.On few-shot ILSVRC-2012, BiT-L trained on JFT-300M even outperforms models trained on the full ILSVRC-2012 dataset.
  • Optimization on Large Datasets: Training on larger datasets requires more computation: applying the standard ILSVRC-2012 budget to ImageNet-21k hurts performance, while longer schedules recover and improve it.The paper emphasizes that sufficient computational budget is crucial for performant training on large datasets.
  • Optimization on Large Datasets: Lower weight decay can accelerate apparent convergence but ultimately produces an under-performing model because growing weight norms diminish the effective learning rate.The authors use weight decay of 10^-4 throughout to avoid this effect.
  • Large Batches, Group Normalization, Weight Standardization: GN alone loses 5.4% ILSVRC-2012 top-1 accuracy versus BN at batch size 4096, while adding WS enables GN to scale and outperform BN.The GN/WS combination also transfers better than BN on the 19 VTAB-1k tasks.

5 Related Work

Prior work explores large-scale supervised, task-specific, self-supervised, and few-shot transfer methods. BiT extends generalist supervised transfer across low-data and diverse-task settings using JFT-300M and a ResNet recipe.

  • Large-Scale Pre-training: Earlier large-scale supervised pre-training studies reported transfer to ImageNet, COCO, and VOC, while BiT also evaluates low-data and 19-task VTAB transfer.The cited prior work includes datasets with up to 100M Flickr images and noisily labelled JFT-300M or Instagram data.
  • Task-Specific Representations: Task-specific representations condition training on the target task, unlike BiT’s generalist pre-training and fine-tuning setup.Prior methods use teacher predictions or target-conditioned importance weights on large support datasets.
  • Self- and Semi-Supervised Learning: Self-supervised and semi-supervised methods leverage unlabelled data or limited labels, including strong results with 40 or 250 labels on CIFAR-10 and SVHN.Unsupervised representations trained on 1B unlabelled Instagram images transferred comparably or better than supervised ILSVRC-2012 features.
  • Few-Shot Learning: Few-shot methods include metalearning and metric learning, but simple linear classifiers or fine-tuning on pre-trained representations can achieve similar or better performance.The reviewed few-shot setups commonly pre-train and adapt within the same domain.

6 Discussion

BiT’s remaining benchmark errors often reflect label ambiguity or noise rather than clear visual misclassification, while more challenging low-data and diverse tasks remain substantially unsaturated.

  • Error Analysis: Around half of BiT-L’s CIFAR-10 mistakes are attributed to ambiguity or label noise, and humans clearly prefer the ground-truth label in only 19.21% of ILSVRC-2012 mistakes.Many mismatches involve valid predictions that do not match the assigned label.
  • Benchmark Saturation: BiT’s error inspection suggests that standard vision benchmark performance is approaching a saturation point.The authors therefore examine tasks farther from saturation.
  • Open Challenges: Few-label adaptation and diverse VTAB tasks remain challenging settings with substantial room for further progress despite BiT-L’s strong performance.VTAB includes spatial localization, simulated environments, medical imaging, and satellite imaging tasks.

A Tuning hyperparameters for transfer

The tuning study tests whether additional hyperparameter search improves BiT-L beyond BiT-HyperRule on diverse VTAB-1k tasks. Search saturates quickly, but selected hyperparameters yield a modest improvement after retraining.

  • Search Setup: VTAB-1k tuning evaluates 40 randomly sampled hyperparameter configurations per task using 800 training images and 200 validation images.The search samples learning rate, updates, dropout, weight decay, MixUp, and image resolution.
  • Search Outcomes: VTAB-1k performance saturates roughly after 20 trials, while further tuning overfits the validation split.This supports limiting the amount of per-task hyperparameter search.
  • Search Outcomes: 78.72% VTAB-1k score after retraining on 1000 images improves by 2.43% over the 76.29% BiT-HyperRule score.The reported improvement uses the selected hyperparameters and the union of training and validation splits.

B Full ObjectNet results

ObjectNet evaluation reports top-5 and top-1 accuracy in standard and cropped single-object settings. Cropping cluttered scenes to one object produces a solid performance improvement, consistent with the original test-set trend.

  • Cropping cluttered ObjectNet scenes to a single object produces a solid improvement in model performance.ObjectNet commonly contains multiple objects, and the cropped evaluation uses one object per crop.
  • The improvement trend is consistent with results on the original ObjectNet test set.
  • ObjectNet results are reported for top-5 and top-1 accuracy, including standard and ground-truth-bounding-box settings.The figure places top-5 accuracy on the left and top-1 accuracy on the right; Table 6 covers both standard and cropped settings.

C Duplicates and near-duplicates

The paper evaluates overlap between upstream training data and downstream test sets through deduplication experiments and duplicate inspection. Near-duplicates barely affect reported performance, while the mistake figures document model errors on several datasets.

  • Less than 50 k duplicate or near-duplicate images were removed from JFT-300M when training BiT-L.The removal targeted duplicates of downstream test images.
  • Near-duplicates barely affect BiT-L’s results across the reported experiments.The comparison uses original Full and cleaned Dedup test sets, with Dups recording removed near-duplicates.
  • Figure 11 shows duplicates between ILSVRC-2012 training data and test splits from four downstream datasets.
  • Figures 12 and 13 show BiT-L’s mistakes on Oxford-IIIT-Pet and Oxford-Flowers102, respectively.The mistake displays pair each prediction with its ground-truth label.
  • The illustrated mistakes use an earlier BiT-L version that reached almost the same accuracy as the latest model.The authors did not rerun the figures or human evaluation with the latest model.

E Object detection experiments

Object detection experiments use RetinaNet with standard training hyperparameters and report median performance across five runs. Training uses fixed 30-epoch optimization at 1024 × 1024 resolution.

  • RetinaNet is used for object detection, with standard hyperparameters and median performance reported over five training runs.
  • Models are trained for 30 epochs with batch size 256, stochastic gradient descent, and an initial learning rate of 0.08.Momentum is 0.9, weight decay is 10^-4, and the learning rate drops by 10× at epochs 16 and 22.
  • Detection training uses 1024 × 1024 input images and random horizontal flips with scale-based augmentation.

F Horizontal flipping and cropping for VTAB-1k tasks

BiT fine-tuning normally uses random flipping and cropping, but these augmentations are disabled for VTAB tasks whose labels are not invariant to them. The out-of-context evaluation instead compares predictions from fine-tuned models across pre-training datasets and architectures.

  • F Horizontal flipping and cropping for VTAB-1k tasks: Random horizontal flipping and cropping can be inappropriate for VTAB tasks whose labels encode angle, location, or object count.
  • F Horizontal flipping and cropping for VTAB-1k tasks: Horizontal flipping is disabled for dSpritesorientation, SmallNORB-azimuth, and dSprites-location.
  • F Horizontal flipping and cropping for VTAB-1k tasks: Random cropping is disabled for Clevr-count, Clevr-distance, DMLab, KITTIdistance, and dSprites-location.
  • F Horizontal flipping and cropping for VTAB-1k tasks: Figure 14 compares top-1 accuracy on ILSVRC-2012 validation images with accuracy on out-of-context objects across pre-training datasets and model sizes.It also shows the top five predictions for an example object, with probabilities plotted on a log scale.

G Robustness: Objects out-of-context

The study tests whether BiT improves robustness when classifying foreground objects pasted onto unusual backgrounds. Larger BiT pre-training improves out-of-context performance, and BiT-L more often focuses confidently on the foreground object rather than contextual distractions.

  • Results: Models pre-trained on ILSVRC-2012 saturate on out-of-context performance, whereas larger models pre-trained with more data transfer better to this setting.The improved out-of-context performance accompanies better ILSVRC-2012 performance for these models.
  • Qualitative behavior: BiT-L tends to classify the foreground object confidently regardless of context, while ILSVRC-2012 models may predict plausible objects absent from the image.The qualitative comparison concerns predictions on the constructed out-of-context images.
  • Dataset construction: The evaluation uses foreground objects from 20 OpenImages classes corresponding to ILSVRC-2012 classes and backgrounds selected for diversity and contextual mismatch.Objects are resized so their longest side spans 80% of the background width.
  • Dataset construction: The dataset combines 81 foreground objects with 41 backgrounds, producing 3321 prominent-object images for out-of-context evaluation.Foregrounds come from OpenImages segmentation masks, while backgrounds are licensed images collected from search-engine results.
Loading 1912.11370v3…