Source-linked AI summary
A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, Neil Houlsby
TL;DR
Representation learning lacks a unified, diverse evaluation tied to few-label adaptation, limiting comparisons of general visual representations. The paper introduces VTAB and uses it to study representation-learning algorithms while controlling architecture, tuning budget, and related confounders. Across tasks, supervised ImageNet representations excel on natural tasks, self-supervision helps structured understanding and can nearly replace most labels, but the study concludes that general visual representations remain distant.
Problem
Representation learning lacks a common evaluation for general visual representations that measures adaptation to diverse unseen tasks with few labels.
Method
The paper introduces VTAB, a 19-task benchmark measuring few-label adaptation while controlling architecture, tuning budget, pre-training data, and other confounders.
Results
Self-supervision can almost, but not quite, replace 90% of ImageNet labels, while supervised ImageNet representations excel on natural tasks and self-supervision can outperform supervision on some structured tasks.
Takeaways & Limitations
Self-supervision is promising for reducing label dependence and improving structured understanding, but the evaluated strategies remain far from general visual representations.
Takeaways & Limitations
VTAB uses fixed hand-picked evaluation tasks, and subsets may correlate less well in other experiments while increasing meta-overfitting risk.
Abstract
from arXiv · showhide
Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNet, CIFAR, Pascal-VOC), or only weakly related to representation quality (ELBO, reconstruction error). We present the Visual Task Adaptation Benchmark (VTAB), which defines good representations as those that adapt to diverse, unseen tasks with few examples. With VTAB, we conduct a large-scale study of many popular publicly-available representation learning algorithms. We carefully control confounders such as architecture and tuning budget. We address questions like: How effective are ImageNet representations beyond standard natural datasets? How do representations trained via generative and discriminative models compare? To what extent can self-supervision replace labels? And, how close are we to general visual representations?
1. Introduction
VTAB addresses the lack of a common evaluation for representation learning by testing adaptation to diverse unseen tasks with few labels. The study compares major training strategies under controlled architecture, tuning, and pre-training conditions.
- Motivation: Representation learning aims to improve sample efficiency so vision systems can address tasks without large labeled datasets.Humans learn many vision tasks from few examples, whereas modern systems often rely on thousands to millions of annotations.
- Motivation: A common benchmark is needed because representation-learning subfields use separate evaluation protocols.Existing evaluations span supervised, self-supervised, semi-supervised, generative, and disentanglement learning contexts.
- VTAB: VTAB defines good representations as those that solve diverse, previously unseen tasks with the fewest possible labels.Its task groups extend beyond standard natural images to sensorimotor control, medical imaging, and scene understanding.
- Findings: The study finds supervised ImageNet pre-training works especially well for natural classification and yields smaller but consistent gains on specialized tasks.Supervised representations are described as extremely limited for tasks requiring structured understanding.
- Findings: Self-supervision is less effective overall than supervision but can improve structured understanding, while combined supervision and self-supervision can substantially replace labels.The study also reports that discriminative representations generally outperform generative ones, except for adversarially trained encoders, and that linear evaluation gives poorer transfer.
2. The Visual Task Adaptation Benchmark
VTAB formalizes representation quality as expected performance across diverse unseen visual tasks given few labeled examples. Its benchmark uses fixed, unseen task samples, standardized adaptation and tuning procedures, and 19 tasks spanning natural, specialized, and structured domains.
- Objective: VTAB seeks algorithms that perform well across many unseen visual understanding tasks with few labels per task.The benchmark formalizes this objective as expected performance over a distribution of tasks.
- Objective: An evaluation task combines a task-specific data distribution and evaluation procedure, while an algorithm maps limited samples to a prediction function.Performance is measured by an evaluation procedure, such as test-set accuracy, and averaged over tasks.
- Benchmark design: VTAB approximates the task expectation with a fixed representative set of hand-picked samples, reducing variance but introducing meta-overfitting risk.Evaluation tasks are treated as unseen, so pre-training cannot use their samples, including unlabeled images.
- Benchmark design: The benchmark uses a homogeneous classification interface and shared architecture and hyperparameter searches to control confounding factors across diverse tasks.Tasks such as detection can be converted into classification problems, although per-pixel prediction tasks are not always efficiently modeled this way.
- Task composition: VTAB contains 19 tasks grouped into NATURAL, SPECIALIZED, and STRUCTURED domains.NATURAL covers classical camera imagery; SPECIALIZED includes remote-sensing and medical images; STRUCTURED measures scene comprehension such as counting and depth prediction.
- Transfer and training: The protocol evaluates representations by adapting pre-trained networks to new tasks with limited in-domain data, using freezing or fine-tuning as transfer strategies.Pre-training data is unconstrained, but representations cannot be pre-trained on the evaluation tasks themselves.
3. Large-Scale Study
VTAB evaluates 18 representation-learning methods across 19 diverse tasks under controlled architecture, data, and tuning conditions. The study finds that supervised and combined supervised/self-supervised representations transfer best overall, while generative models generally underperform and linear evaluation can misrepresent transfer quality.
- VTAB results: Generative representations perform worst overall, with autoencoders below from-scratch training and most GAN discriminators providing unstable or weak representations.BIGBIGAN is an exception, performing similarly to the best self-supervised models, but it remains relatively weaker on STRUCTURED tasks than on NATURAL tasks.
- VTAB results: All self-supervised representations outperform from-scratch training; ROTATION reaches 59.6% on VTAB-1k versus 42.1% for FROM-SCRATCH.Whole-image methods outperform patch-based methods overall, although patch-based methods do slightly better on texture- and fundus-image tasks.
- VTAB results: Supervised ImageNet pre-training performs best overall, especially on NATURAL tasks, while self-supervision narrows the gap with fewer labels and helps on STRUCTURED tasks.SUP-10% reaches 61.6% on VTAB-1k versus 65.6% for SUP-100%; SUP-ROTATION-10% reaches 64.8%, closing 80% of the gap.
- Heavyweight hyperparameter sweeps: Heavyweight tuning improves all evaluated methods, yet pre-trained representations remain better than a tuned from-scratch model across task groups.FROM-SCRATCH is competitive on Clevr-Dist and sNORB-Elev, suggesting that ImageNet representations still miss localization and camera-elevation features.
- Analysis: Linear evaluation lowers performance and can change method rankings, particularly on SPECIALIZED and STRUCTURED datasets.SUP-100% reaches 65.6% with fine-tuning but 57.3% with linear evaluation; Kendall’s ranking correlation with top-1 accuracy is nevertheless always above 0.97 for the aggregation strategies discussed.
- Analysis: Most runs reach optimal performance within 1,000 training steps, with little significant improvement from longer schedules.The evaluation cost is near-linear in schedule length, motivating short schedules for the benchmark’s lightweight mode.
4. Related Work
VTAB is positioned against existing vision benchmarks and representation-learning evaluations that differ in task coverage, data regime, and evaluation protocol. The related work emphasizes that common generative and frozen-feature metrics provide varied, sometimes indirect views of representation quality.
- Vision Benchmarks: The Visual Decathlon covers ten mostly natural-image classification tasks, whereas VTAB includes broader domains and uses a different transfer protocol.Visual Decathlon permits joint optimization across tasks but forbids external data; VTAB forbids downstream multitask learning while permitting transfer from arbitrary datasets.
- Vision Benchmarks: The Facebook AI SSL challenge uses four natural-image tasks, whereas VTAB uses classification-only tasks with more domains and semantics such as localization and counting.The classification-only design is intended to admit task-independent implementations.
- Representation Learning Evaluation: Linear or MLP evaluation freezes a pretrained network and trains a linear layer or MLP, often on the representation-learning dataset itself.This protocol is widely used for self-supervised representations.
- Generative Evaluation: Generative models are assessed with diverse proxy metrics, including reconstruction error, FID, IS, precision-recall, log likelihood, and ELBO.Some generative models are additionally evaluated through linear classification or semi-supervised learning.
5. Discussion
The discussion finds that supervised ImageNet features transfer well to natural and some specialized tasks but remain weaker for structured understanding. Self-supervision can substantially substitute for labels, while generative objectives are generally less effective and VTAB remains far from general visual representations.
- ImageNet representations: Supervised ImageNet representations work well for natural tasks and also transfer to specialized tasks despite domain shift, but can be poorer than unsupervised representations for structured understanding.The authors suggest diverse sources such as video as future training data for moving beyond ImageNet-like tasks.
- Generative and discriminative learning: Generative losses appear less promising than discriminative objectives for learning representations, although BigBiGAN performs on par with self-supervision.BigBiGAN is identified as an exception that warrants further exploration.
- Self-supervision: Self-supervision can outperform supervision on some structured tasks and nearly replace 90% of ImageNet labels, while also adding value above ImageNet labels.The gap between 10% labeled pre-training with self-supervision and 100% labeled pre-training is described as small.
- General visual representations: VTAB remains far from general visual representations, and adding more data on specialized and structured tasks outperforms the evaluated pre-training strategies.The authors identify combining open-domain self-supervision with ImageNet or other label sources as promising future work.
- Benchmark scope: VTAB measures adaptation to diverse unseen tasks while controlling upstream factors such as hyperparameter sweeps, architecture, transfer algorithm, preprocessing, and pre-training data.The benchmark can also be used to analyze and optimize other factors involved in learning generalizable representations.
- Resources: VTAB code, data, and released models are available to support continued progress toward deep learning with limited labeling budgets.The project monitors progress through its public repository.
Supplementary Material: A Large-scale Study of Representation Learning
The supplementary material is associated with the paper’s large-scale study of representation learning using the Visual Task Adaptation Benchmark.
- The supplementary material accompanies the Visual Task Adaptation Benchmark study.
- The material belongs to the paper’s representation-learning evaluation work.
A. Tasks
VTAB comprises task datasets spanning natural images, specialized domains, and structured visual reasoning, with task-specific labels and varied image resolutions. Several structured tasks are constructed from existing datasets by predicting quantities such as object count, depth, location, orientation, or viewpoint.
- NATURAL: Flowers102 classifies 102 flower categories, while Pets classifies cat and dog breeds and Sun397 classifies 397 scenery categories.
- NATURAL: SVHN classifies street-view house numbers into 10 classes using 32x32 images, while CIFAR-100 classifies 100 natural-image categories using 32x32 images.
- SPECIALIZED: Specialized tasks include satellite land-use and remote-sensing scene classification, histopathology metastasis detection, and diabetic-retinopathy grading.
- STRUCTURED: CLEVR-derived tasks predict object count or bucketed depth from images rather than using the dataset’s original visual-question-answering format.
- STRUCTURED: dSprites location and orientation tasks predict bucketed object attributes, while SmallNORB tasks predict azimuth and elevation classes.
- STRUCTURED: DMLab and KITTI-Dist evaluate visual distance or depth reasoning using binned labels from navigational or driving imagery.
B. Human Evaluation
VTAB defines its task universe as visually solvable by humans and checks this assumption using few-shot human evaluations across natural, specialized, and structured tasks.
- Human-evaluated task scope: VTAB tasks are intended to be solvable from visual input alone, spanning natural images, medical and aerial imagery, distance prediction, and object counting.The benchmark evaluates representative tasks including Pets, DTD, Camelyon, EuroSAT, DMLab, and Clevr-count.
- Human evaluation protocol: Human raters receive 20 random training examples per class and classify 50–100 test images without hints about the task.The protocol totals approximately 1K evaluated images per dataset, except DMLab with 534 images.
- Human evaluation protocol: The human DMLab evaluation measures distance prediction alone because learning both object-type groups from 20 examples per class is impractical.This variant contains three classes differing by object distance while allowing many object types.
- Interpretation: Human evaluation verifies learnability of the task types rather than comparing human and machine performance directly.Protocol differences, including the number of training examples, make relative performance incomparable.
- Results: Human performance is measured with mean-per-class accuracy, and raters outperform random guessing on every evaluated task.Task difficulty varies: Camelyon involves subtle distinctions, DMLab contains noise, and Clevr-count is relatively easy.
D. Alternative Metrics
Alternative evaluation metrics and aggregation schemes produce rankings broadly consistent with mean top-1 accuracy, supporting its use as VTAB’s primary ranking metric.
- Alternative metrics: Top-1 accuracy, mean-per-class accuracy, and Cohen’s quadratic kappa produce broadly similar method rankings across VTAB tasks.Figure 7 reports rankings under the three metrics, with only minor changes for Cohen’s quadratic kappa.
- Alternative metrics: Kendall’s ranking correlation with top-1 accuracy is 1.0 for mean-per-class accuracy and 0.97 for Cohen’s quadratic kappa.These results indicate that the alternative metrics leave the overall conclusions unchanged.
- Aggregation concerns: Mean accuracy can implicitly overweight domains sharing input data, larger task groups, or tasks with greater performance headroom.The authors therefore examine dataset-level, group-level, geometric-mean, and rank-based aggregation strategies.
- Primary metric: VTAB uses unweighted mean top-1 accuracy because it is interpretable and can be computed independently for each method.Rank-based aggregation is less convenient for this purpose despite its generally high agreement.
- Aggregation results: Alternative aggregation strategies generally agree strongly with vanilla mean accuracy, with all Kendall correlations above 0.87 and weighted means and geometric mean above 0.95.The study samples evaluation repetitions to account for training and evaluation stochasticity.
F. Representative Subset of Tasks
A small subset of VTAB tasks can often reproduce full-benchmark model rankings, enabling cheaper early experimentation but with reduced reliability for fine-grained comparisons.
- Subset selection: The authors seek representative five-task subsets to reduce the cost of iterating on new representation-learning methods.They exhaustively search subsets of the 20-task benchmark.
- Subset selection: Many different five-task subsets achieve high rank correlation with the full-benchmark mean, despite containing different tasks.Subset quality is evaluated by comparing model rankings against rankings from all tasks.
- Generalization: Subset selection generalizes to ranking an unseen pair of models with high probability.For each model pair, subset selection uses the remaining models before testing agreement on the held-out pair.
- Practical use: Representative subsets support rapid prototyping before running the full VTAB.The benchmark’s full results remain necessary when distinguishing methods within a similar class.
- Limitations: Repeated iteration on only a few tasks risks meta-overfitting, and subset rankings may be less reliable for fine-grained method comparisons.The reported robustness reflects a diverse set of methods spanning weak, supervised, and combined self-supervised approaches.
H. Heavyweight experiments
Heavyweight adaptation uses broad per-task hyperparameter searches and shows that supervised pre-training improves accuracy and tuning stability, while architecture scaling also raises VTAB performance.
- Comparison with prior work: Literature comparisons are mixed because prior results may use more complex architectures or task-specific logic, such as multiple networks and two-eye information for Retinopathy.The paper reports comparable results overall, but trails highly optimized popular tasks such as CIFAR.
- Heavyweight protocol: The heavyweight sweep performs 100 independent random-search trials over relevant fine-tuning hyperparameters for each downstream task.The search varies batch size, training schedule, and other adaptation settings.
- Hyperparameter sensitivity: Supervised pre-training achieves higher accuracy than training from scratch and much smaller variance across hyperparameter values on CIFAR-100 and DMLab.This pattern holds for both small and full dataset variants in the violin-plot analysis.
- Representation comparisons: SUP-100% versus FROM-SCRATCH shows substantial supervised-pre-training gains on NATURAL datasets and some other tasks.SUP-ROTATION-100% versus SUP-100% performs better on STRUCTURED tasks, indicating task-group-specific differences.
- Architecture scaling: Scaling from ResNet50 to a wider and deeper ResNet152 2x architecture increases VTAB performance, while ImageNet accuracy also rises with greater width or depth.The architecture-scaling analysis compares VTAB performance with ImageNet public-validation accuracy.
M. Budget Analysis
Fine-tuning generally improves top-1 accuracy on both 1000-example and full datasets, with performance typically stabilizing after 1000 steps.
- Performance is usually stable after 1000 fine-tuning steps on the 1000-example datasets.
- More fine-tuning steps are usually associated with higher top-1 accuracy in both dataset settings.
- Performance is usually stable after 1000 fine-tuning steps on the full datasets.
N. Comparison to Visual Decathlon
The study compares VTAB and Visual Decathlon as benchmarks for predicting method performance on unseen tasks. VTAB produces rankings that generally correlate better with unseen-task rankings, including for tasks outside VTAB.
- VTAB is evaluated against Visual Decathlon for comparing adaptation abilities to unseen tasks.
- The authors rank 16 baseline methods on each task, then compare VTAB- and Visual-Decathlon-based rankings with each task’s held-out gold ranking using Kendall’s correlation.
- For most datasets, VTAB rankings correlate better with the gold rankings than Visual Decathlon rankings.
- 0.76 average ranking correlation is achieved by VTAB, compared with 0.70 for Visual Decathlon.