PaperlayerView exact revision on arXiv

Source-linked AI summary

CrossTransformers: spatially-aware few-shot transfer

Carl Doersch, Ankush Gupta, Andrew Zisserman

arXiv:2007.11498v5cs.CV

TL;DR

Few-shot transfer degrades under limited labels and task or domain shift because vision representations can lose information beyond the training labels. The paper combines episodic self-supervised learning with spatially aligned CrossTransformers, reporting state-of-the-art performance across nearly every Meta-Dataset dataset while noting important remaining limitations.

  • Problem

    Modern vision systems perform poorly with little labeled data or large distribution shifts, motivating methods that preserve information useful for transfer.

  • Method

    The paper reformulates SimCLR as episodic instance discrimination and introduces CrossTransformers, which align local query-support features before distance-based classification.

  • Results

    The approach achieves state-of-the-art results on nearly every Meta-Dataset dataset, often by large margins.

  • Takeaways & Limitations

    Self-supervised learning and spatially aware local-feature comparisons help representations transfer to unseen classes and domains.

  • Takeaways & Limitations

    The algorithm provides less benefit when less spatial correspondence is available, and current vision systems remain far below human performance for unusual situations.

Abstract

from arXiv · show

Given new tasks with very little data$-$such as new classes in a classification problem or a domain shift in the input$-$performance of modern vision systems degrades remarkably quickly. In this work, we illustrate how the neural network representations which underpin modern vision systems are subject to supervision collapse, whereby they lose any information that is not necessary for performing the training task, including information that may be necessary for transfer to new tasks or domains. We then propose two methods to mitigate this problem. First, we employ self-supervised learning to encourage general-purpose features that transfer better. Second, we propose a novel Transformer based neural network architecture called CrossTransformers, which can take a small number of labeled images and an unlabeled query, find coarse spatial correspondence between the query and the labeled images, and then infer class membership by computing distances between spatially-corresponding features. The result is a classifier that is more robust to task and domain shift, which we demonstrate via state-of-the-art performance on Meta-Dataset, a recent dataset for evaluating transfer from ImageNet to many other vision datasets.

1 Introduction

Modern vision representations transfer poorly when new tasks provide little labeled data or differ substantially in distribution. The paper attributes this to supervision collapse and proposes self-supervised learning plus spatially aware CrossTransformers to improve transfer.

  • With too little labeled data or a large distribution shift, deep-network systems empirically perform poorly on new tasks.
  • Meta-Dataset evaluates few-shot adaptation from ImageNet classes to unseen classes and nine visually distinct fine-grained datasets.Each episode contains a small support set and an unlabeled query set requiring rapid adaptation.
  • Around 50% accuracy for Prototypical Nets contrasts with roughly 84% for a classifier trained on held-out validation classes.The comparison is reported for held-out ImageNet classes in Meta-Dataset’s validation set; chance is roughly 1 in 20.
  • The paper uses SimCLR episodes to encourage embeddings that discriminate between images while remaining invariant to transformations, capturing information beyond classes.SimCLR is reformulated episodically rather than used only as an auxiliary loss.
  • CrossTransformers establish coarse spatial alignment between query and support images, then classify using distances between corresponding local features.The architecture extends Transformers to few-shot fine-grained classification and is intended to make comparisons more agnostic to underlying classes.
  • The paper reports state-of-the-art Meta-Dataset results on nearly every dataset, often by large margins, using self-supervised local features and spatially aware CrossTransformers.

2 Related Work

Related work situates CrossTransformers within meta-learning, attention-based few-shot learning, visual correspondence, and self-supervised transfer. The paper combines these lines through local support-set correspondences and episodic instance discrimination.

  • Few-shot image classification: Meta-learning commonly trains models or update rules through episodes so they can adapt to new data with limited supervision.
  • Attention for few-shot learning: CrossTransformers attend separately to each class’s support set to establish local correspondences, unlike Matching Networks’ attention over the whole support set.
  • Correspondences for visual recognition: Part-based correspondences have been used in deformable-parts models, classification, retrieval, and fine-grained face recognition.
  • Self-supervised learning for few-shot: Self-supervised transfer methods aim to learn representations that capture more than their pretext-task data, and prior work has shown benefits for few-shot learning.The cited prior few-shot methods use self-supervised auxiliary losses rather than integrating instance discrimination episodically.

3 Stopping Collapse: SimCLR Episodes and CrossTransformers

The paper addresses supervision collapse with self-supervised episodic training and CrossTransformers, which preserve transferable information and compare spatially aligned local features. CrossTransformers use attention to build query-aligned class prototypes before distance-based classification.

  • 3.1 Self-supervised training with SimCLR: Self-supervised learning improves embeddings by preserving information beyond the training classes, addressing the limitation that collapsed features cannot recover discarded information.SimCLR uses transformed views of images and encourages representations to distinguish different images, including images within a class.
  • 3.1 Self-supervised training with SimCLR: SimCLR is reformulated as episodic learning rather than an auxiliary loss, allowing it to apply to episodic learners with minimal hyper-parameters.Randomly, 50% of training episodes become SimCLR episodes in which every image is treated as its own class.
  • 3.1 Self-supervised training with SimCLR: In SimCLR episodes, transformed support images define singleton classes, while transformed queries are sampled from the same original support set for |S|-way classification.The original query set is discarded, and each image’s label is its index in the original support set.
  • 3.2 CrossTransformers: CrossTransformers preserve spatial dimensions and use attention to establish soft correspondences between query and support locations, unlike flat Prototypical Net representations.Support images generate keys and values, while query features generate queries; attention is normalized across support images and spatial positions.
  • 3.2 CrossTransformers: For each query location, CrossTransformers aggregate attended support values into a query-aligned class prototype and compute distances between corresponding local features.The resulting scalar distance serves as a negative logit for the class distribution, as in Prototypical Nets.
  • 3.2 CrossTransformers: Shared value heads for query and support images encourage distance-like behavior, while shared key and query heads favor matching corresponding spatial locations.A global feature is also evaluated for datasets with less spatial structure, such as DTD.

4 Experiments

Experiments on Meta-Dataset Train-on-ILSVRC show that SimCLR Episodes and CrossTransformers improve transfer across varied domains and fine-grained tasks, while performance depends on spatial structure and global-feature usefulness.

  • Experimental setup: Meta-Dataset training uses ImageNet’s 712-class train split, then evaluates on 130 held-out ImageNet classes and nine other datasets through few-shot episodes.Test episodes contain 50–500 labeled support images spanning 5–50 classes and an unlabeled query set with 10 images per class.
  • SimCLR Episodes: SimCLR Episodes improve Prototypical Nets on almost all datasets, especially domains distant from ImageNet and tasks requiring distinctions within ImageNet categories.The episodes combine augmentation with instance discrimination; augmentation alone harms performance.
  • Architectural choices: Larger networks and higher resolution provide only small improvements for the Prototypical Nets baseline.The baseline remains below centroid-based state-of-the-art methods that rely more heavily on pretraining and do not use episodic training.
  • CrossTransformers results: CrossTransformers achieve a better average rank than all baselines, with SimCLR Episodes and BOHB-inspired augmentation further improving performance on nearly every dataset.Reported gains over the best baseline include +5% on OmniGlot, +13% on Aircraft, and +5% on Signs.
  • Transfer conditions: CrossTransformers are strongest where test categories have spatial correspondence, whereas DTD textures challenge basic CTX because they provide little such structure.Adding global-feature logistic regression improves DTD but sacrifices performance on some datasets, revealing tension between local and global representations.
  • Spatial correspondence: CrossTransformer attention often matches semantically meaningful parts, including heads, bodies, feet, engines, and strings, even across different fine-grained classes.The inferred matching can be many-to-one rather than strictly one-to-one.

5 Conclusion

The paper identifies supervision collapse as a transfer problem and addresses it with self-supervised learning and spatially aware local-feature matching. The approach improves transfer but remains limited when spatial structure is weak, training categories are useful, or higher-level reasoning is required.

  • Conclusion: Supervision collapse removes intra-class appearance variation that may be necessary for recognizing novel classes.The paper argues that self-supervised learning and CrossTransformers help preserve and use this variation.
  • Conclusion: Self-supervised learning preserves intra-class variation, while CrossTransformers classify with local features that are more likely to generalize.The two techniques target complementary parts of the transfer problem.
  • Limitations: The algorithm provides less benefit when spatial structure is limited, train-time categories are useful, or higher-level reasoning requires conjunctions of multiple objects.The paper leaves selective use of spatial structure as an open problem.

Broader Impact

The algorithm is intended for few-shot recognition in applications that must adapt to unfamiliar objects and new classes. Its factorized approach may also help address dataset biases by separating meaningful traits from spurious correlations.

  • Few-shot recognition can support robotics systems adapting to new objects and photo-organizing software detecting new object classes on-the-fly.The paper connects these applications to the long tail of objects occurring in real scenes.
  • The algorithm factorizes object recognition into feature correspondence and feature comparison sub-problems intended to transfer to new datasets.
  • The authors suggest this factorization may help address dataset biases, including biases involving race, gender, and other attributes.The proposed direction aims to disentangle meaningful traits from spurious correlations.

A Out-of-domain vs. within-domain classification implementation details

The comparison uses Meta-Dataset validation classes to contrast out-of-domain Prototypical Net classification with a supervised within-domain baseline under matched query-set class distributions.

  • 50% accuracy is reported for Prototypical Nets on held-out ImageNet categories, versus 84% for an equivalent classifier trained on those validation classes.The passage frames these as out-of-domain and within-domain comparisons, respectively.
  • 84.2% top-1 accuracy was obtained by a supervised ResNet-34 baseline after restricting its logits to categories present in Meta-Dataset query sets.The baseline used a ResNet-34 trained on all ImageNet classes and matched the Prototypical Net chance distribution.
  • 46.4% accuracy was obtained by the reimplemented Prototypical Net on the Meta-Dataset validation classes.The implementation used 224×224 inputs, ResNet-34, and Normalized SGD.

B Supervision Collapse: nearest neighbor experiments

Nearest-neighbor analysis exposes supervision collapse in default Prototypical Net representations, which are poorly comparable across episodes and retrieve semantically unreliable neighbors.

  • Default Prototypical Net representations are comparable only within a single episode, complicating nearest-neighbor analysis across images.The passage attributes this to episodic training and Batch Norm communication within support sets.
  • Randomly grouped ImageNet training images produce virtually meaningless representations when passed through the default Prototypical Net.The retrieval set included images from both Meta-Dataset ImageNet training and test splits.
  • Training on episodes sampled from all ImageNet categories and replacing Batch Norm with Layer Norm are proposed to reduce this collapse.Such episodes can contain widely different categories, including cars and insects.
  • Test-set queries again produce poor nearest neighbors under the same retrieval procedure, consistent with supervision collapse.The reported experiment uses test-set classes as queries.
  • Figure 4 summarizes 1000 test queries using histograms of top-nine retrieval counts for same-class, training-set, and most frequent training-class neighbors.The x-axis is the retrieval count per query, and bar height is the number of queries with that count.

B.1 Supervision collapse: quantitative analysis

Quantitative nearest-neighbor statistics show that Prototypical Net embeddings frequently miss the correct class and overconcentrate similarity on training categories; SimCLR episodes improve but do not eliminate this pattern.

  • Over 60% of queries had zero nearest neighbors from the correct category, despite 130 correct-category images being present in the retrieval set.The analysis used 1000 Meta-Dataset test queries.
  • 55.3% of queries had at least two neighbors from one training class, compared with 34.1% having at least one neighbor from the correct validation class.The training-class concentration was more frequent than retrieving even one correct-class neighbor.
  • With 50% SimCLR episodes, 43.3% of queries had at least two neighbors from one training class and 48.8% had at least one correct-class neighbor.These results indicate improvement, while the passage states that supervision collapse remains far from solved.

C.1 Training

Training uses specific architectural, optimization, normalization, resolution, and augmentation choices, while CrossTransformer variants combine spatial correspondence with optional globally pooled features and logistic regression.

  • Pretraining: CrossTransformer experiments use no pretraining, whereas Prototypical Net experiments use pretrained representations for consistency with prior work.Prototypical Net pretraining performs direct classification on training categories before episodic training.
  • Training configuration: The main training departures include ResNet-34, 224×224 inputs, normalized gradient descent, random ImageNet category episodes, and no pretraining.The implementation also uses test-mode Batch Norm statistics at test time.
  • Compute: CrossTransformer attention maps require substantial memory, motivating distribution across eight NVIDIA V-100 GPUs and gradient rematerialization.The attention maps contain all pairs of spatial positions between query and support images.
  • Feature-map resolution: Increasing CrossTransformer feature-map resolution from 7×7 to 14×14 via dilation typically provides a small performance boost and almost never harms performance.The largest gains occur on datasets with fine spatial structure, including Aircraft, Birds, and Fungi.
  • Auxiliary loss: The globally pooled auxiliary classification loss provides benefits on some datasets, notably DTD, but not when the embedding network is trained purely as a CrossTransformer.The auxiliary classifier targets the 712 ImageNet training categories.

D Correspondence visualization

CrossTransformers infer spatial correspondences across diverse datasets, often matching semantically related parts despite variation in pose, shape, and appearance. The visualizations also expose cases where correspondences are ambiguous or overly confident.

  • Correspondence visualization: Attention maps show semantically meaningful correspondences across all 10 Meta-Dataset evaluation datasets, often with one query location matching multiple support locations.The visualizations include both 7×7 and 14×14 attention maps for CTX7 and CTX14.
  • Failure cases: Failure cases include confident mismatches, missed parts, location-based matches without clear correspondence, and inaccurate sub-part matching under large shape variation.These cases identify challenging image pairs and motivate fallback to global comparisons or improved correspondence modeling.
  • Five-shot evaluation: The five-shot evaluation fixes exactly five examples per category and produces class-balanced support and query sets.This setting is introduced to aid interpretability alongside the standard Meta-Dataset evaluation.

F Confidence intervals

The paper reports confidence intervals alongside mean accuracies for its quantitative experiments, including five-shot Meta-Dataset evaluations and multiple CTX variants. The reported rows cover CTX models, self-supervised-learning variants, augmentation, learning-rate choices, and ProtoNets baselines.

  • Confidence intervals: Confidence intervals accompany mean accuracy for most experiments, enabling comparisons across the reported models.The intervals summarize episode-to-episode variability in accuracy across 600 test episodes.
  • Five-shot results: Five-shot results report accuracy (%) ± confidence (%) for CTX models with the 14×14 feature grid across Meta-Dataset datasets.The intervals are standard errors computed from variability across 600 test episodes.
  • Experimental settings: The reported model configurations use 224 resolution, ResNet34, and exponential moving average settings unless otherwise specified.Test-time batch normalization is noted as an implementation condition.
  • Reported model variants: The quantitative rows compare CTX variants with SimCLR episodes, augmentation, learning-rate changes, and ProtoNets configurations.The listed model rows include CTX7, CTX14, self-supervised variants, Prototypical Networks, and ProtoNets variants.
Loading 2007.11498v5…