PaperlayerView exact revision on arXiv

Source-linked AI summary

Are We Really Doing Few-Shot Learning? A Critical Examination of Pre-Training Assumptions

Alejandro Galan-Cuenca, Marcelo Saval-Calvo, Antonio Javier Gallego

arXiv:2609.10851v1cs.CVcs.AIcs.LG

TL;DR

Few-shot benchmarks often use class-disjoint pre-training data from the target’s visual domain, raising whether they truly measure low-data learning. The paper compares pre-training regimes, label availability, and source-selection strategies across architectures and datasets. It finds that in-domain pre-training produces an optimistic advantage, while label-free OOD pre-training and descriptor-based source selection offer practical alternatives.

  • Problem

    Class disjointness alone may not define a genuinely low-data problem because standard pre-training and target episodes can share visual and distributional properties.

  • Method

    The paper compares no pre-training, in-domain pre-training, supervised OOD pre-training, and label-free OOD pre-training across eight datasets and three FSL architectures, while testing adaptation and source selection.

  • Results

    In-domain pre-training improves over No-PT by 33.41 pp. on average versus 23.75 pp. for supervised OOD-PT, while UIAug gains 27.71 pp. versus 27.97 pp. for supervised OOD-PT.

  • Takeaways & Limitations

    Realistic FSL evaluation should consider source–target domain compatibility alongside class separation, and label-free pre-training can broaden usable auxiliary data.

  • Takeaways & Limitations

    Future work should test larger, more heterogeneous benchmarks, stronger unsupervised methods, and protocols that explicitly control domain overlap.

Abstract

from arXiv · show

Few-shot learning is commonly evaluated under protocols that pre-train a model on a large auxiliary set whose classes are disjoint from the target episodes yet drawn from the same visual domain. This paper examines whether such protocols truly reflect low-data learning. We systematically compare no pre-training, class-disjoint in-domain pre-training, supervised out-of-domain pre-training, and label-free out-of-domain pre-training across eight datasets, three few-shot architectures, and multiple way-shot settings. Our results show that class disjointness alone is insufficient to remove the influence of target-domain data. In-domain pre-training improves over no pre-training by 33.41 percentage points on average, whereas supervised out-of-domain pre-training yields 23.75 percentage points, revealing a 9.66-point optimistic bias associated with domain overlap. Although out-of-domain pre-training is more realistic in applications where target-domain data are scarce, its effectiveness depends strongly on the compatibility between source and target domains. We further show that labeled source data are not strictly required, with an augmentation-based label-free strategy reaching an average gain of 27.71 percentage points and closely matching supervised out-of-domain pre-training at 27.97 percentage points. Finally, we introduce a descriptor-based source-selection strategy that estimates source-domain suitability before pre-training, reaching a median gap of only 1.37 percentage points to oracle selection. These findings highlight the need to move beyond in-domain pre-training as the default few-shot evaluation protocol, since it can overestimate performance in realistic scenarios where target-domain data are scarce.

1 Introduction

Standard few-shot benchmarks often pre-train on class-disjoint data from the same visual domain, which can make limited-data learning appear easier than it is. The paper compares in-domain, out-of-domain, and label-free alternatives to expose this bias and improve evaluation realism.

  • Motivation: Class-disjoint in-domain pre-training still shares visual characteristics with target episodes, reducing task difficulty and introducing optimistic bias.Standard benchmarks use disjoint classes but retain shared domain properties, so the setup resembles learning under an i.i.d. distribution more than realistic target-domain scarcity.
  • Approach: The study evaluates No-PT, ID-PT, and OOD-PT to establish a more realistic few-shot training and evaluation framework.OOD pre-training reflects settings where abundant labeled in-domain auxiliary data are unavailable and the pre-training distribution may differ from the target.
  • Empirical motivation: 62.3% ID-PT accuracy versus 43.4% OOD-PT average accuracy in Omniglot 5-way 1-shot shows the advantage of in-domain pre-training.In 20-way 5-shot, ID-PT reaches 61.7% versus 53.8% for OOD-PT, with OOD source accuracies ranging from 45.5% to 72.6%.
  • Additional strategies: The paper also studies ID adaptation, label-free pre-training, and domain-similarity measures for selecting suitable OOD sources before pre-training.These strategies target post-transfer mismatch, unavailable source labels, and poor source-domain choices, respectively.
  • Experimental scope: The experiments span Matching Networks, Prototypical Networks, Relation Networks, and eight datasets covering standard benchmarks and additional text, music, and medical-image domains.This design evaluates source–target combinations across multiple architectures and visual domains.

2 Related Work

Related work has addressed few-shot learning through meta-learning, episodic training, and cross-domain transfer, but common cross-dataset protocols may not ensure genuinely out-of-domain evaluation. This paper instead studies the pre-training regime, source choice, source-label availability, and adaptation directly.

  • FSL methods: Meta-learning methods include metric-based architectures that infer query labels from support-set comparisons, while episodic training simulates few-shot evaluation through sampled support and query sets.The paper’s architectural evaluation focuses on Matching Networks, Prototypical Networks, and Relation Networks.
  • In-domain benchmarks: Standard FSL benchmarks use abundant labeled class-disjoint splits from one dataset, preserving visual and semantic similarity between pre-training and target data.This creates an in-domain class-disjoint pre-training regime despite non-overlapping classes.
  • Cross-domain FSL: Cross-Domain FSL studies source and target tasks from different domains, using methods such as invariant representations, disentanglement, cross-domain distances, and multi-source training.Cross-dataset FSL is a common implementation of this setting, typically with disjoint label spaces.
  • Limitations of prior protocols: Different datasets do not guarantee genuine OOD transfer because shared parent collections or semantic overlap can preserve similar visual statistics.miniImageNet and tieredImageNet, for example, both derive from ImageNet, so their apparent dataset shift may remain largely in-domain.
  • Paper perspective: The paper studies ID, truly OOD, and no pre-training while analyzing source choice, source-label availability, and IDA rather than proposing a new cross-domain meta-learner.This perspective treats cross-domain pre-training as the practical default when suitable in-domain data are unavailable.

3 Method

The method formalizes few-shot episodes, contrasts in-domain and out-of-domain pre-training under domain shift, and introduces label-free pre-training and pre-training source-selection strategies.

  • 3.1 Problem formulation: Few-shot episodes contain n target classes with k labeled support examples per class, while disjoint query samples measure performance on those classes.The support set is intended to be the only labeled target-domain information available within an episode.
  • 3.1 Problem formulation: The predictor combines a shared representation fθ with a support-conditioned classifier, covering Matching, Prototypical, and Relation Networks.These metric-based architectures differ in how they compare query and support embeddings.
  • 3.2 Pre-training regimes: ID-PT uses labeled, class-disjoint data from the target visual domain, whereas OOD-PT uses a source with both disjoint labels and a different input distribution.The paper treats ID-PT as closer to an i.i.d. split and OOD-PT as a more realistic, challenging setting.
  • 3.4.1 Clustering-based pseudo-labels: UCKM clusters unlabeled source images into pseudo-classes, providing episodic supervision that encourages the encoder to separate distinct visual groups.The resulting groups may reflect texture, color, or other latent visual attributes rather than human-interpretable categories.
  • 3.4.2 Instance-based pseudo-labeling via augmentations: UIAug derives supervision from augmented views of unlabeled source instances while preserving the standard n-way k-shot episodic structure.The strategy is inspired by instance discrimination but constructs episodes from augmentation-based pseudo-classes.
  • 3.5 Source-domain selection prior to pre-training: Source selection ranks candidate OOD domains using pre-training-independent domain-similarity descriptors, avoiding exhaustive pre-training and evaluation of every candidate.Descriptor associations with transfer gains are calibrated offline, after which a new target requires only descriptor computation.

4 Experimental setup

The evaluation spans heterogeneous datasets, few-shot architectures, episodic settings, and pre-training regimes under standardized experimental conditions. Accuracy gains are measured relative to no pre-training, with adaptation effects isolated separately.

  • 4.1 Datasets: Eight datasets cover music notation, handwritten symbols, medical imaging, and natural images, enabling varied source–target domain comparisons.The datasets include Capitan, Greek, Egyptian, TKH, Omniglot, OrganAMNIST, MiniImageNet, and CIFAR-FS.
  • 4.1 Datasets: For datasets without original class-disjoint splits, each bootstrap run assigns approximately 80% of classes to meta-training and 20% to meta-testing.Target classes are unseen during pre-training.
  • 4.2 Architectures: Three metric-based architectures—Matching, Prototypical, and Relation Networks—provide a common basis for comparing pre-training regimes.
  • 4.2 Evaluation protocol: Experiments evaluate 5- and 20-way classification with 1, 5, and 10 shots across 4,000 test episodes and five bootstrap runs.All compared methods use identical experiment sets and class-disjoint partitions.
  • 4.3 Evaluation metrics: Accuracy gain is defined as mean query accuracy after pre-training minus the corresponding No-PT accuracy, with positive values indicating beneficial transfer.The evaluation also measures adaptation gain after episodic support-set adaptation.

5 Results

Across the experiments, in-domain pre-training provides the largest gains but benefits from domain overlap, while out-of-domain methods are more realistic and sensitive to source–target compatibility. Label-free augmentation nearly matches supervised transfer, and descriptor combinations support effective source selection.

  • 5.1.1 ID-PT vs. OOD-PT: 33.41 pp. versus 23.75 pp. is the average gain over No-PT for ID-PT and supervised OOD-PT, respectively, producing a 9.66 pp. gap.The results identify a substantial advantage for class-disjoint pre-training within the target domain.
  • 5.1.2 Effect of IDA: 7.69 pp. is the average IDA gain under supervised OOD-PT, rising from 3.80 pp. in 1-shot to 10.15 pp. in 10-shot.The largest gains occur mainly on domain-specific datasets.
  • 5.1.2 Effect of IDA: 1.37 pp. is the average IDA gain under ID-PT, including a −0.45 pp. average effect in 1-shot and 3.19 pp. in 10-shot.The reported pattern suggests limited additional information from adapting when pre-training already uses the target domain.
  • 5.1.3 Label-Free OOD-PT: Kc = 1000 yields the largest UCKM gains across all k-shot settings and both IDA configurations, while adaptation reduces sensitivity to cluster count.Without IDA, worst-to-best gaps reach 10.16, 11.73, and 18.34 pp. for 1-, 5-, and 10-shot.
  • 5.1.3 Label-Free OOD-PT: 27.71 pp. versus 27.97 pp. is the mean gain for UIAug and supervised OOD-PT, while UCKM reaches 24.98 pp.UIAug therefore recovers nearly all the benefit of source labels in the reported comparisons.
  • 5.1.3 Label-Free OOD-PT: UIAug reaches 33.59 pp. with best-source selection, compared with 33.40 pp. for supervised OOD-PT and 31.12 pp. for UCKM.The average-versus-best-source gap confirms strong sensitivity to source–target compatibility.
  • 5.2.2 Factor-analysis composite score: A 1.37 pp. median oracle gap makes the Best factor more reliable for source selection than any individual descriptor family.Individual descriptor families have median gaps above 8 pp.

6 Conclusions and Future Work

The paper finds that in-domain pre-training can overestimate few-shot performance, while out-of-domain and label-free alternatives offer more realistic options. It also shows that source selection can reduce transfer risk, although broader evaluation remains needed.

  • 6 Conclusions and Future Work: 33.41 pp. versus 23.75 pp. average gains over No-PT reveal a 9.66 pp. optimistic advantage for ID-PT over supervised OOD-PT.The result identifies pre-training regime and domain overlap as major determinants of few-shot accuracy.
  • 6 Conclusions and Future Work: OOD-PT generally outperforms No-PT, but its effectiveness depends on source–target compatibility and IDA only partially mitigates domain shift.The paper therefore treats realistic evaluation as requiring both class separation and domain compatibility.
  • 6 Conclusions and Future Work: 27.71 pp. versus 27.97 pp. average gains show that UIAug nearly matches supervised OOD-PT without labeled auxiliary data.This expands the datasets that can serve as pre-training sources when annotations are unavailable.
  • 6 Conclusions and Future Work: A descriptor-based Best factor selects suitable sources with a 1.37 pp. median oracle gap and outperforms the worst source by up to 39.8 pp. without IDA.The strategy estimates source suitability before pre-training rather than requiring training on every candidate.
  • 6 Conclusions and Future Work: Future work should test larger, more heterogeneous benchmarks, stronger unsupervised methods, and protocols that explicitly control domain overlap.These directions define the paper’s stated scope boundary.

A Similarity metrics

The similarity framework represents each source–target pair with standardized descriptors and organizes them into complementary families. These descriptors support both individual scores and a factor-analysis score for estimating compatibility.

  • A Similarity metrics: Each source–target pair is represented by a standardized vector of J descriptors, with dissimilarities sign-aligned so larger values indicate higher expected compatibility.The vector feeds both individual-descriptor scores and the factor-analysis score.
  • A Similarity metrics: The framework organizes descriptors into four families: dataset-level statistics, representation-space overlap, distance-based compactness, and reconstruction-based mismatch.The families capture complementary properties of source–target suitability.

A.1 Dataset-level statistics

Dataset-level statistics compare broad visual and structural properties of source and target domains. Numerical differences and semantic-category mismatch provide simple compatibility descriptors, but they do not measure feature-space overlap.

  • A.1 Dataset-level statistics: Dataset-level statistics combine brightness, sharpness, colorfulness, class count, sample count, samples per class, and semantic domain category.These descriptors capture global visual and dataset-structure differences.
  • A.1 Dataset-level statistics: Numerical properties use absolute source–target differences, while semantic categories use a binary mismatch indicator.The category indicator is 0 for matching categories and 1 otherwise.
  • A.1 Dataset-level statistics: These descriptors do not measure source–target overlap or separability in feature space, motivating representation-based descriptors.Their role is therefore complementary rather than comprehensive.

A.2 Representation-space overlap

Representation-space descriptors compare source and target samples using a fixed encoder and treat the target as the reference distribution. The framework combines fidelity, coverage, density, diversity, and recall-oriented measures across generic and task-adapted embeddings.

  • A.2 Representation-space overlap: Feature-based descriptors treat the target domain as the reference distribution and the candidate source as the evaluated distribution.This reframes dataset-auditing measures as source–target compatibility scores.
  • A.2 Representation-space overlap: The descriptor set includes Authenticity, Coverage, Density, GIQA-DS, GIQA-QS, Improved Precision, and Improved Recall.These measures capture related notions of fidelity and coverage using different estimators and aggregation rules.
  • A.2 Representation-space overlap: Descriptors are computed in generic DINOv2 feature spaces and task-adapted representations extracted from the few-shot architectures.The task-adapted encoder is trained using only the target domain’s n-way k-shot support set before extracting representations.

A.3 Distance-based compactness

Distance-based descriptors evaluate candidate sources through a target-adapted few-shot representation, measuring separation, compactness, and source classification behavior. Together, these descriptors characterize how well source samples align with target-induced class structure.

  • A.3 Distance-based compactness: Target-adapted embeddings support three distance-based descriptors that measure source-domain organization relative to the target representation.The descriptors use a fixed few-shot model trained on the target support set to evaluate candidate source samples.
  • A.3 Distance-based compactness: The near-to-far ratio contrasts lower-half and upper-half distances, with lower values indicating stronger separation between nearby and distant candidate classes.The ratio is averaged over query samples and source episodes after splitting distances at the empirical median.
  • A.3 Distance-based compactness: Nearest-prototype compactness aggregates each query’s minimum distance, where lower values indicate source samples lie closer to at least one source representative.
  • A.3 Distance-based compactness: Source accuracy complements geometric descriptors by testing whether target-adapted compactness translates into correct source-domain classification.It uses ground-truth labels for supervised OOD pre-training and pseudo-labels for label-free variants.

A.4 Reconstruction-based mismatch

Reconstruction-based mismatch estimates source–target visual compatibility by training an autoencoder on the target support set and evaluating candidate-source reconstruction error. Lower error indicates that source samples fit reconstruction patterns learned from the target domain.

  • A.4 Reconstruction-based mismatch: A target-support autoencoder estimates source–target visual mismatch by reconstructing candidate-source images as an anomaly-detection problem.The autoencoder is trained without target labels and then kept fixed for evaluating all candidate source domains.
  • A.4 Reconstruction-based mismatch: Low reconstruction error indicates that source samples are compatible with visual patterns learned from the target support set.
  • A.4 Reconstruction-based mismatch: The implementation uses a grayscale convolutional autoencoder with four encoder and four decoder convolutional stages and a 512-dimensional latent representation.

B Robustness of UIAug without IDA

Removing IDA lowers absolute gains but preserves the relative ordering of the pre-training strategies. UIAug remains close to supervised OOD-PT and can slightly exceed it when the best OOD source is used.

  • B Robustness of UIAug without IDA: 20.35 percentage points: supervised OOD-PT’s mean gain without IDA remains close to UIAug’s 19.78 percentage points.The comparison averages across source–target combinations and experimental configurations.
  • B Robustness of UIAug without IDA: Removing IDA reduces absolute gains relative to the main analysis while largely preserving the relative behavior of the three pre-training strategies.
  • B Robustness of UIAug without IDA: 45.78 percentage points: UIAug’s best-source average gain slightly exceeds supervised OOD-PT’s 45.02 percentage points and exceeds UCKM’s 40.04 percentage points.This comparison uses the best OOD source for each target.

C Descriptor-level analysis of source selection

Descriptor-level analysis shows that the Best factor provides the most accurate source selection, while conceptual-domain and representation-coverage descriptors are useful but more variable. The evaluation compares selected sources against oracle sources using accuracy-gain gaps.

  • C Descriptor-level analysis of source selection: 1.37 pp: Factor 1, the Best factor, achieves the lowest overall median gap to oracle source selection.The gap measures downstream accuracy-gain difference between the selected source and the best-performing candidate.
  • C Descriptor-level analysis of source selection: 2.41 pp: Conceptual domain and DINOv2 Coverage have the lowest individual-descriptor median gaps, followed by FSL-encoder GIQA-DS at 4.02 pp.
  • C Descriptor-level analysis of source selection: Conceptual-domain ranking is comparatively robust because it prioritizes sources from the target’s high-level semantic category when available.
  • C Descriptor-level analysis of source selection: Representation-based descriptors are informative because source coverage of the target representation space relates to transfer suitability, but their variability is higher across targets.
Loading 2609.10851v1…