PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning a Universal Template for Few-shot Dataset Generalization

Eleni Triantafillou, Hugo Larochelle, Richard Zemel, Vincent Dumoulin

arXiv:2105.07029v2cs.LGcs.CV

TL;DR

Few-shot dataset generalization asks models trained on diverse datasets to learn classes from unseen datasets using only a few examples. FLUTE uses a universal template with task-specific parameters initialized by a learned network and refined by gradient descent; it sets a new Meta-Dataset state of the art with substantially fewer parameters. The authors also identify test-set underspecification and model-selection difficulty as limitations of the setting.

  • Problem

    Few-shot dataset generalization requires learning classes from unseen datasets using few examples, while training on diverse datasets introduces interference and domain-shift challenges.

    Source: §1 · p. 1

  • Method

    FLUTE trains a partially parameterized universal template and adapts task-specific FiLM parameters from a learned, task-dependent initialization using a few gradient steps.

    Source: §1 · pp. 1–2, §3 · p. 3

  • Results

    FLUTE sets a new state of the art on Meta-Dataset, outperforming prior methods by more than 5% while using approximately eight times fewer parameters.

    Source: §1 · p. 2, §7 · p. 9

  • Takeaways & Limitations

    FLUTE combines parameter efficiency with task-level adaptation by blending dataset-trained parameters across representation levels and fine-tuning a small subset of feature-extractor parameters.

    Source: §1 · p. 2

    The setting is underspecified: validation data may not proxy for test data, and solutions performing well on training and validation may behave differently at test time.

    Source: §6 · p. 9

Abstract

from arXiv · show

Few-shot dataset generalization is a challenging variant of the well-studied few-shot classification problem where a diverse training set of several datasets is given, for the purpose of training an adaptable model that can then learn classes from new datasets using only a few examples. To this end, we propose to utilize the diverse training set to construct a universal template: a partial model that can define a wide array of dataset-specialized models, by plugging in appropriate components. For each new few-shot classification problem, our approach therefore only requires inferring a small number of parameters to insert into the universal template. We design a separate network that produces an initialization of those parameters for each given task, and we then fine-tune its proposed initialization via a few steps of gradient descent. Our approach is more parameter-efficient, scalable and adaptable compared to previous methods, and achieves the state-of-the-art on the challenging Meta-Dataset benchmark.

1. Introduction

Few-shot dataset generalization trains on diverse datasets but must adapt from few examples to classes from unseen datasets. FLUTE addresses this with a universal template whose remaining parameters are specialized for each task.

1. Introduction

  • The setting combines data scarcity with the need to avoid interference across heterogeneous training datasets and adapt to test classes from unseen datasets.The authors frame these as challenges resembling multi-task learning and domain generalization.Source: §1 · p. 1
  • A universal template is a partially parameterized model that can define many dataset-specialized models by supplying values for its remaining parameters.The design is intended to retain knowledge from diverse data while supporting diverse task-solving networks.Source: §1 · p. 1
  • The method trains shared convolutional layers across datasets while assigning each dataset separate batch-normalization parameters, then adapts task-specific parameters from a learned initialization.The initialization combines trained dataset-specific parameters, with weights proposed by a separate network.Source: §1 · pp. 1–2
  • FLUTE significantly outperforms prior methods on Meta-Dataset by more than 5% while using approximately eight times fewer parameters.The benchmark uses diverse datasets, including evaluation-only datasets.Source: §1 · p. 2

2. Background

  • At evaluation, new classes come from held-out datasets, and performance is query-set accuracy averaged across tasks sampled from those datasets.Training and validation data are unions of examples from multiple datasets.Source: §2 · p. 2

3. Introducing FLUTE

FLUTE builds a shared convolutional template with dataset-specific FiLM parameters, then initializes and adapts task-specific FiLM parameters for few-shot tasks from unseen datasets. Its training separates general shared weights from dataset specialization, while its Blender network supplies a task-dependent starting point for gradient-based adaptation.

3. Introducing FLUTE

  • FLUTE uses shared convolutional weights with dataset-specific FiLM parameters to define a universal template that represents dataset-specialized feature extractors.The FiLM parameters are the batch-normalization layers’ scaling and shifting coefficients.Source: §3 · pp. 2–3

3.2. Training the Blender network

  • The Blender network initializes unseen-task FiLM parameters as a convex combination of trained sets, weighted by dataset-classifier estimates of compatibility.The classifier’s probabilities for the task’s support data serve as the combination weights.Source: §3 · p. 3, §3.2 · p. 4

3.1. Training the universal and per-dataset parameters

  • All training examples update shared convolutional parameters, while each example updates only the FiLM parameters associated with its dataset.This division is intended to make the shared template general enough for specialization by changing the parameter-light batch-normalization layers.Source: §3.1 · pp. 3–4

3.3. Tackling few-shot test tasks

  • For each unseen-dataset task, FLUTE initializes FiLM parameters from the support set, adapts them by minimizing nearest-centroid loss, then classifies queries with the nearest-centroid classifier.Other parameters remain frozen during adaptation, and the adapted FiLM parameters are specific to that task rather than reused across tasks.Source: §3.3 · p. 4

4. Related Work

FLUTE is related to multi-task, transfer, and few-shot learning, but targets generalization to new datasets with a universal template and task-specific parameter adaptation. It differs from universal-representation approaches in how it constructs features and shares parameters, with test-time computation that does not grow with the number of training datasets.

4. Related Work

  • FLUTE resembles Puigcerver et al. (2021) in adapting source-trained models to downstream tasks, but fine-tunes only a small set of per-task FiLM parameters.The paper frames this parameter-efficient adaptation as suited to data-scarce settings.Source: §4 · p. 5
  • FLUTE shares the task-specific adaptation idea of CAVIA, but targets across-dataset generalization and trains through joint dataset classification rather than meta-learning.The cited work also discusses metric-based and learning-to-learn approaches and shared warp-layers (Flennerhag et al., 2019).Source: §4 · p. 5
  • FLUTE’s dataset-conditioned FiLM differs from few-shot methods that condition on support-set features.The passage also relates dataset conditioning to residual adapters (Rebuffi et al., 2017), which do not address generalization to new datasets at test time.Source: §4 · p. 5
  • Universal representation, as used by Bilen & Vedaldi (2017), refers to rich features for strong multi-dataset performance; SUR and URT build on that approach.The passage notes that compact SUR-pf and URT-pf variants incur a significant accuracy drop.Source: §4 · p. 5
  • Unlike SUR and URT, which select or weight features from dataset-specific extractors, FLUTE uses a template to define dataset-specialized feature extractors.SUR-pf and URT-pf share most extractor parameters, but train shared parameters only on ImageNet; FLUTE trains them jointly across datasets.Source: §4 · pp. 5–6
  • FLUTE’s test-time computation does not grow with training-dataset count M, unlike SUR(-pf) and URT(-pf), which pass data through M extractors.FLUTE uses 2T + 1 feature-extractor passes per task, with T within-task training steps.Source: Table 1 · p. 6

5. Experimental Setup

The experiments use Meta-Dataset to evaluate few-shot learning across diverse training datasets and previously unseen test datasets. The setup distinguishes strong generalization to new datasets from weak generalization to held-out classes of datasets seen during training.

5. Experimental Setup

  • Meta-Dataset provides diverse training datasets and test classes from previously unseen datasets, making it suitable for evaluating few-shot dataset generalization.The setup includes ImageNet, Omniglot, Aircraft, Birds, Flowers, Quickdraw, and Textures for training, with Traffic Signs and MSCOCO among the test datasets.Source: §5 · p. 6
  • The evaluation calls generalization to new datasets “strong,” while held-out classes from training-seen datasets define the easier “weak” setting.The paper also evaluates weak generalization for completeness and compatibility with prior work.Source: §5 · p. 6
  • In weak generalization, FLUTE uses the same procedure as for unseen datasets because the task’s originating training dataset is not disclosed.The Blender is expected to select nearly exclusively the correct dataset if its classifier is accurate.Source: §5 · p. 6
  • The experiments use ResNet-18 features, cosine classifier readout heads during template training, and SGD with momentum for that training phase.The dataset classifier is trained with Adam and converges in around 14K steps.Source: §5 · p. 6

6. Results

On Meta-Dataset, FLUTE outperforms prior methods on few-shot dataset generalization while using substantially fewer parameters. Results also show that initialization and training-data diversity matter, and that the Blender’s weighted initialization may help on unseen datasets.

Performance on Meta-Dataset

  • >5% on Average SG over previous methods; FLUTE also sets Meta-Dataset’s overall state of the art despite competing models having roughly 8 times more parameters.Table 2 reports average query accuracy over 600 test tasks; SG covers unseen datasets.Source: Table 2 · p. 7, §6 · p. 7

The importance of the Blender network

  • ImageNet FiLM initialization performs well on ImageNet-like datasets but poorly on Omniglot, Quickdraw, and Aircraft, while scratch initialization reverses that pattern.The paper reports that the Blender tailors initialization to each task using predicted compatibility with training datasets; Traffic Signs results are omitted for this method because of an evaluation inconsistency.Source: §6 · p. 7

The importance of diverse training data

  • The all-dataset feature extractor outperforms the ImageNet-only extractor across datasets, including on ImageNet test classes, supporting positive transfer from joint training.The ImageNet-only extractor is evaluated with alternative FiLM-parameter initialization schemes.Source: §6 · pp. 7–8, Table 3 · p. 8

The effect of the number of steps to train Ψd∗

  • Fine-tuning effects vary across unseen datasets; support accuracy decreases for CIFAR-100 and MSCOCO under the reported six steps and 0.005 learning rate.The authors note that these hyperparameters may not suit every test task and that validation datasets may be a poor proxy for test datasets.Source: §6 · p. 8, Figure 2 · p. 8

Inspecting the Blender’s proposals

  • For seen datasets, the Blender strongly selects their dedicated FiLM parameters; for unseen datasets, its choices are more variable and may be less peaked.Examples of reasonable unseen-dataset choices include Quickdraw for MNIST and ImageNet for MSCOCO.Source: §6 · p. 8, Figure 3 · p. 9

The effect of taking a convex combination

  • Hard selection and convex combination perform similarly, especially for weak generalization, while combination may yield small gains on strong generalization.The authors adopt the Blender’s convex-combination scheme as the default because it is more general and may improve performance on new datasets.Source: §6 · p. 9

Discussion

  • Choosing a model remains difficult when the validation set is not a good proxy for test datasets, and results show some variation across dataset-classifier checkpoints.The authors report that their conclusions hold across runs.Source: §6 · p. 9

7. Conclusion

FLUTE uses a universal template to specify feature extractors for held-out datasets by inserting FiLM parameters. It estimates those parameters with a few gradient-descent steps from a task-dependent initialization and sets a new state-of-the-art on Meta-Dataset.

7. Conclusion

  • FLUTE sets a new state-of-the-art on Meta-Dataset for few-shot dataset generalization.It significantly outperforms previous methods on the benchmark.Source: §7 · p. 9
  • The universal template specifies a feature extractor for a held-out dataset by inserting an appropriate set of FiLM parameters.The template is a partially parameterized model.Source: §7 · p. 9
  • FLUTE estimates the parameters with a few gradient-descent steps from a task-dependent initialization.Source: §7 · p. 9

Implementation Details

FLUTE jointly trains a feature extractor across datasets with shared convolutional parameters and per-dataset FiLM parameters, then trains a dataset classifier and fine-tunes task parameters during evaluation. The implementation uses a ResNet-18 feature extractor and cosine-classifier readout heads.

Architecture

  • The feature extractor is a ResNet-18, and the Blender’s dataset classifier uses a permutation-invariant set encoder followed by a linear layer.The encoder has five convolutional blocks, each with a 3x3 convolution, 64 channels, batch normalization, ReLU, and 2x2 max-pooling; global average pooling and averaging across examples produce the set encoding.Source: Implementation Details · p. 11

Training the shared and per-dataset parameters

  • Joint training learns a universal template and per-dataset FiLM parameters using a multi-task objective with one classification readout head per training dataset.Each cosine-classifier head has no bias; its input activations and weight rows are L2-normalized, and it uses a learnable softmax temperature.Source: Implementation Details · p. 11
  • The implementation uses SGD with momentum 0.9, a cosine learning-rate schedule with restarts, and weight decay on convolutional and FiLM parameters.The starting learning rate is 0.01; convolutional and FiLM weight decay are 7e-4 and 0.001, respectively.Source: Implementation Details · p. 11
  • During joint training, examples are sampled from ImageNet half the time and uniformly from the other training datasets for the other half.Source: Implementation Details · p. 11

Training the dataset classifier

  • The dataset classifier is trained with Adam and a cosine learning-rate decay, then early-stopped using validation accuracy on held-out classes from the training datasets.Its initial learning rate is 0.001 and decays over 3000 steps; this phase is shorter than feature-extractor training.Source: Implementation Details · p. 11

Fine-tuning Ψd∗

  • Evaluation fine-tunes task-specific FiLM parameters with Adam, without learning-rate decay; the selected setting uses a 0.005 learning rate for six steps.Source: Implementation Details · p. 11

Hypothesis testing

  • Table entries are ranked using 95% confidence-interval tests on pairwise differences in mean accuracy, with statistically tied entries assigned the same rank.Entries tied for the highest rank are bolded in each row.Source: Implementation Details · p. 11

The effect of the number of steps to train Ψd∗

Fine-tuning Ψd∗ produces a less pronounced accuracy increase on seen-dataset episodes than on held-out-dataset episodes. The paper attributes this to the availability of dataset-specific FiLM parameters for seen datasets, assuming accurate dataset classification.

The effect of the number of steps to train Ψd∗

Inpsecting the Blender’s proposals

The Blender’s proposals are computed from each task’s support set and vary across tasks, particularly for unseen datasets. On seen datasets, it almost exclusively selects the FiLM parameters dedicated to that dataset, while dataset-classifier prediction variance differs across held-out datasets.

Inpsecting the Blender’s proposals

  • Blender coefficients are computed from each task’s support set and are not reused across tasks from the same dataset; dataset identity is unknown at test time.Source: Inpsecting the Blender’s proposals · p. 12
  • For seen datasets, the Blender almost exclusively selects that dataset’s dedicated FiLM parameters; unseen-dataset proposals show more variability.The proposals are visualized for 600 test tasks per dataset.Source: Inpsecting the Blender’s proposals · p. 12
  • The dataset classifier has no prediction variance across tasks for the first eight seen-dataset columns, while held-out datasets show varying levels of variance.MNIST has very low variance, whereas Traffic Signs shows especially large variance in its estimate of whether tasks belong to Flower.Source: Inpsecting the Blender’s proposals · p. 12

Additional confidence intervals

Table 4 reproduces Table 3 with 95% confidence intervals.

Additional runs of the dataset classifier network

The authors test how dataset-classifier checkpoints affect FLUTE, both with and without fine-tuning. Across the runs, performance varies more on unseen datasets, but even the worst fine-tuned variant outperforms the previous state of the art on average.

  • Fine-tuning runs use checkpoint-specific learning rates and step counts selected by validation accuracy over 600 episodes per validation dataset.The five checkpoints were selected from separate dataset-classifier training runs with different hyperparameters.Source: Additional runs of the dataset… · p. 12
  • Without fine-tuning, checkpoint results vary by at most 0.1% on average WG, 1.7% on Average SG, and 0.7% overall.The authors report more variation on unseen datasets, where classifier behavior is underspecified because those datasets are absent from training and validation.Source: Additional runs of the dataset… · p. 12
  • Even the worst fine-tuned variant outperforms the previous state of the art on Average SG.The authors report checkpoint-to-checkpoint differences of at most 0.2% on average WG, 2.7% on average SG, and 1% overall.Source: Additional runs of the dataset… · pp. 12–13
  • Figures 4 and 5 visualize seen-dataset support and query accuracy over fine-tuning steps and the variance of Blender coefficients across test episodes, respectively.Figure 5’s coefficients are predicted from each task’s support set rather than reused across tasks from the same dataset.Source: Figure 4 · p. 13, Figure 5 · p. 13
  • The authors argue that few-shot dataset generalization is underspecified by the gap between training and validation data and test data, and recommend reporting performance across several runs.They caution against reporting only the accuracy of the top-performing run.Source: Additional runs of the dataset… · p. 13

A closer look at the comparison between FLUTE and SUR-pf

The comparison examines FLUTE against SUR-pf variants, the Traffic Signs evaluation setting, and a Hard Blender initialization. FLUTE’s reported advantage over SUR-pf remains when the latter is trained on all datasets, while convex-combination and Hard Blender initializations perform similarly.

A closer look at the comparison between FLUTE and SUR-pf

  • FLUTE significantly outperforms the all-dataset SUR-pf variant, especially on strong-generalization tasks.Because the variant shares FLUTE’s training data, the authors suggest FLUTE’s advantage also reflects its inductive bias, not only training on more data.Source: A closer look at the comparison… · p. 14, Table 8 · p. 16

Shuffled Traffic Signs

  • Earlier Meta-Dataset usage examples defaulted to unshuffled within-class examples, making Traffic Signs evaluations more optimistic; this paper reports results using shuffled data.The unshuffled variant is easier because examples occur in 30-image sequences from the same physical sign, making support and query examples more likely to be close.Source: Shuffled Traffic Signs · p. 14
  • The authors report additional results on the easier unshuffled Traffic Signs variant in Tables 9 and 10.Table 10 includes five dataset-classifier checkpoints; Run 5 is the FLUTE result reported in the main paper.Source: Shuffled Traffic Signs · p. 14, Table 10 · p. 16

Hard Blender: using the dataset classifier but without taking a convex combination

  • The Hard Blender selects the most likely training dataset’s FiLM parameters instead of taking a convex combination; the two initialization schemes perform similarly.The authors adopt the convex-combination Blender, which is more general and may slightly benefit some strong-generalization tasks.Source: Hard Blender: using the dataset… · p. 17, Table 11 · p. 18

Related papers

Loading 2105.07029v2…