A Universal Representation Transformer Layer for Few-Shot Image Classification

1 Introduction

• Multi-domain few-shot classification must leverage or ignore information from different domains while avoiding cross-domain interference.Each task has only a handful of examples per class.Source: §1 · p. 1
• Meta-Dataset work found that training on all available domains often improved some domains at the expense of others.SUR [7] instead uses hand-crafted feature selection, while its classification procedure remains fixed across tasks.Source: §1 · pp. 1–2
• URT uses attention to retrieve or blend domain-specific backbones into task-adapted representations, learning across few-shot tasks from many domains.The layer is inspired by Transformer networks.Source: §1 · p. 2
• URT achieves top performance on 7 Meta-Dataset sources, outperforming SUR on 4 without impairing accuracy on others.The paper also reports a new state of the art on Meta-Dataset.Source: §1 · p. 2

2.1 Problem Setting

• A few-shot task pairs a support set that defines the task with a query set to classify; episodic meta-learning samples such tasks for training.A common episode selects N classes and K support examples per class.Source: §2.1 · p. 2
• Meta-Dataset spans ten datasets, with eight available for training, and varies the number of classes and examples per class across tasks.Unlike single-domain benchmarks, its training and test tasks vary across domains.Source: §2.1 · pp. 2–3

2.2 Background and Related Work

• Prior approaches include fine-tuning pretrained classifiers and meta-learning methods such as Prototypical Networks, MAML, and CNAPS.Simple CNAPS was reported among the strongest Meta-Dataset methods.Source: §2.2 · p. 3
• A universal representation supports performance across domains; SUR [7] weights domain-specific features per task using support-set loss, without meta-learning.URT contributes evidence that meta-learning can replace SUR’s hand-designed inference procedure and improve performance further.Source: §2.2 · p. 3
• Unlike self-attention over support examples, the paper’s Transformer-inspired approach uses attention to select among domain-specific backbones.The comparison distinguishes URT’s selection target from attention mechanisms used in prior work.Source: §2.2 · p. 3
• Figure 1 depicts a single-head URT layer using four backbones, with colors marking their domain-specific subvectors in the universal representation.Source: Figure 1 · p. 4

3 Universal Representation Transformer Layer

• URT uses each task’s support set to retrieve or blend domain-specific backbones into a task-adapted representation.The method is trained across few-shot tasks from many domains.Source: §3 · p. 4

3.1 Single-Head URT Layer

• For each class, learned queries and domain-specific keys produce attention scores that weight the backbone features; scores are averaged across classes for the support set.The resulting representation can select one backbone or blend several.Source: §3.1 · pp. 4–5

3.2 Multi-Head URT Layer

• Multiple URT heads each compute an adapted representation, and their outputs are concatenated to provide diverse representations.A regularizer discourages duplicate attention scores and encourages focused attention on different domain-specific backbones.Source: §3.2 · p. 5

3.3 Training Strategy

• Training uses episodic tasks and a Prototypical Network objective with negative cosine distance; Table 1 reports mean and 95% confidence intervals over 600 Meta-Dataset tasks.The algorithm updates the URT parameters using query-label prediction loss.Source: §3.1 · p. 5, §3.3 · p. 6, Table 1 · p. 6

4.1 Datasets and Setup

• URT is evaluated on ten Meta-Dataset datasets and additional unseen MNIST, CIFAR-10, and CIFAR-100 tasks, with accuracy averaged over 600 sampled tasks.Tasks vary in class count, shot count, and class imbalance.Source: §4.1 · p. 6

4.2 Comparison with Previous Approaches

• On additional datasets, URT matches SUR overall, leads on MNIST, and trails Simple CNAPS on CIFAR-10 and CIFAR-100.Table 2 reports test performance over 600 few-shot tasks.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 Interpreting and Visualizing Attention by URT

• For seen-domain tasks, one head emphasizes the matching-domain backbone while the other blends related domains; for unseen tasks, one weights ImageNet and the other blends four backbones.The attention scores are averaged across test tasks for each domain.Source: Figure 2 · p. 7, §4.3 · p. 8

4.4 URT using FiLM Modulated Backbones

• URT-pf improves performance on three datasets without sacrificing performance on others and beats BOHB-E on VGG Flower.The comparison uses parametric network family backbones.Source: Table 3 · p. 8, §4.4 · p. 8

4.5 Hyper-Parameter and Ablation Studies

• Removing support-set information from keys is more consequential than removing it from queries, and regularization matters for VGG Flower and Birds.The study ablates query/key transformations and the regularizer.Source: Table 4 · p. 8, §4.5 · p. 8
• Multiple heads produce a large performance jump over one, but too many overfit; two to four heads have similar validation performance and average rank.The head count is selected using Meta-Dataset validation tasks.Source: §4.5 · pp. 8–9, Table 5 · p. 9

• URT sets a new state-of-the-art on Meta-Dataset while improving multi-domain few-shot classification.Source: §5 · p. 9
• URT never performs worse than SUR and is 10× more efficient at inference.Source: §5 · p. 9
• The work identifies combining meta-learning with pre-trained universal representations as a promising direction for new few-shot methods.Source: §5 · p. 9
• Future work could investigate richer universal representations beyond one pre-trained backbone per domain and meta-learners adapted to them.Source: §5 · p. 9

• URT may offer a solution for applications where collecting and sharing data is difficult.Source: Broader Impact · p. 9
• One example is a setting where users have limited private data and classification runs directly and solely on their devices.Source: Broader Impact · p. 9
• Before deployment, analyze and correct potential undesirable biases captured by training dataset sources and pre-trained backbones.Source: Broader Impact · p. 9