Source-linked AI summary
Cross-domain Few-shot Learning with Task-specific Adapters
Wei-Hong Li, Xialei Liu, Hakan Bilen
TL;DR
Cross-domain few-shot classification must adapt to unseen classes and domains from few labeled examples, but prior approaches often dynamically predict task-specific weights with an auxiliary network. This paper learns those weights directly from the support set using lightweight residual matrix adapters, which achieve state-of-the-art Meta-Dataset performance with negligible additional parameters. The method is limited by its reliance on existing backbones and fixed adapter structures that may not suit every layer or task.
Problem
Cross-domain few-shot classification must adapt to unseen classes and domains from small support sets, while auxiliary networks face difficulty generalizing across domains and predicting high-dimensional adapter weights.
Method
The method freezes task-agnostic weights and directly optimizes task-specific adapter weights from scratch on the support set, using matrix-form residual adapters across backbone layers.
Results
Lightweight residual adapters achieve state-of-the-art performance on Meta-Dataset with negligible additional parameters and strong adaptation across new tasks and domains.
Takeaways & Limitations
Direct support-set learning with residual adapters provides an efficient approach for adapting few-shot classifiers to new tasks and domains.
Takeaways & Limitations
The method relies on existing ResNet-18 and ResNet-34 backbones and fixed adapter parameterizations that may not be optimal for every layer and task.
Abstract
from arXiv · showhide
In this paper, we look at the problem of cross-domain few-shot classification that aims to learn a classifier from previously unseen classes and domains with few labeled samples. Recent approaches broadly solve this problem by parameterizing their few-shot classifiers with task-agnostic and task-specific weights where the former is typically learned on a large training set and the latter is dynamically predicted through an auxiliary network conditioned on a small support set. In this work, we focus on the estimation of the latter, and propose to learn task-specific weights from scratch directly on a small support set, in contrast to dynamically estimating them. In particular, through systematic analysis, we show that task-specific weights through parametric adapters in matrix form with residual connections to multiple intermediate layers of a backbone network significantly improves the performance of the state-of-the-art models in the Meta-Dataset benchmark with minor additional cost.
1. Introduction
Cross-domain few-shot classification must adapt to unseen classes and domains from small support sets while transferring relevant knowledge across domains. The paper replaces auxiliary-network prediction of task-specific adapter weights with direct support-set learning and evaluates lightweight adapter designs.
- Cross-domain few-shot tasks require learning unseen classes from few samples while selectively transferring knowledge to previously unseen domains.
- Prior methods commonly combine task-agnostic weights learned over multiple tasks with task-specific weights estimated from a small support set.
- Auxiliary networks dynamically predict task-specific adapter weights, but must generalize across unseen tasks and domains while estimating high-dimensional nonlinear feature-space parameters.
- The paper learns lightweight adapter weights directly from scratch on the support set instead of estimating them with an auxiliary network.
- Systematic experiments compare serial and residual connections, matrix and channelwise parameterizations, and alternative task-specific parameter estimation strategies.
- Matrix-form adapters attached to convolutional layers with residual connections improve state-of-the-art performance across most domains, especially unseen Meta-Dataset domains, with negligible computational increase.
2. Method
The method learns task-agnostic representations and adapts them to each target task by optimizing compact task-specific weights on the support set. Residual matrix adapters are attached across backbone layers, followed by a classifier transformation and nearest-centroid classification.
- Few-shot tasks use support samples for adaptation and query samples for classification, with mutually exclusive base and target classes.
- The pipeline first learns a task-agnostic feature extractor from a large dataset, then adapts its representations to target tasks sampled from another dataset.
- Multi-domain learning shares a feature extractor across domains, while URL distills separately trained domain-specific networks into one multi-domain network.
- Task-specific weights are attached directly to the task-agnostic model and estimated from scratch on the support set while the task-agnostic weights remain frozen.
- Adapters transform each backbone layer through either serial composition or parallel residual addition, with the residual configuration performing best when learned during meta-test.
- Adapter transformations use either matrix multiplication with 1 × 1 convolutional kernels or channelwise scaling, with matrix form offering greater power but requiring more parameters.
- The best reported setting uses parallel matrix adapters for feature extraction, followed by a pre-classifier affine transformation and nearest-centroid classification.
3. Experiments
Experiments evaluate the method on Meta-Dataset using frozen task-agnostic ResNet backbones and support-set adaptation, comparing it with state-of-the-art methods and ablations. The method improves especially on unseen domains, while analyses examine optimization, layer placement, parameterization, and efficiency.
- Experimental setup: The evaluation uses Meta-Dataset’s 13 diverse datasets with a ResNet-18 backbone trained on eight subdatasets, whose parameters are frozen as task-agnostic weights.Task-specific weights are learned on support samples during meta-test.
- Comparison to state-of-the-art methods: The method outperforms other methods on 10 of 13 domains, with Average Unseen improving by +7.5 over the second-best approach.Reported gains include +19.5 on Traffic Sign, +8.7 on CIFAR-10, and +6.8 on CIFAR-100.
- Task-specific parameterizations: Compared with auxiliary-network estimation, directly learned residual matrix adapters perform better in almost all cases and use fewer parameters per task.Residual matrix adapters are especially effective, while channelwise adapters provide a performance-computation tradeoff; pre-classifier alignment further improves the best model.
- Further results: In five-shot varying-way experiments, the method remains ahead of URL on unseen domains by +6.2 Average Unseen, while matching URL on seen domains in the one-shot case.The authors attribute the one-shot unseen-domain advantage to residual adapters being less over-fitting.
- Further ablation study: Performance stabilizes after 20 optimization iterations, and attaching residual adapters to more layers improves unseen-domain performance while seen-domain performance remains stable.Adapters on only the last block achieve around 78% average accuracy across domains.
- Further ablation study: Matrix decomposition preserves comparable performance while reducing additional parameters from around 13% to around 4% when N = 32.The decomposition is applied to large residual adapters in the last two blocks.
4. Conclusion and Limitations
Light-weight residual adapters enable strong few-shot adaptation to new tasks and domains with negligible parameter increases, but the method relies on fixed adapter designs and existing backbones.
- Conclusion: Residual adapters achieve strong adaptation to new tasks and domains from only a few samples while attaining state-of-the-art Meta-Dataset performance.The solution remains efficient and adds only a negligible number of parameters across feature extractors.
- Limitations: The method is built on ResNet-18 and ResNet-34 backbones with fixed adapter parameterizations and connection types.These choices may not be optimal for every layer and task in multi-domain few-shot learning.
- Limitations: More flexible adapter structures could be altered and tuned based on the target task.This is identified as a desirable direction for addressing the method's fixed design choices.
A. Dataset
The evaluation uses Meta-Dataset, spanning multiple datasets and both seen- and unseen-domain generalization settings, with randomly sampled few-shot tasks.
- Dataset: Meta-Dataset initially contains ten datasets and was later expanded with MNIST, CIFAR-10, and CIFAR-100.The benchmark includes domains such as ImageNet, Omniglot, Aircraft, Birds, Textures, QuickDraw, Fungi, Flower, Traffic Signs, and MSCOCO.
- Dataset: Training on all datasets uses eight datasets for meta-training and reserves five datasets as unseen domains for meta-test.Evaluation within the meta-training datasets measures generalization to seen domains, while the reserved datasets measure cross-domain generalization.
- Dataset: The ImageNet-only setting trains on ImageNet and evaluates on its test split plus the remaining twelve datasets as unseen domains.
- Dataset: Each dataset is evaluated on 600 randomly sampled tasks with varying numbers of ways and shots, reporting average accuracy and 95% confidence scores.
B.1. Task-agnostic learning
Task-agnostic feature extractors are learned from either multiple domains or a single domain, using established backbones and training protocols.
- Task-agnostic learning: Multi-domain learning trains a shared feature extractor across multiple domains using the protocol of prior work.The passage distinguishes vanilla multiple-domain learning from other multi-domain cases but truncates the detailed formulation.
- Task-agnostic learning: Single-domain learning evaluates ResNet-18 and ResNet-34 feature extractors trained on ImageNet.The resulting backbones are denoted SDL-ResNet-18 and SDL-ResNet-34.
- Task-agnostic learning: The single-domain ResNet-18 uses ImageNet training data, 84 × 84 images, SGD, cosine annealing, momentum 0.9, and weight decay 7 × 10^-4.
B.2. Task-specific learning
Task-specific adaptation learns or predicts adapter parameters on the support set, compares adapter designs, and evaluates residual adapters across backbone baselines.
- Task-specific learning: Adapter parameters α and pre-classifier parameters β are optimized directly on the support set for 40 iterations using Adadelta.β is initialized as an identity matrix, and α uses half β's learning rate.
- Task-specific learning: The experiments compare residual-adapter attachment across SDL-ResNet-18, SDL-ResNet-34, and vanilla multi-domain learning baselines.
- Task-specific learning: Auxiliary-network and direct-adapter approaches are compared with nearest-centroid, Mahalanobis, logistic-regression, SVM, softmax, and k-nearest-neighbors classifiers.The comparison also varies pre-classifier alignment and matrix versus channelwise parameterization.
- Task-specific learning: When α is modulated by an auxiliary network, the task encoder is trained for 10K episodes on eight meta-training domains.Adam with learning rate 1 × 10^-5 is used, with validation every 5K iterations.
C.1. Our method with different feature extractors
The proposed residual adapters remain effective across multiple feature extractors, backbones, and image resolutions, indicating extractor-agnostic adaptation for unseen categories and domains.
- Residual adapters improve feature extractors including SDL-ResNet-18, SDL-ResNet-34, MDL, and URL.The method is evaluated with ImageNet-pretrained single-domain models and multi-domain models.
- The method efficiently adapts models to unseen categories and domains from few support samples.This conclusion is reported across feature extractors with different backbones and image resolutions.
C.2. Task-specific parameterizations
Across Varying-Way Five-Shot and Five-Way One-Shot evaluations, the proposed method outperforms strong baselines on most datasets, including unseen domains.
- Ours outperforms URT and URL on most datasets in Varying-Way Five-Shot and Five-Way One-Shot scenarios.URT and URL surpass Simple CNAPS and SUR in the reported comparisons.
- The evaluations report mean accuracy and 95% confidence intervals for each dataset.The first eight datasets are seen during training, while the last five are unseen test datasets.
C.4. Results evaluated with updated evaluation protocol.
Under the updated Meta-Dataset evaluation protocol, the method retains strong performance and ranks first on average among the compared approaches.
- The updated protocol does not affect the reported results substantially.The authors also standardize the shuffle buffer size to 1000 for evaluation.
- Ours improves over URL on Average Unseen by +7.9 and on Traffic Sign, CIFAR-10, and CIFAR-100 by +20.2, +8.7, and +7.0, respectively.These are reported as significant improvements over the second-best approach.
C.5. Ablation study
Ablations show that the adapter design is stable, effective with few optimization steps, and robust to initialization, while broader layer coverage improves unseen-domain performance and decomposition reduces parameters with modest trade-offs.
- Sensitivity analysis for number of iterations: 20 iterations are sufficient for the proposed method to converge to a stable solution and outperform URL on average performance across domains.The analysis compares 10, 20, 40, and 60 iterations using MDL and URL models.
- Influence of α and β: Both residual adapters α and the linear transformation β improve adaptation, with their combination achieving the best results.Residual adapters particularly improve performance on unseen domains.
- Initialization analysis for adapters: Identity and random initialization obtain similar results, indicating stability across initialization strategies.Identity initialization uses an identity matrix scaled by δ = 0.0001.
- Layer analysis for adapters: Attaching adapters only to block4 obtains around 78% average accuracy across domains and already outperforms URL.Adding adapters to more layers significantly improves unseen-domain performance while seen-domain performance remains stable.
- Decomposing residual adapters: Decomposed residual adapters retain comparable average accuracy with fewer parameters, although unseen-domain accuracy drops slightly as decomposition increases.The drop is especially associated with Traffic Sign when adapters become too small.
- Training time: Meta-training costs are equal to URL, requiring 48 hours in the multi-domain setting and 6 or 33 hours for the single-domain ResNet-18 or ResNet-34 settings.The reported times use one Nvidia V100 GPU.
C.6. Qualitative results
The qualitative analysis compares the proposed method with Simple CNAPS, SUR, URT, and URL across the Meta-Dataset test datasets. The method produces more correct nearest neighbors and adapts to visual variations that distract competing methods.
- C.6. Qualitative results: Figures 12–24 qualitatively compare the proposed method with Simple CNAPS, SUR, URT, and URL across the test datasets.Green denotes correct predictions and red denotes false predictions in each comparison.
- C.6. Qualitative results: The proposed method produces more correct nearest neighbors than the other methods in the qualitative comparisons.
- C.6. Qualitative results: In Traffic Sign, the method retrieves the correct sign despite illumination distortion, whereas other methods mainly retrieve the triangle sign.
- C.6. Qualitative results: In CIFAR-100, the method selects correct shark images while SUR and URT are distracted by the blue background.