Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation

Hung-Yu Tseng; Hsin-Ying Lee; Jia-Bin Huang; Ming-Hsuan Yang

Source-linked AI summary

Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation

Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, Ming-Hsuan Yang

arXiv:2001.08735v3Jan 23, 2020cs.CVcs.LG

TL;DR

Metric-based few-shot classifiers struggle when feature distributions shift between training and unseen domains. The paper adds feature-wise affine transformations and learns their hyper-parameters to simulate domain variation, yielding consistent improvements across metric-based models under domain shift.

Problem
Metric-based few-shot classifiers may fail to generalize to unseen domains because feature distributions differ across domains.
Method
The method inserts feature-wise transformation layers that simulate feature distributions and uses learning-to-learn optimization for their hyper-parameters.
Results
The feature-wise transformation layer applies to various metric-based models and provides consistent few-shot classification improvements under domain shift.
Takeaways & Limitations
Feature-wise transformations offer a method-agnostic way to enhance metric-based few-shot classification frameworks under domain shifts.
Takeaways & Limitations
The learning-to-learn optimization may not find the global minimum, while manual alternatives require meticulous and computationally expensive hyper-parameter tuning.

Abstract

from arXiv · show

Few-shot classification aims to recognize novel categories with only few labeled images in each class. Existing metric-based few-shot classification algorithms predict categories by comparing the feature embeddings of query images with those from a few labeled images (support examples) using a learned metric function. While promising performance has been demonstrated, these methods often fail to generalize to unseen domains due to large discrepancy of the feature distribution across domains. In this work, we address the problem of few-shot classification under domain shifts for metric-based methods. Our core idea is to use feature-wise transformation layers for augmenting the image features using affine transforms to simulate various feature distributions under different domains in the training stage. To capture variations of the feature distributions under different domains, we further apply a learning-to-learn approach to search for the hyper-parameters of the feature-wise transformation layers. We conduct extensive experiments and ablation studies under the domain generalization setting using five few-shot classification datasets: mini-ImageNet, CUB, Cars, Places, and Plantae. Experimental results demonstrate that the proposed feature-wise transformation layer is applicable to various metric-based models, and provides consistent improvements on the few-shot classification performance under domain shift.

1 INTRODUCTION

The paper targets few-shot classification under domain generalization, where feature distributions differ between seen and unseen domains. It proposes feature-wise transformations and learning-to-learn hyper-parameter optimization to improve metric-based model generalization.

Few-shot classification recognizes novel categories from query images using only a few labeled support examples per class.
Domain shifts make metric-based models vulnerable because unseen-domain feature distributions differ significantly from those in seen domains.
Feature-wise transformation layers apply affine activation modulation to simulate diverse image-feature distributions during training.
A learning-to-learn algorithm optimizes transformation hyper-parameters to capture feature-distribution variation across domains instead of relying on exhaustive hand-tuning.
Experiments across MatchingNet, RelationNet, and Graph Neural Networks show improved generalization to unseen domains, with further gains from the learning-to-learn scheme.

2 RELATED WORK

Related work covers metric-based few-shot learning, domain adaptation and generalization, learned augmentation, conditional normalization, and neural-network regularization. The proposed method is positioned as feature-distribution augmentation for domain-shifted few-shot classification.

Few-shot classification: Metric-based few-shot methods encode labeled and unlabeled images, then classify unlabeled images using a learned metric function.
Domain adaptation: Domain adaptation aligns source and target knowledge but commonly requires unlabeled target images and is less suited to novel categories.
Domain generalization: Domain generalization learns from seen domains and evaluates on unseen domains without accessing unseen-domain instances during training.
Learning-based data augmentation: Learned data augmentation increases training diversity by optimizing augmentation strategies rather than using only handcrafted transformations.
Conditional normalization: Conditional normalization modulates activations through learned affine transformations conditioned on external data, especially for style transfer and image synthesis.
Regularization for neural networks: Random affine modulation of intermediate feature activations can also regularize neural-network training.

3 METHODOLOGY

The method targets domain generalization in metric-based few-shot classification by modeling feature-distribution differences across seen and unseen domains. It inserts feature-wise affine transformations into the encoder and learns their hyper-parameters through pseudo-domain evaluation.

3.1 PRELIMINARIES: Metric-based few-shot classification uses a feature encoder and metric function to predict query categories from support labels and encoded support and query images.The training objective is the classification loss on query images.
3.2 FEATURE-WISE TRANSFORMATION LAYER: The feature-wise transformation layers are method-agnostic and can be applied to different metric-based few-shot classification approaches.The paper identifies MatchingNet, RelationNet, and GNN as example metric functions with different design choices.
3.1 PRELIMINARIES: Domain generalization trains on multiple seen domains so a metric-based model can generalize to an unseen domain despite differing feature distributions.The proposed setting uses few-shot classification tasks grouped by domain.
3.2 FEATURE-WISE TRANSFORMATION LAYER: Feature-wise transformation layers augment intermediate encoder activations with affine scaling and bias to simulate feature distributions from various domains.The layers are inserted at multiple levels of the feature encoder, with scaling and bias terms sampled from Gaussian distributions parameterized by θγ and θβ.
3.3 LEARNING THE FEATURE-WISE TRANSFORMATION LAYERS: A learning-to-learn procedure optimizes transformation hyper-parameters by training the metric model on a pseudo-seen domain and evaluating it on a pseudo-unseen domain.Each iteration samples non-overlapping pseudo-seen and pseudo-unseen domains; the model is updated first, then transformation parameters are optimized using pseudo-unseen classification loss.
3.3 LEARNING THE FEATURE-WISE TRANSFORMATION LAYERS: The metric-based model and feature-wise transformation layers are jointly optimized during training.The transformation hyper-parameters are updated because pseudo-unseen loss reflects their effectiveness.

4 EXPERIMENTAL RESULTS

Experiments evaluate feature-wise transformation layers with three metric-based frameworks across seen and unseen domains, including a learning-to-learn variant. The results indicate reduced domain gaps and improved generalization, while learned transformations better capture cross-domain feature variation.

Experimental setup: Experiments apply feature-wise transformations to MatchingNet, RelationNet, and GNN under domain-generalization settings.The study uses five datasets and evaluates models trained on mini-ImageNet or multiple domains.
Feature-wise transformation with manual parameter tuning: Table 1 compares baseline models with feature-wise transformation layers using empirically determined hyper-parameters.The transformations are inserted after the last batch-normalization layer of every residual block.
Experimental setup: Models trained on mini-ImageNet are evaluated on CUB, Cars, Places, and Plantae as unseen domains.The mini-ImageNet domain is used for training, with four other domains used for evaluation.
Generalization from multiple domains: In leave-one-out experiments, learning-to-learn transformations outperform pre-determined transformations because they better capture feature-distribution variation across domains.FT denotes pre-determined transformations, while LFT denotes transformations optimized by the learning-to-learn algorithm.
Generalization from multiple domains: The hyper-parameter search spans a 1920-dimensional space, making manual optimization meticulous and computationally expensive.For n transformation layers, the search dimension is (c1 + c2 · · · + cn) × 2.
Generalization from multiple domains: T-SNE visualizations show that feature-wise transformations reduce distances between features from different domains, with learned transformations further closing the domain gap.The comparison covers the original encoder, pre-determined transformations, and learning-to-learn transformations.
Generalization from multiple domains: Scaling activations tend to become smaller in deeper layers, whereas layer depth has no apparent effect on bias-term distributions.These patterns are visualized using softplus(θγ) and softplus(θβ).

5 CONCLUSIONS

The paper proposes feature-wise transformations and a learning-to-learn procedure to improve metric-based few-shot classification under domain shifts. Across extensive experiments, the method applies to different metric-based algorithms and consistently improves performance over baselines.

Conclusion: Feature-wise transformation layers simulate diverse feature distributions, while learning-to-learn optimizes their hyper-parameters across seen domains.The method is designed for metric-based few-shot classification under domain shifts.
Conclusion: The technique is applicable to different metric-based few-shot classification algorithms and consistently improves performance over baselines.

A.1 DATASET COLLECTION

The experiments use five few-shot classification datasets as domains: mini-ImageNet, CUB, Cars, Places, and Plantae. Cars, Places, and Plantae are additionally collected and split for the study.

Datasets: The study uses mini-ImageNet, CUB, Cars, Places, and Plantae as few-shot classification datasets.
Datasets: Cars, Places, and Plantae are manually processed by randomly splitting their classes.Training, validation, and testing category counts are summarized in Table 3.

A.2 ADDITIONAL IMPLEMENTATION DETAILS

Training uses fixed optimization settings for the metric-based model and transformation layers, with one inner iteration for learning-to-learn optimization. MatchingNet requires a reimplemented LSTM because the available module does not support second-order backpropagation.

Implementation details: The model and transformation layers use learning rate 0.001 for 40,000 iterations, with L2 regularization weight 10^-8.The learning-to-learn scheme uses one inner iteration.
Implementation details: MatchingNet is reimplemented with an LSTM supporting second-order backpropagation for optimizing the feature augmentation layers.The available PyTorch LSTM module does not support second-order backpropagation.

A.3 ADDITIONAL EXPERIMENTAL RESULTS

Additional experiments show that pre-training and learned feature-wise transformation improve cross-domain metric-based few-shot classification, while performance depends on learning transformation hyper-parameters across domains.

Pre-trained metric encoder: Pre-training the metric encoder substantially improves few-shot classification performance in the leave-one-out domain experiment.The encoder is pre-trained on the 64 mini-ImageNet training categories.
Testing-way variation: Learned feature-wise transformation improves generalization to unseen domains across different numbers of testing ways.This experiment evaluates MatchingNet and RelationNet under testing-way counts differing from training.
Transformation hyper-parameters: Feature-wise transformation with {θγ, θβ} = {0.3, 0.5} performs favorably against {1, 1}, which can underperform the original model.The result demonstrates that manually setting transformation hyper-parameters is difficult.
Hyper-parameter initialization: Initialization at {θγ, θβ} = {0.1, 0.3} yields 64.79 ± 0.55% 5-way 5-shot accuracy on CUB, similar to 64.99 ± 0.54%.Cross-domain performance is reported as insensitive when initialization values remain within the same order.
Single-domain learning: Learning the transformation from one domain is less effective than using multiple domains because pseudo-seen and pseudo-unseen feature distributions differ less.The single-domain experiment samples both tasks from mini-ImageNet.
State-of-the-art comparisons: With a pre-trained encoder and carefully hand-tuned transformations, metric-based frameworks achieve competitive mini-ImageNet performance and outperform MetaOptNet under cross-domain evaluation.The cross-domain comparison uses models trained on mini-ImageNet and evaluates on other datasets.