Source-linked AI summary
Feature-Critic Networks for Heterogeneous Domain Generalization
Yiying Li, Yongxin Yang, Wei Zhou, Timothy M. Hospedales
TL;DR
Domain shift harms performance on unseen domains, while heterogeneous domain generalisation further requires features to support novel categories under disjoint label spaces. The paper meta-learns a feature-critic auxiliary loss to train more general feature extractors, and reports favorable performance in both heterogeneous and homogeneous settings. Its approach assumes target-domain training data cannot be accessed for feature learning and avoids directly doubling the base network’s parameters.
Problem
Domain generalisation must handle unseen target domains without target data, and heterogeneous domain generalisation additionally requires source-trained features to support novel categories with disjoint label spaces.
Method
The method simulates domain shift with virtual training and validation domains and meta-learns a non-negative auxiliary loss operating on extracted features.
Results
Feature-Critic trained feature extractors outperform alternatives for fixed downstream image encoding and perform favorably against state-of-the-art in homogeneous domain generalisation.
Takeaways & Limitations
The resulting feature extractor is intended for general-purpose reuse with novel data and categories, including off-the-shelf downstream recognition.
Takeaways & Limitations
Feature learning cannot access the target domain’s training split, and directly conditioning the auxiliary loss on all feature-extractor parameters would double model parameters and increase optimisation and overfitting risks.
Abstract
from arXiv · showhide
The well known domain shift issue causes model performance to degrade when deployed to a new target domain with different statistics to training. Domain adaptation techniques alleviate this, but need some instances from the target domain to drive adaptation. Domain generalisation is the recently topical problem of learning a model that generalises to unseen domains out of the box, and various approaches aim to train a domain-invariant feature extractor, typically by adding some manually designed losses. In this work, we propose a learning to learn approach, where the auxiliary loss that helps generalisation is itself learned. Beyond conventional domain generalisation, we consider a more challenging setting of heterogeneous domain generalisation, where the unseen domains do not share label space with the seen ones, and the goal is to train a feature representation that is useful off-the-shelf for novel data and novel categories. Experimental evaluation demonstrates that our method outperforms state-of-the-art solutions in both settings.
1. Introduction
Domain generalisation seeks models that perform well on unseen domains without target-domain data, including heterogeneous settings with novel label spaces. Feature-Critic uses meta-learning and a learned auxiliary loss to improve feature extractors for both conventional and heterogeneous domain generalisation.
- Domain shift between training and testing statistics can significantly reduce machine-learning performance in real-world applications.
- Domain generalisation trains models to perform on novel target domains without accessing target-domain data during training.
- Heterogeneous domain generalisation targets disjoint label spaces, requiring source-trained features to support novel categories in novel target domains.The paper relates this setting to reusing ImageNet CNN features with shallow classifiers for new problems.
- The method adapts episodic meta-learning by simulating train/test splits among source domains to improve feature-extractor training.
- Feature-Critic decomposes the source model into feature extractor and task networks, then learns a feature-critic network to assess feature robustness to simulated domain shift.The critic supplies a learned auxiliary loss that provides additional feedback during feature training.
2. Related Work
Related work covers domain generalisation, heterogeneous domain generalisation, and episodic meta-learning. Feature-Critic differs from prior approaches by learning an auxiliary loss that produces a general feature extractor usable with any classifier.
- Domain Generalisation: Domain generalisation differs from domain adaptation because target-domain samples are unavailable during training, requiring direct generalisation to the target domain.
- Domain Generalisation: Existing domain-generalisation methods include feature-based, classifier-based, and data-augmentation approaches.
- Heterogeneous Domain Generalisation: Heterogeneous domain generalisation addresses domains with different label spaces by improving source-trained features for representing novel target problems rather than transferring the classifier directly.
- Meta-Learning: Episodic meta-learning commonly simulates few-shot train/test episodes, while prior DG methods applied related strategies through heuristic updates or classifier regularisation.
- Feature-Critic: Feature-Critic learns an auxiliary loss to train a more general feature extractor that can be used with any classifier.
3. Methodology
The method learns a feature extractor for homogeneous and heterogeneous domain generalisation by simulating domain shift and training a feature-critic auxiliary loss on extracted features. The resulting workflow supports off-the-shelf target-domain classification using fixed features and a newly trained classifier.
- The model uses a shared feature extractor with domain-specific classifiers for heterogeneous domain generalisation, or one shared classifier for homogeneous domain generalisation.
- For heterogeneous targets, the learned extractor is fixed, target-training features are used to train an SVM or KNN classifier, and evaluation occurs on the target testing split.
- Training simulates domain shift by randomly splitting source domains into disjoint virtual training and validation sets.
- The optimisation alternates meta-training on virtual training domains, meta-testing on validation domains, and updates to the feature extractor, classifier, and feature-critic.
- The feature-critic learns an auxiliary loss that evaluates feature robustness through validation-domain performance and guides the feature extractor toward domain-agnostic representations.
- Feature-Critic operates on extracted feature matrices rather than directly on extractor parameters, reducing the parameter burden associated with parameter-based regularisation.
- Directly parameterising the auxiliary loss with extractor parameters can double model parameters in large CNNs, increasing optimisation difficulty and overfitting risk.
- The feature-critic input is permutation invariant to examples within a mini-batch and can use set embedding or flattened covariance representations.
4. Experiments
Experiments evaluate Feature-Critic across heterogeneous and homogeneous domain generalisation benchmarks, including limited-data and qualitative analyses. The method generally performs best or comparably to leading alternatives while producing useful off-the-shelf features.
- Visual Decathlon: Feature-Critic generally provides the best performance across Visual Decathlon domains and both evaluated classifier types.
- Visual Decathlon: Feature-Critic outperforms ImageNet and Data Aggregation features across target-domain operating points from few-shot to many-shot regimes.
- Rotated MNIST: On Rotated MNIST, Feature-Critic performs well with both auxiliary-loss variants, while set embedding performs slightly better than covariance embedding.The comparison averages results over ten trials of 1000 digit samples.
- Rotated MNIST: PCA projections for held-out M15 show improved target-domain feature separability for Feature-Critic compared with AGG.Each plotted dot is an image, colored by digit label; AGG appears on the left and Feature-Critic on the right.
- Further Analysis: During training, cross-entropy converges to zero, auxiliary loss stabilizes at a small value, and meta-loss follows an above-zero to below-zero to zero pattern.The late-stage meta-loss approaching zero indicates the learned auxiliary loss no longer helps or hurts because its knowledge has been distilled into the feature extractor.
- PACS: On PACS, Feature-Critic performs comparably to the best-performing state-of-the-art alternative in both evaluated settings.The settings use the train split and train+validation split protocols.
5. Conclusion
Feature-Critic addresses domain generalisation, especially heterogeneous domain generalisation, by meta-learning a regulariser for domain-invariant feature extraction. Its extractors outperform alternatives as fixed downstream image encodings and perform favourably in homogeneous domain generalisation.
- Feature-Critic meta-learns a regulariser to train a domain-invariant feature extractor, focusing particularly on heterogeneous domain generalisation.
- The resulting feature extractor outperforms alternatives for general-purpose use as a fixed downstream image encoding.
- On Visual Decathlon, Feature-Critic’s results suggest potential value for diverse applications.
- Feature-Critic also performs favourably compared to state-of-the-art methods in homogeneous domain generalisation.