Source-linked AI summary
Episodic Training for Domain Generalization
Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, Timothy M. Hospedales
TL;DR
Domain generalization seeks models that generalize to novel domains without target adaptation, while simple source-domain aggregation is already a strong baseline. The paper episodically trains feature extractors and classifiers with badly tuned partners, achieving state-of-the-art benchmark performance and improving fixed ImageNet features for novel downstream problems.
Problem
Domain generalization asks how to generalize to novel domains with different statistics when no target-domain adaptation is possible.
Method
The method decomposes a deep network into feature extractor and classifier modules and episodically trains them with partners that are badly tuned for the current domain.
Results
The method achieves state-of-the-art performance on the main domain-generalization benchmarks and improves an ImageNet CNN used as a fixed feature extractor for novel downstream problems.
Takeaways & Limitations
Explicit domain-generalization training can benefit standard computer-vision practice by improving fixed general-purpose features for novel downstream problems.
Takeaways & Limitations
The Epi-C variant requires training multiple feature extractors, although source-domain subsets can be sampled per batch when many domains are present.
Abstract
from arXiv · showhide
Domain generalization (DG) is the challenging and topical problem of learning models that generalize to novel testing domains with different statistics than a set of known training domains. The simple approach of aggregating data from all source domains and training a single deep neural network end-to-end on all the data provides a surprisingly strong baseline that surpasses many prior published methods. In this paper, we build on this strong baseline by designing an episodic training procedure that trains a single deep network in a way that exposes it to the domain shift that characterises a novel domain at runtime. Specifically, we decompose a deep network into feature extractor and classifier components, and then train each component by simulating it interacting with a partner who is badly tuned for the current domain. This makes both components more robust, ultimately leading to our networks producing state-of-the-art performance on three DG benchmarks. Furthermore, we consider the pervasive workflow of using an ImageNet trained CNN as a fixed feature extractor for downstream recognition tasks. Using the Visual Decathlon benchmark, we demonstrate that our episodic-DG training improves the performance of such a general-purpose feature extractor by explicitly training a feature for robustness to novel problems. This shows that DG training can benefit standard practice in computer vision.
1. Introduction
Domain generalization targets models that work on unseen domains without target-domain adaptation, a difficult setting valuable for robust out-of-the-box visual features. The paper builds on strong data aggregation by episodically simulating domain shift through mismatched network components.
- Domain generalization addresses performance degradation on novel domains without using target-domain data or computation for adaptation.
- Aggregating all source-domain data to train one deep network is a strong, simple, fast baseline that outperforms many published methods.
- Episodic domain-generalization training exposes network layers to neighboring components that are badly tuned for the current domain.
- The method decomposes networks into feature extractor and classifier modules, training them episodically to produce robust, architecture-independent components.
- The approach improves domain-generalization benchmark performance and tests whether explicitly robust features improve fixed ImageNet CNN representations for novel downstream tasks.
2. Related Work
Related work spans multi-domain learning, domain-invariant and hierarchical models, and episodic meta-learning. The paper repurposes Visual Decathlon to evaluate domain generalization with heterogeneous label spaces.
- Multi-domain learning trains one model across several domains, with performance depending on data availability, domain similarity, and the task.
- Visual Decathlon is repurposed for heterogeneous domain generalization by training on some domains and evaluating a feature extractor on unseen disjoint domains.
- Domain-invariant feature methods typically minimize discrepancies among source domains to obtain representations expected to work on the target.
- Hierarchical models combine domain-agnostic and domain-specific parameters to represent individual domains.
- Episodic meta-learning samples source tasks and instances into training episodes that simulate the expected testing condition, whereas this paper applies episodic training to domain robustness.
3. Methodology
The method trains shared feature-extractor and classifier modules for domain generalization using domain-specific partners that are mismatched with current inputs. It combines aggregation with episodic cross-domain and random-classifier regularization, while deploying only the shared modules.
- The baseline aggregates all source-domain data and trains one CNN end-to-end while ignoring domain labels.
- Domain-Specific Models: Each source domain also receives a domain-specific feature extractor and classifier trained only on that domain’s data.
- Episodic Training: Episodic training pairs each shared module with a domain-specific partner mismatched to the current input domain.
- Episodic Training: Feature regularization trains the shared extractor so data from one domain can be classified by a classifier that has not experienced that domain.
- Episodic Training: Classifier regularization trains the shared classifier to classify data encoded by a domain-specific extractor unfamiliar with that data’s domain.
- Random Classifier Regularization: Random-classifier regularization extends feature training to heterogeneous label spaces by using a fixed randomly initialized classifier for each domain.
4. Experiments
Experiments evaluate Episodic across homogeneous and heterogeneous DG benchmarks, comparing it with aggregation and prior methods under multiple architectures and settings. Results show strong performance across benchmarks, cross-domain robustness, practical ImageNet-feature benefits, and implementation trade-offs.
- IXMAS: Epi-FCR improves IXMAS accuracy by 2.4% over AGG and 1.1% over prior state-of-the-art MMD-AAE.
- VLCS: Epi-FCR achieves the best VLCS performance, improving on AGG by 1.7% and on MMD-AAE and MLDG by 0.6%.
- PACS: On PACS with pretrained AlexNet, Episodic performs best overall, improving 3.3% over AGG and at least 1.7% over MLDG, Fusion, and MetaReg.
- PACS: With ResNet-18 on PACS, results improve across the board, while full Episodic remains best overall with a 2.4% improvement over AGG.
- Further analysis: Cross-domain tests show that episodic training improves an agnostic extractor or classifier when paired with a module that has not experienced the incoming domain.
- VD-DG: On Visual Decathlon, Epi-R surpasses AGG and feasible DG competitors in average accuracy and VD score, extending evaluation to larger-scale heterogeneous domains.
5. Conclusion
The paper proposes episodic training that mimics train-test domain shift, achieving state-of-the-art performance on existing DG benchmarks and improving ImageNet feature extraction for novel downstream problems. A concurrent heterogeneous-DG study reported slightly higher results on the proposed VD-DG benchmark using larger images and cross-validated SVM parameters.
- The method achieves state-of-the-art performance on all the main existing domain-generalization benchmarks.
- The Visual Decathlon evaluation demonstrates potential value for improving an ImageNet-pretrained CNN used as a fixed feature extractor on novel downstream problems.
- A concurrent heterogeneous-DG study reported slightly higher VD-DG results using a larger image size and cross-validation of SVM parameters, whereas this paper used sklearn defaults.
A. Additional analysis
The additional analysis examines whether episodic-training benefits arise solely from using extra parameters during training. The current implementation uses domain-specific branches to generate episodes but tests only a single aggregated branch.
- The episodic implementation uses n domain-specific branches alongside the AGG model to generate domain-generalization episodes.
- Training therefore uses n+1 times as many parameters as AGG, while testing uses only a single AGG branch.