Source-linked AI summary
Learning from Extrinsic and Intrinsic Supervisions for Domain Generalization
Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, Pheng-Ann Heng
TL;DR
Neural networks often degrade on unseen domains, while domain adaptation requires target data during training. EISNet jointly learns supervised recognition, momentum metric relationships, and intrinsic self-supervision from multiple source domains, achieving state-of-the-art performance on VLCS and PACS.
Problem
Models can perform poorly on unseen domains, and domain adaptation requires pre-collected target data and retraining for each new target.
Method
EISNet uses a multi-task feature-embedding framework combining supervised recognition, momentum metric learning with K-hard negative selection, and a self-supervised patch-ordering task.
Results
EISNet achieves state-of-the-art performance on VLCS and PACS, including average accuracies of 74.67% on VLCS and 75.86% on PACS.
Takeaways & Limitations
Jointly modeling relationships among images and intrinsic structure within images learns discriminative yet transferable features for domain generalization.
Takeaways & Limitations
The framework assumes its encoder can extract discriminative and transferable features suitable for prompting a classifier on unseen target domains.
Abstract
from arXiv · showhide
The generalization capability of neural networks across domains is crucial for real-world applications. We argue that a generalized object recognition system should well understand the relationships among different images and also the images themselves at the same time. To this end, we present a new domain generalization framework that learns how to generalize across domains simultaneously from extrinsic relationship supervision and intrinsic self-supervision for images from multi-source domains. To be specific, we formulate our framework with feature embedding using a multi-task learning paradigm. Besides conducting the common supervised recognition task, we seamlessly integrate a momentum metric learning task and a self-supervised auxiliary task to collectively utilize the extrinsic supervision and intrinsic supervision. Also, we develop an effective momentum metric learning scheme with K-hard negative mining to boost the network to capture image relationship for domain generalization. We demonstrate the effectiveness of our approach on two standard object recognition benchmarks VLCS and PACS, and show that our methods achieve state-of-the-art performance.
1 Introduction
Domain generalization seeks models that train on multiple source domains and apply directly to unseen targets without target data. EISNet addresses this by jointly learning from relationships among images and intrinsic structure within individual images, achieving state-of-the-art performance on two benchmarks.
- Motivation: Domain adaptation requires target-domain data and retraining for each new target, whereas domain generalization targets direct application to unseen domains.This distinction motivates learning a generalizable network from multiple source domains alone.
- Proposed Framework: EISNet combines extrinsic relationship supervision with intrinsic self-supervision in a multi-task framework for general feature learning.It adds momentum metric learning and a self-supervised auxiliary task to supervised object recognition.
- Proposed Framework: Momentum metric learning uses a K-hard negative selector to identify informative hard triplets and a momentum encoder to stabilize memory-bank embeddings.The selector supports image-relationship learning, while the momentum-updated encoder maintains embedding consistency.
- Proposed Framework: The intrinsic auxiliary task learns spatial relationships by predicting the order of patches within an image.This complements relationship learning across different images with structure learning inside each image.
- Results: The three tasks share one feature encoder and are optimized end-to-end, and experiments on two object-recognition benchmarks report state-of-the-art performance.The framework jointly supports supervised recognition, momentum metric learning, and self-supervision.
2 Related Work
Related work addresses domain adaptation and generalization through target-domain alignment, synthetic data, domain-invariant features, and meta-learning, while EISNet combines metric learning with self-supervision for domain generalization.
- Domain Adaptation and Generalization: Unsupervised domain adaptation uses unlabeled target images to align source and target distributions at input, feature, or output levels.Its setting differs from domain generalization because target images guide optimization.
- Domain Adaptation and Generalization: Domain generalization trains on multiple source domains and applies directly to unseen targets without target-domain guidance during training.Prior approaches increase input diversity with synthetic images or extract domain-invariant features.
- Metric Learning: Metric learning minimizes intra-class distances and maximizes interclass variation, supporting discriminative feature embeddings across several recognition applications.EISNet uses triplet loss with informative negative selection and momentum feature extraction for domain generalization.
- Self-Supervision: Self-supervision creates annotation-free prediction tasks for feature learning, including patch arrangement, color, inpainting, and rotation tasks.This paper uses self-supervision as an auxiliary task to promote the main task and improve generalization.
3 Method
The method uses a shared-encoder multi-task framework combining classification, momentum metric learning, and self-supervised patch-order prediction. Its metric-learning component selects informative hard negatives from a memory bank, while momentum updates stabilize historical representations.
- Overall framework: EISNet trains a shared encoder with classification, extrinsic momentum metric-learning, and intrinsic self-supervised auxiliary tasks.The total objective balances these three supervision sources with hyperparameters α, β, and γ.
- Extrinsic supervision: The framework learns domain-independent yet class-specific features by encouraging same-label embeddings to be close and different-class embeddings to be far apart.This metric-learning objective is intended to reduce ambiguous decision boundaries on unseen domains.
- Extrinsic supervision: K-hard negative selection chooses K negatives that violate the triplet constraint within a margin, providing more informative training examples than random selection.If no hard negatives exist, negatives are sampled without the distance constraint.
- Extrinsic supervision: The resulting triplet objective averages losses over K selected negatives for each anchor-positive pair.The selector is designed to facilitate learning more discriminative features by considering multiple informative hard negatives.
- Efficient metric learning: A memory bank stores historical feature representations to enlarge the negative-sample pool and reduce the computation burden of triplet selection.The memory bank supports diverse triplet pairs, while a momentum-updated encoder improves consistency between current and historical representations.
- Efficient metric learning: The MuEncoder updates its parameters by exponential moving average, θg = δ ∗ θg + (1 − δ) ∗ θf, and slower updates were found to produce better preliminary results.The smoother evolution reduces rapid changes among memory-bank features and eases triplet-loss updates.
- Intrinsic supervision: Intrinsic supervision regularizes embeddings through a jigsaw task that divides each image into nine 3 × 3 patches and predicts their shuffled order.The original unshuffled image is represented as c_a = 0, yielding 31 classes for the auxiliary task.
4 Experiments
EISNet is evaluated on VLCS and PACS across AlexNet and ResNet backbones, with experiments covering benchmark performance, supervision ablations, metric-learning settings, and training cost. It achieves strong generalization results while benefiting from both extrinsic and intrinsic supervision.
- Benchmark results: EISNet achieves 74.67% average accuracy over four VLCS domains, outperforming MASF and producing the best results on Caltech and Sun.It also outperforms JiGen on three domains and is comparable on PASCAL VOC.
- Benchmark results: 75.86% average accuracy across PACS domains gives EISNet the best performance among the compared AlexNet methods.The result is attributed to simultaneously using momentum metric learning and intrinsic self-supervision across source domains.
- Backbone transferability: 1.11% and 3.17% average-accuracy gains over MASF are reported with ResNet-18 and ResNet-50, respectively.The larger gain with ResNet-50 suggests greater benefit from deeper feature extractors in these experiments.
- Ablation analysis: 2.99% and 2.73% average-accuracy improvements over the baseline come from adding extrinsic and intrinsic supervision, respectively.The baseline omits both supervisions and corresponds to DeepAll in the reported PACS comparison.
- Metric-learning settings: K = 256 gives the best Sketch result, whereas K = 512 causes a drastic drop because excessive negatives burden metric-distance calculation.The analysis compares K values of 1, 8, 64, 128, 256, and 512.
- Metric-learning settings: Accuracy is strongest near δ = 0.999, while smaller momentum coefficients degrade performance by weakening feature consistency across memory-bank batches.The study also reports 70.25% accuracy with the K-hard negative selector and a 91% reduction in training time versus MASF.
5 Conclusions
The paper presents a multi-task framework for domain generalization that learns from extrinsic relationships between images and intrinsic self-supervised constraints within individual images. Experiments on two public benchmarks support discriminative, transferable features and state-of-the-art domain-generalization performance, with flexibility across network backbones.
- EISNet learns feature embeddings from extrinsic relationships among different images and intrinsic self-supervised constraints within single images.The framework uses a multi-task learning paradigm for domain generalization.
- The framework includes an effective and efficient momentum metric learning module for compact feature learning.
- Experiments on two public benchmark datasets show that the method learns discriminative yet transferable features for domain generalization.
- The proposed method achieves state-of-the-art performance for domain generalization and can be migrated to various network backbones.