SelfReg: Self-supervised Contrastive Regularization for Domain Generalization

Daehee Kim; Seunghyun Park; Jinkyu Kim; Jaekoo Lee

Source-linked AI summary

SelfReg: Self-supervised Contrastive Regularization for Domain Generalization

Daehee Kim, Seunghyun Park, Jinkyu Kim, Jaekoo Lee

arXiv:2104.09841v1Apr 20, 2021cs.CVcs.AI

TL;DR

Domain shift limits models trained under same-distribution assumptions, while contrastive domain-generalization methods depend on negative-pair sampling. SelfReg instead regularizes representations using only positive pairs and matches or outperforms prior work on PACS and DomainBed benchmarks.

Problem
Domain shift undermines generalization because real-world deployment data rarely follows the training distribution, and learned features remain susceptible to such shifts.
Method
SelfReg learns domain-invariant representations with self-supervised contrastive regularization using positive pairs, supplemented by CDPL, mixup, and gradient-stabilization techniques.
Results
SelfReg matches or outperforms prior work on PACS and DomainBed benchmarks, ranking second among methods evaluated on DomainBed.
Takeaways & Limitations
SelfReg demonstrates that contrastive regularization for domain generalization can achieve competitive benchmark performance without negative-pair sampling.
Takeaways & Limitations
The paper omits the inter-domain curriculum learning strategy because of publication copyright issues and uses SWA only for ablation rather than DomainBed comparison.

Abstract

from arXiv · show

In general, an experimental environment for deep learning assumes that the training and the test dataset are sampled from the same distribution. However, in real-world situations, a difference in the distribution between two datasets, domain shift, may occur, which becomes a major factor impeding the generalization performance of the model. The research field to solve this problem is called domain generalization, and it alleviates the domain shift problem by extracting domain-invariant features explicitly or implicitly. In recent studies, contrastive learning-based domain generalization approaches have been proposed and achieved high performance. These approaches require sampling of the negative data pair. However, the performance of contrastive learning fundamentally depends on quality and quantity of negative data pairs. To address this issue, we propose a new regularization method for domain generalization based on contrastive learning, self-supervised contrastive regularization (SelfReg). The proposed approach use only positive data pairs, thus it resolves various problems caused by negative pair sampling. Moreover, we propose a class-specific domain perturbation layer (CDPL), which makes it possible to effectively apply mixup augmentation even when only positive data pairs are used. The experimental results show that the techniques incorporated by SelfReg contributed to the performance in a compatible manner. In the recent benchmark, DomainBed, the proposed method shows comparable performance to the conventional state-of-the-art alternatives. Codes are available at https://github.com/dnap512/SelfReg.

1. Introduction

Domain generalization addresses deployment-time distribution shifts by learning invariances across multiple source domains for unseen targets [31] [38]. SelfReg uses positive-pair contrastive regularization, CDPL with mixup, and stabilization techniques, achieving comparable or better performance than state-of-the-art methods on DomainBed.

Motivation: Domain generalization targets robustness to unseen domains because real-world data changes across time and space, violating the in-sample/out-of-sample i.i.d. assumption.Although deep convolutional features can be domain-invariant to low-level visual cues [35], they remain susceptible to domain shift.
Method: The model learns domain-invariant representations by bringing same-class samples together in latent space, as illustrated in Figure 1.Different shapes represent classes, while colors represent domains; the approach uses self-supervised contrastive losses.
Method: Loss clipping, stochastic weight averaging, and inter-domain curriculum learning stabilize gradients so network parameters converge toward domain-invariant features.These three techniques are advocated alongside self-supervised contrastive losses because balancing gradients is required for convergence.
Evaluation: SelfReg achieves matched or better performance than alternative state-of-the-art methods on the larger DomainBed benchmark.The study also analyzes the model in detail on PACS before evaluating it on DomainBed.
Method: SelfReg applies metric learning using only positive pairs, avoiding negative-pair sampling challenges that can destabilize training.Negative-pair methods require useful negatives in the same batch, which is often difficult.
Method: CDPL exploits the positive-pair setting, and its combination with mixup improves mixup’s weakness.This design makes mixup augmentation effective even when only positive pairs are used.

2. Related Work

Related work addresses domain generalization through domain-invariant feature learning, distribution alignment, contrastive objectives, mixup, self-supervision, meta-learning, and domain-specific model decompositions. Contrastive approaches align same-class features while separating different classes, but negative-pair sampling can destabilize training.

Domain-Invariant Feature Learning: Domain generalization seeks domain-invariant features for generalization to unseen target domains, using ERM and distribution-matching variants such as adversarial and conditional alignment.These methods optimize shared feature spaces by minimizing distribution discrepancies across domains.
Contrastive Learning: CCSA [30] and MASF [11] align same-class feature distributions while separating different-class distributions, but negative pairs can make training unstable without useful negative samples.Their contrastive objectives depend on positive and negative feature pairs.
Alternative Approaches: Other domain-generalization methods use self-supervised jigsaw solving, meta-learning with MAML, or decompositions into domain-invariant and domain-variant components [5] [27] [14] [26] [32] [12].The decompositions include low-rank parameterization, style-agnostic networks, and domain-specific aggregation modules.

3. Method

SelfReg regularizes domain generalization by bringing same-class representations together in the embedding space through self-supervised in-batch contrastive losses. It combines individualized and heterogeneous dissimilarities with class-specific perturbation, mixup, and training-stabilization techniques.

Method overview: SelfReg uses self-supervised in-batch contrastive losses to learn domain-invariant representations by mapping same-class samples closer and different-class samples farther apart.The method measures individualized and heterogeneous feature-level dissimilarities in the embedding space.
Method scope and limitations: The paper excludes inter-domain curriculum learning from its presented contents because of publication copyright issues.The authors report that SWA provides further performance improvement, possibly by producing a flatter loss surface through ensembling domain-specific models.
Individualized In-batch Dissimilarity Loss: The individualized loss aligns randomly paired same-class latent representations while jointly encouraging uniformity, using a Class-specific Domain Perturbation Layer to prevent representation collapse.The Class-specific Domain Perturbation Layer is an additional MLP, and its benefit is evaluated in an ablation study.
Heterogeneous In-batch Dissimilarity Loss: The heterogeneous loss applies two-domain mixup to same-class representations from different domains, regularizing the model on convex combinations sampled with γ ∼ Beta(α, β).The mixup coefficient satisfies γ ∈ [0, 1] and is controlled by α and β.
Loss Function and Stochastic Weight Average: SelfReg applies its dissimilarity losses to intermediate features and classifier logits alongside classification loss, while loss clipping and stochastic weight averaging mitigate gradient imbalance during training.SWA averages model snapshots from multiple local minima and is used only for ablation studies, not DomainBed, to preserve fair comparison.

4. Proof-of-Concept Experiments

The proof-of-concept experiments evaluate SelfReg on PACS [26] against RSC [22] and analyze whether its regularization produces domain-invariant representations. Results show improved same-class feature and logit alignment, class-relevant attention, and gains from the proposed components, including in single-source generalization.

Main comparison: Table 1 compares SelfReg with RSC [22] using identical ResNet18 backbones, reporting average accuracy and variance across 20 trainings for each test domain.The evaluation uses PACS [26], whose four domains exhibit substantial domain shift.
Qualitative Analysis by t-SNE: SelfReg produces more merged same-class t-SNE clusters across domains, whereas the baseline and RSC [22] produce scattered domain-specific clusters.The visualization uses PACS [26] representations and indicates greater cross-domain similarity for objects of the same class.
Effect of Dissimilarity Loss: SelfReg reduces same-class pairwise distances in both feature and logit spaces relative to the baseline, as shown across training epochs in Figure 5.The distances are measured with Euclidean distance, supporting the intended positive-pair regularization mechanism.
Analysis with GradCAM: GradCAM shows that SelfReg better captures class-invariant regions, such as a giraffe’s long neck, than RSC [22] across target domains.The analysis visualizes the regions receiving attention for the network’s final prediction.
Extreme Domain Generalization: In the single-source setting, SelfReg is compared with RSC [22] across every source-target domain combination, with differences reported in Table 2.Rows represent source domains, columns represent target domains, and positive differences indicate better SelfReg performance.
Ablation Study: Ablations show gains from IDCL (+0.32% average accuracy), SWA (+0.25%), CDPL and Mixup, and logit-level regularization (+1.50% average accuracy with reduced variance).CDPL and Mixup are especially beneficial on Sketch, while logit-level regularization improves accuracy and reduces variance across domains.

5. Experiments on DomainBed

On the DomainBed benchmark, SelfReg generally matches or outperforms alternative state-of-the-art methods and ranks second across the seven benchmarks, despite omitting IDCL and SWA. DomainBed evaluates domain generalization using seven multi-domain datasets and 14 baseline approaches under common conditions.

Experimental setup: The DomainBed evaluation covers seven multi-domain datasets and benchmarks SelfReg against 14 baseline domain generalization approaches.The datasets include ColoredMNIST, RotatedMNIST, VLCS, PACS, OfficeHome, TerraIncognita, and DomainNet.
Results: SelfReg generally matches or outperforms alternative state-of-the-art methods on DomainBed and ranks second in average performance across all seven benchmarks.Table 4 reports average out-of-distribution test accuracies for 14 domain generalization algorithms under the same conditions.
Results: SelfReg does not use IDCL or SWA in DomainBed because of implementation inflexibility, although combining either technique was reported as capable of further improvement.The DomainBed setting trains the domain validation set for model selection.

6. Conclusion

SelfReg is a domain-generalization regularizer using self-supervised contrastive loss with only positive pairs, mitigating problems from negative-pair sampling. On PACS and DomainBed under standard evaluation, it matches or outperforms prior work.

6. Conclusion: SelfReg uses self-supervised contrastive regularization with only positive data pairs, mitigating problems caused by negative-pair sampling.
6. Conclusion: Experiments on PACS and DomainBed show that SelfReg matches or outperforms prior work under the standard domain-generalization evaluation setting.
6. Conclusion: Future work could extend SelfReg with a siamese network.

Appendix

The appendix reports detailed DomainBed scores across seven datasets and notes that SelfReg† achieves state-of-the-art performance on DomainNet. It also documents the evaluation modification used for SelfReg with SWA.

For SelfReg with SWA, the authors modified DomainBed evaluation to test the model after 5000 learning steps with SWA.Table 8 additionally reports SelfReg with SWA, while Table 5–11 SelfReg† excludes IDCL and SWA.
SelfReg† achieves state-of-the-art performance on DomainNet.