PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Domain-Specific Features to Enhance Domain Generalization

Manh-Ha Bui, Toan Tran, Anh Tuan Tran, Dinh Phung

arXiv:2110.09410v1cs.LG

TL;DR

Domain generalization often emphasizes invariant features while underusing domain-specific information that can support prediction across unseen domains. mDSDI disentangles and jointly learns both feature types, using meta-learning to adapt domain-specific representations from source domains. The method achieves competitive domain-generalization results, and Background-Colored-MNIST ablations support the importance of domain-specific information.

  • Problem

    Domain-generalization methods often focus on domain-invariant features while ignoring useful domain-specific information relevant to labels and unseen-domain generalization.

    Source: §1 · p. 1

  • Method

    mDSDI explicitly disentangles domain-invariant and domain-specific representations and uses meta-optimization to learn the domain-specific representation from multiple source domains.

    Source: §1 · p. 2

  • Results

    mDSDI provides competitive results across several domain-generalization benchmarks, while Background-Colored-MNIST ablations support combining domain-invariant and domain-specific information.

    Source: §1 · p. 2, §3.2 · p. 9, §4 · p. 10

  • Takeaways & Limitations

    Domain-specific information can complement domain-invariant information for domain generalization when learned through the proposed disentangling and meta-training framework.

    Source: §3.2 · p. 9, §4 · p. 10

    The paper identifies unresolved challenges including when domain-specific information may hurt unseen-domain performance and how to strengthen the theory–implementation connection.

    Source: §2.2 · p. 5, §4 · p. 10

Abstract

from arXiv · show

Domain Generalization (DG) aims to train a model, from multiple observed source domains, in order to perform well on unseen target domains. To obtain the generalization capability, prior DG approaches have focused on extracting domain-invariant information across sources to generalize on target domains, while useful domain-specific information which strongly correlates with labels in individual domains and the generalization to target domains is usually ignored. In this paper, we propose meta-Domain Specific-Domain Invariant (mDSDI) - a novel theoretically sound framework that extends beyond the invariance view to further capture the usefulness of domain-specific information. Our key insight is to disentangle features in the latent space while jointly learning both domain-invariant and domain-specific features in a unified framework. The domain-specific representation is optimized through the meta-learning framework to adapt from source domains, targeting a robust generalization on unseen domains. We empirically show that mDSDI provides competitive results with state-of-the-art techniques in DG. A further ablation study with our generated dataset, Background-Colored-MNIST, confirms the hypothesis that domain-specific is essential, leading to better results when compared with only using domain-invariant.

1 Introduction and Related work

Domain generalization commonly relies on domain-invariant features, but this can ignore useful domain-specific information. mDSDI addresses this gap by disentangling both feature types and meta-learning domain-specific representations across source domains.

  • Domain generalization trains on multiple source environments without accessing target-domain data, commonly assuming source-shared features transfer to unseen targets.Source: §1 · p. 1
  • Invariant-only learning can ignore domain-specific information that may aid prediction and generalization, particularly when source and target label distributions differ.Source: §1 · pp. 1–2
  • Existing domain-specific approaches require multiple networks or rely on implicit masking, lacking efficient disentanglement or theoretical justification.Source: §1 · p. 2
  • mDSDI explicitly disentangles label-informative domain-specific and domain-invariant representations without training multiple domain-specific networks.Source: §1 · p. 2
  • The framework uses an information-bottleneck analysis and meta-optimization to learn domain-specific representations from multiple source domains without target data.Source: §1 · p. 2
  • mDSDI is evaluated on several domain-generalization benchmarks, including a newly created Background-Colored-MNIST dataset for ablation analysis.Source: §1 · p. 2

2 Methodology

The paper formalizes domain-invariant and domain-specific representations, showing that label-correlated domain-specific information can improve prediction beyond invariant features alone. mDSDI jointly disentangles and learns both representations, using meta-learning to adapt domain-specific information from source domains without target data.

Problem setting and definitions

  • Domain generalization trains on multiple source domains without accessing target-domain data, traditionally relying on features whose distributions remain unchanged across sources.The domain-invariant mapping Q produces ZI under the assumption that its distribution is unchanged across source domains.Source: §2.1 · pp. 2–3

Domain-specific information

  • Domain-specific representations capture information belonging to individual domains, such as sketch shadows or photo backgrounds, but may become redundant when target-domain inputs and labels differ.The paper notes that source-specific information can correlate with labels yet fail to remain relevant in an unseen target domain.Source: §2.1 · p. 3, §2.2 · p. 5

Information-bottleneck analysis

  • Under the label-correlated domain-specificity assumption, source-specific information contributes ε1 > 0 to label-related mutual information beyond the minimal sufficient invariant representation.The paper states I(X1; Y) = I(Zsup; Y) = I(Zsup*; Y) = I(ZI*; Y) + ε1 > I(ZI*; Y).Source: §2.2 · pp. 4–5
  • Theorem 1 therefore states that combining domain-invariant and label-related domain-specific information can outperform learning domain-invariant information alone.This conclusion is tied to the existence of positive label-related domain-specific value ε.Source: §2.2 · p. 5

mDSDI framework

  • mDSDI jointly optimizes domain-invariant mapping Q and domain-specific mapping R by disentangling their latent representations, while using a unified classifier for label prediction.The framework addresses the need to learn both representations without training multiple domain-specific networks.Source: §2.1 · p. 3, §2.2 · pp. 4–5
  • Meta-learning is applied specifically to the domain-specific representation so source-domain information can adapt toward unseen domains while domain-invariant information remains stable.The method uses source domains to learn domain-specific representations without accessing target-domain data.Source: §2.2 · p. 5

3 Experiments

The experiments evaluate mDSDI across seven benchmark datasets and a Background-Colored-MNIST ablation, showing that it preserves domain-invariant information while exploiting useful domain-specific information. mDSDI achieves the highest reported average accuracy across the benchmark comparison and performs best in the ablation setting.

Benchmark evaluation

  • mDSDI is evaluated on seven benchmark datasets using comparisons with 14 related domain-generalization methods.The evaluation reports classification accuracy using DomainBed-style validation and repeated runs.Source: §3.1 · p. 7, Table 1 · p. 8, §3.2 · p. 8

Domain-invariant information

  • 52.2% on Colored-MNIST, 98.0% on Rotated-MNIST, and 48.1% on Terra Incognita show competitive performance where target domains primarily contain domain-invariant information.The authors attribute this to retaining informative domain-invariant features through disentangled representations and adversarial training.Source: §3.2 · p. 8

Domain-specific information

  • 79.0% on VLCS and 86.2% on PACS show that mDSDI captures useful domain-specific information in target domains sharing relevant object-background relations.The paper specifically discusses landscape backgrounds and related object-background patterns as useful domain-specific cues.Source: §3.2 · p. 8

Benchmark results

  • 67.9% average accuracy is the highest reported result across the seven benchmark datasets.The paper states that the difference is statistically significant according to a t-test at α = 0.05.Source: §3.2 · p. 8, Table 1 · p. 8
  • 69.2% on Office-Home and 42.8% on DomainNet indicate that mDSDI also performs strongly on larger and more complex benchmark settings.The paper links these results to balancing informative domain-invariant and domain-specific features.Source: §3.2 · p. 9

Ablation study

  • 89.7% is the best Background-Colored-MNIST ablation setting, while removing disentanglement loss reduces accuracy to 81.4%.The comparison also reports 82.1% for meta-training both representations, 79.0% for meta-training domain-invariant features, and around 80.4% without meta-training.Source: §3.4 · p. 10

4 Conclusion and Discussion

The paper concludes that combining disentangled domain-invariant and domain-specific representations with meta-training supports domain generalization. It also identifies unresolved theoretical, implementation, independence, and computational challenges.

Conclusion

  • mDSDI combines domain-invariant and domain-specific representations, disentangles them, and meta-trains the domain-specific representation for adaptation to unseen domains.The paper presents this combination as supported by its theoretical analysis and experiments.Source: §4 · p. 10
  • The Background-Colored-MNIST ablation demonstrates the efficiency of combining domain-invariant and domain-specific features through mDSDI.The paper states that domain-specific information is useful when combined with domain-invariant information.Source: §4 · p. 10, Table 2 · p. 10

Discussion

  • Open challenges include determining when domain-specific features may hurt unseen-domain performance and strengthening the connection between theory and implementation.The paper also identifies nonlinear independence between representations and the computational cost of the covariance matrix as remaining issues.Source: §4 · p. 10

A.1 Proof of Lemma 1

The proof begins from the case where ZX1 is produced by deterministic functions of X1 and considers events in the sigma-algebra induced by ZX1.

Proof setup

  • The proof analyzes measurable events induced by ZX1 when ZX1 is the output of deterministic functions from X1.This establishes the setting for the lemma’s measure-theoretic argument.Source: §A.1 · p. 14

A.2 Proof of Theorem 1

The proof shows that representations incorporating label-related domain-specific information can retain more label information than domain-invariant representations alone. It separately establishes the mutual-information properties of domain-invariant representations.

  • The proof decomposes into label-related and domain-invariant learned representations.Source: §A.2 · p. 14

Domain-invariant Learned Representations

  • Domain-invariant representations maximize mutual information with the shared domain information I(X1; X2).Source: §A.2 · pp. 14–15
  • The proof preserves the corresponding shared-information and conditional-information maxima involving X2 and Y.These include I(ZX1; X2; Y ) and I(ZX1; X2|Y ).Source: §A.2 · pp. 14–15
  • At the optimum, I(ZI∗; Y |X2) = 0, yielding I(X1; Y ) = I(ZI∗; Y ) + I(X1; Y |X2).Source: §A.2 · p. 15

Label-related Learned Representations

  • I(Zsup; Y ) = I(Zsup∗; Y ) = I(X1; Y ) = I(ZI∗; Y ) + ϵ1 > I(ZI∗; Y ).The result follows by combining the label-related representation result with ϵ1 = I(X1; Y |X2).Source: §A.2 · p. 15

B.1 Domain-specific features may hurt classification performance in the test domain

Domain-specific features can improve label dependence but may hurt unseen-domain performance when their label correlations change. The implementation approximates disentanglement and mutual-information objectives, with computational and optimization costs.

  • Domain-specific features may hurt performance when they correlate with different class labels in the unseen domain than in source domains.The paper gives a target-domain accuracy drop as an example of this failure mode.Source: §B.1 · p. 15
  • The framework approximates its information-theoretic objectives rather than exactly minimizing mutual information because exact computation is costly.Source: §B.2 · p. 16

Connection to theory

  • Cross-entropy is used as a proxy for minimal and sufficient label representations.Source: §B.2 · p. 16
  • Adversarial domain classifiers act as proxies for minimal and sufficient domain-invariant representations.Source: §B.2 · p. 16
  • Disentanglement minimizes covariance between domain-invariant and domain-specific representations as a proxy for minimizing their mutual information.The covariance formulation operates componentwise across the two feature vectors.Source: §B.2 · p. 16

Computational limitation

  • A 2048 × 2048 covariance matrix consumes around 30GB of GPU memory with a ResNet50 backbone on PACS.Source: §B.3 · p. 16
  • An adversarial alternative avoids dependence on feature dimension but produces higher accuracy variance.Source: §B.3 · p. 16

C.1 Dataset details

The datasets vary in how strongly domain-specific information appears alongside domain-invariant object or digit content. Background-Colored-MNIST is constructed to test whether class-correlated background color provides useful domain-specific information.

Colored-MNIST

  • Colored-MNIST constructs color labels by flipping digit-derived labels with a domain-specific noise rate.The preliminary digit label separates digits 0–4 from 5–9 before label and color noise are applied.Source: §C.1 · pp. 16–17

Background-Colored-MNIST

  • Background-Colored-MNIST assigns each class a consistent within-domain background color and uses related colors in the target domain.It contains 1,000 training samples per source domain and 10,000 target-domain test samples.Source: §C.1 · p. 18

PACS

  • PACS contains object-centered domains where artpaint and photo include label-correlated colors or backgrounds, while cartoon and sketches are treated as mainly domain-invariant.Source: §C.1 · p. 18

Office-Home

  • Office-Home combines domains with limited domain-specific information, such as clipart and product, with art and real images containing contextual features.The paper treats object sketches as domain-invariant and contextual features such as rooms as domain-specific.Source: §C.1 · p. 18

Terra Incognita

  • Terra Incognita mainly contains domain-invariant animal information because camera locations change backgrounds without correlating them with animal labels.Source: §C.1 · p. 18

DomainNet

  • DomainNet includes domains assumed to contain mainly domain-invariant conceptual drawings and others with colors, backgrounds, or descriptive text.Quickdraw and sketch are contrasted with clipart, infographic, painting, and photo.Source: §C.1 · p. 18

C.3 Implementation details

The experiments follow DomainBed-like settings, use dataset-specific backbones and preprocessing, and select models with source-domain validation before testing.

  • The implementation uses DomainBed-like settings for data processing, architectures, hyperparameters, and reproducibility.Source: §C.3 · p. 19

Data processing

  • MNIST experiments use 28 × 28 × d inputs, while other datasets are resized to 224 × 224 × 3 after randomized augmentation.Source: §C.3 · p. 19

Evaluation

  • Models are selected by source-domain validation and evaluated after 5000 iterations with three random seeds.Source: §C.3 · pp. 19–20

C.4 Empirical result details

Across DG benchmarks, mDSDI generally performs competitively by combining domain-specific and domain-invariant information, with especially strong results when backgrounds or colors provide useful domain-specific cues.

Colored-MNIST

  • 52.2% average accuracy trails ARM’s 56.2% on Colored-MNIST, although reversed label-color correlations challenge all models.The reported reversal involves an unseen domain with 90% flipped color.Source: §C.4 · p. 21

Rotated-MNIST

  • 98% average accuracy on Rotated-MNIST remains competitive despite the dataset containing limited domain-specific information.The dataset primarily contains domain-invariant information because its images lack backgrounds.Source: §C.4 · pp. 21–22

VLCS

  • 79.0% average accuracy on VLCS is highest, with mDSDI benefiting from background-only samples while retaining useful object information.mDSDI dominates three domains: LabelMe, Sun09, and VOC2007.Source: §C.4 · p. 22

PACS

  • 86.2% average accuracy on PACS combines strong results in background-rich photo and art domains with competitive performance on cartoon and sketch.Reported accuracies are 98.1% for photo, 87.7% for art, 80.4% for cartoon, and 78.4% for sketch.Source: §C.4 · pp. 22–23

Office-Home

  • 69.2% average accuracy on Office-Home outperforms other methods, particularly on background-informative Art and Real-world domains.The reported Art and Real-world accuracies are 68.1% and 80.4%, respectively.Source: §C.4 · p. 23

Terra Incognita

  • 48.1% average accuracy on Terra Incognita is second best, consistent with the dataset’s limited domain-specific features.SagNet achieves 48.6% on average.Source: §C.4 · p. 23

DomainNet

  • 42.8% average accuracy on DomainNet is highest, with mDSDI dominating four domain tests in the 345-class benchmark.The results are attributed to balancing domain-invariant and domain-specific features across environments.Source: §C.4 · p. 24

Training behavior

  • During training, the domain-invariant discriminator is successfully fooled, domain-specific features are extracted, and disentanglement loss decreases to zero.These observations come from the loss visualization for PACS with ResNet-50.Source: §C.5 · p. 24

C.6 Ablation study: Important of mDSDI on the benchmark dataset

The PACS ablation study shows that domain-specific features help on color-rich targets, domain-invariant features help on sketch, and combining both with disentanglement performs best.

Target-domain effects

  • Domain-specific settings perform better for photo, art, and cartoon targets containing color information.DS-Meta reaches 87.1% on art and 79.2% on cartoon.Source: §C.6 · p. 24
  • Domain-invariant settings outperform domain-specific settings on sketch, where color information is absent.DI-Meta reaches 75.1% in the reported sketch setting.Source: §C.6 · pp. 24–25

Combined representation learning

  • mDSDI-Meta DS outperforms DSDI-Meta DI across all target-domain settings and reaches 86.2% average accuracy.The comparison supports meta-training domain-specific features while jointly retaining domain-invariant information.Source: §C.6 · p. 25

Disentanglement

  • 83.9% average accuracy without disentanglement loss is about 2.5% lower than the highest mDSDI-Meta DS result.The ablation attributes this gap to the role of covariance-based disentanglement in separating the two representations.Source: §C.6 · p. 25

Related papers

Loading 2110.09410v1…