PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploiting Domain-Specific Features to Enhance Domain Generalization

Manh-Ha Bui, Toan Tran, Anh Tuan Tran, Dinh Phung

arXiv:2110.09410v1cs.LG

TL;DR

Domain generalization often emphasizes invariant features while underusing domain-specific information that can support prediction across unseen domains. mDSDI disentangles and jointly learns both feature types, using meta-learning to adapt domain-specific representations from source domains. The method achieves competitive domain-generalization results, and Background-Colored-MNIST ablations support the importance of domain-specific information.

  • Problem

    Domain-generalization methods often focus on domain-invariant features while ignoring useful domain-specific information relevant to labels and unseen-domain generalization.

  • Method

    mDSDI explicitly disentangles domain-invariant and domain-specific representations and uses meta-optimization to learn the domain-specific representation from multiple source domains.

  • Results

    mDSDI provides competitive results across several domain-generalization benchmarks, while Background-Colored-MNIST ablations support combining domain-invariant and domain-specific information.

  • Takeaways & Limitations

    Domain-specific information can complement domain-invariant information for domain generalization when learned through the proposed disentangling and meta-training framework.

  • Takeaways & Limitations

    The paper identifies unresolved challenges including when domain-specific information may hurt unseen-domain performance and how to strengthen the theory–implementation connection.

Abstract

from arXiv · show

Domain Generalization (DG) aims to train a model, from multiple observed source domains, in order to perform well on unseen target domains. To obtain the generalization capability, prior DG approaches have focused on extracting domain-invariant information across sources to generalize on target domains, while useful domain-specific information which strongly correlates with labels in individual domains and the generalization to target domains is usually ignored. In this paper, we propose meta-Domain Specific-Domain Invariant (mDSDI) - a novel theoretically sound framework that extends beyond the invariance view to further capture the usefulness of domain-specific information. Our key insight is to disentangle features in the latent space while jointly learning both domain-invariant and domain-specific features in a unified framework. The domain-specific representation is optimized through the meta-learning framework to adapt from source domains, targeting a robust generalization on unseen domains. We empirically show that mDSDI provides competitive results with state-of-the-art techniques in DG. A further ablation study with our generated dataset, Background-Colored-MNIST, confirms the hypothesis that domain-specific is essential, leading to better results when compared with only using domain-invariant.

1 Introduction and Related work

Domain generalization commonly relies on domain-invariant features, but this can ignore useful domain-specific information. mDSDI addresses this gap by disentangling both feature types and meta-learning domain-specific representations across source domains.

  • Domain generalization trains on multiple source environments without accessing target-domain data, commonly assuming source-shared features transfer to unseen targets.
  • Invariant-only learning can ignore domain-specific information that may aid prediction and generalization, particularly when source and target label distributions differ.
  • Existing domain-specific approaches require multiple networks or rely on implicit masking, lacking efficient disentanglement or theoretical justification.
  • mDSDI explicitly disentangles label-informative domain-specific and domain-invariant representations without training multiple domain-specific networks.
  • The framework uses an information-bottleneck analysis and meta-optimization to learn domain-specific representations from multiple source domains without target data.
  • mDSDI is evaluated on several domain-generalization benchmarks, including a newly created Background-Colored-MNIST dataset for ablation analysis.

2 Methodology

The paper formalizes domain-invariant and domain-specific representations, showing that label-correlated domain-specific information can improve prediction beyond invariant features alone. mDSDI jointly disentangles and learns both representations, using meta-learning to adapt domain-specific information from source domains without target data.

  • Problem setting and definitions: Domain generalization trains on multiple source domains without accessing target-domain data, traditionally relying on features whose distributions remain unchanged across sources.The domain-invariant mapping Q produces ZI under the assumption that its distribution is unchanged across source domains.
  • Domain-specific information: Domain-specific representations capture information belonging to individual domains, such as sketch shadows or photo backgrounds, but may become redundant when target-domain inputs and labels differ.The paper notes that source-specific information can correlate with labels yet fail to remain relevant in an unseen target domain.
  • Information-bottleneck analysis: Under the label-correlated domain-specificity assumption, source-specific information contributes ε1 > 0 to label-related mutual information beyond the minimal sufficient invariant representation.The paper states I(X1; Y) = I(Zsup; Y) = I(Zsup*; Y) = I(ZI*; Y) + ε1 > I(ZI*; Y).
  • Information-bottleneck analysis: Theorem 1 therefore states that combining domain-invariant and label-related domain-specific information can outperform learning domain-invariant information alone.This conclusion is tied to the existence of positive label-related domain-specific value ε.
  • mDSDI framework: mDSDI jointly optimizes domain-invariant mapping Q and domain-specific mapping R by disentangling their latent representations, while using a unified classifier for label prediction.The framework addresses the need to learn both representations without training multiple domain-specific networks.
  • mDSDI framework: Meta-learning is applied specifically to the domain-specific representation so source-domain information can adapt toward unseen domains while domain-invariant information remains stable.The method uses source domains to learn domain-specific representations without accessing target-domain data.

3 Experiments

The experiments evaluate mDSDI across seven benchmark datasets and a Background-Colored-MNIST ablation, showing that it preserves domain-invariant information while exploiting useful domain-specific information. mDSDI achieves the highest reported average accuracy across the benchmark comparison and performs best in the ablation setting.

  • Benchmark evaluation: mDSDI is evaluated on seven benchmark datasets using comparisons with 14 related domain-generalization methods.The evaluation reports classification accuracy using DomainBed-style validation and repeated runs.
  • Domain-invariant information: 52.2% on Colored-MNIST, 98.0% on Rotated-MNIST, and 48.1% on Terra Incognita show competitive performance where target domains primarily contain domain-invariant information.The authors attribute this to retaining informative domain-invariant features through disentangled representations and adversarial training.
  • Domain-specific information: 79.0% on VLCS and 86.2% on PACS show that mDSDI captures useful domain-specific information in target domains sharing relevant object-background relations.The paper specifically discusses landscape backgrounds and related object-background patterns as useful domain-specific cues.
  • Benchmark results: 67.9% average accuracy is the highest reported result across the seven benchmark datasets.The paper states that the difference is statistically significant according to a t-test at α = 0.05.
  • Benchmark results: 69.2% on Office-Home and 42.8% on DomainNet indicate that mDSDI also performs strongly on larger and more complex benchmark settings.The paper links these results to balancing informative domain-invariant and domain-specific features.
  • Ablation study: 89.7% is the best Background-Colored-MNIST ablation setting, while removing disentanglement loss reduces accuracy to 81.4%.The comparison also reports 82.1% for meta-training both representations, 79.0% for meta-training domain-invariant features, and around 80.4% without meta-training.

4 Conclusion and Discussion

The paper concludes that combining disentangled domain-invariant and domain-specific representations with meta-training supports domain generalization. It also identifies unresolved theoretical, implementation, independence, and computational challenges.

  • Conclusion: mDSDI combines domain-invariant and domain-specific representations, disentangles them, and meta-trains the domain-specific representation for adaptation to unseen domains.The paper presents this combination as supported by its theoretical analysis and experiments.
  • Conclusion: The Background-Colored-MNIST ablation demonstrates the efficiency of combining domain-invariant and domain-specific features through mDSDI.The paper states that domain-specific information is useful when combined with domain-invariant information.
  • Discussion: Open challenges include determining when domain-specific features may hurt unseen-domain performance and strengthening the connection between theory and implementation.The paper also identifies nonlinear independence between representations and the computational cost of the covariance matrix as remaining issues.

A.1 Proof of Lemma 1

The proof begins from the case where ZX1 is produced by deterministic functions of X1 and considers events in the sigma-algebra induced by ZX1.

  • Proof setup: The proof analyzes measurable events induced by ZX1 when ZX1 is the output of deterministic functions from X1.This establishes the setting for the lemma’s measure-theoretic argument.

A.2 Proof of Theorem 1

The proof shows that representations incorporating label-related domain-specific information can retain more label information than domain-invariant representations alone. It separately establishes the mutual-information properties of domain-invariant representations.

  • The proof decomposes into label-related and domain-invariant learned representations.
  • Domain-invariant Learned Representations: Domain-invariant representations maximize mutual information with the shared domain information I(X1; X2).
  • Domain-invariant Learned Representations: The proof preserves the corresponding shared-information and conditional-information maxima involving X2 and Y.These include I(ZX1; X2; Y ) and I(ZX1; X2|Y ).
  • Domain-invariant Learned Representations: At the optimum, I(ZI∗; Y |X2) = 0, yielding I(X1; Y ) = I(ZI∗; Y ) + I(X1; Y |X2).
  • Label-related Learned Representations: I(Zsup; Y ) = I(Zsup∗; Y ) = I(X1; Y ) = I(ZI∗; Y ) + ϵ1 > I(ZI∗; Y ).The result follows by combining the label-related representation result with ϵ1 = I(X1; Y |X2).

B.1 Domain-specific features may hurt classification performance in the test domain

Domain-specific features can improve label dependence but may hurt unseen-domain performance when their label correlations change. The implementation approximates disentanglement and mutual-information objectives, with computational and optimization costs.

  • Domain-specific features may hurt performance when they correlate with different class labels in the unseen domain than in source domains.The paper gives a target-domain accuracy drop as an example of this failure mode.
  • The framework approximates its information-theoretic objectives rather than exactly minimizing mutual information because exact computation is costly.
  • Connection to theory: Cross-entropy is used as a proxy for minimal and sufficient label representations.
  • Connection to theory: Adversarial domain classifiers act as proxies for minimal and sufficient domain-invariant representations.
  • Connection to theory: Disentanglement minimizes covariance between domain-invariant and domain-specific representations as a proxy for minimizing their mutual information.The covariance formulation operates componentwise across the two feature vectors.
  • Computational limitation: A 2048 × 2048 covariance matrix consumes around 30GB of GPU memory with a ResNet50 backbone on PACS.
  • Computational limitation: An adversarial alternative avoids dependence on feature dimension but produces higher accuracy variance.

C.1 Dataset details

The datasets vary in how strongly domain-specific information appears alongside domain-invariant object or digit content. Background-Colored-MNIST is constructed to test whether class-correlated background color provides useful domain-specific information.

  • Colored-MNIST: Colored-MNIST constructs color labels by flipping digit-derived labels with a domain-specific noise rate.The preliminary digit label separates digits 0–4 from 5–9 before label and color noise are applied.
  • Background-Colored-MNIST: Background-Colored-MNIST assigns each class a consistent within-domain background color and uses related colors in the target domain.It contains 1,000 training samples per source domain and 10,000 target-domain test samples.
  • PACS: PACS contains object-centered domains where artpaint and photo include label-correlated colors or backgrounds, while cartoon and sketches are treated as mainly domain-invariant.
  • Office-Home: Office-Home combines domains with limited domain-specific information, such as clipart and product, with art and real images containing contextual features.The paper treats object sketches as domain-invariant and contextual features such as rooms as domain-specific.
  • Terra Incognita: Terra Incognita mainly contains domain-invariant animal information because camera locations change backgrounds without correlating them with animal labels.
  • DomainNet: DomainNet includes domains assumed to contain mainly domain-invariant conceptual drawings and others with colors, backgrounds, or descriptive text.Quickdraw and sketch are contrasted with clipart, infographic, painting, and photo.

C.3 Implementation details

The experiments follow DomainBed-like settings, use dataset-specific backbones and preprocessing, and select models with source-domain validation before testing.

  • The implementation uses DomainBed-like settings for data processing, architectures, hyperparameters, and reproducibility.
  • Data processing: MNIST experiments use 28 × 28 × d inputs, while other datasets are resized to 224 × 224 × 3 after randomized augmentation.
  • Evaluation: Models are selected by source-domain validation and evaluated after 5000 iterations with three random seeds.

C.4 Empirical result details

Across DG benchmarks, mDSDI generally performs competitively by combining domain-specific and domain-invariant information, with especially strong results when backgrounds or colors provide useful domain-specific cues.

  • Colored-MNIST: 52.2% average accuracy trails ARM’s 56.2% on Colored-MNIST, although reversed label-color correlations challenge all models.The reported reversal involves an unseen domain with 90% flipped color.
  • Rotated-MNIST: 98% average accuracy on Rotated-MNIST remains competitive despite the dataset containing limited domain-specific information.The dataset primarily contains domain-invariant information because its images lack backgrounds.
  • VLCS: 79.0% average accuracy on VLCS is highest, with mDSDI benefiting from background-only samples while retaining useful object information.mDSDI dominates three domains: LabelMe, Sun09, and VOC2007.
  • PACS: 86.2% average accuracy on PACS combines strong results in background-rich photo and art domains with competitive performance on cartoon and sketch.Reported accuracies are 98.1% for photo, 87.7% for art, 80.4% for cartoon, and 78.4% for sketch.
  • Office-Home: 69.2% average accuracy on Office-Home outperforms other methods, particularly on background-informative Art and Real-world domains.The reported Art and Real-world accuracies are 68.1% and 80.4%, respectively.
  • Terra Incognita: 48.1% average accuracy on Terra Incognita is second best, consistent with the dataset’s limited domain-specific features.SagNet achieves 48.6% on average.
  • DomainNet: 42.8% average accuracy on DomainNet is highest, with mDSDI dominating four domain tests in the 345-class benchmark.The results are attributed to balancing domain-invariant and domain-specific features across environments.
  • Training behavior: During training, the domain-invariant discriminator is successfully fooled, domain-specific features are extracted, and disentanglement loss decreases to zero.These observations come from the loss visualization for PACS with ResNet-50.

C.6 Ablation study: Important of mDSDI on the benchmark dataset

The PACS ablation study shows that domain-specific features help on color-rich targets, domain-invariant features help on sketch, and combining both with disentanglement performs best.

  • Target-domain effects: Domain-specific settings perform better for photo, art, and cartoon targets containing color information.DS-Meta reaches 87.1% on art and 79.2% on cartoon.
  • Target-domain effects: Domain-invariant settings outperform domain-specific settings on sketch, where color information is absent.DI-Meta reaches 75.1% in the reported sketch setting.
  • Combined representation learning: mDSDI-Meta DS outperforms DSDI-Meta DI across all target-domain settings and reaches 86.2% average accuracy.The comparison supports meta-training domain-specific features while jointly retaining domain-invariant information.
  • Disentanglement: 83.9% average accuracy without disentanglement loss is about 2.5% lower than the highest mDSDI-Meta DS result.The ablation attributes this gap to the role of covariance-based disentanglement in separating the two representations.
Loading 2110.09410v1…