PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Generalization via Multidomain Discriminant Analysis

Shoubo Hu, Kun Zhang, Zhitang Chen, Laiwan Chan

arXiv:1907.11216v1stat.MLcs.LG

TL;DR

Domain generalization seeks to generalize from source domains to unseen targets without target data, despite distribution differences. MDA learns a class-discriminative feature transformation, and the paper reports strong synthetic and real-benchmark results alongside learning bounds.

  • Problem

    Domain generalization asks how to use multiple source domains to generalize to unseen targets when target data are unavailable and distributions can differ.

    Source: §1 · p. 1

  • Method

    MDA learns a domain-invariant feature transformation by combining measures of within-class domain discrepancy, between-class separability, and instance-level information.

    Source: §1 · p. 2, §3.2.3 · p. 4, §6 · p. 9

  • Results

    MDA performs best in the synthetic base setting and all tested source-class-prior settings, with reported efficacy on synthetic and two real benchmarks.

    Source: §5.2 · p. 8, §6 · p. 9

  • Takeaways & Limitations

    MDA addresses classification DG in a broader setting where both P(Y) and P(X|Y) can change across domains.

    Source: §6 · p. 9

    The theoretical setup assumes each sample set comes from a sampled domain, followed by iid draws from that domain.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Domain generalization (DG) aims to incorporate knowledge from multiple source domains into a single model that could generalize well on unseen target domains. This problem is ubiquitous in practice since the distributions of the target data may rarely be identical to those of the source data. In this paper, we propose Multidomain Discriminant Analysis (MDA) to address DG of classification tasks in general situations. MDA learns a domain-invariant feature transformation that aims to achieve appealing properties, including a minimal divergence among domains within each class, a maximal separability among classes, and overall maximal compactness of all classes. Furthermore, we provide the bounds on excess risk and generalization error by learning theory analysis. Comprehensive experiments on synthetic and real benchmark datasets demonstrate the effectiveness of MDA.

1 INTRODUCTION

Domain generalization uses multiple source domains to train for an unseen target, where target data are unavailable and source and target distributions may differ. MDA addresses this setting with a class-discriminative feature transformation and learning-theoretic bounds.

1 INTRODUCTION

  • Unlike standard supervised learning's same-distribution assumption, source and target data may differ because they were collected under different conditions.The paper gives viewpoints, illumination, and backgrounds as examples.Source: §1 · p. 1
  • Domain generalization (DG) uses multiple source domains to improve prediction on an unseen target, with no target-domain data available during training.A domain is a joint distribution P(X, Y).Source: §1 · p. 1
  • Prior DG approaches commonly seek representations with stable conditional distributions, while kernel methods express these representations as feature transformations.Source: §1 · p. 1
  • Prior work distinguishes covariate, target, and conditional shift by which of P(Y|X), P(Y), and P(X|Y) change across domains.The cited discussion characterizes covariate shift by stable P(Y|X), target shift by changing P(Y) with stable P(X|Y), and conditional shift by the reverse.Source: §1 · pp. 1–2
  • Under conditional shift in the anti-causal direction, changing P(X|Y) while P(Y) remains stable generally changes both P(Y|X) and P(X).Source: §1 · p. 2
  • MDA relaxes the stable-P(Y) assumption by focusing on class separability and combining average class discrepancy with three other measures in a feature-transformation objective.The paper also derives excess-risk and generalization-error bounds for kernel-based methods and reports experimental efficacy on synthetic and real datasets.Source: §1 · p. 2

2 PRELIMINARY ON KERNEL MEAN EMBEDDING

Kernel mean embedding represents probability distributions as elements of a reproducing kernel Hilbert space. The section also describes its finite-sample estimation and the distribution-identifying property of characteristic kernels.

2 PRELIMINARY ON KERNEL MEAN EMBEDDING

  • Kernel mean embedding represents a probability distribution as an element of an RKHS using the expected canonical feature map.The canonical feature map is φ(x) := k(x, ·), and the embedding is the expectation of φ(X).Source: §2 · p. 2
  • For a finite sample, the paper estimates the kernel mean embedding empirically.Source: §2 · p. 2
  • With a characteristic kernel, the embedding captures all information about the distribution, and two distributions have equal embeddings if and only if they are identical.Source: §2 · p. 3

3 MULTIDOMAIN DISCRIMINANT ANALYSIS

MDA learns a transformed representation for classification across unseen domains by combining measures of within-class domain alignment, between-class separation, and instance-level compactness. The section defines these measures and describes their kernel-based transformation, estimation, and joint optimization.

3.1 PROBLEM DEFINITION

  • Domains are joint distributions over features and labels; each sample set is drawn from a domain, then contains i.i.d. instances, and training uses the observed source sets to target an unseen domain.The assumed distribution over domains has finite variance and is unimodal.Source: §3.1 · p. 3

3.2.1 Average Domain Discrepancy

  • Average domain discrepancy measures within-class distribution differences across source domains; with a characteristic kernel, it is zero exactly when those distributions match within each class.The discrepancy is based on distances between kernel mean embeddings.Source: §3.2.1 · p. 3

3.2.2 Average Class Discrepancy

  • Average class discrepancy is proposed because aligning same-class means alone can leave different-class means close; maximizing it tends to distinguish distributions of different classes.The paper relates MMD separation to first-Wasserstein separation when the kernel is bounded.Source: §3.2.2 · pp. 3–4

3.2.3 Incorporating Instance-level Information

  • Between-class scatter pools class instances around the global mean, while within-class scatter sums instance-to-class-mean distances, capturing compactness that distribution means alone miss.The within-class scatter uses information from every instance.Source: §3.2.3 · p. 4

3.3 FEATURE TRANSFORMATION

  • The method maps features into an RKHS, learns a transformation to a q-dimensional space, and expresses the transformation as a linear combination of the data's canonical feature maps.The transformed discrepancy and scatter measures are expressed using the learned transformation.Source: §3.3 · pp. 4–5

3.4 EMPIRICAL ESTIMATION

  • Empirical estimation uses source samples to estimate class-conditional embeddings and class mean representations, assuming equal sampling probabilities for the source domains and using the kernel trick.The class-mean estimation uses Bayes' rule to estimate domain probabilities conditioned on class.Source: §3.4 · p. 5

3.5 THE OPTIMIZATION PROBLEM

  • Four transformed measures are combined with trade-off weights in an eigenvalue problem that preserves class separation while favoring compact, domain-invariant distributions.The stated overall computational complexity is O(n^2 + qn^2), the same as existing kernel-based methods.Source: §3.5 · pp. 5–6

4 LEARNING THEORY ANALYSIS

The analysis derives excess-risk and generalization-error bounds after feature transformation, for a setting where both input marginals and conditional label distributions may vary across domains. The bounds expose how the transformation and sample sizes affect the guarantees.

4 LEARNING THEORY ANALYSIS

  • The analysis allows both P(X) and P(Y|X) to change across domains, with decision functions defined over a domain distribution and an input.It considers functions in an RKHS and uses a product kernel over domain distributions and inputs.Source: §4 · p. 6
  • Bounded kernels satisfy assumptions 2 and 3, and the analysis adopts a Gaussian kernel; the loss is assumed Lipschitz and bounded.Source: §4 · p. 6
  • The excess-risk bound controls test-domain risk under assumptions 2–4 and unit-norm conditions on the empirical and expected risk minimizers.Its distortion term tr(B^T KB) depends on the choice of B, while another term is O(n^-1/2).Source: §4 · p. 6
  • The generalization-error bound assumes equal-sized source sample sets; its first term is O(m^-1/2), and the remaining terms vanish when m and n̄ grow with log m/n̄ = o(1).The bound also contains a term involving tr(B^T KB), so it depends on the choice of B.Source: §4 · p. 7

5 EXPERIMENTS

Experiments compare MDA with established classification and domain-generalization methods on synthetic data and two real benchmarks. MDA is strongest on synthetic class-prior shifts and generally competitive on real datasets, with results varying by benchmark and target domain.

5.1 EXPERIMENTAL CONFIGURATION

  • The comparison includes nine methods spanning 1NN and SVM baselines, feature-extraction, SVM-based, neural-network-based, and kernel-based DG approaches.The listed DG methods include L-SVM, CCSA, DICA, SCA, and CIDG.Source: §5.1 · p. 7
  • For real-data experiments, hyperparameters are selected by five-fold cross-validation on labeled source data, and target-domain classification accuracy is the evaluation criterion.In the synthetic experiments, other methods train on two source sample sets, validate on one target set, and test on the other target set.Source: §5.1 · pp. 7–8

5.2 SYNTHETIC DATA

  • MDA performs best in the synthetic base condition and all tested source class-prior settings, while DICA matches it in two settings but is more affected by prior variation.The results are consistent with MDA working with class-conditional rather than marginal distributions.Source: §5.2 · p. 8, Table 3 · p. 8

5.3 OFFICE+CALTECH DATASET

  • On Office+Caltech, MDA performs best by a smaller improvement margin than in the synthetic experiment; it is the only kernel-based method to beat 1NN for target A,C.L-SVM ranks second among the reported methods; four other target cases are omitted because 1NN already exceeds 90%.Source: Table 4 · p. 8, §5.3 · p. 9

5.4 VLCS DATASET

  • On VLCS, MDA is best among DG methods in 8 of 10 cases, while SVM beats all DG methods in 6 cases and CCSA beats MDA when S is the target.The authors suggest class overlap may help explain SVM’s results and that neural networks may extract better features when S is the target.Source: §5.4 · p. 9, Table 5 · p. 9

6 CONCLUSION

The conclusion presents MDA as a classification DG method for settings where both class priors and class-conditional distributions may shift across domains. It also reports theoretical risk and generalization bounds and experimental verification on synthetic and real benchmarks.

6 CONCLUSION

  • MDA addresses classification DG when both P(Y) and P(X|Y) change across domains, combining four measures in an objective for domain-invariant feature transformation learning.The measures include average domain discrepancy, average class discrepancy, and two measures based on kernel Fisher discriminant analysis.Source: §6 · p. 9
  • The paper proves bounds on excess risk and generalization error for kernel-based DG methods and reports MDA’s effectiveness on synthetic and two real benchmark datasets.Source: §6 · p. 9

A Quantities’ Property Illustration

The illustrations distinguish transformations that reduce within-class domain differences from those that compact classes, and contrast two ways of increasing between-class separation.

A Quantities’ Property Illustration

  • Minimizing average domain discrepancy brings same-class means closer across domains, whereas minimizing multidomain within-class scatter makes each class more compact around its mean.Figure 3 uses colors for classes and markers for domains.Source: Figure 3 · p. 12
  • Average class discrepancy maximizes pairwise distances between class means equally, while multidomain between-class scatter weights class-mean distances by class size.The latter is approximately equivalent to pooling data from different domains within each class and training a classifier.Source: Figure 4 · p. 12

B Derivation of the Lagrangian

The constrained optimization is recast as a Lagrangian, whose stationary condition yields a generalized eigenvalue problem for the transformation.

B Derivation of the Lagrangian

  • Setting the Lagrangian derivative with respect to B to zero yields the generalized eigenvalue problem that determines B for the feature transformation.The equation balances (βF + (1 − β)P)B against (γG + αQ + K)BΓ.Source: B Derivation of the Lagrangian · pp. 12–13

C Proof of Theorem 2

The proof derives a high-probability excess-risk bound by combining a generalization bound with Rademacher-complexity estimates. The bound concerns expected risk on the test domain.

C Proof of Theorem 2

  • Theorem 4 bounds the difference between empirical-risk-minimizer and expected-risk-minimizer test risks under its stated assumptions.The available passages do not include the bound's full expression.Source: C Proof of Theorem 2 · p. 13
  • The proof applies a Rademacher-complexity generalization theorem to the loss class and uses the loss function's Lipschitz property.Theorem 5 supplies the generalization step; the proof then bounds the loss-class complexity.Source: C Proof of Theorem 2 · p. 13
  • A bound for an L2 ball and the kernel-feature norm are used to obtain the complexity estimate and complete the proof.The proof identifies the feature-space norm with the Hilbert-Schmidt norm before applying the L2-ball result.Source: C Proof of Theorem 2 · pp. 13–14

D Proof of Theorem 3

The proof establishes a high-probability generalization bound under assumptions 2–4 and equal-sized source samples. It decomposes the error into terms and bounds them using concentration and complexity arguments.

D Proof of Theorem 3

Bound of term (I)

  • Term (I) is bounded using Lipschitzness, kernel properties, concentration inequalities, and a union bound across source domains.The derivation invokes Hoeffding-type bounds for bounded variables and applies the resulting bound across domains.Source: Bound of term (I) · pp. 14–16

Bound of term (II)

  • Term (II) is decomposed using a conditional idealized test error for a given test distribution.The proof denotes the augmented test input by ˜Xt = (P t X, Xt).Source: Bound of term (II) · p. 16

Bound of term (IIa)

  • For term (IIa), conditioning makes the source samples independent but not identically distributed, enabling a McDiarmid-inequality bound.The proof also uses the loss-class Rademacher-complexity bound.Source: Bound of term (IIa) · pp. 16–17

Bound of term (IIb)

  • For term (IIb), the proof uses McDiarmid's inequality and symmetrization before combining the bounds into the announced result.It again uses a Rademacher-complexity bound for the loss class.Source: Bound of term (IIb) · p. 17

E Experimental Configurations

The experiments tune method-specific hyperparameters and report the highest accuracies across tested transformed-subspace dimensions for several feature-extraction and kernel-based methods.

E Experimental Configurations

  • 1NN uses combined source-domain instances directly, while SVM tunes its regularization coefficient C.SVM validates C values from 0.1 to 10.0.Source: E Experimental Configurations · p. 17
  • KPCA and KFD tune kernel width, while E-SVM and CCSA tune their respective method-specific parameter sets.The passages specify the tested kernel-width, E-SVM, and CCSA grids.Source: E Experimental Configurations · p. 17
  • DICA, SCA, and CIDG tune the listed method-specific parameters in their experiments.DICA tunes λ and ϵ; SCA tunes β and δ; CIDG tunes β, α, and γ.Source: E Experimental Configurations · p. 17
  • MDA tunes β, α, and γ over the specified validation grids.The grids span β values from 0.1 to 0.9, γ from 1e−3 to 1e6, and α from 1 to 1e9.Source: E Experimental Configurations · p. 18
  • For real-data experiments, the highest accuracy is reported across tested leading-eigenvector proportions for KPCA, KFD, DICA, SCA, CIDG, and MDA.The tested proportions of the eigenvalue sum are 0.2, 0.4, 0.6, 0.8, 0.92, 0.94, 0.96, and 0.98.Source: E Experimental Configurations · p. 18

F Synthetic Experimental Results Visualization

The synthetic visualizations compare transformed source and target data across class-prior cases and DG methods; MDA shows the best target-class separation in the reported comparison. The related-work section situates MDA among projection-based and other DG approaches.

F Synthetic Experimental Results Visualization

  • MDA's transformation gives the best separation of target-domain classes, handles source-class overlap slightly better than CIDG, and SCA struggles in some cases.Figures 5 and 6 show class-prior cases by row and DG methods by column, with raw data first; colors distinguish classes, circles sources, and crosses targets.Source: F Synthetic Experimental Results… · p. 18, Figure 5 · p. 19, Figure 6 · p. 20

G Related Work

  • DICA learns a shared subspace that minimizes marginal-distribution differences while preserving the functional relationship between labels and features [Muandet et al., 2013].It was the first feature-projection method to bring shared-subspace learning to domain generalization.Source: G Related Work · p. 21
  • SCA combines domain scatter, KPCA, and KFD; CIDG considers settings where both P(Y |X) and P(X) vary across domains [Li et al., 2018b].The passage describes SCA as a unified framework for domain adaptation and domain generalization, and CIDG as using a causal perspective.Source: G Related Work · p. 21
  • Other DG approaches include max-margin, metric-learning, exemplar-SVM, multitask-autoencoder, and deep-neural-network methods.The passage associates these approaches with Khosla et al. [2012], Fang et al. [2013], Xu et al. [2014], Ghifary et al. [2015], and later deep-learning work [Motiian et al., 2017, Li et al., 2017, 2018a,c].Source: G Related Work · p. 21

Related papers

Loading 1907.11216v1…