Source-linked AI summary
Support and Invertibility in Domain-Invariant Representations
Fredrik D. Johansson, David Sontag, Rajesh Ranganath
TL;DR
The paper addresses gaps in theoretical justifications for domain-invariant representations, especially their treatment of information loss and strict density matching. It develops representation-aware, support-based adaptation bounds and finds that domain invariance is often too strong, while overlapping support can suffice for transfer.
Problem
Existing domain-invariant representation guarantees do not account for information lost in non-invertible transformations and do not characterize when invariance yields consistent estimation.
Method
The paper derives generalization bounds for arbitrary representations that account for non-invertibility and measure support coverage rather than density distance.
Results
Domain invariance is often too strong: overlapping support is sufficient for domain transfer, while equality of densities is not necessary.
Takeaways & Limitations
Support-based criteria and explicit recognition of representation-induced information loss provide alternatives to enforcing domain-invariant densities.
Takeaways & Limitations
The paper notes that its necessary representation assumptions are not guaranteed in practice and are insufficient alone for consistent domain adaptation.
Abstract
from arXiv · showhide
Learning domain-invariant representations has become a popular approach to unsupervised domain adaptation and is often justified by invoking a particular suite of theoretical results. We argue that there are two significant flaws in such arguments. First, the results in question hold only for a fixed representation and do not account for information lost in non-invertible transformations. Second, domain invariance is often a far too strict requirement and does not always lead to consistent estimation, even under strong and favorable assumptions. In this work, we give generalization bounds for unsupervised domain adaptation that hold for any representation function by acknowledging the cost of non-invertibility. In addition, we show that penalizing distance between densities is often wasteful and propose a bound based on measuring the extent to which the support of the source domain covers the target domain. We perform experiments on well-known benchmarks that illustrate the short-comings of current standard practice.
1 Introduction
The paper questions whether domain-invariant representation learning is theoretically justified and argues that both non-invertibility and overly strict invariance can undermine adaptation. It proposes support-based bounds and studies these issues empirically.
- Unsupervised domain adaptation transfers labeled knowledge from a source domain to an unlabeled target domain.
- The paper constructs examples where domain-invariance objectives are minimized despite arbitrarily bad target error, while empirical performance can coexist with surprisingly large bounds.
- Domain-invariant regularization can be too strict when source and target domains partially overlap.The paper gives examples where source-only empirical risk minimization outperforms domain-invariant representation learning.
- The proposed alternative is a generalization bound based on support overlap, applied directly to learned representations.
- Existing domain-invariant representation bounds do not account for information lost by non-invertible transformations.
- The paper evaluates domain-invariant representation learning on a well-known benchmark through the lens of its theoretical findings.
2 Background
This section defines unsupervised domain adaptation, reviews importance weighting and target-risk bounds, and formalizes domain-invariant representations. It also highlights how reducing distributional discrepancy can reduce label-prediction capacity.
- Unsupervised domain adaptation uses labeled source samples and unlabeled target samples to obtain a hypothesis minimizing target risk.
- Under covariate shift, the conditional label distribution given features is stationary across domains, although this assumption is not universally justified.
- Importance weighting: Importance weighting estimates target risk by weighting source observations according to the target-to-source density ratio.
- Importance weighting: Importance-weighted estimators require sufficient source support for the target; otherwise they can have high variance or become inapplicable.
- Upper bounds on target risk: Upper-bound approaches measure disagreement between hypotheses across domains, but reducing the hypothesis class can also reduce its capacity to predict labels.
- Domain-invariant representation learning: A domain-invariant representation is one whose induced source and target densities are equal, commonly learned by minimizing empirical domain variance.
3 Related work
Related work largely studies domain adaptation under covariate shift using invariant representations and distributional distances. The paper challenges whether matching transformed-domain distributions preserves label information or identifies the target labeling function.
- Most prior domain-adaptation work assumes covariate shift, while other work considers target-label and conditional-distribution shifts.
- Common representation-learning approaches minimize adversarial losses, integral probability metrics such as MMD, Wasserstein distance, or other divergences.
- It is unclear when similar transformed feature distributions imply similar conditional label distributions.
- Prior results show that covariate shift and small H∆H-distance alone do not identify labels, whereas sufficient support can be sufficient by counterexample.
- The paper argues that searching for equal transformed-domain distributions can be undesirable and insensitive to information lost in domain-invariant representations.
4 Limitations of domain-invariant representation learning
The paper identifies two major limitations of domain-invariant representation learning: non-invertible representations can lose predictive information, and density-based invariance can be stricter than necessary when supports overlap. These limitations can produce poor target-risk guarantees despite favorable objectives or assumptions.
- 4 Limitations of domain-invariant representation learning: Domain-invariant representation learning can have minimal objectives while yielding arbitrarily bad target error.Example 1 shows two optimal solutions with objective value 0 but radically different target risk.
- 4 Limitations of domain-invariant representation learning: Even an optimal domain-invariant representation is not sufficient for consistent domain adaptation, because variables differing across domains may remain critical for prediction.The paper states that this assumption is necessary for domain-invariant learning to be consistent but is not sufficient for consistent adaptation by any method.
- 4 Limitations of domain-invariant representation learning: A representation may make source and target distributions identical while destroying the conditions needed for reliable prediction.The induced covariate-shift assumption need not hold after representation, and the relevant equality holds generally only when the representation is invertible.
- 4 Limitations of domain-invariant representation learning: Density-distance regularization can prefer a problem with poor support overlap over one whose target support is fully covered by the source.In Example 2, MMD is smaller in Problem B despite insufficient overlap, whereas Problem A satisfies sufficient-support assumptions for identifiability.
- 4 Limitations of domain-invariant representation learning: When domains partially overlap and label marginals differ, requiring domain invariance can be too strict, with source-only training sometimes outperforming invariant representation learning.The paper presents this as a practical consequence of the density-distance problem.
5 A new support-based bound
The paper develops adaptation bounds for arbitrary representations that explicitly account for information loss and measure source support coverage of the target rather than only density distance.
- Support sufficiency divergence: The support sufficiency divergence measures target mass located where source density is below a threshold, and is directional rather than symmetric.It is zero when distributions are equal and can also be zero for unequal densities with identical support; its range is bounded by 1.
- Representation information loss: Both excess information-loss terms vanish for invertible representations, while non-invertibility can create adaptation error that is generally unobservable.The paper notes that the information loss may be negative in some cases, although this is not expected in practice.
- Theorem 2: Theorem 2 bounds target risk using source error, support sufficiency, and excess information loss from non-invertible representations.The bound applies to representations z = φ(x) and predictions h = f ◦ φ under the stated bounded-loss assumptions.
- Bound behavior: Theorem 2 becomes non-informative for completely disjoint supports, whereas partial overlap yields informative adaptation bounds.The bound is tight when source and target distributions match; an alternative bound addresses disjoint supports under stronger assumptions.
- Relation to importance sampling: The bounds connect support-based adaptation with importance sampling, trading off bound terms and estimator variance through the threshold parameter.For common bounded support, minimizing Theorem 2 can reduce to importance sampling; smaller thresholds can increase weights and variance.
- Theorem 3: Theorem 3 replaces the constant out-of-support penalty with an IPM support sufficiency divergence, penalizing extrapolation chiefly where source density is low.This tighter bound requires the representation-induced losses to belong to a known function family and is subject to stronger assumptions.
6 Empirical results
Experiments revisit DANN on MNIST-based shifts, including settings with and without sufficient support, and compare unsupervised adaptation with target-tuned predictions using fixed learned representations.
- MNIST→MNIST-M: The MNIST-M benchmark overlays handwritten digits with random photographs, creating full-color target images with measure-zero support under MNIST.Prior results reported over 55% source-only target accuracy, over 80% with unlabeled target data, and over 95% with labeled target data.
- Controlled support shift: The controlled experiment removes digit classes from the target while keeping the source fixed, so target support is contained within source support.Source-only empirical risk minimization is therefore used as a strong baseline, alongside a similarly perturbed MNIST-M condition.
- DANN setup: The study varies DANN’s domain-adversarial penalty strength α, interpolating between empirical risk minimization, standard DANN, and stronger domain-invariance emphasis.The objective uses α = 0 for empirical risk minimization and α = 0.5 for the standard DANN formulation.
- Results: As label marginals become more unequal, unsupervised DANN performance worsens on MNIST→MNIST and eventually offers no advantage over source-only training on MNIST→MNIST-M.DANN remains beneficial for small label shifts in MNIST→MNIST-M, despite insufficient support.
- Representation quality: Target-tuned predictors using fixed DANN representations perform almost as well as fully target-trained models, while unequal-marginal embeddings show poorer class separation.The comparison tests whether the learned representation itself preserves realizability of the target labels.
7 Discussion
The discussion concludes that standard domain-invariance arguments are flawed because their bounds overlook representation information loss and because equal densities are stricter than necessary for transfer.
- Conclusions: The paper finds that oft-cited generalization bounds are not minimized by the learned representations and do not characterize when domain invariance yields consistent estimation.Examples and benchmark experiments indicate that domain invariance can be too strong both with and without domain overlap.
- Conclusions: Overlapping support can suffice for domain transfer, so equality of source and target densities is not necessary.This conclusion is presented as the explanation for why domain-invariant representations may impose an unnecessarily strict requirement.
- Implications: The proposed bounds measure support distance and explicitly account for loss caused by non-invertible representations.The discussion presents these as alternatives to density-distance penalties.
- Implications: The paper suggests hinge losses, selective regularization in low-source-density regions, and reconstruction losses as possible algorithmic responses.Hinge losses can make derivatives informative, while reconstruction losses aim to keep representation-induced excess adaptation error small.
A.1 Proof of bounds for support sufficiency divergence
The appendix proves basic properties of the support sufficiency divergence, including nonnegativity, boundedness, and tightness.
- Proof properties: The support sufficiency divergence is nonnegative and equals zero when the threshold does not exceed the minimum density of p.The proof also establishes the upper bound using the indicator definition.
- Proof properties: The divergence has a tight upper bound of 1, demonstrated by discrete distributions concentrated on different states.The proof uses q = [1., 0.] and p = [0., 1.] with a positive threshold.
A.2 Proof of Lemma 1
The section states a representation-level domain-adaptation theorem under bounded loss and connects support divergence to kernel-based measurements.
- Proof structure: The proof proceeds through intermediate lemmas before establishing the theorem's bound.
- Representation-level bound: Theorem 2 considers any representation z = φ(x), prediction function f, and induced source and target distributions over representations.It assumes bounded loss for every hypothesis and loss function.
- Support divergence: The support divergence compares density differences where the domain supports differ significantly.
- Kernel formulation: When the function space is Hilbert, the divergence can be decomposed using reproducing kernels, similarly to maximum mean discrepancy.
C Experiments
The experiments examine domain-invariant learning under unequal label marginals and formulate variable-selection conditions for identifiable, sufficient representations. They report that DANN embeddings separate classes less effectively under label-marginal shift and present a theorem for variable-selection estimation under stated assumptions.
- Domain-adaptation experiments: DANN embeddings show worse class separation under unequal than equal label marginal distributions.
- Variable selection: The variable-selection setup asks whether Y is identifiable from domain invariance and source predictive loss when Z = AX is sufficient and has equal source and target distributions.
- Assumptions: The analysis assumes smoothness, identifiable variables, positive bounded source support, bounded norms, and sub-Gaussian additive noise.
- Theoretical result: Under Conditions 1–5 and known parameters, Theorem 4 provides an estimator of the relevant variable set with a stated guarantee.
- Estimator dependence: The constructive estimator in the cited variable-selection result may depend on the source density.