PaperlayerView exact revision on arXiv

Source-linked AI summary

Domain Adaptation with Conditional Distribution Matching and Generalized Label Shift

Remi Tachet, Han Zhao, Yu-Xiang Wang, Geoff Gordon

arXiv:2003.04475v3cs.LGcs.AIstat.ML

TL;DR

Domain adaptation methods that learn domain-invariant representations can be limited when source and target label distributions differ. The paper introduces generalized label shift, derives guarantees and importance-weighted modifications of existing algorithms, and reports consistent improvements, especially under larger mismatches. The framework is simple to implement with barely any additional computational cost, but its weight estimation can fail for some classes.

  • Problem

    Existing domain adaptation methods face limited transfer performance when source and target domains have mismatched label distributions.

  • Method

    The paper introduces generalized label shift, derives performance guarantees, estimates source-target class ratios, and uses them as importance weights in modified JAN, DANN, and CDAN objectives.

  • Results

    The modified algorithms consistently outperform their base versions on classic and artificial tasks, with larger gains under larger label-distribution divergences.

  • Takeaways & Limitations

    The reweighting framework is straightforward to apply and adds barely any computational cost, making robustness to mismatched label distributions relevant to real-world applications.

  • Takeaways & Limitations

    Failure in estimating the weights of some classes might result in poor performance on those classes.

Abstract

from arXiv · show

Adversarial learning has demonstrated good performance in the unsupervised domain adaptation setting, by learning domain-invariant representations. However, recent work has shown limitations of this approach when label distributions differ between the source and target domains. In this paper, we propose a new assumption, generalized label shift ($GLS$), to improve robustness against mismatched label distributions. $GLS$ states that, conditioned on the label, there exists a representation of the input that is invariant between the source and target domains. Under $GLS$, we provide theoretical guarantees on the transfer performance of any classifier. We also devise necessary and sufficient conditions for $GLS$ to hold, by using an estimation of the relative class weights between domains and an appropriate reweighting of samples. Our weight estimation method could be straightforwardly and generically applied in existing domain adaptation (DA) algorithms that learn domain-invariant representations, with small computational overhead. In particular, we modify three DA algorithms, JAN, DANN and CDAN, and evaluate their performance on standard and artificial DA tasks. Our algorithms outperform the base versions, with vast improvements for large label distribution mismatches. Our code is available at https://tinyurl.com/y585xt6j.

1 Introduction

The paper studies unsupervised domain adaptation when source and target label distributions differ, a setting where standard adversarial alignment can have limited transfer performance. It introduces generalized label shift and theoretically motivated reweighting-based modifications to improve robustness.

  • Motivation: Unsupervised domain adaptation learns from labelled source samples and unlabelled target data to generalize to the target domain.Adversarial domain adaptation uses a discriminator to learn source-target-invariant representations alongside a source-trained classifier.
  • Problem: Mismatched label distributions expose an upper bound on existing domain adaptation algorithms, while label shift changes marginal label distributions but preserves class-conditional input distributions.This motivates studying domain adaptation beyond standard invariant-representation assumptions.
  • Contributions: The paper introduces generalized label shift, placing conditional source-target invariance in representation space rather than input space.It derives performance guarantees for feature transformations that enforce GLS and uses them to motivate algorithmic modifications.
  • Contributions: The proposed modifications estimate class ratios between source and target domains and use them as importance weights in adversarial and classification objectives.The weights are estimated by a method of moments formulated as a quadratic program.

2 Preliminaries

The preliminaries define domain adaptation through feature transformations, classifiers, and domain invariance, then describe adversarial implementations and their limitation under label-distribution mismatch. A lower-bound argument links stronger feature alignment to worse joint error in that setting.

  • Notation: The paper considers stochastic k-class classification with source and target domains represented as joint distributions over input and output spaces.The representation space Z is induced from input space X by a feature transformation g.
  • Invariant Representations: In unsupervised domain adaptation, representations are trained to make source and target feature distributions similar while preserving information needed for low source classification error.Feature-space invariance minimizes a distance or divergence between induced distributions.
  • Adversarial Domain Adaptation: Adversarial domain adaptation trains a discriminator to distinguish source from target representations while the feature extractor maximizes discriminator loss and the classifier minimizes source classification loss.DANN applies this procedure to representations, whereas CDAN conditions the discriminator on the outer product of predictions and representations.
  • Lower Bound: Theorem 2.1 lower-bounds the joint source-target classifier error using Jensen-Shannon divergence between domain label distributions.The theorem is stated for arbitrary k and extends to CDAN's prediction-representation input.
  • Lower Bound: When source and target label distributions differ, better feature-distribution alignment can worsen joint error, and zero source error still implies a positive target-error lower bound.The lower bound is algorithm-independent and holds at the population level asymptotically with increasing data.
  • Common Assumptions: Covariate shift is not robust to feature transformation and can cause negative transfer, whereas label shift fails in applications such as synthetic-to-real image transfer with disjoint input distributions.These limitations motivate assumptions that better fit representation-based domain adaptation.

3 Main Results

The paper introduces generalized label shift (GLS), which requires class-conditional invariance in a learned representation, and develops guarantees and conditions for using it in domain adaptation. Its framework estimates label-distribution importance weights and aligns reweighted source features with target features.

  • Generalized Label Shift: GLS requires the learned representation Z to have matching source and target distributions conditioned on every label.When the feature transformation is the identity, GLS reduces to standard label shift; unlike standard label shift, it can remain compatible with a perfect classifier under differing label marginals.
  • Error Decomposition Theorem: The error decomposition separates the source-target error gap into a label-marginal term weighted by source balanced error and a conditional error-gap term.The bound is described as independent of unknown optimal labeling functions in feature space and contrasts with bounds based on marginal feature-distribution distance.
  • Error Decomposition Theorem: Under GLS, the conditional error gap is zero, and any classifier satisfies εS(bY) + εT(bY) ≤ 2BERDS(bY ∥Y).Thus, the joint source and target error is bounded by twice the source balanced error rate.
  • Conditions for Generalized Label Shift: GLS implies matching the target feature distribution with a source feature distribution reweighted by relative class weights.The reweighted alignment is necessary, while clustering structure or additional conditions can make it sufficient; Theorem 3.4 establishes an equivalence under matched features and zero source error.
  • Estimating the Importance Weights w: The weight-estimation method is motivated by a quadratic program, and under zero source error plus perfectly matched reweighted and target marginals, the estimated weights equal the true weights.The framework is designed to reweight domain-adversarial losses and can extend to multiple source or target domains by maintaining one weight vector per domain pair.
  • Estimating the Importance Weights w: The importance weights can be estimated without target labels by solving w = C^-1µ when GLS holds and the source confusion matrix C is invertible.Because direct inversion is numerically unstable with finite samples, the paper instead proposes a constrained quadratic program based on estimated C and µ.

4 Practical Implementation

The paper implements importance-weighted variants of DANN, JAN, and CDAN by estimating relative class weights and reweighting source features or losses. Across standard, subsampled, and artificial tasks, these variants generally improve over their base algorithms, especially under larger label-distribution divergences.

  • Importance-weighted algorithms: IWDAN aligns target features with reweighted source features, while IWJAN and IWCDAN extend the same reweighting principle to JAN and CDAN.The framework modifies domain-adversarial or joint-distribution matching objectives using estimated importance weights.
  • Weight estimation: The method estimates class weights from source-per-class and target-overall classifier predictions, updating them each epoch with an exponential moving average.The reported experiments use λ = 0.5, and the added procedure has minimal computational overhead.
  • Artificial tasks: 6.55% and 8.14% are the mean improvements for IWDAN and IWDAN-O, while IWCDAN and IWCDAN-O improve by 2.25% and 2.81% across 100 artificial tasks.The tasks vary DJS(DY_T) between 0 and 0.1.
  • Original datasets: 1.75%, 1.64%, 1.16%, and 2.65% are IWDAN’s gains over DANN on Digits, Visda, Office-31, and Office-Home, respectively.IWCDAN and IWJAN also show gains, while oracle variants produce larger improvements.
  • Subsampled datasets: 9.3%, 7.33%, 6.43%, and 5.58% are IWDAN’s gains on subsampled Digits, Visda, Office-31, and Office-Home, respectively.For IWCDAN, improvements are 4.99%, 5.64%, 2.26%, and 4.99%; IWJAN gains are 6.48%, 4.40%, and 1.95%.
  • Weight dynamics: IWDAN improves monotonically in accuracy and weight estimation on sDigits, whereas DANN’s performance worsens after several epochs under inappropriate representation matching.IWDAN’s estimated weights do not converge perfectly, suggesting that tuning λ or update frequency could improve performance.
  • Ablation study: DANN benefits mainly from reweighting its adversarial loss, while both losses help CDAN, with an additional +2% gain for the weighted adversarial loss.The ablation study evaluates weighted adversarial and classification losses separately using true weights.

5 Related Work

The paper relates its approach to covariate shift, label shift, invariant-representation domain adaptation, partial adaptation, multi-domain adaptation, and feature-level reweighting. It emphasizes that class-label weighting is more efficient than weighting in a high-dimensional continuous feature space.

  • Shift assumptions and invariant representations: Prior work studies covariate shift and label shift through methods including EM, label-distribution priors, kernel mean matching, and causal or anti-causal analyses.Domain adaptation methods also align marginal feature distributions using divergences such as DJS, MMD, and Wasserstein distance.
  • Related adaptation settings: Partial domain adaptation allows target labels to be a strict subset of source labels, while multi-domain adaptation uses multiple source or target domains.These settings are discussed as related domain-adaptation problems involving differing domain compositions.
  • Importance weighting: The paper distinguishes its class-label importance ratio from prior feature-level ratios, which are described as statistically inefficient and computationally expensive in high-dimensional continuous feature spaces.The label space contains only k distinct classes, making class-based weighting more tractable.

6 Conclusion and Future Work

The paper concludes that GLS-based importance weighting improves domain adaptation under mismatched label distributions while adding little computational cost. It also identifies multi-domain adaptation as a direct extension and domain generalization as a more challenging future direction.

  • Conclusion: Across classic benchmarks and artificial tasks, the importance-weighted algorithms consistently outperform their base versions, with gains correlating with label-distribution JSD.The conclusion notes that real-world JSD may exceed that of deliberately balanced machine-learning datasets.
  • Extensions: The framework can extend to multiple source or target domains by maintaining one importance-weight vector for each source-target pair.The authors suggest it might alleviate performance degradation associated with too many source domains.
  • Future work: Domain generalization remains a more challenging future direction because the learner has no access to target-domain data, including unlabeled data.The paper specifically distinguishes this setting from domain adaptation, where target data are available.

Broader Impact

The paper frames domain adaptation as transferring knowledge across distributions while addressing mismatched source and target label distributions. Its theoretical development characterizes error under these mismatches and introduces GLS-based guarantees for classifiers.

  • Motivation: Domain adaptation transfers knowledge from a source distribution to a target distribution, with potential applications ranging from spam filtering to loan approvals.The paper notes that sensitive applications may involve recidivism prediction and loan approvals.
  • Broader impact: The authors argue that class imbalances can cause poor transfer for standard adaptation methods on some classes, creating a potential source of inequalities.Their method is intended to counterbalance these imbalances, especially under significant shifts.
  • Theoretical analysis: The error decomposition theorem bounds classifier error by label-distribution mismatch and a conditional error-gap term in k-class classification.The theorem applies to any classifier and explicitly includes the L1 distance between source and target label distributions.
  • Generalized label shift: Under GLS, any classifier operating on the invariant representation satisfies a transfer-error guarantee based on its source balanced error rate.The stated bound is εS(bY) + εT(bY) ≤ 2BERDS(bY ∥Y).

A.7 Proof of Theorem 3.3

The proof establishes that a clustered representation satisfying the weighted marginal condition also satisfies GLS. The argument uses a partition of representation space whose regions are perfectly associated with labels in both domains.

  • Theorem 3.3: If DT(Z) = DwS(Z) and representation space has label-specific regions, then the representation satisfies GLS.The theorem assumes DT(Y = y) > 0 for every class and a partition Z = ∪y∈Y Zy.
  • Support structure: Each partition region Zy contains the conditional support of class y in both source and target domains.For y′ ≠ y, both domains assign zero conditional probability to Zy′ given Y = y.
  • Conditional invariance: Applying the weighted marginal identity to measurable subsets E within Zy yields DS(Z ∈ E | Y = y) = DT(Z ∈ E | Y = y).The proof uses the positivity assumption DT(Y = y) ≠ 0 to cancel the target class probability.
  • Approximate sufficiency: The paper also derives a total-variation bound on conditional representation discrepancies from the mismatch between target and weighted source representations.The bound is stated for arbitrary classifiers and measurable subsets of representation space.

B.3 Full results on the domain adaptation tasks

The experiments report best test accuracy during training across domain-adaptation tasks and compare each importance-weighted variant with its base algorithm under matched random seeds.

  • Evaluation protocol: Best test accuracy during a fixed training horizon is reported to ensure fairness against baselines.The paper notes that DANN accuracy can decrease later in training because of inappropriate representation matching.
  • Comparison protocol: The subscript records the fraction of seeds for which an importance-weighted variant achieves higher test accuracy than its base algorithm.Each seed fixes network initialization and the data presented to the model.
  • Comparison protocol: The seed-wise comparison is designed to isolate the effect of the algorithm while holding other experimental factors constant.The protocol compares variants and baselines using the same seed-specific conditions.

B.4 Jensen-Shannon divergence of the original and subsampled domain adaptation datasets

This section compares Jensen–Shannon divergences across original and subsampled domain-adaptation datasets and explains how conditional alignment relates to GLS. Subsampling changes source class proportions, while CDAN’s conditional objective can either support GLS or induce mislabeling under label mismatch.

  • Dataset divergence: Tables 11–13 report Jensen–Shannon divergence with source domains as rows and target domains as columns.The tables cover Digits, Visda, Office-31, and Office-Home datasets and their subsampled versions.
  • Dataset construction: Subsampling retains 30% of the first half of source-domain classes, making divergence asymmetric for the subsampled datasets.This procedure is used to increase label-distribution mismatch.
  • CDAN and GLS: CDAN seeks to match the joint predicted-label–representation distributions DS(bY ⊗Z) and DT(bY ⊗Z), making it suited to conditional alignment.With matching label distributions and perfect predictions, this objective corresponds to matching Z | Y and verifies GLS.
  • CDAN and GLS: When label distributions differ, fooling the CDAN discriminator can require mislabeling certain samples.The paper identifies this effect as quantified by its theoretical error bound.

B.6 Generation of domain adaptation tasks with varying DJS(DS(Z) ∥DT(Z))

The experiments create domain-adaptation tasks with controlled label-distribution divergences, then evaluate importance-weighted variants against unchanged baselines. The weighted methods retain or improve performance as divergences increase, using the original implementations and hyperparameters.

  • Task generation: 50 vectors generate 100 MNIST→USPS and subsampled-MNIST→USPS tasks with varying class fractions and Jensen-Shannon divergences.The divergences range from 6.1e−3 to 9.53e−22.
  • Evaluation setup: The experiments evaluate six algorithms, including importance-weighted versions of DANN, CDAN, and JAN, against their base algorithms.The weighted methods are assessed on both standard and artificially modified tasks.
  • Evaluation setup: The implementations use the baselines’ original hyperparameters, with weight estimation added on top for a direct comparison.The authors report no hyperparameter search and describe the weight estimation as computationally lightweight.

B.8 Weight Estimation

The section examines how importance weights are estimated during training and how estimation quality relates to domain-adaptation behavior. The analysis combines theoretical conditions, numerical stabilization, and empirical weight trajectories.

  • Weight estimation: Importance weights are estimated using Lemma 3.2, but the procedure has no guarantee that GLS holds during training.Thus, the exact training dynamics of the estimated weights remain unclear.
  • Theoretical conditions: Under zero source error and matched weighted source and target marginals, Lemma 3.3 states that the estimated weight vector equals the target reweighting vector.The result provides a sufficient condition linking marginal matching to correct weight recovery.
  • Training dynamics: For DANN, estimated weights tend toward 1 as the marginal distributions are matched, while IWDAN’s estimation improves monotonously during training.The DANN trajectory does not reach 1 because the learning rate is decayed to 0.
  • Training dynamics: The section cautions that weight dynamics are complex and depend on data regularity, neural-network properties, stochastic gradient descent, and adaptation success.The authors explicitly state that they do not fully understand these dynamics.

B.9 Per-class predictions and estimated weights

Per-class analyses show that DANN can confuse classes under source-target label imbalance, whereas importance-weighted methods improve classwise performance. Weight-estimation errors also track prediction errors by class.

  • Per-class predictions: Without domain adaptation, per-class predictions are relatively random, including severe confusion of digit 9 with digits 4 and 8.The reported analysis concerns the sU→M task.
  • Per-class predictions: DANN performs poorly on source-subsampled classes because unweighted marginal matching projects over-represented target classes onto under-represented classes.The reported average accuracy for DANN is 86.71%.
  • Per-class predictions: IWDAN performs rather well across classes, except for confusion between digits 7 and 9.Its reported average accuracy is 94.38%.
  • Per-class predictions: IWDAN-O outperforms the other algorithms on all classes, with a reported average accuracy of 96.8%.This comparison is reported for the sU→M task.
  • Estimated weights: Errors in estimated weights for individual classes strongly correlate with prediction errors for those classes.The examples given include digit 3 for DANN and digit 7 for IWDAN.
Loading 2003.04475v3…