Estimating Generalization under Distribution Shifts via Domain-Invariant Representations

• The central challenge is evaluating generalization to shifted, unlabeled domains when collecting supervised target data is prohibitively expensive or impossible.The motivation includes changing weather, landscapes, and traffic in deployment settings.Source: §1 · p. 1
• Domain-invariant check models proxy unknown target labels, enabling target-error estimates that outperform previous methods.The approach is intended for both domain adaptation and general supervised models.Source: §1 · pp. 1–2
• Representation complexity strongly affects domain-invariant target generalization, creating a model-selection problem for choosing the encoder–predictor split.The encoder’s complexity is described as especially important for selecting check models.Source: §1 · pp. 1–2
• The framework lets domain-invariant models self-tune their complexity to find models with good target generalization.This self-tuning uses the target-error estimation framework to identify an optimal model complexity.Source: §1 · p. 1
• The method estimates the risk of single models or model classes, predicts point-wise error, and supports reliability assessment and model selection.Experiments cover sentiment analysis, digit classification, and general object classification.Source: §1 · p. 2

• The paper differs from earlier risk-estimation methods by using domain-invariant classifiers as proxy target labels.The method applies to both domain adaptation and general supervised models.Source: §2 · p. 2
• Earlier approaches estimate shifted-domain risk through agreement rates, conditional independence, logical constraints, or confidence and H∆H-divergence.The cited methods address settings ranging from multiple data views to domains very different from training data.Source: §2 · p. 2
• Domain-invariant representations align source and target embeddings using divergences such as adversarial objectives, MMD, or Wasserstein distance.Existing DIR approaches differ in the divergence measure used for alignment.Source: §2 · p. 2
• Theoretical analyses include common-support assumptions and H∆H-divergence, but common support fails in most standard domain-adaptation benchmarks.Other work also studies risks when source and target label distributions differ.Source: §2 · p. 2

3. Unsupervised Domain Adaptation

• Unsupervised domain adaptation seeks a hypothesis that minimizes target-domain risk using labeled source and unlabeled target data.The setup considers binary classification with zero-one loss and does not assume common source–target support.Source: §3 · p. 2

3.1. Domain-invariant Representations

• Domain-invariant representations map source and target inputs into a shared latent space, where a predictor is trained after minimizing a distributional divergence.The divergence may be Jensen–Shannon or Wasserstein distance, and the full model is h = f ◦ g.Source: §3.1 · p. 2

3.2. Upper Bounds on the Target Risk

• The H∆H-divergence measures how much hypothesis disagreements differ between source and target distributions and depends on both domain discrepancy and hypothesis-class complexity.It supports upper bounds on target risk.Source: §3.2 · pp. 2–3
• Applying target-risk bounds to a fixed embedding yields an F∆F divergence and a best-hypothesis term λF(g), which becomes large when the representation aligns domains incorrectly.A suitable representation can make the divergence small, but wrong alignment keeps the best hypothesis risk large.Source: §3.2 · p. 3

4. Estimating Target Risk: Main Idea

• The method upper-bounds a model’s target risk by replacing the unknown best hypothesis with candidates from a proxy set of check models.The estimate combines disagreement with a check model and the quality of the check models.Source: §4 · p. 3
• Figure 1 contrasts a smaller check-model set with a larger one, showing that zero bias need not produce a tight proxy-risk estimate.P1 has nonzero bias but tighter estimation, whereas P2 contains htrue but is too broad.Source: Figure 1 · p. 3
• A tight proxy-risk estimate requires check models that include a good hypothesis without adding unnecessarily disagreeing hypotheses.The first term measures disagreement with a check model, while the second measures check-model quality.Source: §4 · p. 3
• The proxy risk becomes useful when the check-model bias is small, meaning at least one check model has low target risk.The proxy risk can then estimate an upper bound on the target risk.Source: §4 · p. 3
• Domain-adaptation models are suitable check models because their target risk affects proxy-risk estimation and they are designed to minimize target risk.The paper later defines the check-model set as DIR models with low DIR objective.Source: §4 · p. 3

5. Understanding the Adaptability of DIR

• A too-powerful nonlinear embedding can achieve zero DIR objective yet maximum target risk, whereas restricting embeddings to linear maps yields minimum target risk in the toy example.The nonlinear map increases ambiguity and variance despite perfect objective alignment.Source: §5 · p. 4
• Empirically, target adaptability is most sensitive to embedding complexity, while predictor complexity and source-domain generalization have weaker effects.The experiments motivate focusing on the complexity of the embedding class G.Source: §5 · p. 4, Figure 3 · p. 4

5.1. Bounds for Domain-invariant Representations

• The FG∆G-divergence measures source-target disagreement while holding the predictor fixed and varying encoders, making embedding variation explicit.It is smaller than the unrestricted (FG)∆(FG)-divergence because the compared hypotheses share the same predictor.Source: §5.1 · p. 4
• The target-risk bound is small when source risk, latent divergence, and embedding complexity are controlled and good joint source-target risk is achievable.Latent divergence decreases with encoder complexity but embedding complexity increases, creating a tradeoff.Source: §5.1 · pp. 4–5

5.2. Experiments

• On sentiment classification, target error initially decreases and then increases with encoder depth; on object classification, an embedding-complexity tradeoff persists even with pretrained ResNet-50 features.Experiments train DANN on Amazon reviews and Office-31 while varying encoder depth and fixing predictor depth.Source: §5.2 · p. 5
• For digit adaptation, adding encoder depth raises target error by 19.8% on MNIST→MNIST-M and 8.8% on SVHN→MNIST relative to the optimal case.Increasing width produces a smaller target-error decrease than changing depth, suggesting depth matters more than width.Source: §5.2 · p. 5, Figure 5 · p. 5

6. Division for Multilayer Neural Networks

• Theoretical decomposition treats each network split as an embedding followed by a predictor, with layer-specific complexity terms.Source: §6 · p. 6
• Deeper embeddings improve alignment and reduce predictor complexity but increase embedding divergence, creating a tradeoff within the fixed network class.Source: §6 · p. 6
• An optimal intermediate layer may minimize the target-risk bound, determining where to apply the domain-invariant loss.Source: §6 · p. 6

6.1. Experiments

• Experiments across sentiment, digit, and object classification show a U-shaped relationship with an optimal division for each task.Source: §6.1 · p. 6, Figure 6 · p. 6
• 19.5% in M→M-M and 10.4% in S→M: different divisions produce large performance gaps despite identical total network size.Source: §6.1 · p. 6

7. Estimating Target Risk

• Proxy risk estimates target risk using the largest target disagreement between the candidate model and a set of domain-invariant check models.Source: §7 · p. 6
• The check-model class contains classifiers with low source risk and well-aligned source and target representations, implicitly assuming one has low target risk.Source: §7 · pp. 6–7

7.1. Connection to Embedding Complexity

• The estimation error is bounded by the target risk of the check models, so excessive proxy risk prevents the objective from identifying a good target classifier.Source: §7.1 · p. 7
• Suitable embedding complexity, often achieved through an optimal network division, is needed to minimize estimation error and can be selected by self-tuning.Source: §7.1 · p. 7

7.2. Computing the Target Risk Estimator

• Computationally, the constrained disagreement problem is relaxed with a Lagrangian penalty and optimized using empirical risks and stochastic gradient descent.Source: §7.2 · p. 7
• Algorithmically, a check model is pretrained for domain invariance, then optimized to maximize disagreement while constraints are checked on validation data.Source: §7.2 · p. 7

8. Experiments

• The experiments cover model selection, target-risk estimation, adaptability assessment, and error detection using shared architectures and labeled source but unlabeled target data.Source: §8 · p. 7, §8.2 · p. 8

8.1. Model Selection for DIR

• Proxy risk accurately identifies the best DIR layer division without supervision and remains robust to different second-level check models.Source: §8.1 · p. 7, Figure 7 · p. 8
• During DANN training, proxy risk tracks target error and provides an early-stopping criterion, nearly matching target risk for MNIST→MNIST-M.Source: §8.1 · p. 8, Figure 8 · p. 8

8.2. Estimating Performance Drop of Supervised Learning under Domain Shift

• For supervised target-risk estimation, the method outperforms both the Ben-David et al. (2010) bound and confidence-score baseline in average absolute error and Pearson correlation.Source: §8.2 · p. 8, Table 1 · p. 8

8.3. Estimating Adaptability between Domains

• The proposed method consistently outperforms baselines for estimating supervised and adaptive models’ target risks across digit and object tasks.For adaptive models, estimates fall within a few percent of true accuracy, while confidence scores and Ben-David et al. (2010) often underestimate or overestimate risk.Source: §8.3 · pp. 8–9, §8.2 · p. 8, Table 1 · p. 8

8.4. Error Detection

• Point-wise error detection treats disagreement between a candidate and check model as a predicted misclassification and evaluates it with target-domain F1.Source: §8.4 · p. 9, Table 2 · p. 9

• Domain-invariant classifiers estimate the target risk of supervised or domain-adaptation models on shifted, unlabeled data.The approach covers single classifiers, point-wise error prediction, and sets of classifiers for model selection.Source: §9 · p. 9
• Embedding complexity is more important for target adaptability than predictor complexity and than its role in non-adaptive supervised learning.Source: §9 · p. 9
• The approach is intended for risk estimation, point-wise error prediction, and model selection.Source: §9 · p. 9
• Future work includes using other domain-adaptation algorithms as check models and extending the approach to detection or segmentation.Source: §9 · p. 9

A.1. Proof of Lemma 3

• Lemma 3’s proof selects a lowest-target-risk hypothesis and applies the triangle inequality to relate proxy risk to target risk.Source: §A.1 · p. 11

A.2. Proof of Lemma 4

• Lemma 4’s proof combines the preceding lemma with further triangle-inequality steps and a supremum over the hypothesis class.Source: §A.2 · p. 11

A.3. Proof of Theorem 6

• Theorem 6 bounds target risk using latent divergence, embedding complexity, source and target risks, and a best-in-class joint-risk term λ_FG(g).Source: §A.3 · pp. 11–12

A.4. Proof of Proposition 7

• Proposition 7 represents an N-layer network as an encoder followed by a predictor and defines q_ij as the mapping between latent spaces at layers i and j.Source: §A.4 · p. 12

B.1. The Role of Inductive Bias

• CNN encoders align source and target representations, whereas MLP encoders produce higher target error and serious label mismatch despite domain overlap.Increasing MLP depth worsens target performance and can be worse than no domain adaptation.Source: §B.1 · p. 12, Figure 10 · p. 12, Figure 11 · p. 13

B.2. Effect of Embedding Complexity

• Target error first decreases and then increases as encoder depth grows on both Amazon reviews and Office-31.The supplied figures vary encoder layers and also examine domain-invariant losses at intermediate layers.Source: §B.2 · pp. 12–13, Figure 12 · p. 13, Figure 13 · p. 13

B.3. Effect of Predictor Complexity

• A seven-layer predictor augmentation decreases target error by only 0.9%, far less than the 19.8% drop associated with varying encoder depth.This contrast motivates focusing on embedding complexity rather than predictor complexity.Source: §B.3 · p. 13, Figure 14 · p. 13

C. Computing Ben-David et al. (2010)’s Bound

• The tightness of the Ben-David et al. (2010) bound depends on the hypothesis class used for the H∆H-divergence and whether relevant candidate and optimal hypotheses belong to it.Choosing a restricted class can tighten the bound while increasing the unobserved best joint risk.Source: C. Computing Ben-David et al. (2010)’s… · p. 14

D.1. Amazon Review Dataset

• The described model uses Adam with learning rate 1×e−3, trains for 50 epochs, and applies progressive training in which the domain-invariant-loss weight α increases from 0 to 1.Source: §D.1 · p. 14

D.2. Digit Classification

• Digit-classification models use Adam at 1 × e−3 for 100 epochs, schedule α from 0 to 0.1, and set λ to 50 for proxy-error computation.Hidden-width experiments scale the relevant widths of the listed architectures.Source: §D.2 · pp. 14–15

D.3. Office-31

• Experiments use ImageNet-trained ResNet-50 features, Adam with learning rate 3×e−4 for 100 epochs, α increased from 0 to 1, and λ set to 50 for all tasks.Source: §D.3 · p. 15