PaperlayerView exact revision on arXiv

Source-linked AI summary

Joint Distribution Alignment for Universal Domain Adaptation

Shizhe Li, Hongshan Pu, Mengying Xie, Yi Xiang, Xiaowei Yang

arXiv:2608.24429v1cs.LGcs.CV

TL;DR

UniDA must handle both distribution shifts and differing label spaces, but existing work offers limited theoretical analysis. The paper derives a Chi-Square-divergence-based generalization-error bound and develops JAUA with joint distribution alignment and progressive pseudo-labeling; experiments on six public image datasets demonstrate its superiority.

  • Problem

    UniDA addresses distribution shifts and differing source-target label spaces, while existing works rarely provide theoretical analysis for this setting.

  • Method

    JAUA minimizes a Chi-Square-divergence-based target generalization-error upper bound for joint distribution alignment and uses progressive pseudo-labeling for unlabeled target samples.

  • Results

    Experiments on six public image datasets demonstrate JAUA's superiority in handling UniDA problems.

  • Takeaways & Limitations

    JAUA provides a UniDA approach that jointly addresses common-class distribution alignment and separation of known, private, and unknown samples.

Abstract

from arXiv · show

Unsupervised domain adaptation (UDA) has been widely concerned in the fields of machine learning, pattern recognition, and computer vision. Traditional UDA learning usually assumes that the label spaces of the source and target domains are exactly the same and only needs to solve the problem of sample distribution drift existing between two domains. However, in real world applications, the label spaces between two domains may be different. In this case, there are both sample distribution drift and class spatial difference between domains, namely Universal Domain Adaptation (UniDA) learning scenario. At present, existing works rarely offer theoretical analysis for universal domain adaptation. In this paper, we provide an upper bound of the generalization error for universal domain adaptation. According to the proposed generalization error bound, we propose a novel UniDA algorithm called Joint Distribution Alignment for Universal Domain Adaptation (JAUA), which aligns the joint distributions by minimizing the distribution discrepancy calculated by Chi-Square divergence. Furthermore, we propose a progressive pseudo-labeling method to assign the pseudo labels to unlabeled target samples. The experiment results on six public image datasets demonstrate the superiority of JAUA in handling the UniDA problem.

I. INTRODUCTION

Universal domain adaptation addresses differing label spaces alongside distribution shifts, while existing approaches lack comparable theoretical analysis. JAUA combines a Chi-Square-divergence-based generalization bound, joint distribution alignment, and progressive pseudo-labeling for unlabeled target samples.

  • UniDA must identify target-private classes as unknown while correctly classifying common classes and aligning their joint distributions across domains.
  • The paper proposes a target generalization-error upper bound based on Chi-Square divergence for UniDA.
  • JAUA builds a UniDA model by minimizing the proposed generalization-error upper bound to align common-class distributions across domains.
  • Progressive pseudo-labeling assigns labels to unlabeled target samples by gradually separating higher-confidence common-class samples from unknown-class samples.
  • Across six public datasets and comparisons with twenty-four state-of-the-art methods, JAUA is reported to show superior performance on UniDA problems.
  • Existing UniDA methods generally lack theoretical interpretability because they are not based on target generalization-error bounds.

III. SYMBOLS AND PROBLEM SETTING

The paper formulates UniDA with differing source and target joint distributions and label spaces, seeking a classifier that minimizes target-domain expected risk. It then introduces a target generalization-error bound under bounded losses and Chi-Square divergence.

  • Problem Setting: UniDA permits different source and target joint distributions and label spaces, including domain-specific unknown classes.The source label space contains private classes y_us, while the target contains private classes y_ut.
  • Problem Setting: The objective is to find a classifier h: X → Y_t that minimizes the target expected risk R_t(h).The loss function measures classification accuracy on the target distribution.
  • Generalization Error Bound: Chi-Square divergence is used to quantify the difference between the source and target joint distributions.The paper states that the divergence underlies the proposed generalization-error analysis.
  • Generalization Error Bound: The proposed analysis gives a target generalization-error upper bound for any hypothesis under bounded-loss assumptions.The theorem assumes bounds M, N_s, and N_T on the loss over relevant domains and class subsets.

B. Proposed Model and Solution Method

The proposed model minimizes an optimization objective derived from the target generalization-error bound, jointly addressing classification losses and joint-distribution discrepancy. It alternates between estimating the classifier and estimating the discrepancy, with kernel-based representations and a unique-minimum condition.

  • Proposed Model: The optimization model minimizes source and target classification losses together with the discrepancy between the domains’ joint probability distributions.This objective is explicitly constructed from the generalization-error bound.
  • Solution Method: The solution uses an alternative iteration scheme that first estimates h and then estimates the joint-distribution discrepancy.The classifier supplies target labels needed to estimate the Chi-Square divergence, while the divergence is needed to optimize h.
  • Solution Method: Quadratic loss is selected because it satisfies the Lipschitz condition on finite closed intervals and permits an analytical solution for h.The paper presents these properties as the theoretical and computational motivations for the choice.
  • Solution Method: The objective has a unique minimum if and only if λ_gst + λ_gss ≤ m_s^k.The condition is stated for optimization problem (28).
  • Solution Method: The joint-distribution discrepancy is estimated with a kernel function whose source and target samples are projected into a high-dimensional space before alignment.The projection matrices W_s and W_t are optimized together with discrepancy minimization, and steepest descent solves the resulting problem.

C. Progressive Pseudo Labeling Method

JAUA progressively assigns target pseudo-labels, separating likely common-class samples from unknown classes through confidence-based selection. The selected common-class samples are incorporated iteratively into joint-distribution alignment, while the paper reports reduced pseudo-label error propagation.

  • Progressive Pseudo Labeling: The initial classifier assigns target samples to source classes and an unknown class, but the classification is progressively updated during training.Samples with confidence E below τ are assigned to class C + 2, and the label set is updated.
  • Progressive Pseudo Labeling: At iteration t of T, target samples pseudo-labeled as common classes are selected at ratio t/T for distribution alignment.Only common-class samples are selected because the alignment objective concerns common classes.
  • Progressive Pseudo Labeling: The selected samples are ordered and chosen using their confidence values within each common class.The selection procedure accounts for the number of available samples in each class.
  • Progressive Pseudo Labeling: Initial pseudo-label errors may propagate iteratively during training and potentially affect algorithmic performance.The paper identifies this as a limitation discussed for JAUA’s progressive pseudo-labeling process.
  • Progressive Pseudo Labeling: The supplementary analysis reports that progressive pseudo-labeling can adaptively reduce error propagation while aligning the domains’ joint distributions.This claim concerns the reported theoretical analysis and discussion of pseudo-label errors.

D. Proposed Algorithm

JAUA alternates pseudo-label initialization, dimensionality reduction, classifier updating, and target-label refinement. Its computational cost is cubic in sample size per iteration in the standard implementation, with scalable approximations available for large datasets.

  • Algorithm Procedure: JAUA first obtains target pseudo-labels with SVM, then uses PCA to reduce source and target feature dimensions from D to d.The reduced representations initialize projection matrices W_s and W_t.
  • Algorithm Procedure: The algorithm updates α and the classification model h using the current projections, then recomputes target pseudo-labels in subsequent iterations.This procedure implements the alternating optimization and progressive labeling strategy.
  • Complexity: For large datasets, Random Fourier Features and Conjugate Gradient reduce complexity to O(T(mdp + mp^2)) time and O(mp) space.The approximation uses random feature dimension p, with p << m.

V. EXPERIMENT

The experiment section specifies JAUA’s iterative implementation, including pseudo-label initialization, joint-distribution alignment, confidence updates, and stopping conditions. It also identifies the experimental computing environment.

  • Experimental setup: The experiments are implemented in Python 3.9 on a computer with a 13th Gen Intel Core i7-13650HX processor.
  • Algorithm implementation: JAUA repeatedly optimizes the alignment objective, maps samples into high-dimensional spaces, computes α∗ and total loss, and updates pseudo-labels.The outer loop continues while the iteration and relative-loss conditions are satisfied.
  • Experimental setup: The experiment section includes a dataset-description table labeled Table II.

A. Datasets

JAUA is evaluated on six public image datasets against multiple categories of UniDA baselines. Performance is measured using common-class accuracy, unknown-class accuracy, and H-score, with ResNet50 used for feature extraction.

  • A. Datasets: The evaluation uses Office-31, Office-Home, Visda, Domainnet, ImageCLEF, and PACS, with settings consistent with baseline algorithms.
  • Baselines: The comparison covers adversarial or feature-alignment methods, classifier or threshold strategies, and mutual-learning or graph-based methods.The listed baselines include UAN, CMU, DANCE, DCC, OSBP, OVANet, CPR, KLS, PCL, STUN, and UACP, among others.
  • Evaluation metrics: Performance is evaluated by common-class accuracy ACC, unknown-class accuracy UNK, and H-score HOS.These metrics range from 0 to 1, and values closer to 1 indicate better performance.
  • Implementation settings: JAUA uses ResNet50 to extract features with an initial feature dimension of 2048.Except for T and E, other JAUA parameters are selected by grid search.

E. Experimental Results and Analysis

JAUA achieves the best average ACC, UNK, and HOS across six datasets, with analyses covering parameter sensitivity, convergence, ablation, and dataset-specific behavior.

  • Overall results: JAUA obtains optimal average ACC, UNK, and HOS across VisDA, Office-31, Office-Home, Domainnet, ImageCLEF, and PACS.The comparison uses average results across the datasets listed in the experimental analysis.
  • Dataset-specific results: JAUA shows particularly significant UNK improvements on D→A and D→W in Office-31, where D contains low-noise source samples.The authors attribute this behavior to easier identification of unknown-class samples when D is the source domain.
  • Dataset-specific results: JAUA improves UNK on several Office-Home tasks and achieves the best ACC on Ar→Pr, Cl→Pr, and Rw→Pr.The analysis associates these results with sketch-based source samples and background-free target images, respectively.
  • Parameter sensitivity: HOS increases as λgss and λgst increase, but drops dramatically at λgst = 0.4 and λgss = 0.1 because this setting violates Theorem 2's restriction.The experiments fix λtt = 1, λs = 0.7, and λt = 0.7 for task D→A on Office-31.
  • Convergence analysis: JAUA stops when the relative loss change between consecutive iterations is below 1%, with a maximum iteration number T ensuring execution.The convergence analysis is reported for D→A on Office-31 and described as similar across other tasks and datasets.
  • Ablation study: All JAUA components improve HOS, while Rtu and the other garbled component are identified as the two most important components.The ablation removes individual loss terms from the whole loss function on D→A in Office-31.

APPENDIX A. PROOF OF THEOREM 1

The proof derives a target-risk upper bound under a bounded-loss assumption, relates risk differences to total variation and Chi-Square divergence, and rewrites the bound using hyperparameters.

  • Assumption: The proof assumes every loss function is bounded above by a constant M.This is stated as Assumption 1 for all inputs and labels in the relevant space.
  • Risk bounds: Under the bounded-loss assumption, differences between target and source risks are bounded using total variation.The proof explicitly identifies DT V as total variation and derives inequalities from the relevant risk expressions.
  • Hyperparameters: The bound introduces λt and λs as hyperparameters that constrain the treatment of target and source risk terms.The proof states that these parameters bound the relevant risk components after the preceding analysis.
  • Final bound: The updated target-risk bound includes a Chi-Square divergence term between source and target joint distributions, together with constants A and B.The proof concludes with the rewritten bound and states that this establishes Theorem 1.

APPENDIX B. PROOF OF THEOREM 2

The proof establishes the minimization condition for L(α), uses a Kronecker-product expression in the derivation, and concludes Theorem 2 after showing msk ≥ 0.

  • Setting ∂α = 0 makes L(α) reach its minimum.
  • The derivation uses ⊗ for the Kronecker product and draws on equations (18) and (21).
  • The condition msk ≥ 0 completes the proof of Theorem 2.
Loading 2608.24429v1…