PaperlayerView exact revision on arXiv

Source-linked AI summary

Heterogeneous domain adaptation: An unsupervised approach

Feng Liu, Guanquan Zhang, Jie Lu

arXiv:1701.02511v5cs.LGstat.ML

TL;DR

Heterogeneous unsupervised domain adaptation without parallel sets lacks a developed theoretical foundation. This paper proposes GLG, combining a transfer theorem and principal-angle metric to construct homogeneous representations; GLG is significantly better than all baselines on seven of ten tasks and statistically matches SFER on the other three.

  • Problem

    The paper addresses the limited theoretical foundation for predicting labels in a heterogeneous, unlabeled target domain without parallel sets.

    Source: §I · p. 1

  • Method

    GLG uses a transfer theorem and linear monotonic maps to construct homogeneous representations, then a principal-angle metric to preserve domain distance relationships.

    Source: E. Discussion of definitions and… · p. 8

  • Results

    GLG is significantly better than all baselines on 7 of 10 tasks and has the same performance as SFER on the remaining 3 tasks from a statistical view.

    Source: D. Experiment II: Overall comparisons · p. 13

  • Takeaways & Limitations

    The paper presents sufficient theoretical conditions for correct transfer to an unlabeled heterogeneous target and a metric for constructing representations that preserve domain geometry.

    Source: §VI · p. 13

    Direct use of GLG for multi-class classification lacks sufficient theoretical guarantees and may yield low target-domain accuracy.

    Source: F. Limitation of GLG · p. 9

Abstract

from arXiv · show

Domain adaptation leverages the knowledge in one domain - the source domain - to improve learning efficiency in another domain - the target domain. Existing heterogeneous domain adaptation research is relatively well-progressed, but only in situations where the target domain contains at least a few labeled instances. In contrast, heterogeneous domain adaptation with an unlabeled target domain has not been well-studied. To contribute to the research in this emerging field, this paper presents: (1) an unsupervised knowledge transfer theorem that guarantees the correctness of transferring knowledge; and (2) a principal angle-based metric to measure the distance between two pairs of domains: one pair comprises the original source and target domains and the other pair comprises two homogeneous representations of two domains. The theorem and the metric have been implemented in an innovative transfer model, called a Grassmann-Linear monotonic maps-geodesic flow kernel (GLG), that is specifically designed for heterogeneous unsupervised domain adaptation (HeUDA). The linear monotonic maps meet the conditions of the theorem and are used to construct homogeneous representations of the heterogeneous domains. The metric shows the extent to which the homogeneous representations have preserved the information in the original source and target domains. By minimizing the proposed metric, the GLG model learns the homogeneous representations of heterogeneous domains and transfers knowledge through these learned representations via a geodesic flow kernel. To evaluate the model, five public datasets were reorganized into ten HeUDA tasks across three applications: cancer detection, credit assessment, and text classification. The experiments demonstrate that the proposed model delivers superior performance over the existing baselines.

I. INTRODUCTION

HeUDA must handle unlabeled heterogeneous targets, often without parallel instances, yet existing approaches have theoretical and practical gaps in this setting. GLG constructs homogeneous representations that preserve variation and distance relationships, then transfers knowledge between them.

I. INTRODUCTION

  • Existing HeUDA models require parallel sets, which are unrealistic in some settings such as confidential credit assessment and limit where the models can be used.The paper targets prediction for a heterogeneous, unlabeled target domain without parallel sets.Source: §I · p. 1
  • The paper’s aim is to establish a theoretical foundation for HeUDA without parallel sets, motivated by heterogeneous domains that may arise as projections of one underlying domain.For example, Amazon reviews and Rotten Tomatoes reviews are distinct representations of human sentiment.Source: §I · pp. 1–2
  • The proposed framework represents domain similarity with variation V between conditional probability densities and distance D between feature spaces.The paper seeks homogeneous representations with VHo = VHe and DHo = DHe.Source: §I · p. 2
  • GLG uses linear monotonic maps to construct homogeneous representations under transfer conditions, a principal-angle metric to assess distance preservation, and a geodesic flow kernel to transfer knowledge.The metric compares the original domain pair with the pair of homogeneous representations.Source: §I · pp. 2–3
  • Five public datasets were reorganized into ten tasks across cancer detection, credit assessment, and text classification to evaluate GLG.The paper presents GLG, its transfer theorem, and its principal-angle metric as its main contributions.Source: §I · p. 3

A. Homogeneous unsupervised domain adaptation

The review distinguishes homogeneous adaptation from heterogeneous settings, where different feature spaces and limited target supervision complicate transfer. GLG addresses this setting through variation-preserving maps and a metric for preserving the distance between original domains.

A. Homogeneous unsupervised domain adaptation

  • Homogeneous unsupervised domain adaptation methods include Grassmann-manifold, integral-probability-metric, pseudo-labeling, and adversarial-training approaches.GFK searches subspaces along a Grassmann-manifold geodesic.Source: A. Homogeneous unsupervised domain… · p. 3

B. Heterogeneous domain adaptation

  • Heterogeneous domain adaptation spans supervised, semi-supervised, and unsupervised settings, commonly with different-dimensional and disjoint source and target features.Many existing approaches rely on target labels, paired instances, or common features.Source: B. Heterogeneous domain adaptation · pp. 3–4

C. Comparison to related work

  • Unlike SSKMDA, which uses target labels to measure feature-space similarity, GLG uses a transfer theorem and principal angles and does not require target labels.The paper also contrasts GLG with paired-instance approaches such as KCCA.Source: C. Comparison to related work · p. 4

III. HETEROGENEOUS UNSUPERVISED DOMAIN ADAPTATION

  • GLG’s transfer theorem and linear monotonic maps preserve the variation factor, while its metric is designed to preserve the distance factor between original domains.The model seeks maps that preserve the original domains’ distance relationship.Source: §III · p. 4

A. Problem setting and notations

  • The HeUDA problem is to label target instances using labeled source data and an unlabeled target domain whose feature space has a different dimensionality.The domains are represented as projections of a common multivariate variable.Source: A. Problem setting and notations · p. 4

B. Unsupervised knowledge transfer theorem for HeUDA

The paper defines a variation factor to characterize label-probability differences between heterogeneous domains and motivates a sufficient condition for reliable unsupervised transfer. Its theorem identifies mapping properties that satisfy that condition.

B. Unsupervised knowledge transfer theorem for HeUDA

  • The variation factor VHe compares conditional label probabilities across the source and target representations, and the paper uses it to formulate its transfer theorem.The setup relates each domain’s conditional probability to the underlying probability for a measurable subset.Source: B. Unsupervised knowledge transfer… · pp. 4–5
  • The paper defines extreme negative transfer as a worst-case failure and notes that lower target-domain classification accuracy can quantify its severity in experiments.The theorem’s condition is intended to avoid this extreme case in advance.Source: B. Unsupervised knowledge transfer… · p. 5
  • The HeUDA condition is sufficient, but not necessary, for correct knowledge transfer and can be satisfied by the proposed linear monotonic map.The paper motivates this condition because extreme negative transfer cannot be quantified without target labels.Source: B. Unsupervised knowledge transfer… · p. 5
  • The theorem guarantees that two maps satisfying monotonicity and invertibility conditions meet the HeUDA condition and can transfer knowledge with theoretical reliability.The maps must be monotonic and recover their original inputs through their inverses.Source: B. Unsupervised knowledge transfer… · p. 5

C. Principal angle-based measurement between heterogeneous feature spaces

The section uses principal angles on Grassmann manifolds to compare heterogeneous feature spaces, including spaces of different dimensions. It extends this geometric comparison to assess whether homogeneous representations preserve relationships between the original domains.

C. Principal angle-based measurement between heterogeneous feature spaces

  • Principal angles allow distances to be calculated between subspaces of different dimensions, using singular value decomposition.For subspaces of dimensions m and n, the number of principal angles is r = min(n, m).Source: C. Principal angle-based measurement… · p. 6
  • The proposed metric compares relationships between two pairs of subspaces, rather than measuring distance between two probability distributions.It is defined using principal angles and differs in aim from KL divergence.Source: C. Principal angle-based measurement… · p. 6
  • The metric can describe geometric relationships between heterogeneous feature spaces when their dimensionalities differ, whereas KL divergence requires equal dimensionalities.The section presents this as a reason for defining a new distance for heterogeneous feature spaces.Source: C. Principal angle-based measurement… · p. 6

D. The proposed HeUDA model

The proposed HeUDA model constructs homogeneous representations using linear monotonic maps and seeks to preserve the geometric relationships between the original feature spaces. It then transfers knowledge between the representations using a geodesic flow kernel.

D. The proposed HeUDA model

  • GLG maps heterogeneous feature spaces into same-dimensional homogeneous representations with linear monotonic maps that satisfy the HeUDA condition.The theorem establishes that these maps meet the condition for HeUDA.Source: D. The proposed HeUDA model · p. 7
  • The proposed measurement compares two pairs of subspaces, capturing whether homogeneous representations preserve the geometric relationship between heterogeneous domains.Unlike KL divergence, it measures relationships between subspace pairs rather than probability distributions.Source: D. The proposed HeUDA model · p. 7
  • GLG minimizes J1 to keep principal angles between original feature spaces and their homogeneous representations close, including under slight input changes.The cost function also smooths the maps; when m = n, its regularization parameters are set to zero.Source: D. The proposed HeUDA model · pp. 7–8
  • GLG transfers knowledge between the learned homogeneous representations using a geodesic flow kernel.The learned representations have dimension r = min(m, n).Source: D. The proposed HeUDA model · p. 8
  • LMMs require positive map-matrix elements, while the GFK supplies a second map without the same rigid restrictions.The authors characterize LMMs as restrictive but state that the composite LMM+GFK map can handle more complex problems.Source: D. The proposed HeUDA model · p. 8
  • When source and target domains have equal feature dimensions, GLG degenerates into the GFK model.Source: D. The proposed HeUDA model · p. 8

E. Discussion of definitions and theorems

The discussion links the paper’s HeUDA condition and mapping theorem to a theoretical route for preventing negative transfer. It also connects the proposed metric and cost function to preserving geometric relationships between heterogeneous feature spaces.

E. Discussion of definitions and theorems

  • The HeUDA condition and mapping theorems establish a theoretical route to correct transfer and preventing negative transfer in HeUDA.Theorem 2 shows that LMMs can map heterogeneous feature spaces into homogeneous representations with a theoretical guarantee.Source: E. Discussion of definitions and… · p. 8
  • The proposed metric and cost function give a way to describe and maintain geometric distances between heterogeneous feature spaces.Theorem 3 establishes that the measurement is a metric, and Theorem 4 connects the homogeneous-domain case to GLG’s degeneration into GFK.Source: E. Discussion of definitions and… · p. 8

F. Limitation of GLG

GLG’s homogeneous-representation construction can be used for multiclass classification, but its theoretical guarantees are insufficient for direct multiclass use. Its optimization derives gradients through changes in subspaces and numerically integrates the cost function.

F. Limitation of GLG

  • GLG’s representation construction does not use source labels, allowing practical extension to multiclass problems, which the authors plan to pursue with label-noise learning.The paper treats an unlabeled target domain with predicted labels as a domain with noisy labels.Source: F. Limitation of GLG · pp. 8–9
  • For multiclass classification, GLG does not provide sufficient guarantees: with 10 classes and P1(xt)=0.1, the mapped probability may lie in [0.1, 0.9).The authors say this interval is too long and that direct GLG use would yield low target accuracy; a mapping keeping P1(ft(xt)) close to P1(xt) is difficult in the unsupervised heterogeneous setting.Source: F. Limitation of GLG · p. 9

IV. OPTIMIZATION OF GLG

  • Minimizing J1 requires derivatives of principal angles with respect to the source and target mappings, plus integration over δ for gradient descent.The optimization analyzes how subspaces change when their feature spaces are disturbed, using first-order derivatives of an Eigen dynamic system.Source: §IV · p. 9, A. Microscopic analysis of an Eigen… · p. 9

A. Microscopic analysis of an Eigen dynamic system

  • The microscopic analysis represents A as span(X) and derives first-order changes in its eigenvectors and eigenvalues when X is disturbed.Lemma 2 gives the first-order derivatives; its expression uses the Moore-Penrose pseudoinverse of XXT−λiI.Source: A. Microscopic analysis of an Eigen… · p. 9

B. Gradients of J1

  • The J1 gradients use chain rules to update the mapping matrices, while Simpson’s rule numerically estimates the integral with Δ=δ0/10.The source mapping is constrained to remain a linear monotonic map.Source: B. Gradients of J1 · pp. 9–10

C. Optimization of GLG

GLG minimizes J1 with a hybrid search that initializes mapping solutions using cuckoo search, then refines them by gradient descent. The algorithm uses Simpson’s rule for integration and selects a step size by minimum cost at each iteration.

C. Optimization of GLG

  • Cuckoo search initializes the mapping solutions, and gradient descent then seeks lower-cost solutions for J1.The step size η is selected from [0.01, 0.05, 0.1, 0.2, 0.5, 1, 5, 20] to obtain the minimum cost at each iteration.Source: C. Optimization of GLG · p. 10
  • The GLG algorithm uses Simpson’s rule to estimate the integral in J1, with 30 nests, discovery rate 0.25, and 100 cuckoo-search iterations.The algorithm also sets MaxIter to 100, err to 10−5, and δ0 to 0.01.Source: C. Optimization of GLG · p. 10
  • After optimization, the homogeneous source and target representations are used to train a model and predict target-domain labels.The algorithm’s stated inputs are the source and target data, and its output includes adapted target data.Source: C. Optimization of GLG · p. 10

V. EXPERIMENTS

The experiments use five public datasets spanning cancer detection, credit assessment, and text classification, reorganized for heterogeneous unsupervised domain adaptation. The paper describes example source-to-unlabeled-target tasks and treats the domains’ observed features as heterogeneous projections of a shared underlying domain.

V. EXPERIMENTS

  • Experiments evaluate GLG on five public datasets spanning cancer detection, credit assessment, and text classification, using an SVM classification engine.The datasets were reorganized because the authors found no real-world datasets directly related to HeUDA.Source: §V · p. 10, A. Datasets for HeUDA · p. 10

A. Datasets for HeUDA

  • The tasks ask whether labeled German credit, Org/People text, or breast-cancer data can provide knowledge to label corresponding unlabeled target data.The examples are G2A, Ope2Opl, and CO2CD, respectively.Source: A. Datasets for HeUDA · p. 10
  • For the cancer and credit datasets, the authors model source and target features as heterogeneous projections of a shared underlying domain.The underlying observations are unavailable; only features in the respective datasets can be obtained.Source: A. Datasets for HeUDA · p. 11

B. Experimental setup

The experimental setup compares GLG with non-transfer, dimensional-reduction, mapping, kernel, and neural domain-adaptation baselines. Evaluation uses target accuracy, with preprocessing and parameter choices adapted to the datasets and the absence of labeled target instances.

1) Baselines:

  • Baselines include non-transfer all-one and clustering models, dimensional-reduction GFK, random mapping variants, KCCA, SFER, and neural domain-adaptation models.The neural baselines are DANN, DAN, and BSWDA; the random LMM GFK baseline does not preserve distance between domains.Source: 1) Baselines · p. 11

2) Implementation details:

  • Text data are reduced with SVD to the top 50% of eigenvalues, while experiments use 1,500 unbiased instances per domain.For German Credit, 600 unbiased instances are sampled per experiment and each model-task experiment is run 50 times to offset dataset imbalance.Source: 2) Implementation details · p. 11
  • The setup uses accuracy as its test metric and trains an SVM on homogeneous source representations before testing on the target domain.Because target labels are absent, the target classifier’s optimal parameters cannot be tuned by cross-validation; LIBSVM defaults are used, and KCCA instances are randomly paired.Source: 2) Implementation details · p. 11
  • Same-domain target accuracy is reported as a reference, and HeUDA accuracy is generally lower because target-domain labels are unavailable.Table III reports five-fold SVM same-domain accuracy for target domains.Source: 2) Implementation details · p. 12, Table III · p. 12

C. Experiment I: RMG

Random mapping can leave heterogeneous domain adaptation unreliable, while GLG is reported to outperform baselines across the evaluated tasks. The conclusion describes GLG’s theoretical conditions, mapping, and evaluation scope.

C. Experiment I: RMG

  • MMD-similar adapted domains yielded CD2CO accuracies of 7.91% and 96.49%, supporting the need for the HeUDA condition and Theorem 1.The paper attributes the disparity to a significant difference in conditional label distributions.Source: C. Experiment I: RMG · p. 12, Table V · p. 13

D. Experiment II: Overall comparisons

  • Across the 10 tasks, GLG produced more stable and higher-accuracy classification results than the other models.The paper also reports that some baselines were unstable because they did not prevent extreme negative transfer.Source: D. Experiment II: Overall comparisons · p. 13
  • GLG outperformed baselines on 9 of 10 tasks and was significantly better than all baselines on 7; it matched SFER statistically on the remaining 3.SFER was not significantly outperformed on OPl2OPe and CO2CD; on OPe2OPl, its higher observed score was not statistically significant.Source: D. Experiment II: Overall comparisons · p. 13

VI. CONCLUSIONS AND FURTHER STUDIES

  • GLG minimizes a principal-angle metric to construct homogeneous representations, then transfers knowledge between them using a geodesic flow kernel.The conclusion reports evaluation on five public datasets spanning cancer detection, credit assessment, and text classification.Source: §VI · p. 13
  • The authors identify multiclass guarantees and a target-loss generalization bound as directions for future work.The paper also notes that direct use of GLG for multiclass classification does not provide sufficient theoretical guarantees.Source: §VI · p. 13

APPENDIX A PROOF OF THEOREM 1

The appendix proofs connect monotonic maps to injectivity and characterize a principal-angle distance through metric properties. They also establish the stated mapping conditions for Theorem 2.

APPENDIX A PROOF OF THEOREM 1

  • Monotonicity of the source map implies a one-to-one map between Xs and Zs in the Theorem 1 proof.The proof then proceeds through an equation to conclude the theorem.Source: Appendix A · p. 16

APPENDIX B PROOF OF LEMMA 1

  • For f(x) = xU^T, the proof relates coordinatewise monotonicity to the signs of the entries uji.It states that the map is increasing when uji > 0 and decreasing when uji < 0.Source: Appendix B · p. 16

APPENDIX C PROOF OF THEOREM 2

  • The Theorem 2 proof uses the Moore–Penrose pseudoinverse to establish the second theorem condition for the LMM maps.The proof says the maps satisfy the first condition and concludes the theorem.Source: Appendix C · p. 16

APPENDIX D PROOF OF THEOREM 3

  • The principal-angle distance is shown to be nonnegative and symmetric, with zero distance exactly when A^T B = C^T D.These are the first three properties listed for the metric.Source: Appendix D · p. 16

D((A, B), (C, D))

APPENDIX E PROOF OF THEOREM 4

Theorem 4’s proof identifies when GLG reduces to ordinary GFK. The following appendix proof derives eigenvalue and eigenvector derivatives for Lemma 2.

APPENDIX E PROOF OF THEOREM 4

  • When m = n, λs = λt = 0, and both mappings are identities, the optimal GLG model reduces to ordinary GFK.The proof associates identity mappings with Us = Is and Ut = It.Source: Appendix E · p. 16

APPENDIX F PROOF OF LEMMA 2

  • The Lemma 2 proof defines matrices and vectors used in the derivation, then proceeds to calculate eigenvalue and eigenvector derivatives.It transforms Eq. (10), derives intermediate equations, and then states the eigenvector derivatives.Source: Appendix F · pp. 16–17
Loading 1701.02511v5…