Source-linked AI summary
Learning with Augmented Features for Heterogeneous Domain Adaptation
Lixin Duan, Dong Xu, Ivor Tsang
TL;DR
Heterogeneous domain adaptation is needed because conventional methods assume source and target data share feature types and dimensions. HFA projects both domains into a common subspace, augments the resulting representations with original features and zeros, and optimizes them with standard learners. On two benchmark datasets, HFA achieves promising results and outperforms existing HDA methods, while its linear formulation motivates kernelization for high-dimensional data.
Problem
Existing domain adaptation methods assume source and target data have the same feature type and dimension, leaving heterogeneous domain adaptation insufficiently addressed.
Method
HFA uses separate projection matrices, augmented feature mappings, standard learners such as SVM or SVR, a transformation metric H, and alternating optimization for heterogeneous domain adaptation.
Results
HFA achieves promising results on two benchmark datasets for object recognition and text classification and outperforms other evaluated HDA methods.
Takeaways & Limitations
Augmented heterogeneous representations provide a reusable way to incorporate patterns from both domains into existing learning methods for HDA.
Takeaways & Limitations
The linear transformation metric can be ineffective for some tasks and computationally infeasible when source and target feature dimensions are very high, motivating kernelization.
Abstract
from arXiv · showhide
We propose a new learning method for heterogeneous domain adaptation (HDA), in which the data from the source domain and the target domain are represented by heterogeneous features with different dimensions. Using two different projection matrices, we first transform the data from two domains into a common subspace in order to measure the similarity between the data from two domains. We then propose two new feature mapping functions to augment the transformed data with their original features and zeros. The existing learning methods (e.g., SVM and SVR) can be readily incorporated with our newly proposed augmented feature representations to effectively utilize the data from both domains for HDA. Using the hinge loss function in SVM as an example, we introduce the detailed objective function in our method called Heterogeneous Feature Augmentation (HFA) for a linear case and also describe its kernelization in order to efficiently cope with the data with very high dimensions. Moreover, we also develop an alternating optimization algorithm to effectively solve the nontrivial optimization problem in our HFA method. Comprehensive experiments on two benchmark datasets clearly demonstrate that HFA outperforms the existing HDA methods.
1. Introduction
Heterogeneous domain adaptation addresses settings where source and target data use different feature types and dimensions, limiting conventional domain adaptation methods. HFA maps both domains into a common subspace, augments the mapped features with original features and zeros, and achieves promising results on two benchmark datasets.
- Research gap: Conventional domain adaptation methods assume source and target data share the same feature type and dimension, so they cannot handle heterogeneous domain adaptation.HDA is motivated by applications where labeled target data are scarce and source-domain labels can provide useful supervision.
- HFA method: HFA uses separate projection matrices to transform source and target data into a common subspace, then augments each representation with its original features and zeros.The augmented features are designed to compare heterogeneous data while retaining domain-specific information.
- HFA method: The augmented feature representations can be incorporated into learning methods such as SVM and SVR to use patterns from both domains for HDA.The paper develops the formulation with hinge-loss SVM and describes both linear and kernelized cases.
- Optimization: HFA introduces a transformation metric H and an alternating optimization algorithm to simplify the optimization without explicitly solving for projection matrices P and Q.The transformation metric combines the two projections, making the common subspace implicit in the optimization.
- Results: Promising results on two benchmark datasets demonstrate HFA's effectiveness for object recognition and text categorization.The introduction reports that HFA outperforms existing methods in the evaluated settings.
2. Kernel Learning for Heterogeneous Domain Adaptation
HFA learns a transformation metric for heterogeneous source and target data, avoiding the need to preselect the latent common-subspace dimension. Kernelization makes this metric independent of feature dimension, while alternating optimization updates the metric and SVM variables.
- Heterogeneous Feature Augmentation: HFA formulates heterogeneous feature augmentation by projecting source and target samples with P and Q, then incorporating original features into the augmented representations.The augmented representations can be used with methods such as SVM and SVR.
- Proposed Method: The linear HFA objective jointly learns P, Q, and the SVM weight vector by minimizing structural risk under complexity constraints on the projection matrices.The regularization parameter C controls training loss, while λp and λq control the complexities of P and Q.
- Proposed Method: Introducing H = [P, Q]′[P, Q] converts the problem into learning a positive-semidefinite transformation metric and eliminates the need to determine the common-subspace dimension beforehand.After this substitution, the optimization involves α and H rather than P and Q, with trace(H) constrained by λ.
- Discussion: The linear formulation is limited because H may be ineffective for some tasks and grows with source and target feature dimensions, motivating the kernelized formulation for very high-dimensional data.The paper specifically identifies text categorization as an example where learning the linear metric can be computationally infeasible.
- Nonlinear Feature Transformation: Kernelization replaces linear inner products with kernel functions and yields a nonlinear transformation metric whose size depends on the number of training samples rather than feature dimension.The kernelized construction defines source and target kernel matrices and corresponding projection matrices in the nonlinear feature space.
- Detailed Solution: An alternating optimization algorithm iteratively updates the transformation metric and dual SVM variables, using projected gradient descent for the metric update and SVM for the α update.The algorithm initializes the metric, alternates updates, and checks convergence of the objective.
3. Related Work
Existing HDA methods either require task-specific transfer information, omit label information, assume manifold structure, or learn a universal transformation that may be inadequate across many classes.
- Task-specific HDA methods require additional information to transfer source knowledge to the target domain.
- HeMap learns a common subspace without using training labels, while manifold alignment uses labels but assumes a manifold structure that may not exist in real applications.
- ARC-t learns a universal asymmetric transformation metric, which may be insufficient when many classes require different feature transformations.
- The object-dataset summary reports 31 categories and separates domain dimensionality, total images, and training images.
4. Experiments
The experiments evaluate HFA for object recognition and multilingual text categorization under heterogeneous source–target settings with limited labeled target data. HFA generally achieves the strongest classification accuracy across the reported comparisons, while its optimization converges within 80 iterations on the object dataset and 40 on Reuters.
- 4. Experiments: Experiments cover object recognition and multilingual text categorization with one source domain, one target domain, limited labeled target samples, and unseen target test data.
- 4.1. Setup: The object dataset contains 4,106 images across 31 categories from amazon, dslr, and webcam domains, with SURF features used for representation.
- 4.1. Setup: For object recognition, dslr is the target; amazon and webcam are separate sources, with 3 randomly selected target training images per category and the remainder used for testing.
- 4.1. Setup: The Reuters multilingual dataset contains about 11K articles from 6 classes in 5 languages, represented using bag-of-words TF-IDF features.
- 4.1. Setup: The Reuters setup uses Spanish as target, four other languages as individual sources, 100 source documents per class, and m = 5, 7, 10, 15, or 20 target documents per class.
- 4.1. Setup: Because heterogeneous source and target dimensions prevent direct classifier training, comparisons use kernelized HFA alongside SVM T, KCCA, HeMap, DAMA, and ARC-t.
- 4.2. Classification Results: Classification accuracy is averaged over ten random training rounds for each method.
- 4.2. Classification Results: HFA outperforms the other methods under both object-recognition settings, while SVM T outperforms KCCA and HeMap using only 3 labeled target samples.
5. Conclusions and Future Work
HFA augments heterogeneous source and target features and learns projection matrices for HDA, achieving promising results on benchmark datasets. Its evaluation includes classification accuracy as target-domain training samples vary across source-language settings.
- Conclusions: HFA augments heterogeneous source and target features with newly proposed feature mapping functions.The augmented representations combine transformed data with original features and zeros.
- Evaluation: Figure 2 reports classification accuracies for m = 5, 7, 10, 15 and 20 target training samples per class, with Spanish as the target domain and different source languages.
- Conclusions: HFA learns two projection matrices using standard SVM hinge loss in both linear and nonlinear cases.The method also uses a transformation metric and alternating optimization algorithm to solve its formulated problem.
- Conclusions: HFA achieves promising results on two benchmark datasets for object recognition and text classification.