Learning Domain-Invariant Subspace using Domain Features and Independence Maximization

Ke Yan; Lu Kou; David Zhang

Source-linked AI summary

Learning Domain-Invariant Subspace using Domain Features and Independence Maximization

Ke Yan, Lu Kou, David Zhang

arXiv:1603.04535v2Mar 15, 2016cs.CVcs.AIcs.LG

TL;DR

The paper addresses domain adaptation when sensor and measurement data undergo instrumental variation or time-varying drift, making source-trained models unsuitable for target conditions. It proposes MIDA and SMIDA, which use background-derived domain features and HSIC-based independence learning, with feature augmentation for background-specific projection. Experiments on synthetic and real-world sensor, measurement, and computer-vision datasets report effective adaptation, while the methods remain flexible and fast.

Problem
Different training and test distributions, including instrumental variation and time-varying drift, make source-trained models unsuitable for target conditions and motivate adaptation using target samples.
Method
MIDA defines domain features describing sample backgrounds and learns a subspace independent of them with HSIC; SMIDA additionally uses labels, and feature augmentation projects samples according to background.
Results
Experiments on synthetic datasets and four real-world sensor, measurement, and computer-vision datasets confirm the effectiveness of MIDA and SMIDA; SMIDA achieves the best average domain adaptation accuracy on the gas sensor array drift dataset.
Takeaways & Limitations
MIDA and SMIDA uniformly handle discrete or continuous distributional change and support multiple domains and supervised, semi-supervised, or unsupervised adaptation.

Abstract

from arXiv · show

Domain adaptation algorithms are useful when the distributions of the training and the test data are different. In this paper, we focus on the problem of instrumental variation and time-varying drift in the field of sensors and measurement, which can be viewed as discrete and continuous distributional change in the feature space. We propose maximum independence domain adaptation (MIDA) and semi-supervised MIDA (SMIDA) to address this problem. Domain features are first defined to describe the background information of a sample, such as the device label and acquisition time. Then, MIDA learns a subspace which has maximum independence with the domain features, so as to reduce the inter-domain discrepancy in distributions. A feature augmentation strategy is also designed to project samples according to their backgrounds so as to improve the adaptation. The proposed algorithms are flexible and fast. Their effectiveness is verified by experiments on synthetic datasets and four real-world ones on sensors, measurement, and computer vision. They can greatly enhance the practicability of sensor systems, as well as extend the application scope of existing domain adaptation algorithms by uniformly handling different kinds of distributional change.

I. INTRODUCTION

Domain adaptation addresses distribution differences between labeled source data and target data, especially instrumental variation and time-varying drift in sensor systems. MIDA introduces domain features and independence-based subspace learning to handle discrete and continuous changes.

Motivation: Different collection conditions create distinct source and target distributions, making target-domain labeling costly for developing new prediction models.Domain adaptation leverages unlabeled, and possibly a few labeled, target samples.
Motivation: Instrumental variation and time-varying drift make models trained on earlier devices or periods unsuitable for new devices or later times.Sensor responses can differ across instruments and change with operating conditions, sensing characteristics, or signal sources.
Motivation: Machine olfaction is strongly affected because e-noses use chemical sensors prone to instrumental variation and time-varying drift.Traditional drift correction requires repeatedly collecting predefined transfer samples for each device and time period.
Existing approaches: Existing feature-level adaptation methods seek domain-invariant representations, but many are designed for discrete source and target domains rather than streaming time-varying change.TCA, for example, minimizes distributional difference using maximum mean discrepancy.
Existing approaches: TCA projects all samples into a common subspace, so samples with similar appearance but different concepts cannot be distinguished.This limitation motivates richer background information in adaptation.
Proposed approach: MIDA defines domain features describing sample backgrounds, then learns a latent space maximally independent of those features using HSIC.Feature augmentation concatenates original features with domain features, while SMIDA additionally exploits label information.

B. Hilbert-Schmidt Independence Criterion (HSIC)

HSIC measures dependence between two sample sets through kernel representations. With characteristic kernels, zero HSIC is equivalent to independence, enabling the paper to minimize dependence between learned features and domain features.

Definition: HSIC measures dependence between sample sets X and Y using kernel functions associated with reproducing kernel Hilbert spaces.It is defined as the squared Hilbert-Schmidt norm of a cross-covariance operator.
Interpretation: With characteristic kernels, HSIC equals zero if and only if the two variables are independent.A large HSIC indicates strong dependence relative to the chosen kernels.
Use in this paper: HSIC has been used for feature extraction and selection, typically to maximize dependence between extracted features and labels.This paper instead uses it to reduce dependence between extracted features and domain features.

III. PROPOSED METHOD

Domain features encode sample background information that correlates with feature distributions and differs between training and test data. For sensor drift correction, the paper represents device identity and acquisition time as domain features.

Domain Feature: Domain features are designed to describe naturally available background information that differs between training and test samples and correlates with original-feature distributions.The paper identifies device label and acquisition time as principal background variables for drift correction.
Domain Feature: For instrumental variation, one-hot domain features encode the device from which each sample was collected.With n_dev devices, each domain feature vector has n_dev components and marks the corresponding device.
Domain Feature: When time-varying drift is considered, acquisition time is added to the device representation.A sample’s vector encodes both its device and collection time.
Domain Feature: The domain-feature kernel matrix K_d is computed for HSIC using a linear kernel.The domain-feature vectors are assembled as columns of D, whose dimension is determined by the domain-feature representation.
Domain Feature: The same one-hot coding scheme can construct domain features for traditional domain adaptation problems with several discrete domains.These problems are treated as analogous to instrumental variation.

B. Feature Augmentation

Feature augmentation concatenates original features with domain features, enabling background-specific subspaces that can distinguish samples with similar appearances but different concepts.

The strategy concatenates original features and domain features to form an augmented representation.
A learned projection of the augmented vector contains components for the original features and background-dependent features.This provides a linear dimensionality-reduction view of the augmentation mechanism.
Augmentation maps samples into a higher-dimensional space before projection, making alignment directions easier to find.
For machine olfaction, background-specific bias lets samples with similar appearances but different concepts receive different treatment as sensor sensitivity changes over time.The strategy also helps align domains in each projected dimension.

C. Maximum Independence Domain Adaptation (MIDA)

MIDA learns a kernel subspace whose projected features are independent of domain features while preserving data variance, using an eigenvalue problem with an orthonormal constraint.

MIDA pools training and test samples, augments their features, and need not explicitly identify each sample’s domain during formulation.
Kernel mappings and a learned projection produce h-dimensional projected samples from the augmented data.
Independence between projected features and domain features is used to diminish inter-domain discrepancy in the learned subspace.
Variance preservation is included because domain adaptation should retain important data properties beyond distribution alignment.
The projection satisfies W^TW = I and uses a trade-off parameter µ between independence and variance preservation.
The solution uses the h largest-eigenvalue eigenvectors of Kx(−HKdH + µH)Kx; kernel choice controls detectable dependence types and requires suitable width selection for powerful kernels.
Unlike TCA, which is feasible only for two discrete domains, MIDA is described as handling a variety of situations.

D. Semi-supervised MIDA (SMIDA)

SMIDA extends MIDA by incorporating labeled samples through label dependence, while retaining domain alignment and supporting classification and regression labels.

SMIDA extends MIDA by incorporating known labels into subspace learning while aligning samples with different backgrounds.
The method uses HSIC to maximize dependence between projected features and labels, including discrete classification labels and continuous regression targets.
Classification labels use one-hot coding, whereas regression targets are centered and represented as continuous values for labeled samples.
The SMIDA solution uses eigenvectors of Kx(−HKdH + µH + γHKyH)Kx, with γ controlling the label-dependence term.
The algorithm constructs domain features, augments samples, computes kernel matrices, obtains W from the leading eigenvectors, and outputs Z = W^TKx.
Manifold regularization can preserve geometry and slightly increase accuracy, but it was not adopted because it requires three additional hyper-parameters.

IV. EXPERIMENTS

Experiments evaluate the proposed methods on synthetic data, sensor and spectroscopy drift correction, and visual object recognition against recent domain-invariant feature methods.

The evaluation covers synthetic datasets, two e-nose datasets, a spectroscopy dataset, and a visual object-recognition dataset.
Comparisons are made with recent unsupervised domain adaptation algorithms that learn domain-invariant features.

A. Synthetic Dataset

Synthetic experiments evaluate MIDA and SMIDA under discrete domains, continuous drift, dimensionality reduction, and nonlinear alignment. Across these settings, the proposed methods generally improve classification and exploit domain or temporal information, with continuous SMIDA strongest on the gas-sensor drift dataset.

Discrete domains: MIDA achieved 88% accuracy versus 70% for TCA and 53% for original features on the 2D discrete-domain dataset.TCA aligned domains only on the first projected dimension, whereas MIDA aligned them in both projected dimensions; feature augmentation contributed to the second-dimension alignment.
Continuous distributional change: MIDA achieved 90% accuracy versus 74% for SSA and 55% for the original data on the 2D continuous-drift dataset.MIDA used chronological sample order as domain features and projected data orthogonally to the drift direction after augmenting the data with time.
Kernel comparison: MIDA with an RBF kernel achieved 87% accuracy versus 57% with a linear kernel and 50% for the original data.The experiment illustrates that nonlinear kernels help when domain distributions are not linearly alignable.
Gas sensor array drift: Continuous SMIDA achieved the best average domain-adaptation accuracy on the gas sensor array drift dataset.Continuous MIDA and SMIDA outperformed their discrete versions and ML-comGFK; removing feature augmentation from continuous SMIDA reduced average accuracy by four percentage points.
Gas sensor array drift: MIDA and SMIDA outperformed other methods when more than 30 features were extracted.The gas sensor array drift evaluation used batch 1 for labeled training and batches 2–10 as unlabeled test data, with 13,910 samples collected over 36 months.

C. Breath Analysis Dataset

The breath analysis dataset contains instrumental variation and time-varying drift that complicate disease screening with e-noses. MIDA and SMIDA address these changes, with SMIDA achieving better results when labeled source information is preserved.

Dataset and drift: E-nose disease screening is hindered by instrumental variation and time-varying drift across devices and acquisition times.The dataset covers five diseases and healthy samples, represented by nine steady-state gas-sensor responses.
Dataset and drift: Sensor sensitivity decayed over time, and severe aging required replacing sensors in both devices around day 200.A 0.3 V response could indicate low concentration on day 0 but high concentration on day 150.
Experimental setup: The experiments used device-based domain features for discrete adaptation and acquisition-time features for continuous adaptation.Continuous domain features used exact acquisition time, while samples were arranged chronologically despite device labels.
Results: MIDA and SMIDA achieved better results than other methods, while SSA produced little improvement on the breath dataset.The authors attribute the limited SSA improvement partly to the stationary criterion not preserving important data properties.
Results: SMIDA outperformed MIDA because labeled samples from the first 50 examples in each class were better preserved.Feature augmentation also compensated for changes in conditional probability in this dataset.

D. Corn Dataset

The corn dataset evaluates domain adaptation for regression across three near-infrared spectrometers. MIDA and SMIDA produced the lowest average errors on both target domains, with SMIDA also outperforming target-domain training results.

Dataset and setup: The corn dataset contains moisture, oil, protein, and starch measurements collected from 80 samples using three near-infrared spectrometers.The experiments target instrumental variation in spectroscopy, where signals indicate analyte concentrations.
Evaluation: Regression performance was evaluated with RMSE for four prediction tasks and their average on two target domains.Overall average RMSE was also examined as a function of subspace dimension h.
Results: MIDA and SMIDA obtained the lowest average errors in both target domains.The methods used device-index domain features and RBF kernels in the reported experiments.
Results: SMIDA outperformed models trained and tested within each target domain, despite those models having no instrument variation between training and testing.The authors associate this with easier discrepancy correction, the RBF kernel, and using unlabeled target samples during subspace learning.

E. Visual Object Recognition Dataset

The visual object recognition evaluation uses four domains and compares unsupervised and semi-supervised adaptation settings. The reported discussion highlights a close relationship between MIDA and TCA under specific domain conditions and a limited role for feature augmentation here.

Dataset and setup: The benchmark comprises Amazon, Caltech-256, DSLR, and Webcam domains with ten shared object classes and 2533 images.Images were represented by 800-bin SURF histograms normalized and z-scored by dimension.
Evaluation: Unsupervised trials used source-domain training samples and unlabeled target samples, while semi-supervised trials added three labeled target samples per class.Averaged accuracies and standard errors were reported for each domain pair.
Baselines and implementation: The comparison included published results for GFK, LTSL, DASC, and IGLDA alongside tuned implementations of other methods.Logistic regression was used as the classifier, with a degree-2 polynomial kernel for the listed kernel methods.
Interpretation: Under one-source, one-target conditions, the HSIC criterion in MIDA and MMD in TCA are identical under certain conditions.The passage also states that feature augmentation is not crucial here because conditional probability does not change.

V. CONCLUSION

The paper concludes that MIDA and SMIDA learn domain-invariant features by maximizing independence from domain features, with feature augmentation helping align domains. Their design supports varied distribution shifts and learning settings while requiring one eigenvalue decomposition.

Core approach: MIDA reduces inter-domain discrepancy by maximizing independence between learned features and sample domain features.Domain features encode background information such as domain labels, device labels, and acquisition times.
Core approach: Feature augmentation adds domain-specific biases to learned features and helps MIDA align domains.This mechanism is intended to account for background-specific changes in the data.
Scope: MIDA and SMIDA support discrete or continuous shifts, supervised, semi-supervised, or unsupervised learning, multiple domains, and classification or regression.The conclusion characterizes the algorithms as flexible across these settings.
Practicality: The algorithms require only one eigenvalue decomposition problem and are described as easy to implement and fast.This computational characterization is stated as part of the conclusion.