Source-linked AI summary
Marginalized Denoising Autoencoders for Domain Adaptation
Minmin Chen, Zhixiang Xu, Kilian Weinberger, Fei Sha
TL;DR
SDAs provide effective domain-adaptation features but are computationally expensive, difficult to tune, and poorly scalable to high-dimensional data. The paper introduces mSDA, which marginalizes corruption using linear denoisers with closed-form training and stacks nonlinear representations. Across benchmark datasets, mSDA matches SDA classification performance while reducing training from up to 2 days to a few minutes.
Problem
SDAs offer effective domain-adaptation features but suffer from high training cost, difficult model selection, non-convex optimization, and scalability challenges for high-dimensional inputs.
Method
mSDA uses linear denoisers, analytically marginalizes random corruption, computes parameters in closed form, and stacks layers with nonlinear feature generation.
Results
mSDA matches SDA classification performance across benchmark datasets while reducing training from up to 2 days for SDA to a few minutes with mSDA.
Takeaways & Limitations
mSDA provides fast training, scalability to large and high-dimensional data, and implementation simplicity within domain adaptation.
Takeaways & Limitations
mSDA removes hidden nodes from SDA layers, which enables closed-form speedups but may entail limitations requiring further investigation.
Abstract
from arXiv · showhide
Stacked denoising autoencoders (SDAs) have been successfully used to learn new representations for domain adaptation. Recently, they have attained record accuracy on standard benchmark tasks of sentiment analysis across different text domains. SDAs learn robust data representations by reconstruction, recovering original features from data that are artificially corrupted with noise. In this paper, we propose marginalized SDA (mSDA) that addresses two crucial limitations of SDAs: high computational cost and lack of scalability to high-dimensional features. In contrast to SDAs, our approach of mSDA marginalizes noise and thus does not require stochastic gradient descent or other optimization algorithms to learn parameters ? in fact, they are computed in closed-form. Consequently, mSDA, which can be implemented in only 20 lines of MATLAB^{TM}, significantly speeds up SDAs by two orders of magnitude. Furthermore, the representations learnt by mSDA are as effective as the traditional SDAs, attaining almost identical accuracies in benchmark tasks.
1. Introduction
Domain adaptation transfers classifiers from data-rich source domains to data-scarce target domains despite distribution differences. The paper proposes mSDA to retain SDA-based representations while reducing their computational burden.
- Domain adaptation aims to generalize classifiers from data-rich source domains to data-scarce target domains.
- Different source and target distributions complicate adaptation, motivating reweighting, subsampling, and joint feature-representation methods.
- SDAs learn robust representations by reconstructing partially and randomly corrupted inputs, then use intermediate outputs as classifier features.
- SDAs are costly because iterative optimization, high-dimensional inputs, and hyperparameter selection can require hours or days of training.
- mSDA uses linear denoisers and marginalizes random corruption, enabling parameter learning in closed form rather than through conventional iterative optimization.
- Across benchmark datasets, mSDA matches SDA classification performance while reducing training from up to 2 days to a few minutes.
2. Notation and Background
The paper frames domain adaptation as learning from labeled source data and unlabeled target data, using denoising autoencoders to derive transferable representations. SDA training is effective but slow, difficult to tune, and non-convex.
- The setup uses labeled source examples and unlabeled target examples, with zero-padding to equalize feature dimensionality.
- An autoencoder maps inputs through an encoder and decoder to reconstruct the original data while minimizing reconstruction error.
- Denoising autoencoders randomly corrupt inputs and reconstruct the originals; this work uses binary masking that sets features to zero.
- SDAs stack denoising autoencoders greedily, feeding each layer’s hidden representation into the next layer.
- SDA features improve source-to-target transfer relative to classifiers trained directly on original inputs.
- SDA representations can reconstruct unseen target-domain features from correlated sentiment features, allowing source-trained classifiers to assign them weights.
- SDA training is slow and hard to parallelize, requires several cross-validated hyperparameters, and is non-convex and initialization-dependent.
3. SDA with Marginalized Corruption
mSDA replaces conventional denoising autoencoder optimization with analytically marginalized linear denoisers. It then adds nonlinear squashing and greedy layer stacking to produce scalable learned representations.
- mSDA addresses SDA concerns through speedups, fewer meta-parameters, faster model selection, and layer-wise convexity.
- Single-layer denoiser: Each mDA layer applies random feature removal and learns a single linear mapping W to minimize squared reconstruction loss.
- Single-layer denoiser: A constant feature can be appended to inputs and left uncorrupted so the mapping incorporates a bias term.
- Single-layer denoiser: Multiple corruption passes reduce variance by learning W from an overall squared loss across differently corrupted inputs.
- Marginalized corruption: The design matrix collects input vectors as columns, while repeated and corrupted versions represent the training copies used in the loss.
- Marginalized corruption: A larger number of corruption samples better approximates the ideal limit of infinitely many noisy copies.
- Marginalized corruption: Expected corruption matrices permit direct computation of W without constructing corrupted inputs, using one data sweep and a convex, non-iterative closed-form optimization.
- Nonlinear feature generation and stacking: mSDA inserts tanh after computing linear weights and stacks layers greedily, feeding each transformed output into the next mDA layer.
4. Extension for High Dimensional Data
The high-dimensional extension avoids learning one full mapping by reconstructing only frequent features through multiple subspace mappings. These mappings are solved in closed form, averaged into a compact first layer, and then further stacked; on the Amazon benchmark, mSDA5 generally achieved the lowest transfer loss.
- Motivation: High-dimensional SDA training can become prohibitively expensive, motivating feature truncation that risks discarding information in rarer features.The workaround retains only the r ≪ d most common features, but this prevents use of rarer features.
- Method: Instead of one mapping W ∈ R^d×(d+1), the method learns multiple mappings while reconstructing only the r ≪ d most frequent features.The described experiments use r = 5000 frequent features.
- Results: With exceptions for B → E and D → E, mSDA5 leads to the lowest transfer loss across the twelve small-Amazon domain-adaptation tasks.The tasks use reviews from Books, Kitchen appliances, Electronics, and DVDs.
- Method: The input features are divided into S random non-overlapping subsets, with an independent mapping W_s learned for each subspace.Each subspace mapping is trained separately using the reconstruction objective.
- Method: Each mapping W_s is solved in closed form, and the first mSDA layer is defined as the average of all reconstructions.This produces a first layer of dimension r ≪ d.
5. Results
Across Amazon domain-adaptation benchmarks, mSDA achieves transfer performance comparable to or better than SDA while substantially reducing training time, including on larger and higher-dimensional data.
- Experimental setup: The benchmark evaluates linear SVMs using raw bag-of-words features and compares mSDA with baseline, PCA, SDA, SCL, and CODA under source-to-target transfer metrics.Transfer ratio is averaged across tasks because task difficulty varies substantially; lower transfer ratio indicates better domain adaptation.
- Comparison per task: mSDA and SDA features clearly outperform other compared methods on the twelve-task benchmark, with mSDA achieving lower transfer loss than SDA in ten tasks.
- Timing: 180× speedup gives mSDA a comparable transfer ratio to SDA on the small benchmark, with five-layer training requiring less than 2 minutes.One-layer mSDA training on 27,677 documents requires less than 25 seconds.
- Large scale results: 230× speedup lets three-layer mSDA match SDA’s transfer error and transfer ratio on 380 large-scale tasks with 14 minutes of training.The large benchmark contains 340,000 reviews from 20 domains; averaged transfer errors are 10.50% for one-layer SDA and 10.47% for three-layer mSDA.
- Scalability: 458× speedup is achieved with 30,000 input features, where SDA requires over one day and mSDA requires only 3 minutes while matching SDA’s transfer ratio.Both algorithms benefit from increasing dimensionality up to 30,000 features, and mSDA’s speed advantage grows with dimensionality.
- General trends: Additional mSDA layers improve results, reaching or surpassing one-layer SDA accuracy while retaining several-hundred-fold speedups across diverse classification tasks.The reported tasks include domain classification and sentiment analysis, and the features remain efficient to train on high-dimensional data.
6. Discussion and Conclusion
mSDA simplifies SDA training by marginalizing corruption with linear denoisers, yielding closed-form parameter estimation and substantial speed-ups. Its scalability and implementation simplicity motivate broader use, while the absence of hidden nodes remains a limitation requiring further investigation.
- Discussion and Conclusion: mSDA marginalizes corruption during training and uses linear denoisers, enabling closed-form parameter estimation instead of conventional iterative optimization.The approach trains each layer to optimally denoise all possible corrupted inputs, which is practically infeasible for standard SDAs.
- Discussion and Conclusion: The absence of hidden nodes enables substantial speed-ups but may limit mSDA's expressiveness and requires further investigation.This architectural difference is identified as a potential limitation of the method.
- Discussion and Conclusion: Fast training, scalability to large and high-dimensional data, and implementation simplicity make mSDA promising beyond domain adaptation.The authors suggest that these properties may support applications across a broad range of machine-learning research problems.