Source-linked AI summary
Frustratingly Easy Domain Adaptation
Hal Daumé
TL;DR
Domain adaptation must use abundant labeled source data together with limited labeled target data without requiring a new learning algorithm. The paper uses feature augmentation as a preprocessing transformation for standard supervised learners and reports performance at least as strong as state-of-the-art methods, with a 10-line implementation. Its formal analysis remains incomplete, though the method does not make learning harder in a minimum Bayes error sense.
Problem
Domain adaptation seeks a model that performs well in a target domain when substantially more labeled data are available from a source domain.
Method
The method augments source and target feature representations, turning domain adaptation into a standard supervised-learning problem applicable to algorithms such as maxent and SVMs.
Results
The approach is reported to perform as well as or better than current state-of-the-art techniques, with a complete transformation implemented in a 10-line Perl script.
Takeaways & Limitations
Feature augmentation lets standard supervised learning algorithms perform the adaptation, and the learned domain-specific weights have plausible interpretations across domains.
Takeaways & Limitations
A formal framework for fully supervised domain adaptation is not yet available, and the analysis establishes only that augmentation does not make learning harder in a minimum Bayes error sense.
Abstract
from arXiv · showhide
We describe an approach to domain adaptation that is appropriate exactly in the case when one has enough ``target'' data to do slightly better than just using only ``source'' data. Our approach is incredibly simple, easy to implement as a preprocessing step (10 lines of Perl!) and outperforms state-of-the-art approaches on a range of datasets. Moreover, it is trivially extended to a multi-domain adaptation problem, where one has data from a variety of different domains.
1 Introduction
Domain adaptation transfers learning from a labeled source domain to a desired target domain. The paper transforms this challenge into standard supervised learning by augmenting source and target feature spaces.
- Domain adaptation seeks models that transfer from a labeled source domain, such as newswire, to a desired target domain, such as biomedical documents.
- The proposed transformation augments both source and target feature spaces, after which any standard supervised algorithm can be applied.
- The approach is presented as a simple alternative to existing domain-adaptation algorithms for the fully supervised case.
- The introduction distinguishes fully supervised adaptation from semi-supervised adaptation using an unannotated target corpus.
- The paper focuses on the fully supervised setting, combining a large annotated source corpus with a small annotated target corpus.
2 Problem Formalization and Prior Work
The formalized task learns a target-domain predictor from source and target data, while prior approaches either combine, reweight, cascade, interpolate, or regularize models across domains.
- Problem Formalization: The goal is to learn h: X → Y with low expected loss on the target distribution from source and target samples, typically with N ≫ M.
- Problem Formalization: The paper assumes X = R^F and Y = {−1, +1} for discussion, while noting that most techniques are more general.
- Baselines: Standard baselines train on source data, target data, or their union; when N ≫ M, union training may wash out target-data influence.
- Baselines: WEIGHTED addresses this imbalance by reweighting source examples, with the weight selected by cross-validation.
- Baselines: PRED adds source-model predictions as target-model features, whereas LININT interpolates source-only and target-only predictions using target development data.
- Prior Work: PRIOR regularizes target-model weights toward source-model weights, and prior work reported that it outperformed baseline approaches on four datasets.
- Prior Work: Another prior approach learns separate source-specific, target-specific, and general models, distinguishing these information types per example.
3 Adaptation by Feature Augmentation
Feature augmentation converts domain adaptation into supervised learning by creating general, source-specific, and target-specific feature copies. The construction supports kernelized models, joint source-target regularization, and extension to multiple domains.
- Each original feature is split into general, source-specific, and target-specific versions, with source examples using general plus source-specific features and target examples using general plus target-specific features.
- For X = R^F, source and target inputs map into an augmented space R^3F using Φs and Φt, with zero vectors masking the opposite domain-specific block.
- The representation lets the classifier assign shared weights to features stable across domains while assigning domain-specific weights when feature-label associations differ.The paper illustrates this with “the,” whose determiner label is shared, and “monitor,” whose noun interpretation is target-specific.
- The first expansion is redundant but is retained because it generalizes directly to K domains, where the augmented space contains K+1 copies of the original feature space.
- 3.1 A Kernelized Version: The approach has a kernelized form, although the experiments use a simple linear kernel.In the expanded representation, same-domain pairs have kernel value 2K(x, x′), while different-domain pairs have K(x, x′).
- 3.2 Analysis: With ℓ2 regularization, feature augmentation induces a penalty proportional to ||ws − wt||2 while jointly optimizing source and target predictive weights.Joint optimization lets one supervised learner regulate the trade-off between general and domain-specific weights without separately selecting task-specific prior parameters.
4 Results
Across varied sequence-labeling domain-adaptation tasks, AUGMENT generally performs best, but its advantage breaks down when source and target domains are especially similar. Weight visualizations show that learned feature behavior varies across domains and reflects domain-specific lexical and capitalization patterns.
- Experimental results: AUGMENT performs best excluding the eight Brown-corpus Treebank-Chunk domains, where its advantage breaks down.When Brown data are lumped into one domain, AUGMENT again performs best, followed by PRIOR.
- Experimental results: On the eight Brown-corpus Treebank-Chunk sections, AUGMENT performs rather poorly and no approach clearly wins.The authors hypothesize these tasks have relatively similar source and target domains, making feature-space expansion less useful.
- Experimental results: MEGAM is roughly comparable to the best tabulated systems on most datasets but requires about ten times longer per training run.MEGAM also needed five-to-ten cross-validation iterations for hyperparameter tuning.
- Model introspection: Hinton diagrams visualize seven domain-specific and one cross-domain weight versions for each feature, with columns as domains, rows as classes, and box size indicating absolute weight.Black boxes denote negative weights and white boxes positive weights.
- Model introspection: The capitalization feature is strongly positive for entity-hood across classes, unavailable in broadcast news, and negative in usenet.The broadcast-news behavior reflects absent capitalization information.
- Model introspection: Domain-specific lexical patterns explain differing weights: “bush” is generally person-related, while “the” and name-list membership behave differently across domains.Broadcast-conversation phrases make previous “the” predictive of entity-hood, while leader names create metonymic effects for name-list membership.
5 Discussion
The paper presents a simple domain-adaptation approach that outperforms previous methods under a common, easy-to-verify condition. It leverages supervised learning by augmenting the feature space, while leaving theoretical analysis and domain-similarity tuning for future work.
- The approach outperforms previous domain-adaptation methods under a common and easy-to-verify condition.
- Feature-space augmentation forces the supervised learning algorithm to perform the adaptation.
- 0.24 percentage points: shallow parsing error on the CoNLL treebank section improves from 5.35 to 5.11.The paper describes this improvement as small but real.
- The authors identify formal theoretical analysis and further exploration of the domain-similarity parameter α as major future directions.Current results show only that augmentation does not make learning harder, not that it makes learning easier.