Source-linked AI summary
Strong Baselines for Neural Semi-supervised Learning under Domain Shift
Sebastian Ruder, Barbara Plank
TL;DR
Neural domain adaptation lacks broad comparisons across tasks, benchmarks, and strong baselines. The paper re-evaluates classic bootstrapping with neural models and introduces a more efficient multi-task tri-training method. Classic tri-training generally performs strongest, while the new method reaches state-of-the-art sentiment performance but is not consistently best.
Problem
Existing domain-adaptation studies often use task-specific methods, limited benchmarks, or weak baselines, hindering broad comparison.
Method
The paper re-evaluates general-purpose bootstrapping methods with neural models and proposes multi-task tri-training to reduce classic tri-training's time and space complexity.
Results
Classic tri-training outperforms recent alternatives overall, while multi-task tri-training establishes a new state of the art for sentiment analysis but not POS tagging.
Takeaways & Limitations
Classic tri-training is a strong baseline for neural semi-supervised learning under domain shift, making comparisons against such baselines important.
Takeaways & Limitations
Self-training can amplify errors under domain shift, and one benchmark has high variance that complicates objective comparison from reported means alone.
Abstract
from arXiv · showhide
Novel neural models have been proposed in recent years for learning under domain shift. Most models, however, only evaluate on a single task, on proprietary datasets, or compare to weak baselines, which makes comparison of models difficult. In this paper, we re-evaluate classic general-purpose bootstrapping approaches in the context of neural networks under domain shifts vs. recent neural approaches and propose a novel multi-task tri-training method that reduces the time and space complexity of classic tri-training. Extensive experiments on two benchmarks are negative: while our novel method establishes a new state-of-the-art for sentiment analysis, it does not fare consistently the best. More importantly, we arrive at the somewhat surprising conclusion that classic tri-training, with some additions, outperforms the state of the art. We conclude that classic approaches constitute an important and strong baseline.
1 Introduction
The paper argues that neural domain-adaptation methods need evaluation across tasks, benchmarks, and strong general-purpose baselines. It re-evaluates bootstrapping methods and proposes multi-task tri-training, finding classic tri-training especially competitive.
- Many domain-adaptation approaches rely on task-specific features, proprietary datasets, single benchmarks, or weak baselines, limiting comparisons across tasks.
- The paper re-evaluates general-purpose self-training, tri-training, and tri-training with disagreement for neural POS tagging and sentiment analysis across multiple domains.The evaluation uses two established benchmarks without additional task-specific measures.
- Classic tri-training outperforms task-agnostic state-of-the-art semi-supervised learning and recent neural domain-adaptation approaches.
- Multi-task tri-training reduces tri-training's time and space complexity, establishes a new sentiment-analysis state of the art, but loses to classic tri-training on POS tagging.
- The paper presents tri-training as a strong, robust baseline and evaluates bootstrapping methods against state-of-the-art approaches on two benchmark datasets.
2 Neural bootstrapping methods
The paper revisits classic bootstrapping methods for neural domain adaptation and introduces multi-task tri-training to reduce tri-training’s time and space costs. The methods rely on pseudo-labeling, model agreement, sampling, confidence selection, and diversity constraints.
- Self-training: Self-training adds high-confidence predictions on unlabeled examples as pseudo-labels, but errors can amplify because the model cannot correct its own mistakes.The paper uses confidence-based selection and later evaluates throttling, which selects the top n examples by relative confidence.
- Tri-training: Tri-training trains three models on bootstrap samples and adds an unlabeled example to one model when the other two agree on its label.Training stops when the classifiers no longer change; sampling candidate unlabeled instances limits the cost of evaluating large unlabeled sets.
- Tri-training with disagreement: Tri-training with disagreement adds examples only when two models agree and the third disagrees, targeting weak points while using data more efficiently than standard tri-training.The modification changes the original agreement condition to require disagreement from the model being updated.
- Multi-task tri-training: MT-Tri shares parameters across three jointly trained models, reducing tri-training’s time and space complexity while preserving model-specific output layers.The shared representation improves convergence, and one forward pass replaces separate forward passes for three independent models.
- Multi-task tri-training: MT-Tri uses an orthogonality constraint to keep output-layer features diverse; identical output parameters would reduce the method to self-training.The constraint discourages models from relying on the same predictive features, with orthogonality enforced between two softmax output layers.
- Multi-task tri-training: MT-Tri trains three outputs jointly on labeled source data, then uses pairwise agreement to pseudo-label target instances for a target-specific third model.Unlike classic tri-training, the labeled source models are trained without bootstrap sampling because the orthogonality constraint supplies diversity.
3 Experiments
The experiments evaluate bootstrapping and neural domain-adaptation methods on POS tagging and sentiment analysis across established multi-domain benchmarks. Classic tri-training is consistently strong, while multi-task tri-training improves efficiency and sentiment adaptation but is not uniformly best.
- Experimental setup: The study evaluates self-training, tri-training, disagreement tri-training, asymmetric tri-training, and MT-Tri on POS tagging and sentiment analysis across two domain-adaptation benchmarks.The evaluation uses multiple domains and compares against published state-of-the-art approaches without additional task-specific measures.
- Sentiment analysis: Tri-training achieves the best average results on two sentiment-analysis target domains and clearly outperforms the state of the art on average.Self-training performs surprisingly well but does not compete with tri-training; disagreement tri-training is only slightly better than self-training.
- Sentiment analysis: Sentiment analysis results show MT-Tri outperforming the state of the art on 3/4 domains and slightly surpassing traditional tri-training overall.The improvement is concentrated in the B→E and D→E scenarios, where classic tri-training struggles with stronger domain shifts.
- POS tagging: Self-training does not work for the sequence-prediction task, while low-data disagreement tri-training reaches 89.70 average accuracy and beats the baseline on 4/5 domains.Newsgroups is the exception, and MT-Tri falls below classic tri-training in this setup.
- POS tagging: In the high-data POS setup, classic tri-training is best, significantly outperforming the baseline in 4/5 cases, while MT-Tri improvements over asymmetric tri-training are not robust across domains.On test sets, tri-training performs consistently best; the authors suggest the MT-Tri model may be too simplistic for this high-data setup.
- POS analysis: Classic tri-training improves OOV accuracy on 3/5 domains but struggles on high-OOV domains, while FLORS is superior for unknown word-tag combinations.Tri-training works best on OOVs and low-frequency tokens, whereas no bootstrapping method performs well on unknown word-tag combinations.
4 Related work
Prior domain-adaptation work includes bootstrapping, shared representations, instance weighting, adversarial learning, and fine-tuning, but bootstrapping is scarcely studied with modern neural NLP under domain shift. Related neural work also includes self-ensembling and multi-task learning.
- Domain-adaptation research has explored bootstrapping, shared feature representations, instance weighting, adversarial learning, and fine-tuning.
- Bootstrapping approaches are rarely evaluated with recent neural NLP models under domain shift, and tri-training remains comparatively understudied.
- Neural self-ensembling includes snapshot and temporal ensembling, with distinctions between explicit and implicit ensembling becoming less clear.
- Multi-task learning supports parameter sharing and has been applied broadly across NLP tasks.
5 Conclusions
The paper argues that classic bootstrapping methods are strong baselines for neural semi-supervised learning under domain shift. It proposes a more efficient multi-task variant, but classic tri-training remains superior overall, especially for POS tagging.
- Classic tri-training works best across the two examined NLP tasks and outperforms a recent state-of-the-art method.
- Multi-task tri-training reduces tri-training’s time and space complexity and outperforms traditional tri-training and recent alternatives for sentiment analysis.
- Classic tri-training is superior for POS tagging, particularly on out-of-vocabulary and low-frequency tokens, suggesting less sensitivity to error propagation.
- The evaluation emphasizes comparing neural approaches with strong baselines and reporting results across several runs.