Source-linked AI summary
Self-training Improves Pre-training for Natural Language Understanding
Jingfei Du, Edouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Ves Stoyanov, Alexis Conneau
TL;DR
The paper investigates whether self-training adds information beyond strong unsupervised pretraining and how to obtain task-relevant unlabeled data without assuming an in-domain corpus. It introduces SentAugment to retrieve and label relevant web sentences, finding complementary gains across tasks, few-shot learning, and knowledge distillation. The approach remains bounded by a retrieval limitation identified for future work.
Problem
Existing semi-supervised methods often require unlabeled data from the downstream domain, limiting broader use, while the relationship between self-training and language-model pretraining remains an open question.
Method
SentAugment computes task-specific sentence embeddings from labeled data to retrieve relevant sentences from a large web corpus, which a teacher model labels for training.
Results
Self-training complements unsupervised pretraining across natural language tasks, improving RoBERTa-Large by 1.2% average accuracy on six standard classification benchmarks and yielding gains in few-shot learning and distillation.
Takeaways & Limitations
Open-domain web data retrieved with SentAugment is sufficient for effective self-training, knowledge distillation, and few-shot learning.
Takeaways & Limitations
The paper identifies retrieval enhancements as future work.
Abstract
from arXiv · showhide
Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.
1 Introduction
The paper asks whether self-training provides information complementary to unsupervised pretraining and addresses the lack of task-domain unlabeled data with SentAugment. SentAugment retrieves task-specific web sentences for self-training, improving strong RoBERTa baselines across classification, few-shot, and distillation settings.
- Self-training uses a labeled-data-trained teacher to create synthetic labels for unlabeled examples, which train a student model.
- Pretraining first trains models on an auxiliary task such as language modeling before fine-tuning on the target task.
- Existing semi-supervised methods often assume same-domain unlabeled data, limiting application to low-resource downstream tasks.
- SentAugment retrieves task-specific in-domain sentences from a large web corpus and uses teacher-generated synthetic labels to train the final model.
- 1.2% average accuracy improvement over RoBERTa-Large was reported across six standard classification benchmarks.
- 3.5% average accuracy improvement was reported for few-shot learning, while knowledge distillation improved distilled RoBERTa-Large by 2.9% on average.
2 Approach
The approach builds task-relevant unlabeled data by querying a large web sentence bank with task embeddings, then applies self-training or distillation using teacher-generated labels. Retrieval, confidence filtering, class-ratio control, and continuous labels make the augmented data scalable for several training settings.
- SentAugment: data augmentation for semi-supervised learning: SentAugment retrieves task-specific in-domain candidates from a large external sentence bank instead of relying on pre-existing in-domain unlabeled data.
- SentAugment: data augmentation for semi-supervised learning: Task embeddings query the sentence bank with cosine similarity; they can average all training samples, average samples per label, or retain one embedding per sentence.
- SentAugment: data augmentation for semi-supervised learning: The procedure fine-tunes a RoBERTa-Large teacher, selects high-scoring retrieved sentences while preserving label ratios, and fine-tunes a RoBERTa-Large student with KL-divergence.
- SentAugment: data augmentation for semi-supervised learning: The method reduces annotation cost by retrieving a few million candidates from the sentence bank and filtering them before synthetic labeling.
- Knowledge-distillation: Knowledge distillation uses a student with an order of magnitude fewer parameters than the RoBERTa-Large teacher and continuous probabilities as synthetic labels.
- Few-shot learning: Few-shot learning augments sparse class-labeled training sets by two to three orders of magnitude and uses a teacher trained on only a few examples per class.
3 Experimental setup
The experiments evaluate SentAugment-based self-training across several natural language understanding benchmarks, using CommonCrawl-derived sentence banks and RoBERTa-Large training procedures. They also compare self-training with baseline and continued-pretraining configurations, including a few-shot setting.
- Data: The external sentence bank is built from filtered and deduplicated CommonCrawl text, with CC-100M, CC-1B, and CC-5B corpus sizes.The corpora contain one hundred million, one billion, and five billion sentences, respectively.
- Evaluation tasks: The evaluation covers SST-2, SST-5, product classification, hate-speech classification, question classification, and named entity recognition.These tasks span sentiment, product, social-comment, question, and sequence-labeling settings.
- Training procedure: The student-model experiments use the open-source RoBERTa-Large pretrained Transformer baseline, Adam optimization, a 1e-5 learning rate, batch size 16, and dropout 0.1.Students are fine-tuned on synthetically annotated data, with KL divergence used in the training procedure.
- Comparisons: Table 2 compares a RoBERTa-Large baseline, in-domain continued pretraining, and self-training on natural language understanding benchmarks.The comparison is designed to distinguish self-training from domain adaptation provided by continued pretraining.
- Few-shot evaluation: Few-shot experiments use five sampled training sets with 20 examples per label, 10 seeds per training set, and mean accuracy across the top three validation-selected models.Final results are reported as the mean and standard deviation across the five training sets.
4 Analysis and Results
The experiments show that SentAugment enables effective self-training across standard, few-shot, and knowledge-distillation settings. Ablations identify task-specific retrieval, sentence embeddings, corpus scale, and continuous labels as important components, while retrieval scale eventually saturates and filtering reduces computational cost.
- Self-training experiments: Self-training improves RoBERTa-Large by 1.2% on average across six standard classification benchmarks, while continued pretraining on the same data decreases performance from 87.4% to 86.2%.Improvements reach 2.6% on SST-5 and 3.1% on IMP.
- Few-shot learning: Few-shot self-training yields 3.5% average gains, increasing performance from 72.0% to 75.5% across tasks and reducing variance.On sequence labeling, the student reaches 58.4 F1 versus 49.0 F1 for the teacher.
- Knowledge distillation: SentAugment raises distilled-model performance to 85.4% average accuracy, only 0.9% below the teacher despite using almost ten times fewer parameters.Using the same amount of data, data-selected augmentation reaches 81.9% average accuracy, compared with 82.5% for ground-truth data.
- Task-specific retrieval: Label-average task queries achieve 83.1% average accuracy, outperforming all-average queries and providing greater diversity for unbalanced classes.Per-sentence queries do not generally improve performance, except for named entity recognition.
- Sentence embedding space: Paraembeddings outperform uSIF and average-word2vec sentence representations, improving over uSIF by 0.4% on average.The comparison evaluates average-word2vec, uSIF-ParaNMT, and the paper’s sentence encoder.
- Scaling bank size: Performance increases substantially as the sentence bank grows from 50 million to 1 billion sentences but appears to saturate from 1 billion to 5 billion.The authors note that the larger bank may still help rare-domain tasks and leave diversity-enhancing retrieval for future work.
- Continuous labels: Continuous synthetic labels outperform discrete labels by 0.9% on average, whereas discrete labels provide very little gain in this setting.The paper contrasts this result with prior computer-vision findings and attributes the difference partly to the number of supervised classes.
- Computational cost: SentAugment prefiltering reduces the data that the teacher must annotate, avoiding the expense of confidence-only filtering over billions of sentences.Annotating 1 billion sentences with RoBERTa-Large is estimated to take 83 hours on one Volta-32GB GPU.
5 Analysis of similarity search
The paper evaluates SentAugment sentence embeddings on semantic textual similarity benchmarks and uses large-scale similarity search to retrieve related sentences from CommonCrawl. The embeddings outperform prior approaches on the reported STS benchmarks, while retrieved neighbors provide paraphrases, reformulations, and additional information across domains.
- Embedding evaluation: SASE embeddings outperform previous approaches by 0.2% on STS 2012–2016 and by 0.9% on the STS-Benchmark test sets.Table 10 reports Pearson’s r × 100 for these comparisons.
- Similarity search: SentAugment uses an embedding space with billions of sentences to retrieve nearest neighbors from CommonCrawl for task-specific similarity search.The examples cover biomedical, financial, and hate-speech domains and use sentence-level or label-level queries.
- Retrieved examples: Retrieved neighbors can preserve meaning, add information, or reformulate the input sentence, illustrating how similarity search supplies related unlabeled text.The passages describe these properties in examples drawn from multiple domains.
6 Conclusion
The paper shows that self-training effectively leverages unlabeled data in natural language understanding, complementing unsupervised pre-training and improving strong RoBERTa baselines across tasks.
- Self-training is complementary to unsupervised pre-training across a range of natural language tasks.
- SentAugment retrieves relevant sentences from a large web corpus for NLP data augmentation.
- Combining self-training with pre-training produces further improvements over a strong RoBERTa baseline.
- Open-domain data retrieved with SentAugment is sufficient for good accuracy in few-shot learning.