Source-linked AI summary
Generating Datasets with Pretrained Language Models
Timo Schick, Hinrich Schütze
TL;DR
High-quality sentence embeddings remain difficult to obtain from PLMs without costly labeled sentence pairs, since standard unsupervised approaches typically trail supervised training. DINO uses instructions and self-debiasing to have large PLMs generate labeled sentence-pair datasets from scratch, then trains smaller models that achieve strong results across semantic textual similarity datasets.
Problem
PLMs do not produce good sentence embeddings out of the box, while supervised approaches require large amounts of difficult and potentially expensive human-annotated sentence-pair data.
Method
DINO provides large PLMs with natural-language instructions and self-debiasing to generate labeled sentence-pair datasets from scratch, without labeled data, finetuning, or parameter updates.
Results
Models trained on DINO-generated datasets achieve strong results on several semantic textual similarity datasets and Sentence-RoBERTa outperforms strong baselines.
Takeaways & Limitations
Large PLMs can be used as data generators for supervised training of much smaller and more efficient sentence-embedding models.
Takeaways & Limitations
Generated examples can be noisy: some pairs labeled fully similar are not similar, while some pairs labeled fully dissimilar remain on related topics.
Abstract
from arXiv · showhide
To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pretraining objectives or finetuned on a large set of labeled text pairs. While the latter approach typically outperforms the former, it requires great human effort to generate suitable datasets of sufficient size. In this paper, we show how PLMs can be leveraged to obtain high-quality sentence embeddings without the need for labeled data, finetuning or modifications to the pretraining objective: We utilize the generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch, which we then use for finetuning much smaller and more efficient models. Our fully unsupervised approach outperforms strong baselines on several semantic textual similarity datasets.
1 Introduction
DINO addresses the difficulty of obtaining high-quality sentence embeddings by using pretrained language models to generate labeled sentence-pair datasets automatically. The resulting datasets support supervised training of smaller models and outperform strong baselines on several semantic textual similarity datasets.
- 1 Introduction: PLMs do not produce good sentence embeddings out of the box, while unsupervised sentence-level objectives typically lag behind supervised training on human-annotated pairs.Obtaining large amounts of high-quality labeled data is difficult and potentially prohibitively expensive.
- 1 Introduction: DINO replaces human crowdworkers with large PLMs to automatically create complete labeled datasets from scratch for training smaller models.The approach also enables use of large PLMs without updating their parameters.
- 1 Introduction: DINO generates labeled datasets of arbitrary size by providing pretrained language models with instructions.The method is named Datasets from Instructions, or DINO.
- 1 Introduction: STS-Dino is the first textual similarity dataset generated completely automatically without human annotation effort.The dataset is released as part of the paper’s contributions.
- 1 Introduction: Sentence-RoBERTa trained on STS-Dino outperforms strong baselines on several semantic textual similarity datasets.This is the paper’s reported benchmark outcome.
2 Related Work
Prior unsupervised sentence-embedding methods use word averaging, sentence-level objectives, or ensembles, while related generation methods generally require labeled data or perform poorly. DINO instead targets sentence-embedding dataset generation without those prerequisites.
- 2 Related Work: Unsupervised sentence embeddings have been obtained by averaging word embeddings, designing sentence-level objectives, or ensembling multiple methods.Recent approaches also supplement BERT and other PLMs with additional unsupervised objectives.
- 2 Related Work: Existing approaches that augment datasets with generated examples require an already labeled dataset to finetune the generator.This requirement distinguishes them from DINO’s from-scratch dataset generation.
- 2 Related Work: Although task descriptions support zero- or few-shot learning with PLMs, those approaches are not suitable for generating sentence embeddings.DINO applies instruction-based generation specifically to sentence-similarity data.
- 2 Related Work: A closely related attempt to replace human crowdworkers with PLMs for example generation performed poorly.The paper positions DINO as a different outcome from that approach.
3 Datasets from Instructions
DINO generates labeled text pairs by prompting pretrained language models with similarity-specific instructions. It supports both existing unlabeled texts and settings where the model must generate the initial texts.
- DINO aims to generate a dataset Z of text pairs and semantic similarity labels from token sequences.The framework defines texts X, labels Y, and model continuation probabilities.
- Given an unlabeled text set X1, each similarity-specific instruction prompts a PLM to generate a suitable second text.The instructions cover three similarity levels: Y = {0, 0.5, 1}.
- DINO uses counterlabels and self-debiasing to favor tokens fitting the target label while disfavoring tokens fitting alternative labels.The token probability is adjusted using α = exp(λ · δy), where λ is a decay hyperparameter.
- When no unlabeled text set exists, generating both texts directly can produce schema mismatches and insufficiently diverse initial texts.The authors instead generate x1 repeatedly, collect the results in X1, and then apply the existing procedure.
4 Experiments
The experiments evaluate DINO-generated datasets on multiple semantic textual similarity benchmarks in an unsupervised setting, comparing them with supervised and fully unsupervised baselines. DINO outperforms fully unsupervised approaches and achieves strong results despite noise in generated labels.
- DINO is evaluated on STS12–16, STSb, and SICK without task-specific training examples.
- The study compares DINO-trained Sentence-RoBERTa with NLI-supervised, fully unsupervised, and other sentence-embedding baselines.
- DINO outperforms all fully unsupervised approaches and supervised approaches on four of six STS datasets.
- STS-x2 performs better than STS-x1x2 on all STS datasets, whereas STS-x1x2 performs better on SICK.
- Removing self-debiasing, label smoothing, or random-pair augmentation hurts DINO performance.Increasing the decay constant to λ = 200 also slightly worsens performance as generated-sentence quality decreases.
- Human evaluation finds that 41% of supposedly different-topic examples retain some similarity, while only 47% of supposedly identical-meaning examples are nearly synonymous.Despite this noise, the authors report sufficient signal for successful training.
5 Conclusion
The conclusion presents DINO as a way to use large PLMs to generate labeled sentence-pair datasets without labeled data or parameter updates. The authors report strong similarity results with noise-handling measures, while identifying further noise reduction as future work.
- DINO uses large PLMs to generate entire labeled sentence-pair datasets from scratch without labeled data or parameter updates.
- Natural-language instructions combined with self-debiasing enable the dataset-generation procedure.
- Generated STS-x2 contains many noisy examples, including dissimilar y = 1 pairs and y = 0 pairs that remain somewhat related.
- With measures for handling noisy data, DINO-trained models achieve strong results on several semantic textual similarity datasets.
- Future work could reduce generated-data noise through alternative instructions or additional filtering.
A Experimental Setup
The experiments used two GPUs for dataset generation and trained Sentence Transformers substantially faster than generation.
- Generating each DINO dataset with two 11GB GPUs took approximately 48 hours, while Sentence Transformer training took less than two hours on average.
B Datasets
The authors publicly release both DINO-generated datasets, which contain over 121,000 filtered examples each.
- STS-x2 contains 121,275 examples and STS-x1x2 contains 143,968 examples after filtering failed generations.
C Additional Results
The additional results use a different fully unsupervised evaluation setup to compare CLEAR, DeCLUTR, and Sentence-RoBERTa. The main results omit CLEAR and DeCLUTR because their evaluation procedures differ from the paper’s primary setup.
- Evaluation differences: CLEAR and DeCLUTR are excluded from the main results because their evaluation setup differs from Reimers and Gurevych (2019).The differences concern supervision and how STS12–16 subsets are aggregated.
- Evaluation differences: For SICK and STSb, CLEAR and DeCLUTR use task-specific training sets as supervised training data.
- Evaluation differences: For STS12–16, CLEAR and DeCLUTR average Spearman correlations computed separately across subsets, whereas the main setup concatenates subsets before computing one correlation.
- Evaluation differences: The paper cannot recompute CLEAR and DeCLUTR under its primary evaluation setup because their implementations were not publicly available.
- Additional results: Table 5 reports mean Spearman correlation across subtasks for CLEAR, DeCLUTR, and Sentence-RoBERTa in a fully unsupervised setting.It evaluates Sentence-RoBERTa trained on STS-x1x2 and STS-x2 using the alternative setup.