Source-linked AI summary
Unsupervised Domain Clusters in Pretrained Language Models
Roee Aharoni, Yoav Goldberg
TL;DR
The paper addresses vague domain definitions and unavailable domain labels in NLP. It shows that pretrained language models cluster sentences by domain without supervision, then uses those representations for domain data selection from small in-domain sets, with high clustering accuracy across five domains.
Problem
Domain definitions are often over-simplistic, while linguistic variation and unavailable labels make it difficult to identify suitable in-domain data.
Method
The paper uses pretrained language-model sentence representations for unsupervised domain clustering and selects training sentences by retrieving those most similar to a small in-domain query vector.
Results
Up to 87.66%, 89.04%, and 89.94% accuracy was achieved with k=5, k=10, and k=15 clusters, respectively, across 10,000 sentences in five domains.
Takeaways & Limitations
Pretrained language models support a more data-driven definition of textual domains and provide representations that can be harnessed for downstream domain data selection.
Takeaways & Limitations
The LDA comparison is limited because its models used only the multi-domain data and lacked the additional pretraining used by the other models.
Abstract
from arXiv · showhide
The notion of "in-domain data" in NLP is often over-simplistic and vague, as textual data varies in many nuanced linguistic aspects such as topic, style or level of formality. In addition, domain labels are many times unavailable, making it challenging to build domain-specific systems. We show that massive pre-trained language models implicitly learn sentence representations that cluster by domains without supervision -- suggesting a simple data-driven definition of domains in textual data. We harness this property and propose domain data selection methods based on such models, which require only a small set of in-domain monolingual data. We evaluate our data selection methods for neural machine translation across five diverse domains, where they outperform an established approach as measured by both BLEU and by precision and recall of sentence selection with respect to an oracle.
1 Introduction
The paper argues that textual domains are difficult to define and label, but pretrained language models can reveal domain structure without supervision. It uses these representations for data selection, achieving stronger neural machine translation results than an established baseline across five domains.
- Textual domains vary across topics, styles, formality, and genres, making exact data requirements difficult to specify.
- Domain labels are often unavailable in large-scale web-crawled corpora, complicating domain-specific data construction.
- Pretrained language models learn sentence representations that cluster by domain without additional supervision.The paper quantifies this structure by fitting Gaussian Mixture Models to the learned representations.
- The proposed selection methods retrieve sentences by embedding-space distance or fine-tune the model for binary classification.Both approaches use positive examples from the domain of interest; retrieval ranks sentences by cosine similarity to an in-domain query vector.
- Across five diverse domains, the methods outperform the cross-entropy difference baseline and achieve over 95% recall relative to an oracle.They require only a small set of monolingual in-domain data and do not require additional domain labels.
- The paper contributes an improved data split for the machine-translation benchmark and releases code and data.
2 Emerging Domain Clusters in Pretrained Language Models
The paper tests whether pretrained language models produce unsupervised sentence representations that cluster by domain, using GMMs over sentence embeddings across five domains. It finds high clustering purity, with masked-language models outperforming alternative representations, while overlap and short or linguistically generic sentences create systematic outliers.
- Method: Sentence embeddings are average-pooled from token hidden states and clustered with a Gaussian Mixture Model using a predefined number of clusters.GMMs permit soft assignments, reflecting that a sentence may express a mixture of domains.
- Unsupervised domain clustering: Pretrained language models generate sentence representations that cluster textual data by domain without domain supervision.The study evaluates this hypothesis across masked-language and autoregressive pretrained models.
- Results: 87.66% accuracy at k=5, 89.04% at k=10, and 89.94% at k=15 were achieved across 10,000 sentences from five domains.The evaluation uses subtitles, medical, legal, Koran, and IT-related text from a German-English multidomain corpus.
- Results: Masked-language models outperformed word2vec and autoregressive models, while all vector-based models substantially outperformed random and LDA baselines.The paper attributes this pattern to the broader sentence context available to masked-language models; PCA improved performance in most cases, especially for autoregressive models.
- Error analysis: Embedding-space overlap produces cross-domain assignments, with shorter outlier sentences averaging 11.62 tokens versus 20.5 tokens overall.Examples include subtitle sentences assigned to Koran clusters and topical sentences assigned across legal, medical, and other clusters.
- Error analysis: The IT cluster attracted 597 outliers, including 340 sentences with numbers, while the Subtitles cluster attracted 372, including 228 with pronouns or question marks.These patterns suggest that numerical and conversational cues can dominate assignments across source domains.
3 Neural Machine Translation in a Multi-Domain Scenario
The paper evaluates domain-specific and general-domain NMT models across five domains using a revised split designed to test generalization. Domain proximity in BERT representation space is strongly associated with cross-domain translation quality.
- 3.1 Multi-Domain Dataset: The dataset contains German-English parallel text from five domains: Medical, Law, Koran, IT, and Subtitles.
- 3.1 Multi-Domain Dataset: Training and development/test overlap was found in both prior dataset splits, motivating a new split for evaluating generalization.Overlap can favor memorization and checkpoint selection on overfitting models.
- 3.2 Cross-Domain Experiments: Domain-specific models were evaluated on every domain test set, alongside a general-domain model trained on all available data.
- 3.2 Cross-Domain Experiments: In most cases, in-domain training produced the best results, while using all data helped mainly on the smaller Koran domain.The largest Subtitles corpus was second to last in performance, indicating that more data was not necessarily better.
- 3.2 Cross-Domain Experiments: A Pearson correlation of 0.81 linked greater centroid similarity between domains to higher cross-domain BLEU.Medical, Law, and IT were close in embedding space and had higher cross-domain BLEU than the more distant Koran and Subtitles domains.
4 Domain Data Selection with Pretrained Language Models
The paper uses pretrained language-model representations to select domain-relevant data from a large unlabeled parallel corpus using only a small monolingual in-domain set. Two selection methods are evaluated, with fine-tuning producing the strongest overall results.
- 4 Domain Data Selection with Pretrained Language Models: The selection scenario uses a small in-domain sample to identify relevant sentences in a larger unlabeled parallel corpus.The general-domain corpus contains 1,456,317 sentences, while 2,000 development sentences per domain provide the in-domain sample.
- 4.1 Methods: Domain-Cosine averages in-domain sentence representations into a query vector and ranks training sentences by cosine similarity.
- 4.1 Methods: Domain-Finetune fine-tunes the pretrained language model as a binary classifier using in-domain positives and randomly sampled general-domain negatives.The classifier then identifies in-domain sentences or ranks them by output probability.
- 4.2 Negative Sampling with Pre-ranking: Pre-ranking negative examples improved F1 in every domain by reducing the sampling of unlabeled in-domain sentences as negatives.The authors therefore use pre-ranking in subsequent experiments.
- 4.3 Results: Domain-Finetune performed best among the selection methods, while all selection methods outperformed random selection in BLEU.Averaged across domains, selection methods also surpassed using all available data or oracle-selected data.
- 4.3 Results: Domain-Finetune achieved minimum domain recall of 97.5, compared with 89.4 for Moore-Lewis and 78.8 for Domain-Cosine.These recalls were obtained using only 2,000 in-domain sentences to select from 1.45 million sentences per domain-selection setting.
5 Related Work
Prior work used n-gram, neural, probabilistic, and vector-based methods for domain adaptation and data selection. This paper distinguishes itself by applying massive pretrained language models to unsupervised domain clustering and NMT data selection.
- Data Selection: Earlier data-selection methods relied on n-gram language models or other count-based approaches that cannot generalize beyond observed in-domain n-grams.
- Data Selection: RNN-based language models improved on n-gram methods but do not capture sentence-level global context as recent self-attention-based masked language models do.The paper also reports autoregressive models as inferior to masked language models for domain clustering.
- Domain Adaptation: Related domain-adaptation work induced topics or latent subdomains with LDA and dedicated probabilistic models rather than the paper’s pretrained-language-model representations.
- Vector-Based and Neural Selection: Other approaches used Bayesian optimization, word2vec, paragraph vectors, NMT representations, retrieved sentences, or neural classifiers for data selection.
- Pretrained Language Models: Unlike work that assumes supervision across tasks or domains, this paper requires only a small set of in-domain data and no domain labels.
- Contribution: The paper claims the first use of massive pretrained language models for both unsupervised domain clustering and NMT data selection.
6 Conclusions and Future Work
The paper concludes that pretrained language models can unsupervisedly map text to data-driven domain clusters and support effective data selection with little in-domain data. It reports comparable or better performance than established selection and oracle training across five domains, while identifying several extensions for future work.
- Pretrained language models map textual data to domains in a fully unsupervised manner using average-pooled sentence representations and GMM-based clustering.
- The proposed selection methods use vector-space ranking and pretrained-language-model fine-tuning, requiring only a small set of in-domain data.
- Across all five benchmark domains, the methods perform similarly or better than an established data-selection method and oracle in-domain training.
- The authors suggest that data-driven clusters may better represent natural-language domains than simplistic labels such as “medical text.”
- Future Work: Future work includes multilingual selection, domain-curriculum training, noisy web-crawled data, additional tasks, and unsupervised neural machine translation.
A.1 NMT Training
The NMT experiments use Transformer Base models with shared 32k-merge BPE embeddings, early stopping, ADAM optimization, and beam-search decoding. Training uses a single NVIDIA GPU and standard Moses preprocessing.
- NMT models use Transformer Base configurations implemented in Fairseq, with a joint 32k-merge BPE vocabulary and tied embedding layers.
- Training uses early stopping when the domain-specific development-set BLEU score fails to improve for 10 consecutive checkpoints.
- Models use ADAM with an initial learning rate of 5 · 10^-4 and a maximum batch size of 4096 tokens.
- All models are trained on a single NVIDIA GPU and decoded with beam search using beam size 5.
- Preprocessing uses the Moses pipeline for tokenization, punctuation normalization, nonprinting-character removal, truecasing, and cleaning.
A.2 Data Split
The data-split analysis reports overlap details for the multi-domain dataset, computed using its English portion.
- Table 8 details overlap between training, development, and test sets across the dataset’s different data splits.Overlap is computed using the English part of the corpus.
A.3 GMM Clustering
The clustering models use full-covariance Gaussian mixture models trained until convergence or for at most 150 EM iterations.
- GMMs use full covariance matrices without constraints on component shapes, as implemented in scikit-learn.
- Training continues until convergence or reaches a maximum of 150 EM iterations.
A.4 Language Model Finetuning
The language model finetuning procedure trains a binary classification head with ADAM and a specified Fairseq NMT configuration. The setup uses four NVIDIA GPUs and a 256-sentence batch, while the Moore–Lewis implementation uses KenLM.
- The binary classification head is fine-tuned for 5 epochs using ADAM with an initial learning rate of 2 · 10−5.
- Training uses 4 NVIDIA GPUs with 256 sentences per batch, or 64 sentences per GPU.
- The Moore–Lewis implementation used in the experiments relies on the KenLM N-Gram language model toolkit.
- Subword processing uses the implementation available from the subword-nmt repository.
- The NMT model is trained with Fairseq using the Transformer WMT English–German architecture and the listed optimization, dropout, token, and scheduling settings.