Source-linked AI summary
BLEURT: Learning Robust Metrics for Text Generation
Thibault Sellam, Dipanjan Das, Ankur P. Parikh
TL;DR
Existing NLG metrics can correlate poorly with human judgments, while human evaluation is costly and slow. BLEURT combines BERT-based rating prediction with synthetic pre-training before fine-tuning on human ratings. It achieves state-of-the-art performance across WMT Metrics Shared Task years and is robust under several distribution shifts.
Problem
Existing metrics such as BLEU and ROUGE may correlate poorly with human judgments, while human evaluation is expensive and high-latency.
Method
BLEURT pre-trains a BERT-based learned metric on large synthetic datasets and fine-tunes it on human ratings.
Results
BLEURT yields state-of-the-art results across WMT Metrics Shared Task years 2017–2019, with pre-training consistently improving performance.
Takeaways & Limitations
BLEURT provides a reference-based English NLG metric that models human assessment and is robust to domain and quality drifts.
Takeaways & Limitations
Fine-tuning BERT requires sizable IID data, which is less ideal for generalization across varied tasks and model drift.
Abstract
from arXiv · showhide
Text generation has made significant advances in the last few years. Yet, evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU and ROUGE) may correlate poorly with human judgments. We propose BLEURT, a learned evaluation metric based on BERT that can model human judgments with a few thousand possibly biased training examples. A key aspect of our approach is a novel pre-training scheme that uses millions of synthetic examples to help the model generalize. BLEURT provides state-of-the-art results on the last three years of the WMT Metrics shared task and the WebNLG Competition dataset. In contrast to a vanilla BERT-based approach, it yields superior results even when the training data is scarce and out-of-distribution.
1 Introduction
NLG has advanced rapidly, but existing automatic metrics can correlate poorly with human judgments. BLEURT addresses this gap by combining BERT-based learning, synthetic pre-training, and human-rating fine-tuning for robust evaluation.
- Human evaluation is informative but expensive and slow, motivating cheap automatic proxies for NLG quality.
- BLEU and ROUGE rely on N-gram overlap, limiting their sensitivity to semantic and syntactic variation.
- Learned metrics must handle domain and quality drifts because training ratings may not match future evaluation settings.
- BLEURT pre-trains a fully learned BERT-based metric on synthetic data, then fine-tunes it on human ratings.Its pre-training uses perturbed Wikipedia sentences with lexical and semantic supervision signals.
- BLEURT achieves state-of-the-art results across WMT Metrics Shared Task years 2017–2019 and is tested for quality and domain generalization.Ablations report that synthetic pre-training improves IID performance and supports robustness with scarce, skewed, or out-of-domain training data.
2 Preliminaries
The paper formulates reference-based evaluation as learning a function that predicts how good a candidate sentence is relative to a reference. Human ratings provide the supervised targets.
- The reference sentence x and prediction sentence ˜x form each evaluation input pair.The reference has length r, while the prediction has length p.
- Each training example contains a reference, prediction, and human rating y indicating candidate quality relative to the reference.
- The learning objective is to estimate a function f(x, ˜x) that predicts the human rating for a sentence pair.
3 Fine-Tuning BERT for Quality Evaluation
BLEURT fine-tunes BERT representations of reference–candidate sentence pairs to predict human quality ratings. This straightforward supervised model performs strongly, but requires substantial IID data for broad generalization.
- BERT encodes the reference and candidate sentences into contextualized vectors for quality evaluation.The model receives both sentences and returns a sequence of contextualized representations.
- A linear layer on BERT’s [CLS] representation predicts the sentence-pair rating.
- BERT and the added linear layer are fine-tuned with supervised rating data, typically comprising a few thousand examples.The model uses a regression loss based on squared prediction error.
- State-of-the-art WMT Metrics Shared Task 2017–2019 results show that vanilla BERT fine-tuning can be a high-performing evaluation metric.However, it requires a sizable amount of IID data to generalize across tasks and model drift.
4 Pre-Training on Synthetic Data
BLEURT pre-trains BERT on large-scale synthetic sentence pairs before fine-tuning on human ratings, using varied perturbations and lexical- and semantic-level signals to improve generalization.
- Synthetic-data pre-training: BLEURT warms up BERT with large-scale synthetic reference-candidate pairs before fine-tuning on rating data.The pre-training phase is an addition to BERT’s initial training, not a replacement.
- Synthetic-data construction: 1.8 million Wikipedia segments are randomly perturbed to create scalable synthetic sentence pairs.Perturbations are designed to expose the metric to errors and alterations produced by NLG systems.
- Synthetic-data construction: Mask-filling introduces lexical alterations while maintaining fluency, using random-token and contiguous-span masking strategies.BERT fills randomly inserted masks in Wikipedia sentences.
- Synthetic-data construction: Backtranslation creates semantically preserving variants and realistic alterations from translation-model mispredictions, while word dropping simulates void predictions and truncation.These perturbations broaden the range of sentence differences represented in synthetic training data.
- Pre-training signals: Nine pre-training tasks provide lexical and semantic supervision signals that are cheap to obtain and scalable to synthetic data.Signals include automatic metrics, backtranslation likelihoods, textual entailment, and a backtranslation-generation flag.
- Modeling: Task-level regression and classification losses are combined with a weighted sum to form the aggregate pre-training objective.Regression uses ℓ2 loss, while classification uses multiclass cross-entropy; task weights are selected by grid search.
5 Experiments
Experiments evaluate BLEURT on WMT translation, quality-drift stress tests, WebNLG adaptation, and pre-training ablations. BLEURT achieves strong benchmark performance, while pre-training improves robustness and adaptation, especially with limited or shifted data.
- WMT Metrics Shared Task: BLEURT-based metrics dominate WMT17 and WMT18 for each language pair and lead WMT19 on Kendall’s Tau for all language pairs.
- WMT Metrics Shared Task: Pre-training consistently improves BLEURT and BLEURTbase, adding up to 7.4 Kendall Tau points for BLEURTbase on WMT17 zh-en.The improvement reaches up to 2.1 points on WMT18, while pre-training generally benefits BERT-base more than BERT-large.
- Robustness to Quality Drift: At α = 3.0, the quality-drift benchmark retains only 11.9% of the original 5,344 training records and tests on excellent translations after training on incorrect ones.The skew factor α controls increasingly left-skewed training and right-skewed test ratings.
- Robustness to Quality Drift: Pre-trained BLEURT remains more robust to training skew than unpretrained BLEURT, falling below the baselines only under the extreme α = 3.0 drift.Without pre-training, BLEURT falls below BERTscore at α = 1.0 and below sentBLEU for α ≥1.5.
- WebNLG Experiments: On WebNLG, BLEURT-wmt is competitive with as little as 836 records, while BLEURT matches BERTscore with zero fine-tuning.The vanilla BERT approach requires about one-third of the WebNLG data to dominate baselines on most tasks.
- Ablation Experiments: Ablations show that BERTscore, entailment, and backtranslation pre-training signals improve BLEURT, whereas BLEU and ROUGE signals have a negative impact.The results support using pre-training signals that correlate well with human judgment.
6 Related Work
Learned metrics expanded beyond handcrafted lexical-overlap measures, while the paper leaves open whether BLEU and ROUGE should be removed from future BLEURT versions.
- BLEU and ROUGE removal may slightly improve WMT 2017 performance but could hurt future tasks where they correlate with human assessments.
- Learned metrics use regression or deep learning, while MoverScore combines contextual embeddings with Earth Mover’s Distance.
- Prior noisy pre-training and synthetic perturbation work addressed related tasks, but generally did not combine noisy pre-training with synthetic data.
7 Conclusion
The conclusion presents BLEURT as an English reference-based metric whose end-to-end training models human assessment, with pre-training improving robustness to domain and quality drift.
- BLEURT is a reference-based text generation metric for English trained end-to-end to model human assessment.
- Pre-training makes BLEURT particularly robust to domain and quality drifts.
- Future research directions include multilingual NLG evaluation and hybrid methods involving humans and classifiers.
A.1 Data Generation
The pre-training data-generation pipeline creates synthetic sentence pairs and derives supervision from masking, translation, word dropping, automatic metrics, and translation likelihoods.
- Random Masking: Random masking replaces sampled individual words or contiguous spans with masks, using up to 15 masks per sentence.
- Backtranslation: Backtranslation uses forward and backward translation models to generate synthetic sentence variants.
- Word dropping: Word dropping randomly removes words from synthetic candidates, applying the transformation to about 30% of previously generated data.
- Automatic Metrics: Automatic supervision uses BLEU, ROUGE, and BERTscore signals, including precision, recall, and F-score for ROUGE and BERTscore.
- Backtranslation Likelihood: Backtranslation likelihood losses come from custom Transformer models trained on English-French and English-German language pairs.
- Task weights: Pre-training task weights are selected by grid search on the WMT 17 validation set, with related tasks grouped under shared weights.
B.1 Training Setup for All Experiments
The experiments use BERT checkpoints with fixed optimization settings, evaluate checkpoints during training, and retain the best validation-performing checkpoint.
- Training uses BERT public checkpoints with Adam, learning rate 1e-5, and batch size 32.
- Unless specified otherwise, pre-training runs for 800,000 steps and fine-tuning for 40,000 steps.
- Evaluation runs every 1,500 steps, and the checkpoint performing best on a held-out validation set is stored.
B.2 WMT Metric Shared Task
The WMT Metrics Shared Task uses different correlation measures across years, with robustness adjustments for noisy judgments and comparisons against established metrics.
- 2017 uses Pearson correlation on standardized human judgments, while 2018 and 2019 use DARR, a custom Kendall’s Tau variant on raw judgments.DARR compares translation pairs sharing a reference segment and filters pairs according to the organizers’ procedure.
- The 25 points filter is intended to improve robustness to noisy WMT 2018 and 2019 judgments, whereas ordinary Kendall’s Tau omits the filter.
- 0 pre-training steps yield Kendall Tau values of 0.555 for BLEURTbase and 0.580 for BLEURT on WMT Metrics Shared Task 2017.
- Models are selected by highest validation Kendall Tau after leak-free train–validation separation, using 10% validation data in 2017–2018 and 5% in 2019.
- The comparison baselines are sentenceBLEU, BERTScore, and MoverScore.Reference and candidate segments are Moses-tokenized before sentenceBLEU computation.
B.3 Robustness to Quality Drift
The robustness experiment induces quality drift by reweighting records across ten quality bins, with stronger drift reducing the available dataset size.
- Training and test records are sampled separately across 10 equal-sized bins, using complementary probabilities controlled by skew factor α.α = 0 produces no drift, while α = 3.0 produces extreme differences between training and test ratings.
- Most downstream accuracy gains occur within the first 400,000 pre-training steps, corresponding to about two epochs over the synthetic dataset.The analysis fine-tunes BLEURT variants on WMT17 data while varying pre-training duration.