PaperlayerView exact revision on arXiv

Source-linked AI summary

BERTScore: Evaluating Text Generation with BERT

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi

arXiv:1904.09675v3cs.CL

TL;DR

Common surface-overlap metrics do not account for meaning-preserving variation in generated text. BERTScore instead compares contextual token embeddings, achieving stronger human-judgment correlations across translation and captioning, better model selection than BLEU, and greater robustness to adversarial paraphrases.

  • Problem

    Text-generation evaluation aims to measure semantic equivalence, but common surface-overlap metrics do not account for meaning-preserving lexical and compositional diversity.

    Source: §1 · p. 1

  • Method

    BERTScore evaluates generated text through contextual-embedding similarity between candidate and reference tokens, using token matching to compute precision, recall, and F1.

    Source: §1 · p. 1, §3 · p. 4

  • Results

    BERTScore achieves stronger human-judgment correlations across translation and captioning, better model selection than BLEU, and greater robustness to adversarial paraphrases than other metrics.

    Source: §1 · pp. 1–2

  • Takeaways & Limitations

    BERTScore is particularly suitable for analyzing specific examples in machine translation.

    Source: §5 · p. 7

    WMT16 segment-level experiments cover only translation into English because of dataset errors.

    Source: §4 · p. 6

Abstract

from arXiv · show

We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.

1 INTRODUCTION

BERTScore evaluates generated text using cosine similarities between contextual token embeddings rather than surface overlap. It shows stronger agreement with human judgments and greater robustness to adversarial paraphrases than comparison metrics.

1 INTRODUCTION

  • Surface-form metrics miss meaning-preserving lexical and compositional variation, while n-gram matching can mishandle paraphrases, distant dependencies, and semantically critical ordering.Source: §1 · p. 1
  • BERTScore sums cosine similarities between token embeddings from pretrained BERT (Devlin et al., 2019) to compare candidate and reference sentences.Contextualized embeddings replace string matching or matching heuristics and are trained to capture distant dependencies and ordering.Source: §1 · p. 1
  • BERTScore achieves stronger machine-translation system- and segment-level human correlations than existing metrics and surpasses SPICE (Anderson et al., 2016) in image-captioning human agreement.Experiments cover outputs from 363 systems across both tasks. BERTScore also demonstrates strong model selection performance compared with BLEU.Source: §1 · pp. 1–2
  • BERTScore is more robust than other metrics to adversarial examples on the PAWS paraphrase dataset (Zhang et al., 2019).Source: §1 · p. 2

2 PROBLEM STATEMENT AND PRIOR METRICS

Reference-based generation metrics are judged by their correlation with human judgments and fall into four broad families: n-gram matching, edit distance, embedding matching, and learned functions. BERTScore uses contextual token embeddings without external linguistic structures or optimization for a specific evaluation task.

2 PROBLEM STATEMENT AND PRIOR METRICS

  • Reference-based evaluation assigns a real-valued score to a reference–candidate pair, with better metrics correlating more strongly with human judgments.The section organizes existing metrics into n-gram matching, edit-distance-based, embedding-based, and learned approaches.Source: §2 · p. 2

2.1 n-GRAM MATCHING APPROACHES

  • N-gram matching captures more word order as n increases, but becomes increasingly constrained to the reference’s exact wording.BLEU combines clipped corpus-level n-gram precision with a brevity penalty and geometric averaging; BERTScore instead uses contextual embeddings with potentially unbounded dependencies. Related approaches vary matching units, weighting, and precision versus recall, including character-based CHRF (Popovi´c, 2015), recall-based ROUGE-n, and tf–idf-weighted CIDER, whose weighting approach informs BERTScore.Source: §2.1 · pp. 2–3
  • 104 languages have BERT embeddings, whereas resource-dependent METEOR supports its full feature set in five languages and partial features in eleven.METEOR relaxes unigram matching using stems, synonyms, and paraphrases supplied by external resources. BERTScore instead uses BERT embeddings for relaxed matching and simple corpus statistics for importance weighting.Source: §2.1 · p. 2

2.2 EDIT-DISTANCE-BASED METRICS

  • Edit-distance-based metrics measure reference–candidate similarity through required edit operations, with variants addressing normalization, stemming, word order, and character-level matching.TER normalizes by reference length, while CDER treats block reordering as an edit. Character-level CHARACTER and EED achieve higher human-judgment correlations on some languages.Source: §2.2 · p. 3

2.3 EMBEDDING-BASED METRICS

  • BERTScore uses contextual token embeddings and greedy matching, avoiding external linguistic structures while allowing importance weighting at the token level.Contextual embeddings capture a token’s specific sentence usage and potentially sequence information. Alternative embedding metrics use optimal matching based on earth mover’s distance or sentence-level representations.Source: §2.3 · p. 3

2.4 LEARNED METRICS

  • Supervised evaluation metrics require costly human judgments and risk poor domain generalization, whereas BERTScore’s underlying model is not optimized for a specific evaluation task.BEER (Stanojevi´c & Sima’an, 2014) regresses on character n-grams and word bigrams; BLEND combines 29 metrics; RUSE combines three pretrained sentence-embedding models. Human-generation classifiers also risk fitting existing data and generalizing poorly, as discussed for Cui et al. (2018) and Lowe et al. (2017).Source: §2.4 · p. 3

3 BERTSCORE

BERTScore uses contextual embeddings and greedy token matching to compute precision, recall, and F1 from candidate–reference cosine similarities. Optional inverse-document-frequency weighting incorporates token importance, while baseline rescaling improves score readability without changing rankings or human correlations.

Token Representation

  • Contextual embeddings represent tokens according to surrounding words, allowing the same word to receive different representations in different sentences.Each embedding model uses its own tokenizer. The main model, BERT, uses word pieces (Wu et al., 2016) represented through a Transformer encoder.Source: §3 · pp. 3–4

Similarity Measure

  • Token similarity uses cosine similarity, reduced to an inner product through vector normalization, while contextual embeddings retain information from the surrounding sentence.Similarity compares individual token vectors, but those vectors encode sentence context.Source: §3 · p. 4

BERTSCORE

  • BERTScore greedily matches each token to its most similar counterpart, computing reference-to-candidate recall, candidate-to-reference precision, and their combined F1.Each token independently selects the most similar token in the other sentence.Source: §3 · p. 4

Importance Weighting

  • Optional inverse document frequency weighting uses test-set reference sentences to incorporate token importance, with identical weights across systems evaluated on that test set.Plus-one smoothing handles unknown word pieces.Source: §3 · p. 4

Baseline Rescaling

  • 1M random candidate–reference pairs per language and embedding model estimate the baseline used to linearly rescale BERTScore for readability.The baseline averages BERTScore over randomly paired Common Crawl sentences; original scores theoretically range from −1 to 1 but occupy a narrower empirical range. Rescaled precision, recall, and F1 are typically between 0 and 1, without changing rankings or human correlations.Source: §3 · p. 5

4 EXPERIMENTAL SETUP

The experimental setup evaluates BERTScore on machine translation and image captioning through agreement with human judgments and translation model-selection tests. It also specifies language-dependent contextual encoders and validation-based layer selection.

Contextual Embedding Models

  • Twelve pretrained contextual embedding models are evaluated, with each model’s embedding layer selected using WMT16 validation data.The selected encoders are 24-layer RoBERTalarge for English, 12-layer BERTchinese for Chinese, and 12-layer cased multilingual BERTmulti for other languages. The evaluated families include BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), XLNet, and XLM (Lample & Conneau, 2019).Source: §4 · p. 5

Machine Translation

  • 149 translation systems across 14 language pairs form the main WMT18 evaluation corpus, with segment-level and system-level human judgments.Metric quality uses absolute Pearson correlation |ρ| and Kendall rank correlation τ, with Williams tests and bootstrap resampling following Graham & Baldwin (2014), respectively. System scores average reference-candidate BERTScores; hybrid systems sample candidates from available systems and average corresponding human judgments, enabling larger system-level comparisons against canonical metrics.Source: §4 · p. 5
  • 100K repeated trials rank 100 randomly selected systems from 10K WMT18 hybrid systems, measuring agreement with human selection through Hits@1.Hits@1 is the percentage of trials in which metric and human rankings select the same best system. Additional measures are the selected system’s reciprocal human rank and its human-score gap from the human-preferred system.Source: §4 · pp. 5–6
  • 202 additional systems extend evaluation to WMT17 and WMT16, with results reported in Appendix F.Dataset errors restrict WMT16 experiments to segment-level evaluation on to-English language pairs.Source: §4 · p. 6

Image Captioning

  • Twelve COCO 2015 Captioning Challenge entries support image-captioning evaluation using Pearson correlations with human assessments.M1 measures captions judged better than or equal to human captions, while M2 measures captions indistinguishable from human captions; BERTScore returns the highest score across available references. Comparisons include eight task-agnostic metrics and two task-specific metrics: SPICE (Anderson et al., 2016), using parsed scene-graph similarity, and LEIC (Cui et al., 2018), predicting human authorship given the image.Source: §4 · p. 6

5 RESULTS

BERTSCORE is consistently a top performer in machine-translation evaluation and substantially outperforms task-agnostic baselines in image captioning.

Machine Translation

  • BERTSCORE consistently performs among the best for machine-translation system-level correlation, hybrid-system evaluation, and model selection, while achieving significantly higher segment-level correlations than other metrics.Segment-level improvements are especially large over BLEU, and BERTSCORE significantly outperforms RUSE. RUSE is competitive for to-English translation but requires additional data and training when pretrained models are unavailable, including from-English experiments.Source: §5 · p. 7
  • F1 FBERT is recommended because it performs reliably across machine-translation settings, while idf weighting offers inconsistent benefits and is omitted from subsequent experiments.Recall RBERT, precision PBERT, and F1 FBERT alternate as the best measure across settings; understanding when idf helps remains an open question.Source: §5 · p. 7

Image Captioning

  • BERTSCORE outperforms all task-agnostic baselines on COCO captioning, while image-informed, COCO-optimized LEIC (Cui et al., 2018) performs best overall.idf weighting significantly improves captioning performance, suggesting greater human emphasis on content words. Strict n-gram metrics, including BLEU and ROUGE, correlate weakly with human judgments.Source: §5 · pp. 7–8, Table 5 · p. 8

Speed

  • 192.5 candidate-reference pairs/second are processed on a GTX-1080Ti GPU, making BERTSCORE relatively fast despite its large pretrained model.The 2,998-sentence WMT18 en-de test set takes BERTSCORE 15.6 seconds versus 5.4 seconds with SacreBLEU. The authors consider this overhead relatively marginal for validation and testing.Source: §5 · p. 8

6 ROBUSTNESS ANALYSIS

BERTScore shows greater robustness to adversarial paraphrases than other automatic metrics: its performance drops only slightly, while most metrics fall nearly to chance.

6 ROBUSTNESS ANALYSIS

  • Adversarial testing uses QQP and word-swapped PAWSQQP examples from PAWS (Zhang et al., 2019).QQP contrasts real duplicate questions with related but different questions. PAWS word swapping creates challenging pairs, such as reversing a flight’s origin and destination.Source: §6 · p. 8
  • 667 PAWSQQP development sentences and the first 5,000 QQP training sentences support evaluation without paraphrase-detection training data for automatic metrics.QQP training sentences are used because test labels are unavailable. The first sentence is the reference and the second the candidate, with higher scores interpreted as stronger evidence of paraphrasing.Source: §6 · p. 8
  • Supervised classifiers trained on QQP perform worse than random guessing on PAWSQQP, tending to classify adversarial examples as paraphrases.With adversarial training examples, DIIN (Gong et al., 2018) and fine-tuned BERT identify adversarial examples, but their performance still decreases significantly relative to QQP.Source: §6 · pp. 8–9
  • BERTScore’s performance drops only slightly on PAWSQQP, whereas most automatic metrics fall nearly to chance despite decent performance on QQP.Performance is evaluated using area under the ROC curve (AUC).Source: §6 · pp. 8–9

7 DISCUSSION

BERTSCORE achieves better correlation than common metrics and supports effective model selection, but configuration choices should reflect the domain and languages. Future directions include task-specific metrics built on BERTSCORE and training losses that exploit its differentiability.

7 DISCUSSION

  • BERTSCORE achieves better correlation than common metrics, supports effective model selection, and addresses limitations exposed by challenging adversarial examples.The metric evaluates generated text against gold-standard references and is designed to be simple, task agnostic, and easy to use.Source: §7 · p. 9
  • No BERTSCORE configuration clearly outperforms all others, making domain and language important selection considerations despite often small differences between top configurations.The authors recommend FBERT for machine translation evaluation and 24-layer RoBERTalarge for English text generation. Multilingual BERTmulti is suitable for non-English languages, but its performance is less stable on low-resource languages.Source: §7 · p. 9
  • Integrating improvements from Zhao et al. (2019) makes BERTSCORE equivalent to or better than their concurrently developed earth mover’s distance approach.Their approach combines contextual embeddings with earth mover’s distance rather than BERTSCORE’s simple matching process. The improvements have a more modest effect on BERTSCORE than on their method.Source: §7 · p. 9
  • YiSi-1 improved after adopting BERT embeddings, further corroborating the authors’ findings.Related approaches train a model atop BERT to maximize correlation with human judgments or evaluate generation using BERT fine-tuned on paraphrasing.Source: §7 · p. 9
  • Recent work explores BERTSCORE for summarization training (Li et al., 2019) and SciBERT-based domain-specific evaluation of abstractive summarization (Beltagy et al., 2019).SciBERT is attributed to Beltagy et al. (2019), and its application to abstractive summarization evaluation to Gabriel et al. (2019).Source: §7 · p. 9
  • BERTSCORE could underpin task-specific metrics and, because it is fully differentiable, provide training losses that reduce the mismatch between optimization and evaluation objectives.The proposed task-specific metrics would use BERTSCORE as a subroutine while accommodating individual tasks’ needs.Source: §7 · p. 9

A QUALITATIVE ANALYSIS · B REPRESENTATION CHOICE

Qualitative examples show that BERTSCORE handles synonyms and word-order changes but struggles with factual errors and equivalent measurements. Representation comparisons indicate that intermediate layers generally correlate better with human judgments than final layers.

A QUALITATIVE ANALYSIS

  • 560 candidate-reference pairs from WMT16 German-to-English are ranked separately by human scores, BERTSCORE, and SENTBLEU to examine ranking agreement.The analysis uses WMT16 German-to-English data.Source: A QUALITATIVE ANALYSIS · p. 15
  • BERTSCORE handles synonyms and word-order changes better than SENTBLEU in qualitative examples and avoids rewarding misleading word overlap.SENTBLEU ranks some meaning-preserving translations too low and a semantically different pair with substantial phrase overlap too high.Source: A QUALITATIVE ANALYSIS · p. 15
  • BERTSCORE struggles to detect factual substitutions and recognize equivalent measurements; SENTBLEU shares these limitations.Examples include substituting English for German language, Tuesday for Monday, and failing to recognize that 5ft 11in is equivalent to 1.80 metres.Source: A QUALITATIVE ANALYSIS · p. 16
  • The matching visualization illustrates BERTSCORE aligning synonymous phrases and words despite substantial ordering differences.Examples include matching imported cars with foreign cars and aligning people despite its changed position; word-match cosine similarities are color-coded.Source: A QUALITATIVE ANALYSIS · p. 16, Figure 2 · p. 16

B REPRESENTATION CHOICE

  • Layer selection uses WMT16 segment-level human judgments as development data, with WMT17 English-Chinese data used for Chinese models.English-Chinese is unavailable in WMT16; the authors tune layer choices across publicly available models.Source: B REPRESENTATION CHOICE · p. 17
  • Intermediate representations generally yield better FBERT correlations with human judgments, while correlations drop significantly in final layers.The layer comparisons cover BERT, RoBERTa, XLNet, and XLM models.Source: B REPRESENTATION CHOICE · p. 17, Figure 3 · p. 18

C ABLATION STUDY OF MOVERSCORE

The ablation finds no consistent advantage for WMD over BERTSCORE’s greedy matching, while confirming benefits from MNLI fine-tuning and power-mean layer aggregation. Token removal and separate candidate idf weighting contribute less reliably.

C ABLATION STUDY OF MOVERSCORE

  • The study compares MOVERSCORE with BERTSCORE using 12-layer uncased BERT on WMT17 to-English segment-level data.It evaluates unigram WMD1 and bigram WMD2 using Pearson correlations. Reported performance was replicated except for fi-en and lv-en, whose implementations Zhao et al. (2019) had not released at publication.Source: C ABLATION STUDY OF MOVERSCORE · p. 19, Table 9 · p. 20
  • 3,005 system-level reference sentences replace 560 segment-level sentences when computing reference-token idf weights in [IDF-L], rather than [IDF-S].The vanilla setting corresponds to F_BERT, while its importance-weighted variant corresponds to [IDF-S].Source: C ABLATION STUDY OF MOVERSCORE · p. 19
  • MNLI fine-tuning and power-mean layer aggregation are effective improvements.[MNLI] uses a BERT model fine-tuned on MNLI, while [PMEANS] aggregates information across different layers.Source: C ABLATION STUDY OF MOVERSCORE · p. 19
  • Candidate-specific idf recomputation [SEP] and excluding punctuation and noninitial subword tokens [RM] make less stable contributions.[SEP] retains reference-token weights from [IDF-S]. Complete MOVERSCORE combines [IDF-S], [SEP], [PMEANS], [MNLI], and [RM].Source: C ABLATION STUDY OF MOVERSCORE · p. 19
  • Replacing greedy matching with WMD does not consistently improve performance; BERTSCORE often performs better under the same setup.BERTSCORE remains among the best-performing metrics for every language pair in the comparison.Source: C ABLATION STUDY OF MOVERSCORE · p. 19, Table 9 · p. 20

D ADDITIONAL EXPERIMENTS ON ABSTRACTIVE TEXT COMPRESSION · E BERTSCORE OF RECENT MT MODELS

In abstractive text compression, RBERT correlates most strongly with human meaning judgments, PBERT correlates highly with grammar judgments, and FBERT balances both. For recent machine translation models trained on different data, BERTSCORE shows a larger score gap than BLEU.

D ADDITIONAL EXPERIMENTS ON ABSTRACTIVE TEXT COMPRESSION

  • RBERT has the highest correlation with human meaning judgments, PBERT correlates highly with grammar judgments, and FBERT balances these aspects in abstractive compression.The supplied prose reports these findings from Table 10 without providing numerical correlations.Source: D ADDITIONAL EXPERIMENTS ON… · p. 21
  • Compression evaluation follows Toutanova et al. (2016), measuring Pearson correlations against meaning, grammar, and combined human scores from the MSR Abstractive Text Compression Dataset.Meaning measures preservation of the original text’s meaning; grammar measures grammatical correctness. The combined score averages meaning and grammar.Source: D ADDITIONAL EXPERIMENTS ON… · p. 21

E BERTSCORE OF RECENT MT MODELS

F ADDITIONAL RESULTS … F.3 IMAGE CAPTIONING ON COCO

Additional translation and image-captioning experiments show that BERTSCORE correlates effectively with human judgments and demonstrates better model-selection performance. The results also motivate selecting contextual embeddings suited to the target language and domain.

F ADDITIONAL RESULTS

F.1 WMT CORRELATION STUDY

  • 50 systems across 6 language pairs supply WMT16’s to-English segment-level evaluation and hyperparameter-validation data.Missing released data restricts the evaluation to this subset. Pearson correlations include participating metrics, but significance tests compare only BERTSCORE variants because original baseline-metric outputs are unavailable.Source: §F.1 · p. 23
  • 152 translations in WMT17 and 159 translation systems in WMT18 cover 14 language pairs each.WMT17 evaluates segment- and system-level correlations in both translation directions, excludes erroneous fi-en segment data, and applies standard significance testing against participating metrics. WMT18 extends the main results with all participating metrics and alternative contextual models.Source: §F.1 · p. 23
  • BERTSCORE’s improvements in human-judgment correlation are more pronounced at segment level than system level.More optimized or larger BERT models provide better contextual representations, whereas smaller XLNet outperforms larger XLNet. The authors hypothesize that permutation-language-model pretraining need not produce informative embeddings for generation evaluation.Source: §F.1 · pp. 23–24
  • 24-layer RoBERTa fine-tuned on MNLI is recommended for English evaluation; multilingual BERT and XLM trained on 100 languages are suitable non-English candidates.Fine-tuning on related tasks can improve correlation with human judgments. Domain- or language-specific embeddings, such as Chinese BERT for Chinese tasks, are recommended when available.Source: §F.1 · p. 24

F.2 MODEL SELECTION STUDY

  • BERTSCORE demonstrates better model-selection performance, while the supervised metric RUSE also shows strong selection ability.Experiments repeatedly rank 100 randomly selected systems from 10K WMT18 hybrid systems, repeating the procedure 100K times. Evaluation uses Hits@1, mean reciprocal rank of the metric-selected system in human rankings, and absolute human-score difference from the human-selected system.Source: §F.2 · p. 24

F.3 IMAGE CAPTIONING ON COCO

  • RBERT surpasses task-specific SPICE Anderson et al. (2016) in human-judgment correlation, while precision-based BLEU and PBERT correlate weakly on image captioning.The authors hypothesize that human judges prefer captions capturing the picture’s main objects. Fine-tuned RoBERTa-Large remains among the best metrics but does not achieve the highest correlation.Source: §F.3 · p. 24

F.4 ROBUSTNESS ANALYSIS ON PAWS-QQP

BERTScore is generally more robust than commonly used metrics, with the 24-layer RoBERTa version performing best. Fine-tuning RoBERTa-Large on MNLI improves robustness, whereas fine-tuning BERT on MRPC worsens it.

F.4 ROBUSTNESS ANALYSIS ON PAWS-QQP

  • Fine-tuning RoBERTa-Large on MNLI can significantly improve robustness against adversarial sentences.Source: §F.4 · p. 24
  • BERT fine-tuned on MRPC (Microsoft Research Paraphrasing Corpus) performs worse than its counterpart in the robustness study.Source: §F.4 · p. 24
  • BERTScore is generally more robust than commonly used metrics, with the 24-layer RoBERTa version performing best.The study evaluates area under the ROC curve (AUC) on QQP and PAWS-QQP.Source: §F.4 · p. 24, Table 30 · p. 43

Related papers

Loading 1904.09675v3…