PaperlayerView exact revision on arXiv

Source-linked AI summary

BERTScore: Evaluating Text Generation with BERT

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, Yoav Artzi

arXiv:1904.09675v3cs.CL

TL;DR

Existing text-generation metrics often miss semantic equivalence because they rely on surface-form overlap. BERTScore instead compares contextual token embeddings and correlates better with human judgments across translation and captioning while improving robustness to adversarial paraphrases.

  • Problem

    Common text-generation metrics rely on surface-form similarity, limiting their ability to capture meaning-preserving lexical and compositional diversity.

  • Method

    BERTScore evaluates candidate-reference similarity by matching tokens through contextual embeddings and combining precision and recall into an F1 score.

  • Results

    Across 363 machine-translation and image-captioning systems, BERTScore correlates better with human judgments, supports stronger model selection, and is more robust to adversarial examples than existing metrics.

  • Takeaways & Limitations

    BERTScore provides a task-agnostic metric that is effective for evaluating generated text and selecting models, including on challenging paraphrase examples.

  • Takeaways & Limitations

    WMT16 segment-level experiments were limited to to-English language pairs because of dataset errors.

Abstract

from arXiv · show

We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.

1 INTRODUCTION

BERTScore is introduced as a contextual-embedding metric that evaluates semantic similarity between candidate and reference sentences. Across machine translation, image captioning, and adversarial paraphrase detection, it correlates strongly with human judgments and improves robustness and model selection over existing metrics.

  • Metric: BERTScore computes sentence similarity by summing cosine similarities between tokens’ pretrained BERT contextual embeddings.It is designed to move beyond the surface-form similarity used by common n-gram metrics.
  • Motivation: BERTScore addresses n-gram metrics’ failure to robustly match paraphrases, which can penalize semantically correct phrases that differ in surface form.The introduction illustrates this problem with contrasting BLEU and METEOR scores for paraphrased sentences.
  • Experiments: 363 systems were evaluated across machine translation and image captioning by correlating BERTScore and related metrics with available human judgments.The experiments cover outputs from both task families.
  • Results: In machine translation, BERTScore shows stronger system-level and segment-level correlations with human judgments than existing metrics on multiple common benchmarks.The introduction also reports strong correlation with human evaluations overall.
  • Results: BERTScore achieves stronger model selection performance than BLEU, surpasses SPICE for image captioning, and is more robust to adversarial examples than other metrics on PAWS.PAWS is an adversarial paraphrase dataset used to test robustness.

2 PROBLEM STATEMENT AND PRIOR METRICS

Text generation metrics compare candidate and reference sentences, with better metrics showing higher correlation with human judgments. Existing approaches rely on n-gram or edit matching, embeddings, or learned functions, each with distinct trade-offs.

  • Evaluation setup: Generation metrics compute a score between tokenized reference and candidate sentences, and better metrics correlate more strongly with human judgments.Metrics are represented as functions f(x, ˆx) ∈ R.
  • N-gram matching: N-gram metrics capture more word order at higher n but become increasingly restrictive because they require the reference’s exact form.Common metrics count n-grams occurring in both the reference and candidate.
  • Existing metrics: BLEU modifies exact n-gram precision with clipped matches, corpus-level aggregation, and a brevity penalty, typically across multiple n values.METEOR instead backs off from exact unigram matching to stems, synonyms, and paraphrases using external resources.
  • Learned metrics: Learned metrics optimize correlation with human judgments but require costly dataset-specific supervision and may generalize poorly to new domains.Examples include BEER, BLEND, and RUSE, which use regression or combinations of pretrained representations and existing metrics.

3 BERTSCORE

BERTScore represents tokens with contextual embeddings and matches reference and candidate tokens by cosine similarity rather than exact-string matching. It combines greedy token matching into precision, recall, and F1 scores, with optional inverse-document-frequency weighting and baseline rescaling for readability.

  • Token Representation: Contextual embeddings assign different vectors to the same word across sentences based on surrounding context.The method uses contextual representations such as BERT and ELMo, rather than prior static word embeddings.
  • Similarity Measure: Cosine similarity provides a soft token-matching measure that uses contextual information instead of exact-string or heuristic matching.With pre-normalized vectors, cosine similarity reduces to an inner product.
  • BERTSCORE: BERTScore greedily matches each reference token to the most similar candidate token for recall and reverses the direction for precision, then combines them into F1.Greedy matching maximizes matching similarity by selecting the most similar token in the other sentence.
  • Importance Weighting: Inverse document frequency weighting can emphasize rare words and is computed from the reference sentences of each test corpus.The scores remain fixed across systems evaluated on a specific test set, with plus-one smoothing for unknown word pieces.
  • Baseline Rescaling: Baseline rescaling typically maps BERTScore values to between 0 and 1 without affecting system rankings or human correlation.The rescaling is intended solely to make scores more readable.

4 EXPERIMENTAL SETUP

The experiments evaluate the approach on machine translation and image captioning using contextual embedding models, human-judgment datasets, and model-selection studies.

  • Contextual Embedding Models: The study evaluates twelve pre-trained contextual embedding models, using RoBERTalarge for English, BERTchinese for Chinese, and multilingual BERTmulti for other languages.The models include variants of BERT, RoBERTa, XLNet, and XLM.
  • Machine Translation: Machine translation evaluation uses WMT18, covering 149 systems across 14 language pairs with segment-level and system-level human judgments.The study also reports results on WMT17 and WMT16 in Appendix F, adding 202 systems; WMT16 is limited to segment-level to-English experiments.
  • Model Selection: Model selection samples 100 systems from 10K hybrid systems 100K times and measures Hits@1, mean reciprocal rank, and the human-score gap between top human- and metric-rated systems.The 10K hybrid systems are super-sampled from WMT18 and ranked using automatic metrics.
  • Image Captioning: Image captioning evaluation uses human judgments from twelve COCO 2015 Captioning Challenge submissions, with approximately five reference captions per validation-set image.The study computes Pearson correlations with two system-level metrics, including M1, the percentage of captions judged better than or equal to human captions.

5 RESULTS

BERTScore is consistently a top performer for machine translation and outperforms task-agnostic baselines by large margins for image captioning. Despite using a large pre-trained model, it remains relatively fast to compute.

  • Machine Translation: BERTScore is consistently a top performer across machine-translation system correlation, hybrid-system correlation, and model-selection evaluations.RUSE is competitive in to-English results but requires supervised training data and unavailable models cannot be used in some from-English experiments.
  • Image Captioning: BERTScore outperforms all task-agnostic baselines by large margins on the COCO Captioning Challenge.Metrics based on strict n-gram matching, including BLEU and ROUGE, show weak correlations with human judgments.
  • Image Captioning: LEIC outperforms all other methods on COCO captioning.LEIC is trained specifically for the COCO data and system set and uses images as additional inputs.
  • Speed: 192.5 candidate-reference pairs/second are processed by BERTScore on a GTX-1080Ti GPU.The complete WMT18 en-de test set of 2,998 sentences takes 15.6sec, compared with 5.4sec using SacreBLEU.

6 ROBUSTNESS ANALYSIS

The robustness analysis evaluates BERTScore on adversarial paraphrase classification using QQP and PAWS, comparing automatic metrics and supervised models. Supervised QQP classifiers and most metrics degrade substantially on PAWSQQP, indicating difficulty distinguishing adversarial examples.

  • Adversarial paraphrase classification: BERTScore robustness is tested on paraphrase classification using QQP and adversarial paraphrases from PAWS.Both datasets contain sentence pairs labeled as paraphrases or non-paraphrases; QQP positives are duplicate questions, while negatives are related but different questions.
  • Adversarial paraphrase classification: QQP-trained supervised classifiers perform worse than random guessing on PAWSQQP, predicting adversarial examples are more likely to be paraphrases.Table 6 reports area under the ROC curve (AUC) for existing models and automatic metrics.
  • Adversarial paraphrase classification: Most metrics perform decently on QQP but drop significantly on PAWSQQP, nearly reaching chance performance.The result suggests these metrics fail to distinguish harder adversarial examples.
  • Adversarial paraphrase classification: DIIN and fine-tuned BERT identify adversarial examples when they are included during training, but their performance still decreases significantly from QQP to PAWSQQP.These models are described as state-of-the-art in the passage.

7 DISCUSSION

The discussion presents BERTSCORE as a simple, task-agnostic metric that addresses limitations of common metrics, particularly on adversarial examples. It also outlines future extensions using task-specific metrics and differentiable training losses.

  • Contributions: BERTSCORE is designed as a simple, task-agnostic, and easy-to-use metric for evaluating generated text against gold-standard references.The authors conduct extensive experiments with configuration choices, including the contextual embedding model and importance weighting.
  • Contributions: BERTSCORE addresses limitations of commonly used metrics, especially on challenging adversarial examples.
  • Future work: Future work will develop task-specific metrics that use BERTSCORE as a subroutine and incorporate its differentiable score into training losses.This aims to reduce the mismatch between optimization and evaluation objectives.

A QUALITATIVE ANALYSIS · B REPRESENTATION CHOICE

BERTScore’s contextual matching captures synonyms and word-order variation, but can miss factual errors and semantic equivalences. Representation choice uses held-out human-judgment data, with correlations declining in final BERT layers.

  • A QUALITATIVE ANALYSIS: BERTScore and SentBLEU are compared by ranking 560 WMT16 German-to-English candidate-reference pairs against human similarity rankings.The ideal ranking from each metric would resemble the human-score ranking.
  • A QUALITATIVE ANALYSIS: BERTScore captures synonyms and word-order changes that cause large ranking disagreements with SentBLEU.One example replaces “opinion in Hungary” with “Hungarian view” and changes quotation-mark ordering.
  • A QUALITATIVE ANALYSIS: BERTScore struggles to detect factual errors, assigning high similarity when translations replace German with English or Monday with Tuesday.It also fails to recognize that 5ft 11in is equivalent to 1.80 metres.
  • A QUALITATIVE ANALYSIS: BERTScore matches synonymous phrases such as imported cars and foreign cars, even under substantial ordering distortion.Figure 2 also illustrates matching the token people despite high word-order distortion.
  • B REPRESENTATION CHOICE: Selecting BERT’s representation layer or layer combination is important, so BERTScore tunes this choice using WMT16 segment-level human judgments.Chinese models instead use WMT17 English-Chinese data because that language pair is unavailable in WMT16.
  • B REPRESENTATION CHOICE: Recommended BERTScore representation layers are selected using a held-out WMT16 validation set.The table reports the recommended layer for representation in BERTScore.
  • B REPRESENTATION CHOICE: Correlation with segment-level human judgments drops significantly in BERT’s final layers across the evaluated models.Layer 0 corresponds to BPE embeddings, and WMT17 English-Chinese data is used for the Chinese BERT model.

C ABLATION STUDY OF MOVERSCORE

The ablation study isolates differences between MOVERSCORE and BERTSCORE, including model choice, layer aggregation, IDF weighting, and token matching. Results compare WMD variants and BERTSCORE settings, finding that WMD does not consistently improve BERTSCORE.

  • Ablation factors: The study isolates five MOVERSCORE differences: MNLI fine-tuning, power-mean layer aggregation, alternative IDF computation, candidate-specific IDF recomputation, and restricted token matching.The restricted matching excludes punctuation and most sub-word tokens.
  • Experimental setup: The experiments use Zhao et al.’s setup and released fine-tuned BERT model, comparing WMD1, WMD2, vanilla FBERT, importance-weighted FBERT, and complete MOVERSCORE.The complete MOVERSCORE configuration combines [IDF-S], [SEP], [PMEANS], [MNLI], and [RM].
  • Results: Table 9 reports Pearson correlations on WMT17 to-English segment-level data and compares ablation settings across language pairs.Correlations not outperformed under the Williams Test are bolded.
  • Results: WMD does not consistently improve BERTSCORE across the evaluated language pairs.This observation is stated for the WMT17 to-English segment-level ablation results.

D ADDITIONAL EXPERIMENTS ON ABSTRACTIVE TEXT COMPRESSION · E BERTSCORE OF RECENT MT MODELS

The additional experiments apply BERTScore to abstractive text compression using human judgments and evaluate recent publicly available fairseq machine translation models across three translation tasks.

  • D ADDITIONAL EXPERIMENTS ON ABSTRACTIVE TEXT COMPRESSION: BERTScore is applied to abstractive text compression using human judgments from the MSR Abstractive Text Compression Dataset.The experiment follows the setup of Toutanova et al. (2016).
  • D ADDITIONAL EXPERIMENTS ON ABSTRACTIVE TEXT COMPRESSION: The compression dataset provides human scores for meaning preservation, grammatical correctness, and their combined average.Meaning evaluates preservation of the original text’s meaning, while grammar evaluates grammatical correctness.
  • E BERTSCORE OF RECENT MT MODELS: Table 11 reports BLEU and BERTScore results for publicly available pre-trained fairseq machine translation models.The table includes both rescaled and raw BERTScore values, with markers identifying training-data provenance and models trained by the authors.
  • E BERTSCORE OF RECENT MT MODELS: The machine translation evaluation covers WMT14 English-to-German, WMT14 English-to-French, and IWSLT14 German-to-English.The models are publicly available pre-trained fairseq models, except that the authors trained the IWSLT model because a pre-trained Transformer was unavailable.
  • E BERTSCORE OF RECENT MT MODELS: Multilingual cased BERTbase is used for the English-to-German and English-to-French pairs, while English uncased BERTbase is used for German-to-English.The passage specifies distinct BERT configurations for the evaluated language pairs.
  • E BERTSCORE OF RECENT MT MODELS: The experiment uses fairseq code and identifies the BERT model versions through hash codes in the accompanying implementation notes.The notes also provide the fairseq repository and a ParaCrawl download link.

F ADDITIONAL RESULTS … F.3 IMAGE CAPTIONING ON COCO

Additional experiments extend BERTScore evaluation across WMT correlation, model selection, COCO image captioning, robustness, and multiple pretrained contextual embeddings. The reported results support stronger human-correlation and model-selection performance, with model choice affecting contextual representations.

  • F ADDITIONAL RESULTS: The additional-results section covers WMT16–18 correlation studies, WMT18 model selection, COCO captioning, and PAWS-QQP robustness.These four experiments are explicitly listed as the section’s scope.
  • F ADDITIONAL RESULTS: Experiments compare four pretrained contextual embedding families: BERT, XLM, XLNet, and RoBERTa.The study evaluates multiple released Transformer-based pretrained embeddings.
  • F.1 WMT CORRELATION STUDY: The WMT correlation experiments use to-English segment-level validation data from WMT16 and evaluate segment- and system-level data from WMT17 and WMT18.The datasets contain outputs from 50 systems on 6 language pairs in WMT16, 152 translations on 14 language pairs in WMT17, and 159 systems on 14 language pairs in WMT18.
  • F.1 WMT CORRELATION STUDY: Tables 12–22 show that BERTScore correlates with human judgments, with larger gains at the segment level than the system level.The results also indicate that more optimized or larger BERT models can produce better contextual representations.
  • F.1 WMT CORRELATION STUDY: For English evaluation, fine-tuning pretrained models on related tasks such as natural language inference can improve human correlation.The authors hypothesize that permutation language modeling may produce less informative embeddings for generation evaluation, despite useful downstream fine-tuning weights.
  • F.2 MODEL SELECTION STUDY: BERTScore demonstrates better model-selection performance in the additional WMT18 hybrid-system experiments.The study ranks randomly selected systems from 10K hybrids across 100K repetitions and also finds strong model-selection ability for RUSE.
  • F.3 IMAGE CAPTIONING ON COCO: On COCO image captioning, precision-based BLEU and PBERT are weakly correlated with human judgments, while RBERT generally has high correlation.RBERT even surpasses the task-specific metric SPICE, according to the reported observation.

F.4 ROBUSTNESS ANALYSIS ON PAWS-QQP

The PAWS-QQP robustness analysis finds BERTScore more robust than commonly used metrics, with 24-layer RoBERTa performing best and MNLI fine-tuning improving adversarial robustness.

  • Robustness results: BERTScore is generally more robust than other commonly used metrics on the PAWS-QQP robustness study.The full results are reported using area under the ROC curve (AUC) on QQP and PAWSQQP datasets.
  • Robustness results: BERTScore computed with the 24-layer RoBERTa model performs best.
  • Robustness results: Fine-tuning RoBERTa-Large on MNLI significantly improves robustness against adversarial sentences, whereas fine-tuned BERT on MRPC performs worse than its baseline.
Loading 1904.09675v3…