Source-linked AI summary
BARTScore: Evaluating Generated Text as Text Generation
Weizhe Yuan, Graham Neubig, Pengfei Liu
TL;DR
Generated-text evaluation must assess qualities such as fluency, accuracy, and effectiveness, while existing neural metrics can disconnect generation-oriented pre-training from evaluation. The paper formulates evaluation as conditional text generation with BARTScore, whose variants support different perspectives without supervised human judgments. BARTScore outperforms existing top-scoring metrics in 16 of 22 test settings across 16 datasets and 7 perspectives.
Problem
Generation evaluation needs to judge qualities such as fluency, accuracy, and effectiveness, but using pre-trained models as feature extractors disconnects evaluation from their text-generation objectives.
Method
BARTScore evaluates generated text through conditional generation probabilities from a pre-trained BART seq2seq model, with variants based on generation direction, prompting, and fine-tuning.
Results
16 of 22 test settings favored BARTScore over existing top-scoring metrics across 16 datasets and 7 evaluation perspectives.
Takeaways & Limitations
BARTScore provides an unsupervised metric that can flexibly evaluate generated text from perspectives including informativeness, coherence, and factuality.
Takeaways & Limitations
BARTScore is less effective at distinguishing extractive summarization systems, and prompting effectiveness for factuality and linguistic quality remains insufficiently demonstrated.
Abstract
from arXiv · showhide
A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.
1 Introduction
The paper identifies a mismatch between generation-oriented pre-training and feature-extractor-based evaluation, then formulates evaluation itself as text generation using BARTScore. Across 7 perspectives and 16 datasets, BARTScore performs best in 16 of 22 test settings.
- Motivation: Pre-trained neural representations benefit NLP tasks and have also supported automatic text-generation metrics through matching, ranking, or regression.BERTScore and MoverScore use unsupervised matching, while other methods learn from human evaluation scores.
- Problem: Generation evaluation disconnects models’ text-generation pre-training objectives from their downstream use as feature extractors.This may underuse pre-trained parameters, including the output prediction layer.
- Approach: BARTScore evaluates generated text through the probability of generating one textual input or output from another with a pre-trained seq2seq model.The formulation directly matches text-generation pre-training tasks.
- Approach: BARTScore is unsupervised and parameter-efficient, requiring no human judgments for training or parameters beyond those used during pre-training.Its conditional generation inputs and outputs can be adjusted for perspectives including informativeness, coherence, and factuality.
- Results: 16 of 22 test settings favored BARTScore over existing top-scoring metrics across 7 perspectives and 16 datasets.Adding “such as” produced a 3% point absolute correlation improvement for German-English machine-translation evaluation.
2 Preliminaries
The paper frames generated-text evaluation around human judgments of multiple quality perspectives and categorizes automated metrics by their task formulation. It introduces text generation as a broader evaluation formulation than common matching, regression, and ranking approaches.
- Evaluation Setup: Conditional generation evaluation assesses a hypothesis produced from a source, often using one or more human-created references.Human evaluation remains the gold-standard method for assessing generated-text quality.
- Evaluation Perspectives: Human judgments can target informativeness, relevance, fluency, coherence, and factuality.These perspectives concern source coverage, source consistency, readability, discourse structure, and factual correctness.
- Existing Metrics: Existing metrics usually cover only a small subset of evaluation perspectives, with BLEU targeting translation adequacy and fluency and ROUGE targeting semantic coverage.Trainable metrics may support multiple perspectives but generally optimize correlation with each judgment type separately.
- Scope: BARTScore is designed to evaluate text from the great majority of the listed perspectives, expanding its applicability relative to existing metrics.Its broader coverage is presented as a contrast with metrics designed for narrower evaluation targets.
- Task Formulations: Neural evaluation metrics include unsupervised matching, supervised regression, supervised ranking, and text-generation formulations.Figure 1 presents evaluation metrics as different tasks involving source, hypothesis, and reference words.
- Task Formulations: BARTScore formulates evaluation as text generation, assigning higher scores when a hypothesis is more easily generated from source or reference text, or vice versa.This formulation is less extensively explored than prior matching, regression, and ranking approaches, with PRISM identified as a notable exception.
3 BARTScore
BARTScore uses a pre-trained sequence-to-sequence model to score one text by its conditional generation probability from another, with directions selected for different evaluation perspectives. Prompting and task-specific fine-tuning extend this formulation.
- Seq2seq Models: Sequence-to-sequence pre-trained models suit conditioned generation because they combine an encoder and decoder with autoregressive predictions.BART is used as the backbone, with preliminary comparisons to T5 and PEGASUS reported in the appendix.
- Scoring Function: The model factorizes a target sequence’s generation probability conditioned on a source sequence, enabling text-quality metrics based on these probabilities.The source has n tokens and the target has m tokens.
- Scoring Function: BARTScore is a weighted log probability of one text given another, with equal token weights used in this work.The general weighting scheme can accommodate alternatives such as inverse document frequency.
- Evaluation Directions: Faithfulness scores p(h|s, θ), precision scores p(h|r, θ), and recall scores p(r|h, θ) for source, hypothesis, and reference texts.These directions target factuality or relevance, precision-focused evaluation, and recall-focused evaluation, respectively.
- Extensions: BARTScore can change its inputs and outputs through prompting or its parameters through task-specific fine-tuning.Both extensions aim to bring the pre-training task or domain closer to the evaluation task.
- Prompting: Prompting adds short phrases to inputs or outputs to encourage specific model behavior, and BARTScore implements this as BARTScore-PROMPT.Prompts can be appended to the source or prepended to the target before applying the scoring equation.
- Fine-tuning: Fine-tuning explores summarization and paraphrasing downstream tasks to bring BART’s pre-training domain closer to evaluation.The paper uses BART fine-tuned on CNN/DailyMail and then continued on ParaBank2.
4 Experiment
Experiments evaluate BARTSCORE and competing metrics across summarization, machine translation, data-to-text, and factuality datasets using human-correlation measures. BARTSCORE generally performs strongly, with prompting and fine-tuning helping selectively across tasks and perspectives.
- Machine Translation: On machine translation, CNN+Para BARTSCORE outperformed all other unsupervised methods on five language pairs and was comparable on two others.Adding the prompt “such as” improved de-en Kendall’s Tau by 0.033 and surpassed supervised BLEURT and COMET there.
- Text Summarization: On summarization, vanilla BARTSCORE exceeded BERTScore and MoverScore on eight settings, with improvements of 0.251 and 0.265 respectively.Fine-tuning further helped REALSum and SummEval, while prompt combinations improved informativeness by up to 0.072 Spearman correlation on NeR18.
- Factuality: On factuality, BARTSCORE+CNN nearly matched the human baseline on Rank19 and outperformed FactCC and QAGS by a large margin.Paraphrase fine-tuning reduced performance, and prompting decreased performance rather than improving it.
- Data-to-text: On data-to-text, paraphrase fine-tuning produced gains up to 0.083 Spearman correlation on BAGEL, while prompt combinations improved correlation by 0.028 on average.These results surpassed existing top-performing metrics.
- Analysis: BARTSCORE was more robust for high-quality texts and varying input lengths, but less effective at distinguishing extractive summarization systems than abstractive systems.Prompting consistently helped semantic overlap, improved factuality only for a few prompts, and had less predictable effects on linguistic quality.
5 Implications and Future Directions
BARTSCORE formulates generated-text evaluation as text generation and supports evaluation from multiple perspectives. The paper highlights prompting and connections between metric and system design as directions for future work.
- BARTSCORE evaluates generated text as a text generation task and achieves the best performance in 16 of 22 settings without human-judgment supervision.
- Prompt-augmented metrics: Prompting improves performance particularly for semantic overlap-based perspectives, but its effectiveness for factuality and linguistic quality remains insufficiently demonstrated.
- Co-evolving evaluation metrics and systems: BARTSCORE connects metric design with system design, allowing advances in a BART-based summarization system to support automated summary evaluation.
A.1 Summary of Commonly Used Metrics for Text Generation
Table 7 summarizes commonly used text-generation metrics by their available input combinations and evaluated task categories.
- Table 7 records whether metrics use source, hypothesis, and reference texts, and groups applications into summarization, machine translation, multiple tasks, and factuality.
A.2 Pre-trained Model Selection
The paper compares PEGASUS and T5 with BART for generation-based scoring on WMT19. Table 8 reports the resulting correlations, with the highest values bolded.
- PEGASUS and T5 do not improve over BART for generation scores on the WMT19 dataset.
- Table 8 reports PEGASUS and T5 experiment results on WMT19, highlighting the highest correlations in bold.
A.3 Prompt Set
The full prompt set is listed for both the source-to-hypothesis direction and the hypothesis-reference directions.
- Table 9 lists the complete prompt set for the s →h and h ↔r scoring directions.
A.4 Prompt Combination
Prompt combination evaluates a prompted target sequence under a seq2seq model across a set of prompts, combining the resulting generation scores. The prompt set is documented for both s →h and h ↔r directions.
- A prompted target sequence [y : z_i] is defined for each prompt z_i in a prompt set.
- Table 9 lists the full prompt set used for both s →h and h ↔r.
- The method combines generation scores from different prompts under a seq2seq model parameterized by θ.
- n denotes the number of prompts considered, while m_i is the target length after adding prompt z_i.
A.5 Robustness to Language Pair Distance
The analysis examines whether metric performance varies with language-pair distance. It uses language vectors and Pearson-correlation heatmaps across six distance types.
- Six language-pair distances are considered: syntactic, geographic, phonological, genetic, inventory, and featural.
- Language vectors are used to measure distances between languages, and Pearson correlation heatmaps are plotted in Figure 4.
- The reported correlations do not change much with respect to language-pair distance.