PaperlayerView exact revision on arXiv

Source-linked AI summary

AlignScore: Evaluating Factual Consistency with a Unified Alignment Function

Yuheng Zha, Yichi Yang, Ruichen Li, Zhiting Hu

arXiv:2305.16739v1cs.CL

TL;DR

Factual consistency evaluation must handle diverse errors and text settings, but existing metrics often rely on narrow NLI or QA functions. AlignScore unifies diverse training data into an information-alignment metric and performs strongly across broad benchmarks, including unseen datasets, while remaining much smaller than ChatGPT- and GPT-4-based metrics.

  • Problem

    Existing factual consistency metrics depend on narrowly trained NLI or QA functions and struggle across diverse errors, tasks, inputs, and outputs.

  • Method

    AlignScore trains a unified information-alignment function on 4.7M examples from 7 tasks and aggregates scores over context chunks and claim sentences.

  • Results

    AlignScore substantially outperforms previous metrics across 22 evaluation datasets, generalizes to 19 unseen datasets, and matches or outperforms ChatGPT- and GPT-4-based metrics.

  • Takeaways & Limitations

    A unified alignment function can support broad factual consistency evaluation across tasks and unseen data with a 355M-parameter metric.

  • Takeaways & Limitations

    The synthetic-data generation method can produce noisy samples that may not accurately model real-world generative-system error types and distributions.

Abstract

from arXiv · show

Many text generation applications require the generated text to be factually consistent with input information. Automatic evaluation of factual consistency is challenging. Previous work has developed various metrics that often depend on specific functions, such as natural language inference (NLI) or question answering (QA), trained on limited data. Those metrics thus can hardly assess diverse factual inconsistencies (e.g., contradictions, hallucinations) that occur in varying inputs/outputs (e.g., sentences, documents) from different tasks. In this paper, we propose AlignScore, a new holistic metric that applies to a variety of factual inconsistency scenarios as above. AlignScore is based on a general function of information alignment between two arbitrary text pieces. Crucially, we develop a unified training framework of the alignment function by integrating a large diversity of data sources, resulting in 4.7M training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization). We conduct extensive experiments on large-scale benchmarks including 22 evaluation datasets, where 19 of the datasets were never seen in the alignment training. AlignScore achieves substantial improvement over a wide range of previous metrics. Moreover, AlignScore (355M parameters) matches or even outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude larger.

1 Introduction

ALIGNSCORE addresses the limited generalizability of task-specific factual consistency metrics with a unified information-alignment approach. It trains on diverse tasks and performs strongly across broad evaluation benchmarks, including unseen datasets.

  • Motivation: Previous metrics based on narrow NLI or QA functions struggle with diverse factual errors, input lengths, domains, and evaluation scenarios.The paper motivates a more holistic metric that generalizes across varying contexts and claims.
  • Approach: ALIGNSCORE models factual consistency with a unified text-to-text information alignment function for arbitrary text pieces.The function estimates alignment between a context and a claim.
  • Training: 4.7M training examples from 15 datasets across 7 language tasks support the alignment function.The tasks include NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization.
  • Metric design: ALIGNSCORE splits contexts into coarse-grained chunks and claims into fine-grained sentences, then aggregates alignment scores.This design handles long inputs and different context and claim roles.
  • Results: Across 22 challenging evaluation datasets, ALIGNSCORE substantially outperforms previous metrics and generalizes strongly to 19 zero-shot datasets.The 355M-parameter metric is on par with, and sometimes better than, metrics based on much larger models such as ChatGPT and GPT-4.

2 Related Work

Prior factual consistency metrics commonly rely on task-specific functions or language-model scoring, while newer work seeks unified evaluation frameworks. ALIGNSCORE instead emphasizes a general alignment function trained from diverse data and uses a smaller model than LLM-based metrics.

  • Factual Consistency Metrics: Traditional factual consistency metrics increasingly use task-specific NLI or QA capabilities to evaluate generated text.Other approaches use long-premise training, synthetic data, or sentence-level evaluation for long-text generation.
  • Factual Consistency Metrics: Other methods use embedding matching, regression to human scores, weighted language-model probabilities, or alignment-based evaluation.These approaches differ in how pretrained language models provide consistency signals.
  • Factual Consistency Metrics: ALIGNSCORE defines alignment more generally to integrate diverse training data and focus on factual consistency.The paper contrasts this with prior alignment-based evaluation and concurrent LLM-prompting metrics.
  • Unified Training: ALIGNSCORE reports stronger performance than LLM-based metrics while using a much smaller model size.This comparison is stated in the related-work discussion.
  • Unified Training: Unified training frameworks convert different tasks into common formats to improve transfer or evaluation across tasks.Prior work includes text-to-text generation and unified automatic evaluation formulations.

3 Methods

The method unifies heterogeneous language-understanding data into an alignment task, trains multiple alignment heads, and converts alignment predictions into factual consistency scores. Its context-chunk and claim-sentence strategy addresses long inputs, distributed evidence, and the need for fine-grained scores.

  • Method overview: ALIGNSCORE combines a unified alignment function with a context/claim splitting and aggregation strategy.The alignment function is trained from diverse data, then used as the metric’s building block.
  • Unified Alignment Function: The alignment task maps two text pieces, a and b, to a label y characterizing their level of information alignment.The paper treats b as aligned when its information is present in and does not contradict a.
  • Unified Alignment Function: 15 datasets spanning 7 tasks are converted into a unified alignment corpus, preserving binary, 3-way, or continuous labels where appropriate.The tasks include NLI, fact verification, paraphrase, semantic textual similarity, QA, information retrieval, and summarization.
  • Unified Alignment Function: The model encodes text pair (a, b) and uses separate 3-way classification, binary classification, and regression heads with a joint loss.The heads are trained with cross-entropy losses for classification and mean squared error for regression; λ1 = λ2 = λ3 = 1.
  • The ALIGNSCORE Metric: 350-token context chunks and independently evaluated claim sentences let the metric use longer evidence spans while respecting the 512-token model limit.For each claim sentence, the highest alignment score across context chunks is selected and then averaged into the final factual consistency score.
  • The ALIGNSCORE Metric: Fine-grained sentence-level scoring addresses the tendency of the classification-heavy alignment model to produce scores near two extremes.Humans typically assign consistency scores on a continuous spectrum reflecting the amount of factual error.

4 Experiments

ALIGNSCORE is evaluated across diverse factual-consistency benchmarks using RoBERTa-based models and compared with established metrics and LLM-based evaluators. It generally achieves leading performance, while ablations show benefits from its 3-way head, chunk-level splitting, and diverse task training.

  • Evaluation Setup: 22 evaluation datasets across TRUE, SummaC, and other testbeds are used to assess factual-consistency metrics.The evaluation covers summarization, dialogue, paraphrase, fact verification, and other settings.
  • Benchmark Results: ALIGNSCORE-large achieves the best average AUC-ROC on SummaC, ranking first on 4 of 6 datasets.Higher AUC-ROC indicates better detection of factual consistency errors; balanced accuracy results also establish new state-of-the-art performance.
  • Benchmark Results: ALIGNSCORE-large obtains the highest average AUC-ROC on TRUE and outperforms baselines on 7 of 11 tasks.Its performance remains strong when excluding training-overlap datasets, indicating generalization to unseen data such as the dialogue-domain DialFact dataset.
  • Benchmark Results: ALIGNSCORE-large outperforms previous metrics overall on 6 of 7 additional factual-consistency datasets.DAE and QuestEval perform better on XSumFaith, which the authors speculate may reflect that dataset’s token-level annotation and fine-grained metrics.
  • LLM Comparisons: ALIGNSCORE is comparable to LLM-based metrics on SummEval and outperforms them on QAGS-XSum and QAGS-CNNDM.These comparisons include metrics based on substantially larger language models.
  • Ablation Study: The 3-way classification head consistently outperforms regression and binary heads, while chunk-level splitting performs best among the tested splitting methods.The splitting ablation indicates that chunk-level processing helps capture salient information from long contexts.
  • Ablation Study: Removing QA causes the worst performance, followed by removing fact verification, whereas removing NLI improves performance on most benchmarks.This ablation measures the contribution of each upstream task to alignment-model training.

5 Conclusion

AlignScore is a holistic factual consistency metric built on a unified alignment function trained from diverse language-understanding tasks. It achieves state-of-the-art results on major benchmarks, correlates better with human judgments than competing metrics, and generalizes to unseen data.

  • AlignScore is a holistic factual consistency metric based on a unified alignment function.
  • 4.7M diverse training samples adapt seven established language-understanding tasks into a unified alignment task.
  • AlignScore achieves state-of-the-art performance on the SummaC and TRUE benchmarks.
  • AlignScore has higher correlation with human judgments than competing metrics and generalizes well to unseen data.

Limitations

The paper identifies interpretability, synthetic-data quality, and language coverage as limitations or boundaries of AlignScore. It also motivates future work on explanations, better synthetic data, and multilingual evaluation.

  • Interpretability: AlignScore’s predictions are difficult to interpret, motivating metrics that identify error-containing spans or provide human-readable explanations.
  • Synthetic data: Synthetic-data generation may produce noisy examples that do not accurately model real-world error types and distributions.
  • Future directions: Future work includes analyzing synthetic-data quality, improving its generation, and extending AlignScore to multiple languages.
  • Language coverage: AlignScore generalizes to unseen data but currently covers only English.

Ethics Statement

The paper adapts datasets from seven NLP tasks into a common information-alignment format. These adaptations use task-specific transformations to create aligned and non-aligned text pairs for training.

  • Datasets from seven NLP tasks are adapted into a common information-alignment format.
  • NLI, fact verification, and paraphrase datasets map their original labels to binary or three-way alignment labels.
  • Semantic textual similarity scores are normalized to the 0–1 range for regression.
  • QA samples become aligned or non-aligned pairs by converting question-answer pairs into declarative sentences and pairing them with contexts.
  • Information-retrieval samples are labeled aligned when a passage answers the query and non-aligned otherwise.

A.2 Synthetic Data

The paper augments alignment training with synthetic data derived from WikiText-103 and WikiHow. It creates aligned examples through paraphrasing or summaries, then generates non-aligned examples by masking and infilling.

  • Synthetic training data is built from the WikiText-103 corpus and the WikiHow summarization dataset.
  • WikiText-103 aligned samples use back translation to create paraphrases of sentences.
  • WikiHow aligned samples pair source documents with ground-truth and extractive summaries.
  • Masking 25% of tokens in aligned text and infilling them produces semantically different samples used as non-aligned data.

A.3 Training the Alignment Model

The alignment model is implemented and trained with standard deep-learning tooling and optimization, while evaluation preprocessing addresses dataset artifacts before scoring factual consistency.

  • ALIGNSCORE is implemented with Transformers and trained using PyTorch Lightning.
  • The alignment model uses AdamW with a 1e-5 peak learning rate after warmup, followed by linear decay.
  • Training does not split contexts and claims into chunks for simplicity.
  • Rule-based preprocessing removes escape sequences and corrects claim capitalization to match contexts.
  • Inputs are split into sentences, after which ALIGNSCORE computes instance-level factual consistency scores and correlation statistics.

B.1 SummaC Benchmark

The evaluation uses SummaC and TRUE benchmarks plus other datasets to assess factual consistency across summarization and several related tasks. The reported setup also specifies how errors, class imbalance, and correlations are handled.

  • SummaC Benchmark: SummaC contains 6 summarization datasets, standardizes labels by binarization, and evaluates metrics as classifiers.
  • SummaC Benchmark: PolyTope results classify intrinsic inaccuracy, extrinsic inaccuracy, and positive-negative aspect errors as factually incorrect, excluding addition and omission.
  • SummaC Benchmark: Balanced accuracy is additionally reported for SummaC to address imbalanced datasets.
  • TRUE Benchmark: TRUE evaluates factual consistency metrics across summarization, dialogue, fact-verification, and paraphrasing using 11 datasets and AUC-ROC.

C.1 Converted QA Samples

The section illustrates converting question-answering examples into alignment-format inputs by turning questions and answers into declarative claims. The examples pair source contexts with generated claims that can be checked against them.

  • Conversion procedure: Questions and answers are combined into declarative claims using a sequence-to-sequence model.
  • Examples: A question about the Times Literary Supplement is converted into a claim locating its editorial offices at Times House, Pennington Street, London.
  • Examples: A context stating that 25,000 U.S. cotton growers are subsidized is paired with a generated claim expressing that fact.
  • Examples: A context about IBM’s Weather Company acquisition is paired with a generated claim stating that the sale closed on January 29, 2016.
  • Examples: A Neptune magnetic-field question and answer are converted into a declarative claim reporting 14 microteslas (0.14 G).
  • Examples: A Tibet question is shown with a generated claim about a Qing commander publicly executing rebel supporters in 1732 and 1728.
Loading 2305.16739v1…