PaperlayerView exact revision on arXiv

Source-linked AI summary

MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents

Liyan Tang, Philippe Laban, Greg Durrett

arXiv:2404.10774v2cs.CLcs.AI

TL;DR

LLM fact-checking against grounding documents is important but expensive because generated responses contain many facts requiring verification. MiniCheck uses structured synthetic training data and a unified benchmark to train small models for this task, with GPT-4-comparable accuracy at much lower cost. Its strongest results support efficient verification without separate claim decomposition, though evaluation remains limited for widely separated evidence and non-English outputs.

  • Problem

    Fact-checking LLM outputs against grounding documents requires checking many facts and cross-sentence inferences, making existing LLM-based verification computationally expensive.

  • Method

    MiniCheck trains small Flan-T5-based fact-checkers on synthetically generated, challenging instances designed to teach atomic-fact verification and cross-sentence reasoning.

  • Results

    MiniCheck matches GPT-4-level aggregate accuracy and outperforms prior specialized fact-checkers, with MiniCheck-FT5 achieving a 4.3% improvement over AlignScore.

  • Takeaways & Limitations

    LLM-AGGREFACT and MiniCheck provide a unified, substantially cheaper route to fact-checking across closed-book and grounded-generation settings without separate claim decomposition.

  • Takeaways & Limitations

    The benchmark does not require reasoning over evidence that is significantly separated or spread across multiple documents, and the models were trained exclusively on English data.

Abstract

from arXiv · show

Recognizing if LLM output can be grounded in evidence is central to many tasks in NLP: retrieval-augmented generation, summarization, document-grounded dialogue, and more. Current approaches to this kind of fact-checking are based on verifying each piece of a model generation against potential evidence using an LLM. However, this process can be very computationally expensive, requiring many calls to a model to check a single response. In this work, we show how to build small fact-checking models that have GPT-4-level performance but for 400x lower cost. We do this by constructing synthetic training data with GPT-4, which involves creating realistic yet challenging instances of factual errors via a structured generation procedure. Training on this data teaches models to check each fact in the claim and recognize synthesis of information across sentences. For evaluation, we unify datasets from recent work on fact-checking and grounding LLM generations into a new benchmark, LLM-AggreFact. Our best system MiniCheck-FT5 (770M parameters) outperforms all systems of comparable size and reaches GPT-4 accuracy. We release LLM-AggreFact, code for data synthesis, and models.

1 Introduction

Fact-checking LLM outputs against grounding documents is difficult and costly because responses contain many facts and may blend information incorrectly. MiniCheck addresses this with synthetic training data and a unified benchmark, achieving GPT-4-comparable performance at far lower cost.

  • Motivation: Past work treated closed-book verification and grounded-generation attribution largely as separate problems.The paper identifies statement checking against grounding documents as their shared primitive operation.
  • Motivation: Grounding-document fact-checking must detect subtle errors while maintaining a low false-positive rate across many generated facts.A 110–150 word biography in FActScore can require 130–205 entailment checks.
  • Approach: MiniCheck synthesizes challenging fact-checking instances from scratch so small models can verify multiple facts against multiple grounding sentences.The system is a Flan-T5 model fine-tuned on synthetic data plus standard entailment data.
  • Evaluation: LLM-AGGREFACT unifies factual evaluation across closed-book and grounded-generation settings by aggregating 10 existing datasets.The constituent datasets contain sentence-level factual-error labels from human annotators.
  • Results: MiniCheck beats prior specialized systems by 4% to 10% in absolute performance, matches GPT-4-level accuracy, and costs 400 times less.The system achieves these results without a separate claim-decomposition step.

2 Background and Motivation

The task classifies each claim as supported or unsupported by grounding documents, but claims may contain multiple facts and require reasoning across sentences. MiniCheck targets these challenges with synthetic data generation methods.

  • Problem Setup: Each sentence claim is classified as supported or unsupported according to an associated grounding document.The setup allows different evidence sets per sentence, shared evidence, or a single document.
  • Decision Rule: The sentence-level decision is positive if any associated document supports the claim.This is implemented by taking the maximum model score across documents.
  • Challenges: A claim may contain several atomic facts, each of which must be checked even when those facts are not explicitly separated.The paper illustrates this with an LLM-generated summary sentence containing four atomic facts.
  • Challenges: Verification may require inferences spanning multiple document sentences, as later dialogue can alter the interpretation of an earlier statement.The airline example supports weather as uncontrollable while suggesting labor problems are within airlines’ control.
  • Motivation: The paper argues that existing specialized fact-checkers struggle with atomic-fact coverage and cross-sentence reasoning.It introduces two synthetic data-generation methods to improve both abilities.

3 Methodology: Training Data Synthesis

The paper synthesizes challenging fact-checking training instances because existing entailment data does not reflect LLM-generated claims and real-error annotation is difficult to scale. Its C2D and D2C procedures create examples requiring atomic-fact checking across sentences and documents.

  • Motivation: Existing MNLI and ANLI instances lack LLM fact-checking complexity, while real-error annotation is difficult to scale.Datasets containing such errors, including LLM-AGGREFACT, are largely test-only.
  • Claim to Document (C2D): C2D decomposes claims into atomic facts, expands each fact into a sentence pair, and generates a supporting document from those sentences.The generated pair is designed so that support requires combining both sentences.
  • Claim to Document (C2D): Removing one sentence from a pair yields nonsupporting documents when the remaining context cannot entail the omitted atomic fact.GPT-4 performs an entailment check before retaining such examples, and redundancy can prevent nonsupport.
  • Claim to Document (C2D): C2D pairs supporting and nonsupporting documents with augmented subclaims formed from nonempty subsets of a claim’s atomic facts.This produces contrastive examples in which the same subclaim can be supported or unsupported depending on document contents.
  • Document to Claim (D2C): D2C starts from human-written documents, summarizes chunks into claims, decomposes those claims into atomic facts, and augments document-claim pairs by removing sentences or using other chunks.The design seeks greater document diversity and realism while preserving multi-sentence support challenges.
  • Model Training: Three models are fine-tuned on synthetic data, with MiniCheck-DBTA and MiniCheck-FT5 also using a selected 21K-example ANLI subset.All models use standard cross-entropy loss.

4 LLM-AGGREFACT Benchmark

LLM-AGGREFACT is a unified benchmark of 10 factual-consistency datasets spanning closed-book and grounded generation. It uses human-annotated document-claim-label tuples and supports zero-shot cross-dataset evaluation without threshold tuning.

  • Benchmark Construction: LLM-AGGREFACT aggregates 10 publicly available datasets covering factual-consistency evaluation in closed-book and grounded-generation settings.The benchmark is presented as a fact-verification resource.
  • Dataset Characteristics: Its datasets contain human-annotated document-claim-label tuples drawn from Wikipedia, interviews, and web text across news, dialogue, science, and healthcare.Most claims were generated by recent generative models without human intervention in error injection.
  • Data Splits: Validation and test splits are inherited for AGGREFACT, TOFUEVAL, WICE, and CLAIMVERIFY, while four other datasets are randomly split 50%/50% by unique queries.The split procedure prevents responses to identical queries from appearing in both sets.
  • Evaluation Protocol: The evaluation avoids per-dataset threshold tuning to focus on zero-shot deployment across multiple downstream tasks.For M(d, c) → z ∈ [vmin, vmax], the threshold is set at the midpoint of the output score range.
  • Evaluation Metric: Fact-checkers are evaluated using balanced accuracy, averaging true-positive and true-negative rates.Balanced accuracy is defined from TP, TN, FP, and FN.

5 Experimental Setup

The experiments compare specialized fine-tuned fact-checkers with LLM-based fact-checkers. The setup includes established baselines and several contemporary general-purpose language models.

  • Specialized Fact-Checkers: Specialized baselines include T5-NLI-Mixed, DAE, QAFactEval, SummaC-ZS, SummaC-CV, AlignScore, and FT5-ANLI-L.A meta-comparison of these systems and the authors’ models appears in Table 3.
  • LLM Fact-Checkers: LLM-based fact-checkers include Gemini-Pro, PaLM2-Bison, Mistral variants, Claude variants, GPT-3.5, and GPT-4.These models use a zero-shot prompt adapted from prior work.

6 Results

MiniCheck models perform strongly across architectures and approach frontier LLM fact-checkers while substantially reducing inference cost.

  • MiniCheck-FT5 improves overall performance by 4.3% over AlignScore, outperforming it on 6 of 10 datasets and matching it on the other 4.The authors attribute an additional 2% gain over MiniCheck-RBTA and -DBTA to FT5’s larger model size.
  • Synthetic training data improves performance across RoBERTa, DeBERTa, and Flan-T5 backbones.The results indicate that training-data selection matters in addition to model capacity.
  • Table 2 reports BAcc on LLM-AGGREFACT without per-dataset threshold tuning, grouping LLM-based, specialized, and MiniCheck systems.Green highlights indicate the best or statistically indistinguishable performance by dataset.
  • MiniCheck-FT5 matches Claude-3 Opus and approaches GPT-4 performance with a much smaller model.MiniCheck-RBTA and MiniCheck-DBTA also surpass non-frontier LLM fact-checkers such as Mistral-8x7B and GPT-3.5.
  • More than 400 times cheaper, MiniCheck-FT5 achieves almost the same performance as GPT-4.Specialized models generally have much lower inference costs than LLM-based fact-checkers.

7 Rethinking LLM Fact-Checking

The paper examines whether claim decomposition and decontextualization improve fact-checking, finding limited benchmark gains despite added computational cost.

  • Decontextualization: Decontextualization does not help on the benchmark, although the authors believe it is generally needed.
  • Claim decomposition: Claim decomposition does not consistently improve performance, while increasing inference time and cost by a factor of 2-4.GPT-4 shows near-zero performance change and specialized fact-checkers show mixed changes.
  • Table 5 compares average performance changes from decomposing claims and decontextualizing applicable claims against original-claim predictions.
  • Decontextualization: Decontextualization changes 11% to 57% of claims across the evaluated context-dependent datasets.The reported percentages are 33%, 33%, 39%, 11%, 35%, 47%, and 57%, respectively.
  • Decontextualization: For context-dependent claims, models may make decent guesses when retrieval already implicitly enforces shared context between claims and documents.The authors still view decontextualization as crucial for meaningful document retrieval in tasks such as retrieval-augmented generation.

8 Related Work

Related work covers hallucination detection, specialized and LLM-based fact-checkers, and entailment datasets; MiniCheck targets multi-fact and multi-sentence reasoning without decomposition.

  • Hallucinations in LLMs: LLMs hallucinate in closed-book, retrieval-augmented, summarization, and simplification settings, producing information unsupported by sources or inconsistent with documents.
  • Methods in Detecting Hallucinations: Existing methods verify claims directly against documents when sources are available, while closed-book settings may require retrieving evidence before checking.
  • Methods in Detecting Hallucinations: Fact-checking research uses specialized evaluators or LLMs as fact-checkers, with MiniCheck matching GPT-4 performance across settings without sentence decomposition.
  • Entailment Datasets: The paper’s synthetic dataset differs from human-authored, collected, and automatically generated entailment resources by targeting multi-sentence and multi-fact reasoning.

9 Conclusion

The paper introduces synthetic data generation methods and LLM-AggreFact to improve factual consistency evaluation across generation settings.

  • MiniCheck’s synthetic data methods encourage verification of each atomic fact and reasoning across multiple sentences.
  • LLM-AggreFact unifies factual consistency evaluation for closed-book and grounded generation settings.
  • A model fine-tuned on the synthetic data outperforms prior specialized fact-checkers while costing much less than LLM-based fact-checkers.

Limitations

The paper’s fact-checkers have interpretability, multi-document reasoning, synthetic-data quality, and language-scope limitations. Evaluation also relies on specific synthetic-data and experimental settings.

  • Interpretation: The models do not expose their internal decisions, making it difficult to localize errors to mismatched claim or document spans.Claim decomposition can improve localization, while reliable generated explanations remain future work.
  • Multi-Document Reasoning: The benchmark does not require reasoning over evidence that is widely separated across multiple documents.The authors identify such scenarios as a target for future evaluation and model development.
  • Synthetic Data: The synthetic data uses a simple pair-of-sentences construction, leaving more complex and higher-quality generation strategies unexplored.The authors note that improving synthetic-data quality is a future direction.
  • Language: The models are trained exclusively on English data, and performance in other languages has not been systematically assessed.The missing evaluation reflects the absence of human-annotated factual-consistency datasets for non-English LLM outputs.

A.2 Ablation of D2C/C2D

The ablations show that carefully constructed synthetic data is central to cross-domain performance, while simplified construction fails to produce the desired reasoning behavior. The models also remain robust without threshold tuning, and claim decomposition provides no consistent accuracy benefit despite added cost.

  • Cross-domain evaluation: AlignScore is the strongest of four specialized metrics, but FT5-C2D and FT5-D2C outperform it on the held-out evaluations.Fine-tuning AlignScore’s backbone on 14K synthetic examples also improves its benchmark performance.
  • Training-data ablation: Nearly 10% performance is lost when MiniCheck-FT5 is trained only on the ANLI subset without threshold tuning, while adding either 7K synthetic examples substantially improves performance.The -BOTH ablation drops from 69.1 to 59.9 without threshold tuning.
  • Synthetic-data quality: 80% and 78% are the automatic-label accuracies on C2D and D2C, compared with 85% and 88% for the average human annotator accuracy.Disagreements often reflect subjectivity in natural-language inference, yet the authors report the data remains useful for training.
  • Threshold-tuned results: 72.2% is AlignScore’s highest overall performance on LLM-AGGREFACT, while MiniCheck-RBTA surpasses it by 1.5% using 14K synthetic examples.The 14K examples constitute less than 0.3% of AlignScore’s original training data.
  • Robustness without tuning: MiniCheck-FT5 drops 0.4% without threshold tuning, compared with a 9.2% drop for SummaC-CV.The authors interpret this as evidence that synthetic data improves robustness across benchmark domains.

C.3 Excluded Datasets

The benchmark excludes synthetic-error datasets because their intentionally induced errors are considered unnatural for detecting true LLM generation errors. It also includes datasets whose evidence sources can create label-alignment concerns.

  • Exclusion rationale: HALUEVAL and SUMMEDITS are excluded because their summaries contain errors intentionally induced through instruction prompts.These errors are described as synthetic and unnatural for the target task.
  • Evidence alignment: FACTSCORE uses naturally generated biographies with human-annotated atomic-fact labels, but human verifiers may search articles different from the models’ retrieved evidence.This creates a mismatch between annotation evidence and the evidence used for grounded fact-checking.
  • Benchmark composition: LLM-AGGREFACT aggregates nine factual-consistency datasets on CNN or XSum, including FactCC, SummEval, FRANK, and CLIFF.The dataset statistics table reports dataset size, average document and claim lengths, and unsupported-claim proportions.

D Synthetic Data Details

The synthetic-data pipeline combines claim-to-document and document-to-claim generation with filtering to create challenging fact-checking examples. Training first on synthetic data and then on realistic documents consistently improves performance across the three backbone models.

  • Data sources: The C2D method starts from about 400 Wikipedia claims with cited web articles, while D2C uses about 300 diverse Google News articles.D2C documents cover science, politics, world, entertainment, business, and technology, and average approximately 500 words.
  • Synthetic-data characteristics: The synthetic dataset uses human-written or naturally generated claims rather than targeting only predefined error types such as entity swaps or sentence negation.This distinguishes the construction approach from earlier synthetic summarization data methods.
  • Quality filtering: GPT-4 filtering rejects 6% of C2D sentence pairs, 5% of supporting documents, and 53% of non-supporting documents that still support the claim.The filtering steps are described as crucial for improving the training dataset.
  • Simplified generation: C2D-SIMP generates a supporting document from a claim, then minimally revises it so that only some atomic facts remain supported.Four revision types produce diverse non-supporting documents; the simplified dataset contains 7K datapoints.
  • Training strategy: Training sequentially on C2D and then D2C consistently outperforms training on all data simultaneously across all three backbone models.The authors hypothesize that ending with human-written D2C documents helps models adapt to a realistic distribution.
Loading 2404.10774v2…