PaperlayerView exact revision on arXiv

Source-linked AI summary

DecMetrics: Structured Claim Decomposition Scoring for Factually Consistent LLM Outputs

Minghui Huang

arXiv:2509.04483v1cs.CLcs.AI

TL;DR

Fact-checking systems use claim decomposition, but research has paid limited attention to evaluating the quality of the resulting atomic claims. The paper introduces DecMetrics and uses them as reinforcement-learning rewards for a claim decomposition model, alongside a benchmark for diverse evaluations.

  • Problem

    Fact-checking decomposes generated text into atomic claims, but existing methodologies often overlook variability in decomposition quality.

    Source: §1 · p. 1

  • Method

    DecMetrics assesses completeness, correctness, and semantic entropy, and DecModel uses these metrics as reinforcement-learning rewards for atomic-claim generation.

    Source: §1 · p. 2, §7 · p. 8

  • Results

    The authors report that DecModel uses DecMetrics rewards to generate accurate atomic claims, while Claim2Atom supports diverse evaluations of claim decomposition.

    Source: §7 · p. 8

  • Takeaways & Limitations

    Claim2Atom integrates existing datasets with DecData to provide a benchmark for diverse claim-decomposition evaluations.

    Source: §7 · p. 8

    Synthetic data may not capture real-world complexity, and the 200 selected subjects may not represent all topics.

    Source: §8 · p. 9

Abstract

from arXiv · show

Claim decomposition plays a crucial role in the fact-checking process by breaking down complex claims into simpler atomic components and identifying their unfactual elements. Despite its importance, current research primarily focuses on generative methods for decomposition, with insufficient emphasis on evaluating the quality of these decomposed atomic claims. To bridge this gap, we introduce \textbf{DecMetrics}, which comprises three new metrics: \texttt{COMPLETENESS}, \texttt{CORRECTNESS}, and \texttt{SEMANTIC ENTROPY}, designed to automatically assess the quality of claims produced by decomposition models. Utilizing these metrics, we develop a lightweight claim decomposition model, optimizing its performance through the integration of these metrics as a reward function. Through automatic evaluation, our approach aims to set a benchmark for claim decomposition, enhancing both the reliability and effectiveness of fact-checking systems.

1 Introduction

Claim decomposition underlies fact-checking, but its quality can affect which claims are evaluated and the resulting scores. The introduction proposes metrics for decomposition quality, a reward-trained model, and a benchmark.

1 Introduction

  • Decomposition can change the number and scope of evaluated claims, making fact-checking results and metrics dependent on the decomposition method.The introduction notes that altered claims may appear factual while misrepresenting the original claim.Source: §1 · pp. 1–2
  • An ideal atomic claim is complete, correct, and semantically distinct, capturing necessary information without misinformation or redundant paraphrasing.The paper names these dimensions COMPLETENESS, CORRECTNESS, and SEMANTIC ENTROPY and uses them to evaluate atomic claims.Source: §1 · p. 2
  • DecModel uses reinforcement learning with the three DecMetrics as reward signals to generate atomic claims aligned with the proposed quality standards.The introduction presents this as a model-development contribution.Source: §1 · p. 2
  • Claim2Atom combines FActScore, WICE, and the newly curated DecData into a benchmark for evaluating LLMs and specialized decomposition models across several dimensions.The benchmark is also proposed as a resource for future claim-decomposition research.Source: §1 · p. 2

2 Related Works

Prior work decomposes claims using prompted language models and specialized methods, but the quality of the resulting atomic claims has received limited evaluation. The section identifies gaps in ClaimScore and motivates metrics that can also serve as training rewards.

2 Related Works

3 DecMetrics: Evaluation Metrics for Claim Decomposition Quality

DecMetrics evaluates claim decompositions with three entailment-based measures. The section also describes synthetic-data generation and model training for these evaluations.

3.1 DecMetrics Overview

  • All three DecMetrics metrics are framed as supported-versus-unsupported entailment classification tasks, and comprehensive assessment requires high scores across all three.Source: §3.1 · p. 3, §3.1.3 · p. 3

3.1.1 COMPLETENESS Metric

  • COMPLETENESS measures whether the atomic claims collectively cover the original claim by computing entailment probability between the merged atoms and original claim.Its granularity is claim-to-claim, with the probability derived using an NLI model.Source: §3.1.1 · p. 3

3.1.2 CORRECTNESS Metric

  • CORRECTNESS is the fraction of decomposed atomic claims entailed by the original claim, capturing factual faithfulness and limiting fabrication.The entailment decision uses an NLI model; higher scores indicate less factual deviation.Source: §3.1.2 · p. 3

3.1.3 SEMANTIC ENTROPY Metric

  • SEMANTIC ENTROPY clusters mutually entailing atomic claims and quantifies their diversity, encouraging non-redundant, highly atomic decompositions.Arbitrary unrelated atoms can yield high entropy, while COMPLETENESS and CORRECTNESS can expose discrepancies.Source: §3.1.3 · p. 3

3.2 Synthetic Data Generation

  • Wikipedia summaries are iteratively split into non-splittable claims, with entities filtered after 10 iterations; trees are reverse-checked for coverage, faithfulness, and non-overlap.Source: §3.2 · pp. 4–5

3.2.1 Data Synthesis for DecMetrics

  • Completeness positives combine leaves while negatives drop some; correctness negatives use outside-subtree nodes; entropy labels overlapping atom pairs supported and non-overlapping pairs unsupported.Source: §3.2.1 · p. 5

3.2.2 Training and Evaluation Synthesized Data for DecMetrics

  • Training uses an 80/20 tree split, Qwen-generated data from 778 retained trees, and DeBERTa-v3-large fine-tuned with Localized Contrastive Estimation.Source: §3.2.2 · p. 5, §3.3 · p. 6

3.4 Performance Evaluation of DecMetrics

  • NLI models perform well on SNLI and MultiNLI but struggle on DecMetrics test data, with some showing high recall and low precision and others the reverse.The authors conclude that models need adaptation and training on the synthetic DecMetrics dataset for consistent performance across classification metrics.Source: §3.4 · p. 6

4 DecModel: A Standardized Claim Decomposition Model

DecModel uses DecMetrics to train a standardized claim-decomposition model. Its training combines supervised learning, reward modeling, and reinforcement learning, with experiments across three T5 sizes.

4 DecModel: A Standardized Claim Decomposition Model

  • DecModel is trained to produce atomic claims meeting COMPLETENESS, CORRECTNESS, and SEMANTIC ENTROPY criteria.The metrics are integrated into model training.Source: §4 · p. 6

4.1 Training Pipeline of DecModel

  • Training proceeds through supervised fine-tuning, composite reward modeling, and PPO optimization using the learned reward.The reward metrics are used as reinforcement-learning signals while generation diversity is preserved.Source: §4.1 · p. 6

4.2 Configuration of DecModel

  • Experiments use T5-small (80 million), T5-base (250 million), and T5-large (780 million), with reward-component ablations zeroing one coefficient at a time.The default coefficients α, β, and γ are each set to 1.Source: §4.1 · p. 6, §4.2 · p. 7

5 Claim2Atom: Claim Decomposition Benchmark

Claim2Atom combines public datasets with newly curated DecData to evaluate claim decomposition. Its construction filters existing examples for decomposition quality and supports assessment of LLMs and specialized models.

5 Claim2Atom: Claim Decomposition Benchmark

  • Claim2Atom combines FActScore, WICE, and newly curated DecData for claim-decomposition evaluation.The benchmark is intended to support rigorous evaluation across several dimensions.Source: §5 · p. 7

5.1 FActScore and WICE: Public Datasets for Claim Decomposition Evaluation

  • Examples with incomplete, incorrect, or semantically overlapping atomic claims are filtered from FActScore and WICE.The source datasets do not guarantee atomic-claim quality.Source: §5.1 · p. 7

5.2 DecData: A New Dataset for Claim Decomposition Evaluation

  • DecData pairs claims with atomic claims and is split into training and test sets for model training and evaluation.Source: §5.2 · p. 7

5.3 Benchmark Details

  • The benchmark supports evaluation of LLM capabilities and specialized decomposition models, and is intended as a foundation for future research.Source: §5.3 · p. 7

6 Experimental Setup

Experiments compare DecModel with contemporary LLMs, assess decompositions in downstream fact-checking, and ablate reward components. DecModel is parameter-efficient, while downstream aggregation can slightly reduce performance.

6 Experimental Setup

  • The experimental setup compares decomposition quality with contemporary LLMs, evaluates downstream fact-checking utility, and ablates individual metric components.Source: §6 · p. 7

6.1 Metric Performance Comparison

  • DecModel performs comparably to contemporary LLMs and is more parameter-efficient; Qwen3-32B, Mistral-NeMo-12B-Instruct, and Llama-3.2-3B-Instruct show high SEMANTIC ENTROPY but low CORRECTNESS.The comparison evaluates decomposition on Claim2Atom and LLM-AGGREFACT.Source: §6.1 · pp. 7–8, Table 4 · p. 8

6.2 Downstream Fact-Checking Performance

  • Downstream fact-checking shows a slight performance decrease attributed to aggregation that requires support for every atomic claim.Including non-essential claims that are difficult to verify can diminish recall; more sophisticated aggregation is proposed for exploration.Source: §6.2 · p. 8

6.3 Ablation Study

  • Removing any one DecMetrics component lowers its corresponding score, while paired components increase COMPLETENESS, CORRECTNESS, and SEMANTIC ENTROPY.Without the RL stage, omitting DecMetrics reward from SFT decreases COMPLETENESS and SEMANTIC ENTROPY.Source: §6.3 · p. 8, Table 5 · p. 8

7 Conclusion

The conclusion presents DecMetrics, DecModel, and Claim2Atom as contributions to claim decomposition and its evaluation.

7 Conclusion

  • DecMetrics comprises COMPLETENESS, CORRECTNESS, and SEMANTIC ENTROPY metrics for evaluating claim decomposition.Source: §7 · p. 8
  • DecModel uses reinforcement learning with DecMetrics to generate accurate atomic claims.The conclusion describes the model as lightweight.Source: §7 · p. 8
  • Claim2Atom combines existing datasets with DecData to support diverse claim-decomposition evaluations.Source: §7 · p. 8

8 Limitations

The paper identifies limits in the synthetic-data generation process, including constraints on generalization and scalability. It frames robustness, generalizability, and efficiency as areas for further improvement.

8 Limitations

  • Synthetic data may miss real-world complexity and nuances, limiting model generalizability.The selected 200 subjects may not represent all topics, and specific decomposition prompts may perpetuate LLM biases.Source: §8 · p. 9
  • The decomposition and verification process is resource-intensive, limiting larger-dataset and cross-domain experimentation.The computational demand can also hinder refinement and adaptation efforts.Source: §8 · p. 9
  • The authors identify robustness, generalizability, and efficiency as areas needing further improvement in synthetic-data generation.Source: §8 · p. 9

A Claim Decomposition Algorithm

Algorithm 1 recursively decomposes a claim into subclaims and collects claims that are no longer split. Its output is a list of atomic claims, with nested results retained for subclaims.

A Claim Decomposition Algorithm

B Data of Reverse Checking Process

The reverse-checking data are categorized by support status, and claims with more atomic statements tend to have higher failure rates on the three quality metrics. The section also describes decontextualization and verifiability strategies used in post-processing.

B Data of Reverse Checking Process

C Post-Processing Strategies

C.1 Decontextualization

  • SIMPLE-DECONTEXT is used to clarify or modify context in decomposed atomic claims.The passage describes coreference resolution and LLM-based methods as approaches to decontextualization.Source: §C.1 · p. 12

C.2 Verifiability

  • VERISCORE is used to assess whether each decomposed atomic claim can be substantiated.The passage cautions that treating all claims as verifiable can include examples or hypotheticals and unfairly penalize models.Source: §C.2 · p. 12

C.3 Results

The results discuss edits and removals made during post-processing, along with limits of decontextualization and claims that cannot be verified. The accompanying prompts guide decomposition and reverse checking.

C.3 Results

  • Wikipedia-derived claims often need decontextualization for standalone clarity, but it may not significantly enhance fact-checking performance.The passage notes that systems may infer context-dependent information when retrieval provides context alignment.Source: §C.3 · pp. 12–13
  • Figure 5 presents samples of unsupported claims and their corresponding atomic claims from reverse checking.One sample is labeled “not complete.”Source: Figure 5 · p. 13, §C.3 · p. 13
  • Some atomic claims are unverifiable yet must be generated during decomposition and can be disregarded or labeled unverifiable during verification.The paper gives “Betacyanin is like a superhero cape” as an example and notes potential for quality improvement through post-processing.Source: §C.3 · p. 13

D Prompt for Claim Decomposition and Reverse Checking

Loading 2509.04483v1…