PaperlayerView exact revision on arXiv

Source-linked AI summary

A Closer Look at Claim Decomposition

Miriam Wanner, Seth Ebner, Zhengping Jiang, Mark Dredze, Benjamin Van Durme

arXiv:2403.11903v1cs.CL

TL;DR

Textual-support metrics depend on how claims are decomposed, but prior work has largely left decomposition quality untested. The paper introduces DECOMPSCORE and an LLM approach using theory-inspired examples; among the methods considered, that approach yields the greatest number of subclaims supported by the original claim.

  • Problem

    Claim decomposition can change the number and scope of evaluated subclaims, yet prior work has largely left decomposition quality untested.

    Source: §1 · p. 2, §3 · p. 3

  • Method

    The paper introduces DECOMPSCORE to measure supported subclaims and prompts an LLM with manually decomposed examples inspired by logical atomism and neo-Davidsonian semantics.

    Source: §1 · p. 2, §3.2 · p. 3, §4.1 · p. 4

  • Results

    Among the methods considered, the theory-inspired approach outperformed others and produced the greatest number of subclaims supported by the original claim.

    Source: §4.1 · p. 4, §10 · p. 8

  • Takeaways & Limitations

    Textual-support metrics such as FACTSCORE are sensitive to decomposition, so assessing decomposition quality helps select an appropriate method.

    Source: §10 · p. 8

    The study covers entity biographies in English, and the authors do not know whether the results hold across other languages.

    Source: Limitations · p. 9

Abstract

from arXiv · show

As generated text becomes more commonplace, it is increasingly important to evaluate how well-supported such text is by external knowledge sources. Many approaches for evaluating textual support rely on some method for decomposing text into its individual subclaims which are scored against a trusted reference. We investigate how various methods of claim decomposition -- especially LLM-based methods -- affect the result of an evaluation approach such as the recently proposed FActScore, finding that it is sensitive to the decomposition method used. This sensitivity arises because such metrics attribute overall textual support to the model that generated the text even though error can also come from the metric's decomposition step. To measure decomposition quality, we introduce an adaptation of FActScore, which we call DecompScore. We then propose an LLM-based approach to generating decompositions inspired by Bertrand Russell's theory of logical atomism and neo-Davidsonian semantics and demonstrate its improved decomposition quality over previous methods.

1 Introduction

The paper argues that claim decomposition affects textual-support evaluation and introduces both a way to assess decomposition quality and a theory-inspired LLM approach. It also situates this work among localized-support methods that score individual subclaims against external sources.

1 Introduction

  • Prior work leaves decomposition itself largely untested, despite its role in shaping downstream textual-support analysis and metrics.Source: §1 · p. 2
  • FACTSCORE’s textual-support result is sensitive to decomposition method because the method determines the number and scope of evaluated subclaims.The paper argues that better decompositions make it easier to attribute measured factual precision to the generator rather than decomposition artifacts.Source: §1 · p. 2
  • DECOMPSCORE measures decomposition quality by counting subclaims supported by the original claim, favoring higher scores alongside qualitative evidence of atomicity and coverage.Source: §1 · p. 2
  • The proposed LLM approach uses decompositions inspired by logical atomism and neo-Davidsonian semantics, producing more subclaims while maintaining high coherence.The paper presents the approach as outperforming previous methods.Source: §1 · p. 2
  • The paper reports empirical evidence that decomposition method affects textual-support metrics and compares methods quantitatively and qualitatively.Source: §1 · p. 2

2 Localized Textual Support

  • Localized-support systems split claims into subclaims, score them against external sources, and aggregate results while retaining information about which parts are supported.Source: §2 · p. 2

3 Evaluating Decomposition Quality

Decomposition quality affects how reliably support metrics measure generated text: decompositions should be faithful, comprehensive, and atomic. The section introduces DECOMPSCORE and compares prompting, semantic parsing, and parse-conditioned prompting as decomposition approaches.

3.1 Qualitative Evaluation

  • For precise localization of discrepancies, decompositions should cohere with the original claim while covering all its parts and remaining as atomic as possible.FACTSCORE can vary for the same generated text, and decomposition errors can add unclaimed information or omit claims.Source: §3.1 · p. 3, §3 · p. 3

3.2 Quantitative Evaluation: DECOMPSCORE

  • DECOMPSCORE estimates quality by using an LLM to judge whether each subclaim is supported by the sentence it came from.The optimal score for a passage is difficult to determine without reference decompositions.Source: §3.2 · pp. 3–4
  • DECOMPSCORE is the average number of subclaims per passage that a decomposition method produces and that the original sentence supports.It uses the original sentence, rather than an external knowledge source, to judge support; methods with higher atomicity and coverage generally score higher.Source: §3.2 · pp. 3–4
  • DECOMPSCORE correlates highly with the average number of subclaims entailed by the original claim under a strong NLI model.The authors report this as evidence that entailment can also evaluate whether subclaims are valid parts of a decomposition.Source: §3.2 · p. 4

4.1 LLM prompting

  • Prompted LLM methods use prior instructions and static or retrieved in-context examples, including manually decomposed examples based on philosophical and linguistic theories.The authors compare three prior-work instructions and one of their own.Source: §4.1 · p. 4
  • LLM prompting produces fluent, flexible text but permits hallucinations and offers no guarantee that instructions will yield desired decomposition traits.Retrieved examples with high similarity may provide some controllability.Source: §4.1 · p. 4

4.2 Shallow semantic parsing

  • PredPatt extracts predicate-argument structures from dependency parses, which an LLM then converts into natural-language subclaims.PredPatt outputs contain only original-sentence words and are often not grammatical; the LLM does not guarantee grammatical output either.Source: §4.2 · p. 4

4.3 LLM prompting with parse

  • Parse-conditioned prompting gives the LLM a dependency parse to ground its response while retaining prompting’s fluency and flexibility.The added structure is intended to improve controllability, but generated subclaims may still fail to cohere with the claim.Source: §4.3 · pp. 4–5

5 Russellian and Neo-Davidsonian decomposition

The proposed decomposition draws on Russellian atomic facts and neo-Davidsonian semantic structure to break claims into small propositions about entities and their relations.

5 Russellian and Neo-Davidsonian decomposition

  • The authors choose Russell’s theory as actionable inspiration, while noting that they do not require subclaims to be independent.They contrast this choice with Wittgenstein’s similar theory of elementary propositions.Source: §5 · p. 5
  • The approach treats Russellian atomic facts as properties of individuals or relations between individuals, and takes individuals to include entities and eventualities mentioned in a sentence.It aligns this analysis with neo-Davidsonian decomposition into unary and binary predicates.Source: §5 · p. 5
  • The authors manually decomposed 21 in-context examples into propositions, then further split them into unary and binary relations to obtain the smallest claimed units.Each subclaim designates a property of one individual or a relation between two individuals; existence is not included as an entity property.Source: §5 · p. 5

6 Data

Experiments on static biographies show that decomposition methods change FACTSCORE, while DR-ND produces the most supported subclaims. Qualitative and quantitative analyses examine the decomposition properties behind these differences.

6 Data

  • The study uses unchanged biographies for 500 individuals generated by each of 12 language models, treating them as static documents for sentence-level decomposition.Evaluation uses Inst-LLAMA for DECOMPSCORE and Inst-LLAMA with retrieval and NPM for FACTSCORE.Source: §6 · p. 5, §7 · p. 5

7 Experiments

  • The experiments apply several decomposition methods to the biographies' sentences and evaluate both decomposition quality and downstream textual support.Generating decompositions took 120 GPU-hours; computing DECOMPSCORE and FACTSCORE took 250 and 450 GPU-hours, respectively.Source: §7 · p. 5

8 Results

  • FACTSCORE values vary across decomposition methods even when the same biographies are evaluated, making the metric sensitive to decomposition choice.The paper argues that the highest-quality decomposition offers the most reliable estimate of generated text's factual precision.Source: Figure 2 · p. 6, §8 · p. 6, §8.2 · p. 7
  • 42.3 DECOMPSCORE: DR-ND has the highest average number of supported subclaims per biography, ahead of DChen et al. and DFACTSCORE (around 32); DWICE scores 20.0.DPredPatt scores 29.2, slightly above DCoNLL-U at 27.1.Source: §8 · p. 5, Figure 3 · p. 6

8.1 Qualitative Analysis

  • DR-ND separates “suspenseful” from “thrilling,” while DR-ND and DFACTSCORE split “captivate and inspire”; other methods often leave these as combined units.For the Nash example, DFACTSCORE and DChen et al. also produce incremental, non-atomic subclaims that can make support harder to localize.Source: §8.1 · p. 6

8.2 Quantitative Analysis

  • Restricting FACTSCORE to supported subclaims raises the score in most cases, suggesting that subclaims incoherent with the sentence are often unsupported by the knowledge source.The filtering removes 0.2 subclaims per biography on average for DWICE and 1.2 of 43.5 for DR-ND, compared with 4 for DPredPatt.Source: §8.2 · p. 7
  • A higher FACTSCORE does not necessarily indicate a better decomposition: DR-ND has lower FACTSCORE than most methods but much higher DECOMPSCORE, which the authors hypothesize reflects its manually decomposed examples.Despite high coherence and coverage, DWICE has the lowest DECOMPSCORE, attributed to low atomicity; using fewer in-context examples has little apparent effect on decomposition quality or factual precision.Source: §8.2 · pp. 7–8

9 Related Work

Related work evaluates textual support through decomposed subclaims, structured representations, and entailment judgments. These strategies also support error analysis, sub-sentence inference, and fact verification.

Evaluation

  • Question-answering approaches evaluate summaries by asking questions, often about noun phrases, then comparing extracted answer spans with lexical heuristics.These methods require question generation as the decomposition step.Source: §9 · p. 8
  • Other evaluation methods compare extracted relation or fact tuples, or assess entailment on dependency-parse arcs, which assumes syntactic arcs correspond to semantic units.Fan et al. (2023) use semantic role labeling to extract fact tuples.Source: §9 · p. 8
  • Textual-error and ambiguous-support evaluations can also be applied to subclaims, alongside approaches that categorize generated-text errors.The passage cites Pagnoni et al. (2021), Devaraj et al. (2022), Mishra et al. (2024), and Glockner et al. (2024).Source: §9 · p. 8

NLI Decomposition

  • Sub-sentence NLI methods identify propositions or combine their entailment judgments to make sentence- and paragraph-level judgments more interpretable and robust.PROPSEGMENT identifies subclaims by marking their tokens in a claim.Source: §9 · p. 8

Fact Verification

  • Fact-verification systems decompose complex claims into questions or align generated claims and propositions with documents to retrieve supporting or contradicting evidence.The passage also describes proposition-level contrastive learning for retrieval.Source: §9 · p. 8

10 Conclusion

FACTSCORE varies with the decomposition method, motivating DECOMPSCORE and a logical-atomism- and neo-Davidsonian-inspired LLM approach that outperforms the alternatives considered. The authors also identify limits in what the metrics evaluate, the study’s domain and language, and LLM reliability.

10 Conclusion

  • FACTSCORE depends on decomposition method, so the authors use DECOMPSCORE to compare decomposition quality and select among the methods considered.Source: §10 · p. 8
  • The proposed LLM approach yields the greatest number of supported subclaims among methods considered, while analyses find that all methods miss subclaims and often produce non-atomic ones.It uses manually decomposed in-context examples inspired by logical atomism and neo-Davidsonian semantics.Source: §10 · p. 8

Limitations

  • FACTSCORE and DECOMPSCORE assess only information stated in generated text, so they cannot account for relevant information absent from that text.Source: Limitations · p. 9
  • The study covers English-language entity biographies, so its findings are not established for other use cases or languages.Source: Limitations · p. 9
  • Running LLMs can be financially infeasible for some users, even with the less costly LLAMA validator used in the study.Source: Limitations · p. 9

Ethics Statement

  • LLM-based decomposition can introduce factual errors that may persist despite filtering, warranting caution when relying on model-generated text.Source: Ethics Statement · p. 9

A Full Results

The appendix reports FACTSCORE and DECOMPSCORE-related results across decomposition methods and language-model subjects, including results after filtering unsupported subclaims. It also notes data and knowledge-source conditions that affect interpretation.

A Full Results

  • Results include FACTSCORE by language-model subject and decomposition method, plus FACTSCORE after filtering subclaims judged unsupported by their original claims.Source: A Full Results · p. 13
  • DECOMPSCORE results report supported subclaims per biography; separate results report the percentage of subclaims judged coherent with the original claim.Source: A Full Results · p. 13
  • Invalid language-model responses add noise to decomposition-quality evaluation, while differences in Wikipedia versions can affect FACTSCORE but not DECOMPSCORE.Source: A Full Results · p. 13

B Model Details

The appendix documents model settings, decomposition procedures, and the scope of reported tables and figures. It also describes a rare context-window fallback in the DCoNLL-U approach.

B Model Details

  • The study uses gpt-3.5-turbo-instruct for decomposition and specifies its context window, output limit, and token costs; Inst-LLAMA is used for FACTSCORE and DECOMPSCORE evaluations.Source: B Model Details · p. 13
  • For DPredPatt, the appendix specifies Trankit parsing, PredPatt flags, and a language-model prompt for converting outputs into natural-language sentences.Source: B Model Details · p. 13
  • When DCoNLL-U prompts exceed the context window, examples are removed; the approach falls back to the original sentence in 0.05% of cases.Across 6,000 passages, the fallback occurred three times; the authors leave reducing parse size to future work.Source: B Model Details · p. 13
  • The appendix reports DECOMPSCORE by decomposition method and language-model subject, with averages of supported subclaims per biography.Source: Table 2 · p. 14
  • The appendix also reports average subclaims per biography, coherence percentages by method and subject, and correlations with results reported by Min et al. (2023).Source: Table 3 · p. 14, Table 6 · p. 17, Table 7 · p. 18
  • FACTSCORE results are reported by biography-generation model and decomposition method, while a separate table reports scores after filtering unsupported subclaims.Source: Table 4 · p. 15, Table 5 · p. 16

C NLI Entailment

The number of subclaims entailed by the original sentence closely tracks the number judged supported by DECOMPSCORE. The reported Pearson correlation is 0.9978.

C NLI Entailment

D Decomposition Examples

The section presents decompositions of two example sentences, produced manually and by all evaluated claim-decomposition methods.

D Decomposition Examples

E Russellian/Neo-Davidsonian In-context Learning Examples

This section presents manually decomposed DR-ND examples for in-context learning, covering biographical claims about attributes, events, dates, roles, relationships, career, office tenure, and professional teams.

E Russellian/Neo-Davidsonian In-context Learning Examples

  • The listed decomposition separates the acting debut, identifies The Moon is the Sun’s Dream as its film, gives its 1992 release year, and places small-role appearances after the debut.Source: E Russellian/Neo-Davidsonian… · p. 21
  • The producer-and-engineer example distinguishes occupations and success from the broader claim of working with artists and specific collaborators.Its claims separately identify Willie Nelson as an artist and state that he worked with him.Source: E Russellian/Neo-Davidsonian… · p. 21
  • The Gemini example splits Collins’s 1963 NASA selection and astronaut-group membership from his backup Command Module Pilot role and mission participation.The decomposition makes the selection date, group, role, and participation separate claims.Source: E Russellian/Neo-Davidsonian… · p. 21
  • The Bateman example expresses the number of short films written and directed as separate count claims and relates those films and acting roles to his feature debut.It also states that he is currently developing his feature debut.Source: E Russellian/Neo-Davidsonian… · p. 21
  • The Apollo 11 example separates Michael Collins’s birth date (October 31, 1930) and biographical status as a retired American astronaut and test pilot from his participation in Apollo 11 in 1969 and his Command Module Pilot role.Source: E Russellian/Neo-Davidsonian… · p. 21
  • Other examples break descriptions into claims about a series’s genre and premiere year or a musician’s listed styles and charity ambassadorship.The examples identify Love and Destiny as a romantic comedy series that premiered in 2019, and separately name musical styles and the charity Leonard Cheshire Disability.Source: E Russellian/Neo-Davidsonian… · p. 22
Loading 2403.11903v1…