Source-linked AI summary
DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation
Miriam Wanner, Benjamin Van Durme, Mark Dredze
TL;DR
Decomposition and decontextualization have been studied separately, although they interact in factuality verification and can create uncertainty about what a verifier should judge. This paper evaluates methods that combine them and introduces DNDSCORE, which verifies subclaims with contextual information. The authors report that contextualized verification changes judgments and handles context better than previous measures.
Problem
Prior work validated decomposition and decontextualization independently, without considering their interaction in a complete factuality-verification system.
Source: §1 · p. 2Method
The paper evaluates sequential and joint approaches to decomposition and decontextualization and introduces DNDSCORE, which verifies an atomic subclaim using its decontextualized form as context.
Source: §1 · p. 2Results
Takeaways & Limitations
The authors identify ambiguity handling as especially important when verifying generations with many entities.
Source: §8 · p. 9The evaluation is limited to generated biographies with corresponding Wikipedia source documents; production retrieval errors and other generation types are not evaluated.
Source: §8 · p. 9
Abstract
from arXiv · showhide
The decompose-then-verify strategy for verification of Large Language Model (LLM) generations decomposes claims that are then independently verified. Decontextualization augments text (claims) to ensure it can be verified outside of the original context, enabling reliable verification. While decomposition and decontextualization have been explored independently, their interactions in a complete system have not been investigated. Their conflicting purposes can create tensions: decomposition isolates atomic facts while decontextualization inserts relevant information. Furthermore, a decontextualized subclaim presents a challenge to the verification step: what part of the augmented text should be verified as it now contains multiple atomic facts? We conduct an evaluation of different decomposition, decontextualization, and verification strategies and find that the choice of strategy matters in the resulting factuality scores. Additionally, we introduce DnDScore, a decontextualization aware verification method which validates subclaims in the context of contextual information.
1 Introduction
Decomposition can strip context needed to verify atomic claims, while decontextualization can add facts that blur what should be checked. The paper evaluates ways to combine both steps and proposes DNDSCORE, which verifies subclaims with their context.
1 Introduction
- Decomposed subclaims can lose information needed for verification; “He was a director” does not identify who “He” refers to.The authors say these ambiguities can prevent successful verification and could lead to false positives.Source: §1 · p. 1
- Decontextualization can make a claim less atomic, obscure which added facts need verification, and produce redundant claims.Added information can also mask whether the original subclaim is correct, while the resulting claims may still depend on the original context.Source: §1 · p. 1
- Decontextualizing before decomposition can repeat context across claims and then leave each added subclaim without enough context.The authors give an example in which the same Quentin Tarantino claim appears for two sentences.Source: §1 · p. 1
- The paper evaluates decomposition followed by decontextualization, the reverse order, and a joint method called DnD.DnD jointly performs decomposition and decontextualization.Source: §1 · p. 2
- DNDSCORE uses both an atomic subclaim and its decontextualized form to specify what to verify and supply relevant context.The authors report that this contextualized method changes verification judgments and handles context better than previous factuality measures.Source: §1 · p. 2
2 Related Works
Prior work developed claim decomposition and decontextualization methods, along with related factuality-scoring approaches. The paper situates its work across these two research areas.
2.1 Decomposition
- FACTSCORE and related systems decompose claims for verification; VERISCORE also extracts only verifiable claims but does not use context in verification.Other approaches include claim splitting, question generation, and sub-sentence representations for fact retrieval or attribution.Source: §2.1 · pp. 2–3
- Prior decomposition research includes methods that select claims for uniqueness and informativeness, and reports a trade-off between accuracy and noise.The paper adopts the decomposition technique of Wanner et al. (2024), which is grounded in Russellian and Neo-Davidsonian theory.Source: §2.1 · p. 3
2.2 Decontextualization
- Decontextualization rewrites sentences to stand alone, using edits such as name completion, pronoun replacement, discourse-marker removal, scope bridging, and additions.The last two edit types can add text, including prepositional phrases or background information.Source: §2.2 · p. 3
- Work on decontextualization examines minimality and ambiguity in biographies, and also applies the technique to factuality annotation and long-form response evaluation.Other cited uses include making scientific-document snippets stand alone and generating summaries of long-form answers.Source: §2.2 · p. 3
3 Methods: Claim Decomposition and Decontextualization
The methods section adopts prior approaches to decomposition and decontextualization, then introduces DnD to perform both jointly. It also defines a measure for evaluating decomposition outputs.
3 Methods: Claim Decomposition and Decontextualization
- The paper adopts validated methods for decontextualization and decomposition, then proposes a prompt-based joint method called DnD.The section covers Molecular Facts, DR-ND, and DnD.Source: §3 · p. 3
3.1 Decontextualization: Molecular Facts
- Molecular Facts uses a two-step prompt: an LLM identifies ambiguities, then uses them to decontextualize the text.The authors adopt it because it was developed for human biographies and use GPT-4o mini for both steps.Source: §3.1 · p. 3
3.2 Decomposition: DR-ND
- The authors use DR-ND for all decompositions, drawing on prior work that found it highly atomic while faithful and coherent to the original claim.Their experiments use gpt-3.5-turbo-instruct, as in Wanner et al. (2024).Source: §3.2 · pp. 3–4
- Table 1 reports that the DnD and DR-ND subclaim sets are similar, as are the decontextualized sets, while the latter two methods contain more facts than the former.The table gives examples of each decomposition and decontextualization method.Source: Table 1 · p. 4
3.3 Joint Decontextualization and Decomposition: DnD
- Sequential decomposition and decontextualization can constrain decisions across steps and double LLM calls; DnD produces paired subclaims and decontextualized forms in one call.Each subclaim has a corresponding decontextualized claim.Source: §3.3 · p. 4
3.4 Decomposition Evaluation
- DECOMPSCORE measures the average number of supported subclaims produced per passage, indicating how many generated subclaims cohere with the original sentence.Support is highly correlated with NLI entailment.Source: §3.4 · p. 4
4 Methods: Claim Verification
The verification methods include FACTSCORE and a context-aware alternative, DNDSCORE. DNDSCORE uses the atomic subclaim to identify what to judge and the augmented claim and source document to provide context.
4 Methods: Claim Verification
- The paper considers FACTSCORE and a new method that uses decontextualization to better isolate the specific claim being verified.Both are decompose-then-verify methods.Source: §4 · pp. 4–5
4.1 FACTSCORE
- FACTSCORE is the percentage of decomposed claims supported by a trusted reference source, and the paper uses it as a baseline.The implementation uses Inst-LLAMA with retrieval and NPM.Source: §4.1 · p. 5
4.2 DNDSCORE
- DNDSCORE provides the source document, subclaim, and augmented claim, then asks the verifier to judge the specific subclaim using relevant context.The authors motivate this design because an augmented statement contains multiple atomic facts, making the verification target unpredictable.Source: §4.2 · p. 5
5 Results
The results compare sequential and joint decomposition–decontextualization strategies, then evaluate FACTSCORE against contextualized DNDSCORE. The scores vary by strategy and verification method.
5 Results
- The evaluation uses fixed biographies generated by 12 language models, applying sequential DR-ND and Molecular Facts methods and one LLM call for joint DnD.The biographies were released by Min et al. (2023); the authors did not generate additional biographies or modify the existing ones.Source: §5 · p. 5, §5.1 · p. 5
5.1 Methods of Decomposition and Decontextualization
- DECOMPSCORE remains high across methods, with only a small increase after adding decontextualized information.The authors suggest the additions are supported by the original claim or too minimal to change the entailment judgment.Source: §5.1 · p. 5, Table 2 · p. 5
- In the illustrated example, DnD’s decomposition and subclaims are nearly identical, while decontextualize-then-decompose produces more subclaims than decomposition-then-decontextualization.The example only replaces a pronoun and adds no external information.Source: §5.1 · p. 5
5.2 Fact Verification
- DNDSCORE averages higher than FACTSCORE; Decontextualization → Decomposition reaches 61.51% DNDSCORE, while DnD’s subclaim and decontextualized sets score 51.60%.The compared verification setups use the same number of subclaims by design.Source: Table 3 · p. 6, §5.2 · p. 6
- Decomposition → Decontextualization has almost 13% higher FACTSCORE than decomposition-only; DnD’s corresponding claim sets have similar FACTSCOREs with fewer subclaims.Decomposition-only averages 33.00% FACTSCORE and 43.48 subclaims per paragraph.Source: §5.2 · p. 6
6 Analysis
The analysis measures how context changes verification judgments and examines examples where DNDSCORE handles added context. It also finds that decontextualized claims can be redundant after filtering.
6.1 Verification Changes: Quantitative Evaluation
- 19.11% of DnD subclaim and decontextualized-claim FACTSCORE judgments change; 16.25% switch false-to-true, and 48.52% of those involve pronoun replacement.Only 3.26% switch true-to-false, and 11.82% of those involve pronoun replacement.Source: §6.1 · p. 7
- 16.97% of DnD subclaim judgments change between FACTSCORE and DNDSCORE; 16.50% switch false-to-true, compared with 0.48% true-to-false.The authors report that added context rarely changes judgments to false and can add background or pronoun replacement that makes claims easier to verify as correct.Source: §6.1 · p. 7
6.2 Verification Changes: Qualitative Evaluation
- Examples show added context disambiguating Prince Daniel and replacing “She” with Susan Sarandon; DNDSCORE judges both claims true.Other examples switch from true to false, and DNDSCORE correctly judges those claims false.Source: §6.2 · p. 7
- When decontextualization adds false information about Fuerza Guerrera, DNDSCORE still judges the atomic subclaim true using context without letting it overshadow the claim.FACTSCORE instead judges the decontextualized claim false because the added name refers to a different Mexican wrestler.Source: §6.2 · pp. 7–8
6.3 CORE for Subclaim Deduplication
- CORE filtering leaves an average of 19–25 claims across methods, and removes more claims from DnD’s decontextualized set than its subclaim set.The authors describe the decontextualized set as more redundant because claims include information from other subclaims.Source: §6.3 · p. 8
- Decontextualize-then-decompose retains the most subclaims after filtering, as repeated essential context can survive when wording differs slightly.FACTSCORE changes by less than two percent for the other deduplicated sets, but drops almost four percent for this method.Source: §6.3 · p. 8
7 Conclusion
The paper introduces DnD for extracting subclaims with decontextualized forms and DNDSCORE for verifying those subclaims with their context; it reports that contextualized verification handles context better than previous factuality measures.
7 Conclusion
- DnD is a prompt-based method for extracting subclaims alongside corresponding decontextualized forms.Source: §7 · p. 8
- DNDSCORE verifies subclaims using their paired decontextualized forms as context.Source: §7 · p. 8
- The authors demonstrate differing verification judgments and report that DNDSCORE handles context better than previous factuality measures.Source: §7 · p. 8
8 Limitations
The evaluation is bounded to generated biographies with available Wikipedia sources, static reference documents, and English text; the authors also identify claim types outside DNDSCORE’s intended scope.
8 Limitations
- The evaluation considers fact verification for generated biographies and does not establish DNDSCORE’s use beyond that setting.Source: §8 · p. 9
- DNDSCORE is not intended for debatable claims such as opinions; the authors describe filtering to verifiable claims as a possible integration.Source: §8 · p. 9
- The study uses static reference documents, while retrieval from decontextualized subclaims is future work for extending the method to arbitrary rather than predetermined-topic generations.Source: §8 · p. 9
- The evaluation assumes the correct Wikipedia sources are available; the authors hypothesize retrieval errors could increase verification confusion, and note that multi-entity outputs may also increase confusion.Source: §8 · p. 9
- The work evaluates English only, although the authors expect the results to hold in other languages.Source: §8 · p. 9
9 Ethics Statement
The ethics statement flags hallucinated information as a risk of decomposition and decontextualization; the appendix describes the prompts used for these methods and reports the computational estimate.
9 Ethics Statement
- Decomposition and decontextualization may inject information absent from the original claim, so the authors say untrue generations need evaluation and warrant caution.Source: §9 · p. 9
A.1 Decomposition Prompt
- The decomposition experiments use Wanner et al.’s DR-ND prompt, which uses dynamically retrieved in-context decompositions.Source: §A.1 · p. 11
A.2 Decontextualization Prompt
- The decontextualization setup uses Molecular Facts, a two-step prompt that identifies claim ambiguities before using them to decontextualize the claim.Source: §A.2 · p. 11
A.3 DnD Prompt
- The DnD prompt combines Molecular Facts ambiguity criteria with decomposed in-context examples from DR-ND to extract subclaim–decontextualized-claim pairs.Source: §A.3 · p. 11
A.4 FACTSCORE Prompt
- The FACTSCORE prompt supplies a reference document and asks whether an atomic fact is true or false.Source: §A.4 · p. 11
A.5 DNDSCORE Prompt
- The DNDSCORE prompt adapts FACTSCORE by providing the decontextualized claim as context for judging the atomic claim against a reference document.Source: §A.5 · p. 11
A.6 Compute
- The decomposition and decontextualization experiments ran on a Quadro RTX 6000 GPU cluster and were estimated to take around 400 GPU-hours.Source: §A.6 · p. 11
B.1 FACTSCORE Results
The appendix points to separate full results tables for FACTSCORE and DNDSCORE, each reporting results by language-model split.
B.1 FACTSCORE Results
- Table 8 contains the full FACTSCORE results for each language-model split.Source: §B.1 · p. 11
- The appendix presents split-specific full results separately for FACTSCORE and DNDSCORE.Source: §B.1 · p. 11, §B.2 · p. 11
B.2 DNDSCORE Results
- Table 9 contains the full DNDSCORE results for each language-model split.Source: §B.2 · p. 11
B.3 DECOMPSCORE Results
DECOMPSCORE remains high despite additional information in decontextualized subclaims. Across split-level FACTSCORE and DNDSCORE tables, language-model rankings stay consistent while scores differ.
B.3 DECOMPSCORE Results
- Tables 5–7 present the DnD method’s extracted subclaims alongside their corresponding decontextualized subclaims.Each table contains one of three parts of the extraction method.Source: Table 5 · p. 12, Table 6 · p. 13, Table 7 · p. 14
- The language-model factuality ranking stays consistent across splits, although FACTSCORE and DNDSCORE scores differ.Table 8 reports FACTSCORE and Table 9 reports DNDSCORE; both tables describe results aggregated in Table 3.Source: Table 8 · p. 15, Table 9 · p. 16
- DECOMPSCORE remains high despite additional new information for decontextualized subclaims.Table 10 reports results aggregated in Table 2.Source: Table 10 · p. 17