Source-linked AI summary
Molecular Facts: Desiderata for Decontextualization in LLM Fact Verification
Anisha Gunjal, Greg Durrett
TL;DR
Atomic facts may lack context needed for reliable interpretation, while larger chunks are difficult to fact-check. The paper defines molecular facts and a two-stage generation method, finding that they balance minimality and verification accuracy for ambiguous entities. It also reports that decontextualization can produce non-minimal claims, with scope limited by its biography-focused evaluation.
Problem
Atomic claims may lack the context needed for correct interpretation, making the granularity of fact-checking a central challenge.
Source: §1 · pp. 1–2Method
Results
Molecular facts balance minimality with fact-verification accuracy for ambiguous entities, while human evaluation finds non-minimal cases in 1.7% to 9.6% of decontextualizations.
Source: §4.5 · p. 6, §8 · pp. 8–9Takeaways & Limitations
Molecular facts offer a balance between minimal claims and accurate fact verification in the evaluated setting of ambiguous entities.
Source: §8 · pp. 8–9The main evaluation is on English-language biographies and focuses on entity ambiguity, leaving other datasets and ambiguity types for future work.
Source: Limitations · p. 9
Abstract
from arXiv · showhide
Automatic factuality verification of large language model (LLM) generations is becoming more and more widely used to combat hallucinations. A major point of tension in the literature is the granularity of this fact-checking: larger chunks of text are hard to fact-check, but more atomic facts like propositions may lack context to interpret correctly. In this work, we assess the role of context in these atomic facts. We argue that fully atomic facts are not the right representation, and define two criteria for molecular facts: decontextuality, or how well they can stand alone, and minimality, or how little extra information is added to achieve decontexuality. We quantify the impact of decontextualization on minimality, then present a baseline methodology for generating molecular facts automatically, aiming to add the right amount of information. We compare against various methods of decontextualization and find that molecular facts balance minimality with fact verification accuracy in ambiguous settings.
1 Introduction
Atomic claims can lose the context needed for reliable attribution, while adding too much context can make claims harder to verify. The paper proposes molecular facts to balance standalone clarity with minimality and evaluates this trade-off in fact verification.
1 Introduction
- Atomic claims can mislead verification when removed from context, including by merging information about similarly named entities or matching evidence that does not support the intended claim.The paper notes that current evaluation methods struggle with ambiguities in atomic claims.Source: §1 · p. 1
- Molecular facts balance standalone specificity with minimality, adding only enough context to make a claim unambiguous without complicating verification.The paper frames decontextualization as adding context while retaining the original meaning.Source: §1 · p. 2
- In a synthetic fact-checking experiment, an average of 6% of claims may pose error-localization problems when decontextualization adds unsupported nuances.The introduction notes this could affect a large fraction of responses with five sentences and three claims per sentence.Source: §1 · p. 2
- Molecular fact generation balances accuracy under ambiguous entity references with minimality of claims.The evaluation uses a dataset of fact-checking with ambiguous entity names.Source: §1 · p. 2
- Molecular claims are reported as more performant and minimal for long-form generations than existing decontextualization methods.Source: §1 · p. 2
2 Desiderata for Decontextualization
The section argues that useful fact-checking units must be interpretable on their own without unnecessary added information. It defines criteria for that balance and sets out how claims, evidence, and human labels are used for fact-checking.
2 Desiderata for Decontextualization
- Different decomposition methods produce varying numbers of subclaims, illustrating that an atomic fact is not rigorously defined from a semantic standpoint in prior NLP work.Reported macro-averages range from 20.2 to 32.9 subclaims per biography across two methods.Source: §2 · p. 2
2.1 Desiderata
- Decontextuality requires a standalone claim to preserve the original truth-conditional meaning by specifying relevant entities, events, and context.The paper describes this as uniquely specifying the interpretation of the claim.Source: §2.1 · p. 3
- Minimality favors the decontextualized statement supported by the largest set of evidence documents, avoiding details that unnecessarily narrow support.For example, a broadly recognizable description may be supported by more documents than a highly specific one.Source: §2.1 · p. 3
2.2 Task Definition: Fact-checking LLMs
- The fact-checking task compares model judgments on decontextualized claims against human SUPPORTED or NOT_SUPPORTED labels and their associated evidence documents.The framework augments each atomic claim to a molecular claim before making supportedness predictions.Source: §2.2 · pp. 3–4
3 Method: Producing Molecular Facts
The method generates molecular facts through two stages: identifying ambiguity in a claim’s subject, then using the resulting criteria and claim context to disambiguate it. Its stated assumption is that ambiguity is typically restricted to one entity.
3 Method: Producing Molecular Facts
- The method assumes ambiguity is typically restricted to a single entity, as in the datasets studied in the paper.Source: §3 · p. 4
- The method first identifies a claim’s primary subject and whether it may refer to multiple entities, then uses the identified disambiguation criteria and claim context to generate a molecular fact.The criteria can be absent when the subject is unambiguous or can specify a type such as profession, birth year, or location.Source: §3 · p. 4
3.1 Baselines
- The baselines include atomic claims, simple decontextualization using the response as context, and SAFE decontextualization using a revision prompt.Baseline outputs are generated with gpt-4-turbo-2024-04-09.Source: §3.1 · p. 4
- MOLECULAR-DECONTEXT applies the paper’s two-stage process to identify disambiguation criteria and then decontextualize the atomic claim.Source: §3.1 · p. 4
4 Experiment: Minimality & Localization
This experiment tests how adding context to atomic claims affects minimality and error localization, then measures how often flagged cases are genuinely non-minimal. It also examines how much context is needed to verify claims about ambiguously named people.
4.1 Controlled Dataset Construction
- The experiment uses 812 Factcheck-Bench claims with human-annotated factuality labels, extracting atomic facts and revising them with SIMPLE-DECONTEXT and SAFE-DECONTEXT.The dataset consists of long-form ChatGPT responses.Source: §4.1 · p. 5
- From revisions entailing a core and at least one auxiliary atomic fact, the authors generated evidence supporting response facts while withholding a sampled banned fact.This construction tests whether a multi-fact revision can be marked unsupported when its extra fact lacks evidence.Source: §4.1 · p. 5
4.2 Evaluation Criteria
- On E′, roberta-large from AlignScore serves as Check() to measure supported-to-not-supported label changes after decontextualization.Preliminary analysis with GPT-4 produced very similar results.Source: §4.2 · p. 5
4.3 Results
- Table 1 reports 8.49% for SAFE-DECONTEXT and 23.39% for SIMPLE-DECONTEXT in claims included in E′; auto non-minimal cases occur at 3.94%–13.42%.The pipeline flags revisions when the evidence supports the core fact but not the decontextualization and banned fact.Source: §4.3 · p. 6, §4.2 · p. 6
4.4 Human Evaluation
- Human review found that 43.8% of SAFE-DECONTEXT auto flags were truly non-minimal, representing 1.7% of the dataset; for SIMPLE-DECONTEXT, the figures were 72.5% and 9.6%.Some revisions flagged as non-minimal instead add necessary context to make an atomic claim standalone.Source: §4.4 · p. 6
4.5 Conclusion: Problem of Non-minimality
- 1.7%–9.6% of decontextualizations were judged genuinely non-minimal, and the authors warn that added facts can impair error localization.The paper notes that even a low fraction may produce several localization problems in a response containing dozens of atomic facts.Source: §4.5 · p. 6
5 Experiment: Ambiguous Biographies
The ambiguous-biography experiment evaluates claim revisions against evidence for similarly named entities, considering both accuracy and minimality. The reported results compare the revisions’ accuracy with atomic claims and examine their length and human-rated minimality and ambiguity.
5 Experiment: Ambiguous Biographies
- Table 3 evaluates each baseline’s claim revisions using accuracy measured by Check(Di, m) against the ambiguous document set associated with the claim’s main entity.Source: Table 3 · p. 7
- The dataset contains claims requiring entity specification or noun completion and multiple same-name entities requiring disambiguation by location, occupation, or time period.The study samples 726 human-annotated SUPPORTED or NOT_SUPPORTED claims and compares method revisions with the human labels.Source: §5 · p. 7
- Evaluation requires both agreement with the human SUPPORTED/NOT_SUPPORTED label and support from the correct entity’s evidence document.The accuracy measure applies Check() to each ambiguous subject-related document and the revised claim.Source: §5 · p. 7
- Human analysis found SIMPLE-DECONTEXT had a larger fraction of non-minimal and ambiguous claims than MOLECULAR-DECONTEXT; SAFE-DECONTEXT was more minimal than SIMPLE but struggled with ambiguity.The human analysis sampled 25 claims per baseline.Source: Table 5 · p. 7
6 Results: Ambiguous Biographies
In ambiguous biographies, decontextualization improves verification accuracy over atomic claims, while molecular claims balance accuracy with minimality. SAFE-DECONTEXT is more minimal than SIMPLE-DECONTEXT but struggles with ambiguity.
6 Results: Ambiguous Biographies
- All decontextualization methods achieve higher accuracy than atomic claims across every subset, and all three lie on a length–accuracy Pareto frontier.Molecular and simple methods modify claims more and produce longer sentences on average than SAFE-DECONTEXT; greater modification generally accompanies higher accuracy.Source: §6 · p. 7
- SAFE-DECONTEXT is more minimal than SIMPLE-DECONTEXT, but it struggles with ambiguity.Source: §6 · p. 8
- Molecular claims balance minimality with ambiguity removal and improve accuracy; they are more minimal than SIMPLE-DECONTEXT and more performant than SAFE-DECONTEXT on ambiguous generations.The human evaluation sampled 25 claims to assess minimality.Source: §6 · p. 8
- All decontextualization methods have lower error rates than atomic claims, with errors analyzed by evidence count and whether the evidence matches the correct entity.Source: §6 · p. 8
7 Related Work
Prior work spans factuality-verification pipelines, claim decomposition, and decontextualization methods. The paper situates its contribution in the decontextualization step and the sensitivity of verification evaluation to claim decomposition.
7 Related Work
- Factuality-verification pipelines decompose generations into claims and may include extraction, revision, checkworthiness, evidence retrieval, and verification.The cited work also uses recently released datasets of generation errors.Source: §7 · p. 8
- Related approaches include citation generation, retrieval-based revision, retrieval-supported explanations, alignment scoring, and sentence-level factuality labels.Source: §7 · p. 8
- Prior decontextualization work covers anaphora resolution, question answering, and extract-then-decontextualize summarization; fact verification uses it or claim revision before evidence retrieval.Other work uses decontextualization to resolve ambiguity, a focus shared by this paper.Source: §7 · p. 8
8 Conclusion
The paper introduces molecular facts and defines decontextuality and minimality as desiderata for decontextualization in LLM fact verification. Its method improves precision for claims about ambiguous entities while balancing minimality and accuracy.
8 Conclusion
- The paper introduces decontextuality and minimality as criteria for decontextualization and molecular facts as a proposed approach to LLM fact verification.Source: §8 · p. 8
- Molecular facts improve fact-verification precision for claims about ambiguous entities.Source: §8 · pp. 8–9
- Molecular facts balance maintaining minimality with fact-verification accuracy.Source: §8 · p. 9
Limitations
The section records evaluation and scope limitations, alongside appendix material on related work and criteria for judging minimality and ambiguity.
Limitations
- Evaluation centers on English-language biographies and high-performing LLMs; end-to-end testing, decomposition errors, and broader domains and languages remain outside the study.Synthetic minimality experiments also use English outputs and GPT-4 parametric knowledge, which the authors say may limit topic and domain breadth.Source: Limitations · p. 9
A Prompts
- Appendix A provides prompts for molecular-fact generation, SIMPLE-DECONTEXT, and controlled evidence generation.The prompts cover ambiguity detection and disambiguation, molecular decontextualization, and the minimality experiment.Source: A Prompts · p. 11
B Additional Related Work
- Summarization work uses weighted Summarization Content Units, parser-generated Semantic Triplet Units, Atomic Content Units, and sentence propositions for evaluation [Nenkova and Passonneau, 2004; Zhang and Bansal, 2021; Liu et al., 2023b; Chen et al., 2023b].These approaches represent or evaluate content units in generated summaries.Source: B Additional Related Work · pp. 11–12
- Decontextualization resolves missing context while preserving meaning, while specificity concerns whether a sentence can stand alone without ambiguity.Specificity includes identifiable references, assessable truth, and fully expressed participant and event information.Source: B Additional Related Work · p. 12
C Human Annotation Criteria for Categorizing the Non-minimal Subset
- The annotation treats added information as minimal when it is necessary to make the atomic claim standalone, including context that resolves ambiguity.The examples distinguish necessary disambiguating context from an entailment error in which a claim adds unsupported information.Source: C Human Annotation Criteria for… · p. 12
D Human Analysis Criteria for Categorizing Minimality and Ambiguity
- Human analysis labels a decontextualization non-minimal if it adds beyond standalone needs, or ambiguous if it fails to clarify potentially different referents.A claim is minimal when neither condition applies; excess information can potentially impair error localization.Source: D Human Analysis Criteria for… · p. 12
E Models, Datasets and Computation Cost
This section reports the models and computation used for the experiments, along with the datasets and software resources.
E Models, Datasets and Computation Cost
- The ambiguous-biography baselines and outputs used gpt-4-turbo-2024-04-09, while gpt-3.5-turbo evaluated them through FActScore; generation temperature was 0.75 and the experiment cost approximately $120.For minimality experiments, gpt-3.5-turbo extracted atomic facts and gpt-4-turbo-2024-04-09 handled decontextualization and generation, at around $100; AlignScore and WANLI were used for evaluation and entailment computation.Source: E Models, Datasets and Computation Cost · p. 12
- The authors used ChatGPT to improve writing formatting and generate boilerplate code for figure generation.Source: E Models, Datasets and Computation Cost · p. 12
- The experiments use Wang et al.'s open-source dataset and FactScore code, plus AlignScore code, with Apache 2.0 and MIT licenses respectively.The dataset is published under Apache 2.0; FactScore and AlignScore are published under MIT licenses.Source: E Models, Datasets and Computation Cost · pp. 12–13
F Controlled Experiment on Minimality Generation Details
The section specifies filtering in the controlled minimality experiment and examines decontextualization at ambiguous-biography entity switches, where molecular claims show relative robustness.
F Controlled Experiment on Minimality Generation Details
- Before filtering claims supported by more than two atomic facts, the procedure excludes cases where one atomic fact is a substring of another.Source: F Controlled Experiment on Minimality… · p. 13
- Evidence-generation filtering removes cases where a key fact resembles the banned fact or where the generated evidence article supports that banned fact.The procedure samples key facts excluding the banned fact before generating the evidence article.Source: F Controlled Experiment on Minimality… · p. 13
G Remaining Challenges
- The entity switch point is where a claim draws on entity B although preceding sentences refer to entity A, making decontextualization important for identifying its referent.Source: G Remaining Challenges · p. 13
- Molecular claims improve accuracy by 3.5% over SAFE-DECONTEXT at the entity switch point, where basic decontextualization methods perform worst.The results are reported for baselines tested with gpt3.5-turbo.Source: G Remaining Challenges · p. 13
- Molecular claims generated with weak human-in-the-loop supervision and gold Wikipedia disambiguations significantly outperform automated decontextualization methods at the switch point.Source: G Remaining Challenges · p. 13
- The accompanying prompts cover disambiguation selection, ambiguity detection, molecular and simple decontextualization, and controlled evidence generation that incorporates key facts while avoiding banned facts.Source: Figure 5 · p. 13, Figure 6 · p. 14, Figure 7 · p. 15