Language Models with Conformal Factuality Guarantees

1 Introduction

• LLM hallucinations make factuality and correctness an active research concern, and some domains require known, user-specified correctness probabilities.The paper notes that model complexity and opacity make precise performance guarantees challenging.Source: §1 · p. 1
• Prior conformal methods have had limited success with LMs, and the paper identifies a gap in exact guarantees for open-ended outputs.The cited challenges include intractable scoring of the output space and confidence sets too large to use.Source: §1 · p. 1
• Conformal factuality defines each LM output’s uncertainty set as the statements that entail it, linking output correctness to set coverage.The sets are represented implicitly, avoiding instantiation of uncountably large sets while providing interpretable guarantees directly on outputs.Source: §1 · p. 2
• The method forms a chain of increasingly less-specific outputs and uses conformal prediction to select a level with high-probability correctness.Its implementation decomposes outputs into subclaims, scores them, and removes claims according to uncertainty.Source: §1 · p. 2
• 30% → 80% on FactScore, 78% → 93% on NaturalQuestions, and 75% → 95% on MATH, while retaining most output subclaims.The approach is demonstrated on GPT-4 outputs and is described as supporting any target correctness level.Source: §1 · p. 2

2 Related Work

• Factuality methods include retrieval augmentation, training changes, ensembling, and decoder modification; this work is a decoding-time method aiming for probabilistic correctness guarantees.The paper distinguishes its goal from methods that seek to reduce fabrication without precise guarantees.Source: §2 · p. 3
• Uncertainty estimators such as SelfCheckGPT and prompting-based estimates are complementary: conformal factuality uses them as a meta-algorithm for guaranteed outputs.The authors expect better uncertainty quantification to improve performance and say their algorithms work with any such method.Source: §2 · p. 3
• Conformal prediction provides marginal coverage without strong modeling assumptions, but its language-model applications have been limited.The paper describes successful confidence-set use for black-box models such as deep neural networks.Source: §2 · p. 3
• LM conformal methods use token sets, multiple-choice reductions, or sequence-level sets; conformal factuality instead returns a natural-language sequence that implicitly represents a confidence set.The paper notes token coverage does not translate to sequence correctness, multiple-choice methods are constrained to QA-style domains, and direct sequence sets require approximations.Source: §2 · pp. 3–4
• Compared with Angelopoulos et al. (2023), this work defines sets through entailment, avoids enumerating hierarchy edges, and guarantees output correctness rather than risk.The cited work uses hierarchical labels and an intermediate tree node to represent a confidence set of leaves.Source: §2 · p. 4

3 Preliminaries

• For any user-specified α, the goal is to guarantee that an LM output is correct with probability at least 1 − α over a distribution P.Correctness is defined relative to a reference response or knowledge source through entailment.Source: §3 · p. 4
• An LM output’s entailment set serves as an uncertainty set: containing the reference response in that set is equivalent to the output being correct.This connects the correctness target to the coverage guarantee of conformal prediction.Source: §3 · p. 4
• Split conformal prediction uses exchangeable calibration data to select a set that meets a target coverage constraint.The standard construction described here uses a sequence of nested sets.Source: §3 · p. 4
• The LM construction forms a sequence of outputs and associated entailment sets, without requiring those sets to be nested.Nested sets can correspond to increasingly generic outputs, but enforcing this property may be difficult for a language model.Source: §3 · p. 5
• The back-off function removes unreliable claims as its threshold rises, with a sound function eventually abstaining by returning an empty output.The conformal procedure applies to a base LM, target error rate, back-off function, and calibration dataset to produce an α-conformally factual LM.Source: §3 · p. 5

4 Theoretical analysis

• The guarantee applies to sound back-off functions and exchangeable calibration and test examples.Soundness requires that sufficiently high thresholds accept no claims.Source: §4 · p. 6
• For α ∈ [1/(n+1), 1], Algorithm 1 achieves α-conformal factuality under the theorem’s assumptions.When the entailment sets are nested, the theorem also gives an upper bound of 1 − α + 1/(n+1).Source: §4 · p. 6
• The guarantee does not ensure useful outputs: the selected threshold may produce uninformative or empty sequences.The implementation that follows aims to keep the threshold small.Source: §4 · p. 6
• The accompanying example depicts an output decomposed into three subclaims: Lincoln’s birthplace, notable job, and what he was best known for.The figure illustrates subclaims in the back-off construction, rather than a quantitative result.Source: Figure 2 · p. 7

5 Implementation of Ft via sub-claims

• The implementation extracts subclaims, scores their factuality, retains those scoring at least t, and merges the retained claims into an output.A larger score is intended to indicate a higher probability that a subclaim is factual; sufficiently large thresholds accept no claims, making the back-off function sound.Source: §5 · p. 7
• Under Assumption 5.1, entailment checks can be limited to the original subclaims, substantially reducing annotation effort.This preserves guarantees when the merger does not add or remove subclaims and thus preserves entailment relations.Source: §5 · pp. 7–8
• Assumption 5.1 also yields the nested-set condition needed for the upper bound in Theorem 4.1.Removing accepted subclaims enlarges the corresponding intersection of entailment sets.Source: §5 · p. 8

5.1 Partial entailment

• The partial-entailment extension targets an acceptable fraction of factual subclaims rather than requiring full factuality.It modifies the score definition to account for the entailed fraction of accepted subclaims.Source: §5.1 · p. 8
• Empirically, allowing partial factuality yielded small gains, and its guarantee concerns the subclaims rather than the merged output.Providing the subclaims alongside the output can enable users to verify the merged output’s correctness.Source: §5.1 · p. 9

6.1.1 Models

• For the proof-of-concept implementation, GPT-4 supplies the base outputs and prompted sub-claim separator and merger.Source: §6.1.1 · p. 9

6.1.2 Sub-claim scoring functions

• Scoring options include random and output-order baselines, GPT-4 confidence, frequency across five alternate outputs, and an oracle based on true entailment.The oracle is not usable without ground truth and is inefficient when entailment must be checked by a human.Source: §6.1.2 · pp. 9–10
• Results are reported at the sub-claim level because there is no agreed-upon separator for measuring what fraction of an output sequence is factual.Source: §6.1.2 · p. 9

6.1.3 Datasets and Annotation

• The study uses FActScore biographies, simplified-training-set Natural Questions answered in long form, and MATH word problems whose reasoning steps are treated as sub-claims.GPT-4 generations on the first 50 inputs per dataset were manually annotated with four labels; Factual and Subjective were treated as entailed.Source: §6.1.3 · p. 10

6 Experiments

• The experiments evaluate whether the guarantees hold empirically and whether outputs remain useful, using question-answering and reasoning tasks.Source: §6 · p. 9, §6.2 · p. 10

6.2.1 Empirical factuality

• With sub-claims frequency scoring, results showed very tight factuality control, guaranteed marginally over calibration sets and test points; additional high-probability calibration-set guarantees added empirical-factuality variation (standard deviation ∼0.09).Factuality was measured on the second half after using the first half.Source: §6.2.1 · p. 11

6.2.2 Utility

• Frequency scoring raises FActScore factuality from ∼25% to ∼80% while retaining around half the claims; NQ and MATH each gain ∼15% after removing about one-quarter and ∼10%, respectively.Figure 4 summarizes factuality against the fraction of sub-claims removed across datasets.Source: §6.2.2 · p. 11, Figure 4 · p. 11
• At α = 0.2, most FActScore examples lose about half their claims yet remain informative; examples also show removal of false claims and an incorrect MATH step.In the MATH example, the result is a partial proof with one step left for the user.Source: §6.2.2 · p. 12, Table 1 · p. 12

• Conformal factuality connects conformal prediction and language modeling and yields a practical algorithm for guarantees on LM outputs.Source: §7 · p. 12
• The demonstrated guarantees are restricted to a prespecified distribution P and are marginal with respect to calibration and test-data draws.Source: §7 · p. 13

Limitations

• Coverage is average over inputs, so factuality for a particular domain or user may differ significantly from the target when calibration pools tasks or users.The guarantee is not conditional on each input.Source: Limitations · p. 13
• Coverage is also marginal over calibration-set draws, so repeatedly reusing one threshold from a small fixed calibration set may deviate from target coverage.Source: Limitations · p. 13
• Under distribution shift, a threshold computed from past calibration data may fail to maintain desired coverage, potentially losing the factuality guarantee if inputs change drastically.Source: Limitations · p. 13
• The authors identify split-conformal extensions as possible approaches to these challenges and expect better uncertainty quantification to improve experimental results.They describe the work as a step toward guaranteeing LM factuality.Source: Limitations · p. 13

A Prompts used in experiments

• Table 2 presents prompts for sub-claim separation, merging, GPT-4 confidence scoring, and frequency scoring; frequency scoring evaluates five GPT-4 alternates generated at temperature 1.0.Source: Table 2 · p. 17

B Partial factuality

• Under partial factuality, the relaxed entailment conditions use a = 0.7 for FActScore, a = 0.9 for NQ, and a = 0.8 for MATH, with a lower percentage of sub-claims removed.Figure 6 plots factuality against the percentage removed across datasets; tick marks represent target α and error bars show standard error.Source: B Partial factuality · p. 17, Figure 6 · p. 18

C Ranking-based scoring functions

• Ranking-based scoring converts a sub-claim scoring function into a ranking intended to remove a fixed percentage of facts, which may suit roughly uniform output modification.The ranking score is divided by the number of sub-claims to account for varying-length outputs.Source: C Ranking-based scoring functions · p. 18
• Ranking-based versions of the scoring functions remove a higher percentage of sub-claims and have smaller standard deviations.Figure 7 plots factuality against the percentage removed for these ranking-based functions across datasets.Source: C Ranking-based scoring functions · p. 18, Figure 7 · p. 18

D More conformal factuality output examples

• The appendix provides additional conformal-factuality output examples for FActScore, NQ, and MATH in Tables 3, 4, and 5, respectively.Source: D More conformal factuality output… · p. 18

• The remaining sub-claim scoring functions are evaluated using the same procedure, with results presented in Figure 3.Source: E Empirical factuality for all scoring… · p. 18
• Tables compare original GPT-4 outputs with conformal-factuality outputs on FActScore, NaturalQuestions, and MATH, highlighting incorrect claims in red.The captions specify α = 0.2 for FActScore and α = 0.1 for NaturalQuestions and MATH, all with frequency scoring.Source: Table 3 · p. 19, Table 4 · p. 20, Table 5 · p. 21
• Figure 8 presents empirical factuality plots for all scoring functions.Source: Figure 8 · p. 22