PaperlayerView exact revision on arXiv

Source-linked AI summary

Large language model validity via enhanced conformal prediction methods

John J. Cherian, Isaac Gibbs, Emmanuel J. Candès

arXiv:2406.09714v2stat.MLcs.LGstat.ME

TL;DR

LLM conformal filters must balance conditional validity against retaining useful claims. The paper introduces level-adaptive guarantees and conditional boosting, reporting improved claim retention across datasets.

  • Problem

    Precise conditional guarantees can require larger prediction sets and smaller filtered outputs, creating a trade-off between validity and retaining useful claims.

    Source: §1.1 · p. 2

  • Method

    The paper introduces level-adaptive conformal prediction and conditional boosting, which differentiates through a conditional conformal procedure to improve claim scores.

    Source: §1.1 · pp. 2–3

  • Results

    39% vs. 24% mean claim retention across datasets for boosted versus unboosted scores, while level-adaptive prediction retains at least 70% of original claims in most MedLFQA examples.

    Source: §1.1 · p. 3

  • Takeaways & Limitations

    The methods mitigate the trade-off between conditional validity and useful filtered outputs, supporting practical conformal prediction for LLMs.

    Source: §1.1 · p. 2

    Theoretical guarantees assume i.i.d. prompt-response tuples; robustness to shifts beyond a pre-specified class requires additional research.

    Source: §5 · p. 11

Abstract

from arXiv · show

We develop new conformal inference methods for obtaining validity guarantees on the output of large language models (LLMs). Prior work in conformal language modeling identifies a subset of the text that satisfies a high-probability guarantee of correctness. These methods work by filtering claims from the LLM's original response if a scoring function evaluated on the claim fails to exceed a threshold calibrated via split conformal prediction. Existing methods in this area suffer from two deficiencies. First, the guarantee stated is not conditionally valid. The trustworthiness of the filtering step may vary based on the topic of the response. Second, because the scoring function is imperfect, the filtering step can remove many valuable and accurate claims. We address both of these challenges via two new conformal methods. First, we generalize the conditional conformal procedure of Gibbs et al. (2023) in order to adaptively issue weaker guarantees when they are required to preserve the utility of the output. Second, we show how to systematically improve the quality of the scoring function via a novel algorithm for differentiating through the conditional conformal procedure. We demonstrate the efficacy of our approach on biography and medical question-answering datasets.

1 Introduction

LLM responses can be unreliable, and existing claim-filtering methods provide only marginal guarantees while sometimes removing valuable text. The paper introduces conditional boosting and level-adaptive conformal prediction to improve conditional validity and claim retention.

1 Introduction

  • Prior filtering methods censor claims below a calibrated score threshold, but their guarantees hold only marginally and weak scores can remove valuable claims.LLMs can confidently hallucinate facts, and the resulting reliability concerns hinder deployment in settings such as legal work or customer service.Source: §1 · pp. 1–2

1.1 Summary of contributions

  • The paper addresses the apparent trade-off between stronger conditional guarantees and useful output with methods intended to mitigate the need for smaller filtered outputs.Prior work establishes that precise conditional guarantees in black-box uncertainty quantification require larger prediction sets.Source: §1.1 · p. 2
  • Conditional boosting differentiates through the conditional conformal procedure to discover improved claim-scoring functions.The paper proves the filtering cutoff is differentiable under mild assumptions and gives an efficient derivative computation method.Source: §1.1 · pp. 2–3
  • On each MedLFQA benchmark dataset [13], level-adaptive conformal prediction retains at least 70% of original output claims in most examples, with issued factuality probabilities from 50% to 85%; fixed-level prediction guaranteeing 90% correctness retains very little original output.Source: §1.1 · pp. 3–4
  • 39% vs. 24% mean claim retention across datasets for boosted scores versus the frequency-score baseline.Boosting learns a linear combination of four candidate scoring functions.Source: §1.1 · p. 3
  • Level-adaptive conformal prediction adjusts the claimed probability of correctness to prompt characteristics to target retaining at least 70% of the original claims.For the shingles example, the level is 63%, compared with 90% for fixed-level filtering.Source: §1.1 · p. 3

2 Background and related work

Split conformal prediction gives marginal coverage, while conditional conformal methods target guarantees across covariate groups. Conformal factuality instead filters LLM claims using a score threshold calibrated to retain claims with high probability of being error-free.

2.1 Conformal prediction with conditional guarantees

  • Marginal split-conformal coverage can leave performance systematically worse on safety-critical examples, while exact distribution-free covariate-conditional coverage requires vacuous sets.Gibbs et al. [10] target guarantees over a user-specified function class, which yields group-conditional coverage for subgroup indicator functions.Source: §2.1 · pp. 4–5
  • Gibbs et al. [10] estimate a high-probability score bound with augmented quantile regression and use a smaller randomized prediction set to achieve exact coverage.The construction assumes exchangeable calibration and test data and symmetric solutions to the optimization problem.Source: §2.1 · p. 5

2.2 Conformal factuality

  • Conformal factuality filters LLM responses to claims whose prompt-and-claim confidence scores meet a calibrated threshold, aiming for a high-probability error-free retained set.The method relies on exchangeability of the annotated prompt-response-claim tuples.Source: §2.2 · pp. 5–6

3 Methods

The methods extend conditional conformal control to broader alignment losses, adapt the target error level to each prompt, and optimize claim scores while retaining conditional constraints.

3.1 Generalization to alternative targets

  • The framework extends conditional conformal control to monotone losses for general alignment targets, assuming the method can abstain with zero loss.It supports high-probability guarantees across groups defined by prompt features, while the guarantee remains marginal over calibration data.Source: §3.1 · p. 6

3.2 Level-adaptive conformal prediction

  • Prompt-specific error levels use data-dependent pinball losses to adapt filtering, with α(·) estimated on one training-data fold and calibration run on another.The target can be sufficient claim retention, and the function class controls the adaptivity-computation trade-off.Source: §3.2 · p. 7

3.3 Conditional boosting

  • Conditional boosting tunes a parameterized claim-scoring function to maximize retained claims on held-out data under conditional conformal filtering.The filtering threshold depends on the scores through a quantile-regression problem.Source: §3.3 · pp. 7–8
  • For a linear function class, the threshold can be differentiated through a basis-defined linear system when the basis is locally stable and the score is differentiable.The stated technical condition is usually satisfied; small random perturbations can secure uniqueness when needed.Source: §3.3 · p. 8

4 Experiments

The experiments test conditional boosting and level-adaptive conformal prediction on synthetic regression, medical question-answering, and Wikipedia biographies. Results show improved conditional coverage in the synthetic and biography settings, with claim retention varying by topic popularity in the biography experiment.

4.1 Synthetic data

  • Conditional boosting retains near-perfect conditional coverage, whereas marginal boosting can produce shorter intervals but has extremely poor coverage at high conditional variance.The comparison uses the marginal boosting procedure of Stutz et al. [27].Source: Figure 3 · p. 8, §4.1 · p. 9
  • Adaptive α(Xn+1) brings interval lengths below the target maximum of 500 and closely tracks true conditional coverage, unlike many intervals at fixed α levels.The synthetic example uses the naive absolute-value score, which can yield very large intervals.Source: §4.1 · p. 9, Figure 4 · p. 9

4.2 Medical long-form question-answering

  • The experiment uses MedLFQA, which combines HealthSearchQA (n=3047), K-QA (n=1077), LiveQA (n=100), and MedicationQA (n=627); each prompt is queried with GPT-3.5-Turbo and parsed into self-contained sub-claims by GPT-4o.GPT-3.5-Turbo evaluates whether each claim is substantiated by the prompt’s pseudo-ground-truth response.Source: §4.2 · pp. 9–10

4.3 Wikipedia biographies

  • Conditional conformal prediction provides accurate coverage across Wikipedia page-popularity levels, retaining more claims for frequently viewed topics and fewer for rare topics.The comparison is against Mohri and Hashimoto's [21] marginal method; Figure 6 reports miscoverage and claim retention across view-count bins.Source: §4.3 · p. 11, Figure 6 · p. 11

5 Limitations

The paper identifies assumptions and scope boundaries for its guarantees, and the appendix details the randomized conditional-conformal procedure. The method assumes i.i.d. prompt-response tuples, while weak claim scores can undermine the usefulness of filtered outputs.

5 Limitations

  • The theoretical results assume i.i.d. prompt-response tuples; robustness to distribution shifts beyond a pre-specified class of covariate shifts requires further research.The authors note that changing user interactions can make test prompts incomparable to earlier prompts.Source: §5 · p. 11
  • A non-zero error allowance may suit tasks where errors are low-cost or high error rates appear unavoidable, but the authors caution it is not always desirable.Examples include vacation recommendations and lists of already-approved drugs that might be repurposed to treat COVID.Source: §5 · p. 11
  • When claim scores correlate weakly with factuality, filtered outputs may have vacuous nominal guarantees, so the wrapper's utility depends on score quality.The authors note that the wrapper can use new features that better predict hallucinations as they are developed.Source: §5 · pp. 11–12

A Additional details about conditional conformal

  • The appendix derives a randomized filtering threshold by randomizing over the boundary event, using an independent uniform variable U on [−α(Xi), 1 − α(Xi)].The appendix states coverage guarantees under assumptions including exchangeability, strong duality, and a symmetric algorithm for computing dual solutions.Source: A Additional details about conditional… · pp. 14–15

A.1 Proof of Theorem 3.1 and Theorem 3.2

The appendix proves the main theorems as special cases of Theorem A.1. Both proofs use finite-dimensional linear classes and establish strong duality through Slater's condition.

A.1 Proof of Theorem 3.1 and Theorem 3.2

  • The appendix presents Theorems 3.1 and 3.2 as corollaries of Theorem A.1.Source: §A.1 · p. 16
  • Theorem 3.1 is the Theorem A.1 special case with zero penalty, constant α, and a finite-dimensional linear class.The resulting primal and dual programs are linear, and the dual has the interior feasible solution η = 0, so Slater's condition gives strong duality.Source: §A.1 · p. 16
  • Theorem 3.2 is the Theorem A.1 special case with zero penalty and a finite-dimensional linear class; Slater's condition again gives strong duality.Source: §A.1 · p. 16

A.2 Interpretation of the guarantees in terms of covariate shifts

The appendix interprets the guarantees as robustness to covariate shifts that reweight prompt-response features while leaving the conditional label distribution unchanged. Under the stated conditions, Theorem 3.1's loss guarantee holds under such reweightings.

A.2 Interpretation of the guarantees in terms of covariate shifts

  • The guarantees can be interpreted as robustness to covariate shifts affecting PX but not PY | X.The features are functions of the prompt and response, and the shift reweights their distribution.Source: §A.2 · p. 16
  • Under Theorem 3.1's setting and assumptions, the loss guarantee remains 1 − α for every reweighting function f in F with positive, finite expectation under Q.Source: §A.2 · p. 16

B.1 Method for learning α(·)

The method selects a prompt-dependent level α(·) to meet a quality criterion, then estimates that level from held-out data using quantile regression.

B.1 Method for learning α(·)

  • The quality criterion Q takes a data point and cutoff and returns 1 or 0 according to whether the criterion is satisfied.Source: §B.1 · p. 16
  • Figure 7 plots optimal level thresholds for a hold-out set of 424 observations.Source: Figure 7 · p. 17
  • For each candidate α, the method evaluates the quality criterion on held-out data, records the largest α that meets it, and fits a quantile regression to predict that level.The estimated level is truncated above 0.1 to avoid vacuous guarantees.Source: §B.1 · p. 17

B.2 Dependence of the level-adaptive guarantee on the function class

The function class determines whether level-adaptive guarantees track the claimed level; the appendix also details conditional boosting and a faster approximation used in experiments.

B.2 Dependence of the level-adaptive guarantee on the function class

  • With a constant-function class, the guarantee controls only the expected error rate, not whether the nominal level is close to the true conditional coverage.Source: §B.2 · p. 17
  • Adding α(X) to the function class empirically aligns realized coverage with the nominal level, whereas a constant-only class produces spurious guarantees on the synthetic example.The paper leaves the trade-off between function-class complexity, calibration, and efficiency for future study.Source: §B.2 · p. 18, Figure 8 · p. 18

C Details of the Boosting Procedure

  • Conditional boosting differentiates the conditional-conformal cutoff under finite, unique, non-degenerate quantile-regression solutions to improve claim scores.The appendix derives the cutoff derivative and gives the boosting algorithm.Source: C Details of the Boosting Procedure · pp. 18–19
  • The experiments use a linear-time gradient approximation based on the quantile-regression fit to the first split alone, omitting test-point augmentation.This replaces the per-test-point basis computation, which otherwise scales quadratically in the size of the second split.Source: C Details of the Boosting Procedure · pp. 19–20

D.1 Additional results for Section 4.2

MedLFQA ablations show that boosting improves retention or strengthens guarantees at matched retention, while level-adaptive guarantees are approximately calibrated. Score choice also affects the guarantee-retention trade-off.

D.1 Additional results for Section 4.2

  • The level-adaptive estimate uses a 0.85-quantile regression over the function class and is truncated to the interval [0.1, 0.5].The estimate is based on a data split that identifies levels preserving at least 70% of claims.Source: §D.1 · p. 20
  • Boosted scores improve claim retention at fixed levels and yield stronger guarantees at the same retention when paired with level adaptation; reported levels approximately match realized error rates.Source: §D.1 · p. 20, Figure 9 · p. 20
  • Figure 9 compares retention across MedLFQA datasets and shows calibration, while Figure 10 isolates level-adaptive performance by claim-scoring approach without boosted scores.Source: Figure 9 · p. 20, Figure 10 · p. 21
  • Log-probability and frequency scores appear to offer the strongest guarantees, but lower retention with log-probability makes direct comparison difficult.Source: §D.1 · p. 21, Figure 10 · p. 21

D.2 Additional results for Section 4.3

On FActscore, conditional boosting improves retention or shifts issued guarantees upward, and the reported levels remain calibrated. The experiments also compare scoring approaches and test a stricter zero-false-claim target.

D.2 Additional results for Section 4.3

  • The biography experiments trim Wikipedia view counts at the 0.95 quantile to reduce optimizer issues from highly popular articles.Source: §D.2 · p. 21
  • The biography function class uses Wikipedia view counts, and level-adaptive fitting adds a quadratic term in α(·)−0.1 to model variation above the truncated floor.The adaptive level is estimated with a 0.75-quantile regression and truncated to at least 0.1.Source: §D.2 · p. 21
  • On FActscore, conditional boosting improves retention and/or shifts guarantees rightward, and the reported guarantees are well-calibrated; targeting zero false claims rarely permits nominal probabilities above 50%.The authors therefore motivate a target of at most three false claims instead.Source: §D.2 · p. 21, Figure 11 · p. 22, Figure 12 · p. 22
  • For FActscore, log-probability and frequency scores again appear to offer the strongest guarantees.Figures 13 and 14 compare the nominal guarantees across scoring methods.Source: §D.2 · p. 21

E.1 Claim scores

This section describes claim-scoring options and prompting procedures for parsing and assessing claims in responses.

E.1 Claim scores

  • The default frequency score averages five GPT-3.5-Turbo resampled-generation judgments per claim, coded −1 for contradiction, 0 for absence, and 1 for support.A claim present in three of five responses and absent in two receives 0.6.Source: §E.1 · pp. 22–23
  • Figures 13–14 display level-adaptive performance for each claim-scoring approach in isolation, without boosted scores.Source: Figure 13 · p. 22, Figure 14 · p. 23
  • Alternatives include self-reported correctness probabilities, original-response claim order, and internal token probabilities from T/F assessments; the first two follow Mohri and Hashimoto [21].Internal probabilities are computed by exponentiating the log probability of the returned token.Source: §E.1 · p. 23

E.2 Prompts

  • GPT-4o splits responses into sentences and independent facts, then uses BM25-matched examples from Min et al. [20] as in-context demonstrations to improve parsing.The prompt asks the model to break a sentence into independent facts.Source: §E.2 · p. 23
  • For annotation, MedLFQA responses or BM25-retrieved biography Wikipedia passages are paired with prior annotated claims; prompts assess claims as supported, contradicted, or unrelated.The annotation prompt uses the context and previous answers to assess each claim.Source: §E.2 · p. 23
  • Prompts elicit model-internal T/F judgments and self-reported correctness probabilities, with the latter requested to three significant figures.Source: §E.2 · pp. 23–24

Related papers

Loading 2406.09714v2…