PaperlayerView exact revision on arXiv

Source-linked AI summary

Truth in Contrast: Calibrated Factual Evidence from Language Model Likelihoods

Ponhvoan Srey, Jason Lucas, Pittawat Taveekitworachai, Junyoung Lee, Anh Tuan Luu

arXiv:2610.23527v1cs.CLcs.AI

TL;DR

Reliable factual confidence must distinguish factual from non-factual claims while providing calibrated probabilities across languages. The paper contrasts likelihoods under factual and non-factual conditions, derives a probabilistic confidence score, and analyzes shared-error cancellation. CFP improves aggregate calibration while retaining competitive discrimination, and selective cross-lingual alignment improves target-language discrimination and factual generation.

  • Problem

    Factual confidence estimators need to distinguish factual from non-factual claims and provide probabilities interpretable as correctness despite variation across languages and input forms.

    Source: §1 · p. 2

  • Method

    CFE contrasts the likelihood of the same claim under factual and non-factual conditions, while CFP is its probabilistic counterpart; the paper also aligns target-language claims selectively along the factual-evidence direction.

    Source: §1 · p. 2, §5.2 · p. 7

  • Results

    CFP achieves mean ECE 0.115 and Brier’s Score 0.241, with AUROC 0.689 versus 0.694 for P(True)-norm; selective alignment improves target-language discrimination and factual generation.

    Source: §4.2 · p. 6, §6 · p. 8

  • Takeaways & Limitations

    Shared error between surrogate conditionals can cancel in the contrast, and selective alignment transfers stronger English factual evidence without target-language factuality labels.

    Source: §1 · p. 2, §5.2 · p. 7

    The multilingual benchmark mainly uses short, atomic factual statements, so it does not capture more complex or long-form factuality.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large Language Models (LLMs) encode substantial factual knowledge. Yet, estimating confidence in factual claims reliably remains challenging. We study likelihood-based factual confidence from the complementary perspectives of discrimination and calibration across multilingual factual statements. Specifically, we instantiate contrastive generative likelihoods as Contrastive Factual Evidence (CFE), which compares the likelihood of the same claim under competing factual and non-factual conditions, and derive its probabilistic counterpart, the Contrastive Factual Posterior (CFP). We show analytically that CFP calibration depends on the differential error between the two surrogate conditionals, allowing errors shared across them to cancel and yield tighter calibration bounds. Empirically, CFP substantially improves calibration while maintaining factual discrimination comparable to its strongest competitor. Nonetheless, discrimination is uniformly weak across methods in lower-resource languages, which simultaneously undermines the utility of the measured calibration. To this end, we interpret a likelihood geometry, within which an evidence-bearing direction is induced, along with an orthogonal evidence-invariant direction. In our investigation, we selectively align target-language claims with stronger English evidence only along the factual-evidence direction, improving discrimination and open-ended factual generation without target-language labels.

1 Introduction

Reliable factual confidence should distinguish factual from non-factual claims and yield probabilities interpretable as correctness across languages and input forms. The paper compares factual and non-factual generative likelihoods, studies calibration and discrimination, and motivates selective transfer of factual evidence to lower-resource languages.

1 Introduction

  • Confidence in factual claims should both rank factual above non-factual claims and provide probabilities interpretable as correctness, despite variation across languages and input forms.Existing estimators may degrade across languages and input forms, and discrimination and calibration need not coincide.Source: §1 · pp. 1–2
  • Prior approaches use predictive uncertainty or elicited truth judgments, but neither is explicitly designed to isolate factuality-specific evidence from unrelated variation.Likelihood-based confidence offers a training-free, probabilistically interpretable alternative to internal-representation approaches.Source: §1 · pp. 1–2
  • CFE compares the same claim’s likelihood under factual and non-factual conditions, and CFP is its probabilistic counterpart.The paper also analyzes how shared surrogate errors can cancel, interprets likelihoods geometrically, and motivates evidence-selective cross-lingual alignment without target-language factuality labels.Source: §1 · p. 2
  • The paper jointly examines calibration and discrimination across multilingual settings, where lower-resource languages can show weak discrimination despite low ECE.This mismatch motivates selective transfer of stronger-language factual evidence.Source: Figure 1 · p. 1, §1 · p. 2

2 Related Work

Related work connects factual confidence to generative classification, uncertainty estimation, truth representation, and cross-lingual alignment. This paper’s distinction is to compare a fixed claim’s likelihood under factual and non-factual conditions as a continuous confidence signal.

Generative Classification

  • Generative classification compares how well observed text is explained under alternative class-conditioned models, while this work holds the claim fixed and contrasts factual versus non-factual conditions as continuous confidence.Prior generative classification primarily targets categorical prediction.Source: §2 · p. 2

Uncertainty and Confidence

  • Confidence estimators use predictive-distribution quantities or elicited truth judgments, whereas CFE contrasts claim likelihoods under factuality conditions without verbalized self-assessment.Prior evaluations also report sensitivity to prompt and cross-lingual variation.Source: §2 · pp. 2–3

Truth Representation and Alignment

  • Truth-related information has been decoded from hidden activations and used in interventions, while this paper derives a factual-evidence direction from conditional likelihoods for cross-lingual transfer.The likelihood geometry aligns only the factual-evidence component, unlike approaches that learn truth directions from high-dimensional hidden states.Source: §2 · p. 3

3 Factuality as Contrastive Generative Evidence

The paper defines factual confidence by contrasting language-model likelihoods for the same claim under factual and non-factual conditions. Its calibration analysis shows how correlated surrogate errors can cancel in the contrast.

3.1 Problem Setup

  • The framework uses class-conditional likelihoods for claims labeled factual or non-factual, with class priors defining the corresponding factual posterior.The model’s natural-language conditions induce surrogate distributions for each class.Source: §3.1 · p. 3

3.2 Factual Evidence and Confidence from Conditional Likelihoods

  • CFE contrasts the model’s surrogate conditional likelihoods, and CFP converts the contrast into probabilistic factual confidence.Scores are generally token-length normalized and averaged across five paraphrased condition pairs.Source: §3.2 · pp. 3–4

3.3 Calibration under Shared Surrogate Error

  • CFP’s posterior-logit error depends on the difference between the two surrogate log-errors and prior misspecification, so shared error components are suppressed.Under equal error means and variances and a correctly specified prior, the calibration bound tightens as the error correlation rises, vanishing as correlation approaches 1.Source: §3.3 · p. 4
  • The calibration analysis also provides a general bound that allows unequal surrogate-error means and variances and prior misspecification.Positive error correlation is plausible because both conditionals score the same statement with the same underlying model, though whether this yields shared error is an empirical question.Source: §3.3 · p. 4

4 Evaluating Calibration and Discrimination

The evaluation compares calibration and discrimination across multilingual factual statements, model families, and formulation choices. CFP has the lowest average ECE, but its calibration advantage varies by language and depends on the class prior.

4.1 Experimental Setup

  • The evaluation uses 48,680 matched statements across eight languages and 12 base/instruction-tuned model pairs; it compares CFP with nine baselines using ECE, AUROC, and Brier’s Score.Table 1 reports language-level results and an across-language mean; Table 2 compares formulation choices.Source: §4.1 · pp. 4–5

4.2 Results and Discussion

  • 0.115 mean ECE and 0.241 Brier’s Score make CFP best on average, while its AUROC of 0.689 is competitive with P(True)-norm’s 0.694.CFP’s lower aggregate ECE is driven primarily by Hindi and Thai, where weak discrimination and Brier’s Score near 0.25 complicate its interpretation.Source: Table 1 · p. 5, §4.2 · p. 6
  • Contrasting the conditional likelihoods reduces ECE from approximately 0.29 to 0.14 and raises AUROC from 0.60/0.63 to 0.69; length normalisation and paraphrase averaging have smaller effects.Mapping CFE to CFP with a sigmoid substantially reduces ECE while leaving AUROC unchanged.Source: Table 2 · p. 5, §4.2 · p. 6
  • Post-training and chat templating consistently reduce discrimination; for CFP, ECE falls from 0.140 to 0.088 as AUROC falls from 0.736 to 0.652.CFP’s discrimination drops less than P(True)-norm’s across these configurations.Source: §4.2 · p. 6
  • CFP’s calibration is sensitive to class prevalence: ECE rises from 0.1 in the balanced setting to 0.4 in the extreme case, but known priors reduce ECE to as low as 0.04 without retraining.The prior correction leaves AUROC unchanged.Source: §4.2 · p. 6, Figure 2 · p. 6

5 Geometry-Guided Cross-Lingual Factuality Alignment

The section develops a likelihood geometry for transferring English factual evidence to target languages, then evaluates selective alignment against full consistency. Selective CFE alignment improves target-language discrimination while largely preserving general-task performance.

5 Geometry-Guided Cross-Lingual Factuality Alignment

  • The alignment addresses degraded CFE discrimination in lower-resource languages while allowing language-specific differences in absolute likelihood.The evaluation uses parallel claims and measures held-out True-False AUROC, Belebele accuracy, and factual precision in biography generation.Source: §5 · pp. 6–7, §5.3 · p. 8

5.1 Likelihood Geometry of Factual Evidence

  • CFE is the likelihood-ratio direction that changes factual evidence, while the orthogonal direction captures evidence-invariant variation.The geometry treats the two log-likelihoods of the same claim under factual and non-factual conditions as coordinates.Source: §5.1 · p. 7

5.2 Factuality Alignment

  • Selective alignment uses stronger English CFE as an anchor and constrains target-language claims only along the factual-evidence direction.The English CFE is treated as constant during LoRA adaptation; the full-consistency baseline aligns both likelihood coordinates and constrains the evidence-invariant direction too.Source: §5.2 · pp. 7–8

5.3 Experimental Results

  • Hindi AUROC rises from 57.0 to 70.5 with Qwen-3.5-4B CFE alignment, closing 44% of the gap to English’s 87.9; full consistency does not reliably narrow the gap.CFE alignment also approximately preserves generated claim volume while increasing correct claims; full consistency produces fewer claims and fewer correct claims per entity.Source: Table 3 · p. 7, §5.3 · p. 8

6 Conclusion

The paper studies CFE and CFP as likelihood-based factual confidence, then uses likelihood geometry to guide cross-lingual alignment. Its results highlight the need to assess both calibration and discrimination.

6 Conclusion

  • Selective alignment improves target-language discrimination and factual generation while largely preserving general-task performance.The approach transfers stronger English evidence while leaving evidence-irrelevant variation unconstrained.Source: §6 · p. 8
  • CFP is competitive with likelihood baselines in calibration and discrimination, but its relative advantage varies by language and evaluation setting.In lower-resource languages, low ECE coincides with weak discrimination.Source: §6 · p. 8
  • The paper derives CFE from contrastive generative likelihoods and its probabilistic counterpart, CFP.Its analysis identifies differential error between surrogate conditionals as relevant to calibration.Source: §6 · p. 8

Limitations

The authors identify limits in the evaluated claims, model sizes, alignment experiments, and capability benchmarks.

Limitations

  • The multilingual True-False benchmark mainly uses short, atomic claims, so complex or long-form factuality remains untested.Extending evaluation to those settings is future work.Source: Limitations · p. 9
  • The main evaluation covers models only up to 14B parameters, leaving generalization to much larger models untested.Source: Limitations · p. 9
  • Cross-lingual alignment is a proof of concept on 2B–4B models, with general-task retention and biography factuality assessed through limited benchmarks.Biography factuality uses LLM-as-a-judge, which correlates well but imperfectly with human evaluation.Source: Limitations · p. 9

A Theoretical Proofs and Analysis

The analysis connects CFP’s calibration to approximation of the Bayes posterior and the difference between the two surrogate log-errors.

A Theoretical Proofs and Analysis

B Experimental Details

CFE scores claims by comparing their likelihood under factual and non-factual prompts. The default averages evidence across paraphrased condition pairs, and the appendix also describes prompt templates for CFE and truth-judgment baselines.

B.1 Language Model Formulation of CFE

  • CFE scores a claim by teacher-forcing it after factual and non-factual prompts, using the model’s next-token distribution to compute conditional likelihoods.The formulation includes an end-of-turn token or EOS token and considers a length-normalised variant because summed log-likelihood differences can scale with claim length.Source: §B.1 · p. 13
  • The default CFE averages over five paraphrased condition pairs to reduce sensitivity to prompt wording.The ablation compares raw versus length-normalised evidence and one condition pair versus the five-pair default.Source: §B.1 · p. 13

B.3 Factuality Alignment Setup

The alignment experiments train separate adapters for target-language topics using paired True–False claims and evaluate both factuality discrimination and general-task retention. They also assess target-language biography generation, with factual precision estimated by an LLM judge.

B.3 Factuality Alignment Setup

  • Each target-language True–False dataset contains 6,085 claims paired with English translations across six topics; separate adapters use topic-specific splits.The setup uses 3,830 training, 428 validation, and 1,827 test pairs across the six topic splits, with 386–918 training pairs per adapter and a small LoRA parameter subset.Source: §B.3 · p. 13
  • General-task retention is evaluated on 900 Belebele questions per language before and after finetuning, using five English demonstrations for each query.The evaluation covers English and the target-language subsets and reports accuracy over all 900 questions.Source: §B.3 · p. 13
  • Biography generation is evaluated on 50 people, with factual precision measured as the mean proportion of extracted claims judged correct across biographies containing claims.The LLM judge decomposes each biography into atomic claims and assigns correct, incorrect, or uncertain verdicts; the measure corresponds strongly but imperfectly with human judgment.Source: §B.3 · pp. 13, 15

C Supplementary Experimental Results

Supplementary experiments examine scaling, prompt language, model scale, and human validation. They report robust calibration patterns alongside method-dependent trade-offs and limitations in factual discrimination and judge agreement.

Held-out Scaling

  • Held-out min-max scaling does not change the substantive conclusions, so the main experiments use full-dataset scaling given the limited dataset size.The robustness comparison evaluates ECE against the main results; AUROC remains unchanged.Source: Table 5 · p. 15, C Supplementary Experimental Results · p. 15

English versus Native Prompting

  • Native prompting modestly improves CFP discrimination for base and IT models while slightly increasing ECE; P(True)-norm shows small AUROC decreases and mixed calibration effects.The effects of prompt language depend on the method, and calibration and discrimination can shift differently.Source: C Supplementary Experimental Results · p. 15, Table 7 · p. 16

LLM-Judge Human Validation

  • LLM-generated biography labels show good but imperfect agreement with human labels, and the two human annotators also disagree on some factuality judgments.The validation uses eight biographies each from English, German, and Hindi, with claims translated into English for annotators who did not speak German or Hindi.Source: Table 8 · p. 16, C Supplementary Experimental Results · p. 16

Performance Across Model Scale

  • CFP has the lowest ECE at every evaluated scale, while its AUROC remains competitive with P(True)-norm and can be near chance despite low ECE.AUROC generally improves with scale for Qwen-3 and Qwen-3.5 but not universally; Gemma-4’s E4B model outperforms 12B in AUROC.Source: C Supplementary Experimental Results · p. 16, Figure 6 · p. 17

Related papers

Loading 2610.23527v1…