ChainUQ: Reasoning Consistency-Aware Uncertainty Quantification for Large Language Models

1 Introduction

• Response-level UQ should assess whether reasoning steps support the final answer, because intermediate claims can be unsupported or inconsistent even when the conclusion is correct.Such errors may remain difficult to detect when the response appears fluent.Source: §1 · p. 1
• Reasoning-consistency evidence helps identify flaws that surface fluency or final-answer confidence may not reflect.The framework evaluates intrinsic confidence and chain consistency as complementary signals.Source: §1 · p. 2
• ChainUQ combines conclusion-aligned frozen features for raw intrinsic confidence with reasoning-chain consistency evidence to calibrate that confidence.The two modules evaluate a single response without retraining the backbone LLM or repeated generation.Source: §1 · p. 2
• An average 3.1% relative gain in AUROC and up to 45.0% relative reduction in ECE are reported across four datasets, alongside robust transferability.The evaluation includes in-distribution and out-of-distribution scenarios.Source: §1 · p. 2

2 Preliminary

• The task is to quantify the trustworthiness of one generated response, using a frozen white-box LLM with accessible internal states and reasoning chain.The goal is calibrated confidence, not selecting among multiple sampled answers.Source: §2 · p. 2

2.1 Reasoning Chain Decomposition

• The claim-level decomposition separates the final conclusion from intermediate reasoning claims, centering confidence on the answer while using intermediate claims as consistency evidence.It provides a unified representation across reasoning chains with different textual formats.Source: §2.1 · p. 2

2.2 Consistency-Aware Labels

• Conclusion labels mark factual correctness, while intermediate-step labels assess contextual support, logical coherence, and usefulness in supporting the conclusion; the pipeline then calibrates raw confidence with consistency evidence.The labels are obtained automatically using a stronger LLM as judge.Source: §2.2 · p. 2, Figure 1 · p. 3

3 Methodology

• ChainUQ processes a generated reasoning chain after generation, producing the conclusion as the answer and a calibrated confidence score through two sequential UQ modules.The first estimates intrinsic confidence in the conclusion; the second calibrates it using reasoning-chain consistency evidence.Source: §3 · p. 3

3.1 Response-Level Reasoning Consistency-Aware UQ Formulation

• The formulation decomposes the response into reasoning claims and a final conclusion, then uses conclusion-masked token states to estimate raw confidence and consistency evidence to calibrate it.This separates conclusion confidence from chain reliability and reuses one generated chain without repeated sampling or backbone fine-tuning.Source: §3.1 · p. 3

3.2 Alignment-Aware Lightweight UQ

• The lightweight estimator encodes conclusion-span information in token representations, combines conclusion-centered evidence with global reasoning context, and predicts intrinsic confidence.The estimator is trained with binary cross-entropy on response correctness.Source: §3.2 · pp. 4–5

3.3 Reasoning Consistency-Aware Calibration

• The calibrator combines an anchor score with a reasoning-conditioned score using quantile-bin weights, relying more on reasoning evidence where it is more reliable.Quantile binning keeps score regions comparably populated, making mixing weights less sensitive to sparse tails and improving calibration stability.Source: §3.3 · pp. 5–6

4.1.1 Datasets

• HotpotQA is the in-distribution source; MuSiQue, StrategyQA, and bAbI are out-of-distribution transfer benchmarks, with Llama-3.1-8B-Instruct as the primary frozen backbone.Mistral-Small-24B-Instruct-2501 is the primary judge model.Source: §4.1.1 · p. 6, §4.1.2 · p. 6

4.1.3 Baselines

• Comparisons include unsupervised and supervised uncertainty baselines, evaluated using response accuracy, discrimination metrics, and UQ metrics.Metrics include AUROC, PRAUC, ECE, Brier score, and Risk@90.Source: §4.1.3 · p. 6, §4.1.4 · p. 6

4.3 Calibration & Consistency (RQ 2)

• On HotpotQA, R-Log is best on ECE, Brier, and Risk@90; on OOD datasets, it remains competitive, especially on Brier and Risk@90.The comparison covers Platt scaling, temperature scaling, isotonic regression, binwise hybrid, and R-Log.Source: §4.3 · p. 6, Table 2 · p. 7

4.2 Main Results (RQ 1)

• 3.1% average relative AUROC gain; ChainUQ achieves the strongest Accuracy, AUROC, and PRAUC across four datasets.The comparison includes supervised baselines matched to roughly 2 million trainable parameters.Source: §4.2 · p. 6, Table 1 · p. 7

4.4 Cross-Distribution Generalization (RQ 3)

• Most model–OOD pairs gain in both discrimination and calibration; R-Log’s ranking gain tends to grow with OOD hardness (Pearson r = 0.707).The transfer protocol tunes thresholds and calibrator parameters on validation, then transfers them unchanged to test.Source: §4.4 · p. 7, Figure 2 · p. 7

4.5 Ablation Study (RQ 4)

• On HotpotQA and MuSiQue, every module simplification degrades calibration; removing the learned bank worsens ECE most, while removing context fusion most harms Brier and Risk@90.Removing conditioned selection and the final residual refiner also performs worse than the full system.Source: §4.5 · pp. 7–8, Table 3 · p. 7

4.6 Efficiency (RQ 5)

• ChainUQ’s best HotpotQA head is marginally slower but reduces peak memory relative to UHead, while R-Log is lighter than BWH and preserves reasoning-aware calibration.Among supervised baselines, Factoscope and Lookback are fastest, while UHead is most memory-intensive.Source: §4.6 · p. 8

5.1 Uncertainty Quantification Heads

• Most supervised heads aggregate representations without explicitly separating the final conclusion from its supporting reasoning; ChainUQ splits conclusion-focused prediction from reasoning-aware calibration.Source: §5.1 · p. 8

5.2 Reasoning-Aware Uncertainty Estimation

• Unlike verifier-based reranking or answer selection, ChainUQ uses reasoning consistency as post-hoc calibration evidence and preserves the original prediction.It combines internal-state probing with efficient single-trace calibration rather than modifying outputs.Source: §5.2 · p. 8

• ChainUQ combines a lightweight conclusion-aligned confidence module with reasoning-consistency calibration, achieving a 3.1% relative AUROC gain and up to 45.0% relative ECE reduction across four datasets.The framework supports robust ID–OOD transfer without additional fine-tuning, larger auxiliary heads, or repeated generation.Source: §6 · p. 8

• The single-trace post-generation pipeline depends on judge verification and inherits judge-model noise or bias, primarily affecting calibration.Source: Limitations · p. 9
• ChainUQ neither repairs generation errors nor searches alternative chains; weakly structured or question-misaligned reasoning makes conclusion-aligned features less informative and transfer harder.Source: Limitations · p. 9
• The main evaluation is fully comparable English-language testing under one protocol; multilingual, interactive, and retrieval-augmented settings remain future work.Source: Limitations · p. 9

Ethical Considerations

• Confidence scores and judged reasoning labels may be misused as guarantees, so ChainUQ is intended for abstention, triage, and human review rather than replacing expert verification.Deployment should account for possible annotation bias, overconfidence, and domain mismatch in both generator and verifier models.Source: Ethical Considerations · p. 9

A Pipeline Details

• The frozen backbone generates a reasoning chain; a judge supplies correctness and step-consistency labels, then a trained head and calibrator produce the final confidence score.At inference, the full calibrated setting uses judged reasoning metadata to construct u(r).Source: A Pipeline Details · p. 11

B.1 Datasets and Preprocessing

• StrategyQA uses deterministic exact-match evaluation, while HotpotQA, MuSiQue, and bAbI use an external judge for free-form response correctness and reasoning verification.The shared judge sees task inputs and generated text, not the baseline or confidence score.Source: A Pipeline Details · p. 11
• The evaluation spans HotpotQA, MuSiQue, StrategyQA, and bAbI; HotpotQA supplies the source training split, while the other datasets provide transfer validation and test splits.Official train, validation, and test splits are used where the dataset provides them.Source: §B.1 · p. 11

B.2 Models and Agreement Analysis

• The 24B judge achieves agreement above 91% and Cohen’s κ of at least 0.80 with both reference judges across datasets for conclusion correctness and step-level consistency.The authors report this agreement for Mistral-Small-24B against Qwen2.5-72B and Llama-3.3-70B using identical verification prompts.Source: §B.2 · p. 12

B.3 Baselines

• Baselines include untrained probability-based scores and supervised lightweight classifiers over frozen target-model activations, all evaluated as response-level confidence estimators.The probability-based set includes MCP, perplexity, mean token entropy, and claim-conditioned probability; supervised examples include SAPLMA, Factoscope, Lookback Lens, and UHead.Source: §B.3 · p. 13

B.4.1 Thresholded Classification Metric

• Accuracy measures agreement between predicted and ground-truth correctness labels, while the reported metrics use the final calibrated confidence score and lower ECE, Brier, and Risk@90 values are better.Higher Accuracy, AUROC, and PRAUC values are better.Source: §B.4 · p. 13, §B.4.1 · p. 13

B.4.2 Discrimination Metrics

• AUROC measures correct-versus-incorrect ranking, while PRAUC summarizes precision and recall across confidence thresholds and is informative under highly skewed correctness distributions.Both are discrimination metrics that do not depend on the absolute confidence-score scale.Source: §B.4.2 · p. 13

B.4.3 Calibration and Selective-Risk Metrics

• ECE measures binned confidence–correctness mismatch, Brier measures squared confidence error, and Risk@90 reports residual error among the 90% most confident predictions.Lower values are better for all three metrics.Source: §B.4 · p. 13, §B.4.3 · p. 14

B.5 Reasoning-Aware Calibration Features

• The calibrator’s u(r) feature vector combines base confidence, reasoning summaries, token uncertainty, structure and feature-gap statistics, and judge-derived consistency signals without using the response-correctness label as a feature.Preprocessing retains samples with at least one usable reasoning score, segmented claim, and valid judged reasoning label.Source: §B.5 · pp. 14–15

B.6 Transfer Protocol

• Thresholds and calibration parameters are fit on held-out validation data and frozen before test evaluation, so OOD results do not use test-set-specific retuning.The protocol repeats validation-based fitting on each target dataset; reported quantitative results are means over 10 random-seed runs.Source: §B.6 · p. 15

C Hardware and Efficiency

• 408 vs. 48.96 samples/s on UHead and 204 vs. 51.72 samples/s on ChainUQ: R-Log is faster than BWH while retaining the structured-calibration regime.GPU usage stays near 15 GB across almost all rows, while latency varies by nearly two orders of magnitude.Source: C Hardware and Efficiency · p. 16

D Hyperparameter Space

• The final head uses width 512, a two-layer scoped encoder with 8 attention heads, dropout 0.1, balanced BCE, and label smoothing 0.02.The feature recipe fuses hidden representations from layers [−1, −2, −4, −8] with weights [0.4, 0.3, 0.2, 0.1], plus top-6 token-probability features and layer −1 attention summaries.Source: D Hyperparameter Space · p. 16, Table 9 · p. 16

E Additional Evaluation Results

• Across Mistral-7B, Gemma-2, and Phi-4, ChainUQ is typically strongest or near-strongest in discrimination, with stable gains on HotpotQA and StrategyQA and competitive results on MuSiQue.Calibration tables show reduced ECE, Brier score, and often Risk@90 versus simpler scalar calibrators across most settings; MuSiQue remains the hardest OOD target.Source: E Additional Evaluation Results · pp. 16–17

F Documentation Of Artifacts

• The documented artifacts cover four English reasoning settings, from natural-text multi-hop QA to commonsense, synthetic state-tracking, and compositional reasoning benchmarks.The datasets are task-oriented rather than demographic surveys, and demographic attributes are not used as supervision, evaluation data, or calibration features.Source: F Documentation Of Artifacts · p. 17

H Prompt Templates

• A shared prompt family elicits short atomic reasoning steps followed by one conclusion, while separate judge prompts handle conclusion correctness and step-level consistency.The two judge prompts expose different information because their outputs supervise different cache fields.Source: H Prompt Templates · p. 19

H.1 Generation-side templates

• The generation-side contract standardizes a Reasoning/Conclusion split so claims can be segmented uniformly and the final conclusion span isolated for the alignment-aware UQ module.Dataset-specific objectives and answer formats remain configurable within the shared prompt skeleton.Source: §H.1 · p. 19

H.2 Judge-side templates

• Stage 1 compares the conclusion with the reference answer; Stage 2 checks reasoning steps against the question, context, and conclusion while omitting the gold conclusion.The omission keeps step-level labels from trivially encoding the gold answer.Source: §H.2 · p. 20

I Case Studies

• Across HotpotQA and OOD examples, calibration raised confidence for coherent reasoning and lowered it when conclusions were mismatched or multi-hop reasoning drifted.The examples include both correct but under-confident responses and fluent chains with errors.Source: I Case Studies · p. 21

Example A: correct response rescued; Example B: wrong abstraction suppressed

• On HotpotQA, calibrated confidence rose from 0.089 to 0.998 for the correct entity-count comparison, but fell from 0.649 to 0.096 for the mismatched abstraction.The alignment-aware UQ scores were 0.089 and 0.649, respectively.Source: I Case Studies · p. 21

Examples C and E: coherent reasoning transfers

• For coherent OOD reasoning, calibrated confidence rose from 0.325 to 0.919 for the country-location chain and from 0.326 to 0.997 for the short director-to-spouse composition.Both examples began with lower alignment-aware UQ scores despite correct conclusions.Source: I Case Studies · pp. 21–22

Example D: final comparison fails

• In the OOD film comparison, calibrated confidence fell from 0.758 to 0.242 after the chain incorrectly treated 1976 as earlier than 1960.The entity links and birth-year facts were correct, but the final comparison was not.Source: I Case Studies · p. 22

Example F: OOD drift lowers confidence

• In the MuSiQue OOD case, calibrated confidence fell from 0.714 to 0.375 after an unsupported county assignment led to a neighboring-county hop.The chain appeared geographically coherent but compounded the initial unsupported assignment.Source: I Case Studies · p. 22