Credal Large Language Models for Semantic Commitment under Uncertainty

Shireen Kudukkil Manchingal; Sofiia Nikolenko; Fabio Cuzzolin

Source-linked AI summary

Credal Large Language Models for Semantic Commitment under Uncertainty

Shireen Kudukkil Manchingal, Sofiia Nikolenko, Fabio Cuzzolin

arXiv:2608.23244v1Aug 24, 2026cs.CLcs.AIcs.LGstat.ML

TL;DR

LLMs can be confidently wrong because a single predictive distribution does not expose whether support is robust or merely sharp. CLLM uses LoRA ensembles to form credal sets and derives token- and semantic-level commitment scores. Across QA, hallucination, calibration, selective prediction, and reasoning evaluations, it reports strong accuracy and calibration, with CTC offering a generation-free hallucination signal.

Problem
Single-softmax LLMs can represent weak evidence as sharp confidence, motivating uncertainty estimates that distinguish epistemic ignorance from robust predictive support.
Method
CLLM forms a credal set from LoRA adapters and derives CTC from credal geometry, while SCC and SCC-Gap incorporate sampled semantic completions.
Results
CLLM is best on QA accuracy at competitive calibration; CTC is within 1.5 pp of the best on five of eight hallucination settings, SCC reaches 99.0% OpenBookQA accuracy at 80% coverage, and CLLM+Csem achieves under 0.6% ECE on ARC-Challenge across three backbones.
Takeaways & Limitations
Credal representations support selective prediction and abstention by exposing agreement across plausible predictors and enabling either generation-free or semantic commitment assessment.
Takeaways & Limitations
The five-adapter credal set is an empirical approximation rather than a calibrated posterior, and the hallucination protocol and sample sizes limit evaluation scope and precision.

Abstract

from arXiv · show

Large language models (LLMs) often produce fluent but incorrect answers with unwarranted confidence. A central limitation is that standard LLMs represent uncertainty through a single predictive distribution, conflating epistemic ignorance with genuine ambiguity. We introduce Credal Large Language Models (CLLMs): an ensemble of LoRA adapters induces a credal set whose lower and upper probabilities expose the spread of plausible predictive distributions rather than collapsing to a single softmax output. From this representation we derive two complementary commitment scores. Credal Token Commitment (CTC) is a token-space score that combines lower-bound support, credal width, and intersection entropy, computed without additional generation. Semantic Commitment Consistency (SCC) extends commitment to semantic space using sampled completions, with SCC-Gap measuring the mismatch between token-level and semantic-level support. We evaluate hallucination detection, calibration, selective prediction, and reasoning on Gemma-2-9B, Llama-3.1-8B, and Qwen2.5-7B across OpenBookQA, CoQA, TriviaQA, and ARC-Challenge. CLLM is the best method on QA accuracy at competitive expected calibration error, and CTC tracks the best hallucination AUROC within 1.5 pp on most settings without additional generation. On selective prediction at 80% coverage, CLLM with SCC reaches 99.0% accuracy on OpenBookQA, and on ARC-Challenge CLLM with Csem confidence achieves <= 0.6% ECE across the three backbones.

1 Introduction

The paper argues that single-softmax uncertainty can mistake weak evidence for confident commitment, and introduces credal-set representations and commitment scores to expose and assess support across plausible predictions.

Motivation: LLMs can produce fluent but incorrect answers with unwarranted confidence, especially under incomplete, conflicting, or adversarial context.This is particularly consequential in safety-critical healthcare, legal, and scientific assistance settings.
Motivation: Standard LLMs collapse uncertainty into one softmax distribution, forcing precise probabilities even when evidence is weak.Its sharpness reflects one distribution rather than robust support across multiple plausible hypotheses.
Credal perspective: A credal set represents epistemic uncertainty as a closed convex set of plausible distributions with lower and upper probabilities.These bounds provide worst- and best-case summaries, while the intersection-probability transform supplies a representative distribution.
Proposed framework: CLLMs use a LoRA-adapter ensemble on a frozen backbone to induce lower and upper next-token probability bounds.The framework brings credal-set representations to next-token prediction in instruction-tuned LLMs.
Commitment scores: CTC, SCC, and SCC-Gap address cheap token-level commitment, semantic support, and mismatch between token- and semantic-level evidence.CTC combines lower-bound support, credal width, and intersection entropy without additional generation; SCC uses sampled generations.

2 Related Work

Related work distinguishes predictive uncertainty, ensemble disagreement, and semantic variability as complementary approaches to uncertainty and selective prediction in language models.

Uncertainty in LLMs: Prior work formalizes the aleatoric/epistemic split at the predictive-distribution level and reports systematic miscalibration in QA and few-shot tasks.Sequence-level uncertainty can also be decomposed into token-level contributions through ensembling.
Semantic uncertainty: Semantic entropy addresses the weakness of token dispersion by clustering sampled generations according to meaning and scoring cluster diversity.This reflects that multiple surface forms can express the same answer.
Selective prediction: Hallucination and abstention are studied within selective prediction with a reject option, alongside semantic-uncertainty and self-consistency methods.These approaches frame uncertainty estimates as inputs to decisions about prediction or rejection.

3 Methodology

CLLM represents next-token uncertainty with a credal set induced by LoRA adapters, preserving lower and upper support across plausible predictors. It derives token- and semantic-level commitment scores for prediction, selective prediction, and abstention.

Credal Set Construction: An ensemble of M LoRA adapters on a frozen backbone forms a convex-hull credal set over next-token distributions, retaining lower and upper probability bounds.The intersection-probability transform supplies a representative distribution for selecting a prediction while preserving the full credal representation for uncertainty analysis.
Credal Set Construction: Lower probability captures robust support across plausible predictors, upper probability captures most-favourable support, and their gap measures unresolved epistemic variability.Mean-pooling can discard disagreement when a token has high average probability but lacks stable support across predictors.
Credal Uncertainty Measures: The credal set yields intersection entropy and credal width, separating diffuse predictions within one distribution from disagreement across predictive distributions.High entropy can coexist with low width, while low entropy can coexist with large width.
Credal Token Commitment: CTC combines lower-bound token support, credal width, and intersection entropy multiplicatively, so failure of any condition drives commitment toward zero.It is computed entirely from the credal set without sampled completions or semantic clustering.
Credal Token Commitment: When all ensemble members agree, the credal set degenerates to one distribution, and CTC reduces to familiar single-distribution counterparts except for its disagreement-sensitive width factor.The width term distinguishes the credal regime from single-distribution scoring.
Semantic Commitment: Semantic commitment clusters K sampled completions by meaning, while SCC multiplies token commitment by semantic commitment and SCC-Gap measures disagreement between the two evidence sources.This addresses cases where local token confidence conflicts with semantic stability across completions.

4 Experiments

Across reliability evaluations, CLLM provides competitive accuracy and calibration, while different credal commitment scores excel in dataset- and regime-specific selective prediction and hallucination detection. Results also show clear limits: semantic scores benefit from first-token resolution, SCC-Gap is regime-dependent, and CTC trades semantic sensitivity for generation-free operation.

QA calibration: CLLM is best on accuracy across all three QA datasets and is at or within 0.3 pp of the best ECE.This performance is measured before commitment-based selective prediction.
Selective prediction: At 80% coverage, CLLM with SCC matches or beats evaluated baselines, with the winning score varying by dataset.Ctok wins on OpenBookQA, SCC on CoQA, and Csem on TriviaQA, matching the properties of each answer space.
Reasoning: On ARC-Challenge, CLLM+Csem achieves ECE under 0.6% across all three backbones, while Standard-LLM ECE reaches 26−81%.On Qwen, accuracy matches Standard-LLM while ECE falls from 26.3% to under 0.1%.
Hallucination detection: CTC tracks intersection entropy within 1.5 pp on 7/8 hallucination settings without sampled completions, while intersection entropy is best on 4/8.Credal scores expose predictive spread that single-distribution entropy averages away.
Semantic comparisons: First-token semantic clustering closes most of the gap with credal scores, whereas full-sequence semantic scores underperform every credal score by up to 20 pp.The first decoded token preserves informative disagreement while avoiding downstream paraphrastic variation.
Operational limits: SCC trails CTC by 4−12 pp, and SCC-Gap is anti-correlated with hallucination on 5/8 settings under joint token- and semantic-level degradation.The paper characterizes SCC-Gap as a regime-specific diagnostic rather than a universal hallucination ranker.
Bayesian comparison: Intersection-probability entropy beats Bayesian-LoRA mutual information by 3.6−10.1 pp across three Qwen settings.This indicates that credal summaries can preserve signal that scalar Bayesian summaries average away.
Limitations: The commitment scores trade cost for richness: CTC requires no generation, whereas SCC and SCC-Gap require sampling and clustering and depend on the embedding model and threshold τ.The five-adapter credal set is an empirical approximation rather than a calibrated posterior, and the hallucination protocol uses one corruption variant per dataset.

5 Conclusion

CLLM represents epistemic uncertainty with credal sets induced by LoRA ensembles and derives token- and semantic-level commitment scores. Across hallucination detection, selective prediction, and ARC-Challenge, these scores provide strong reliability performance with limited generation or calibration costs.

CLLM exposes lower and upper predictive probabilities through credal sets induced by ensembles of LoRA adapters.
CTC combines lower-bound support, credal width, and intersection entropy without additional generation, while SCC uses sampled completions and SCC-Gap measures their disagreement.
CTC is within 1.5 pp of the best hallucination detector on five of eight settings, while intersection entropy is best on four of eight.
At 80% coverage, CLLM with SCC reaches 99.0% accuracy on OpenBookQA, and Csem reaches 79–88% accuracy at ≤0.6% ECE on ARC-Challenge.

A Proofs

The singleton-credal-set proposition shows that when all ensemble members agree, credal uncertainty collapses to ordinary predictive quantities. Consequently, CTC reduces to a sharpness-weighted tempered-softmax margin, while improvements beyond that baseline require ensemble disagreement.

Singleton-credal-set limit: When all ensemble distributions equal p∗, the credal set becomes the singleton {p∗}, and lower and upper probabilities both equal p∗.
Singleton-credal-set limit: Under unanimous ensemble predictions, credal width vanishes and the intersection-probability transform reduces to p∗.
Singleton-credal-set limit: Intersection entropy reduces to standard predictive entropy H(p∗(· | x)) in the singleton limit.
Singleton-credal-set limit: CTC reduces to a tempered-softmax margin weighted by predictive sharpness when the ensemble has no disagreement.
Implication: Benchmark gains over tempered-margin or low-entropy scoring must exploit non-trivial credal width, and corrupted-context results support this pattern.

B Implementation Details

The implementation freezes three open-weight backbones and trains small LoRA ensembles with controlled optimization settings. Inference computes exact coordinate-wise credal bounds, while semantic scores add sampled completion clustering.

Backbones: The experiments use frozen Gemma-2-9B-Instruct, Llama-3.1-8B-Instruct, and Qwen2.5-7B-Instruct backbones.
LoRA training: Each setting trains five rank-8 LoRA adapters with α=16, dropout 0.1, one epoch, learning rate 1e−4, batch size 2, gradient accumulation 4, and maximum length 1024.
Inference: At inference, token-wise minima and maxima across adapters provide exact lower and upper probabilities for the finite convex-hull credal set.
Semantic clustering: Semantic clustering uses K=16 completions at temperature 0.8 with bge-base embeddings and cosine thresholds τ=0.5 for free-form QA and τ=0.8 for OpenBookQA.

C Experimental Setup Details

The experimental design tests credal uncertainty across backbones, datasets, corruption regimes, and reliability tasks. It combines exact credal transforms with semantic clustering while explicitly varying scoring sharpness and documenting sampling choices.

Credal inference: The intersection-probability transform supplies a representative prediction that respects both lower and upper bounds rather than only their average.
Datasets: OpenBookQA, CoQA, and TriviaQA span constrained multiple-choice, free-form conversational, and lexical-versus-semantic QA regimes relevant to CTC and SCC.
Corruption protocol: Corrupted-context evaluation perturbs evidence while preserving the question to isolate disagreement caused by inconsistent support.
Sample sizes: Hallucination settings use 250 clean and 250 corrupted prompts, while QA and ARC settings use N=500 queries.
Scoring choices: CTC uses β=1 for hallucination ranking and β=10 for selective prediction at the 80% coverage threshold.

D Additional Results and Tables

The additional analyses examine selective prediction, uncertainty metrics, score variants, ensemble-size sensitivity, semantic-threshold robustness, and adversarial detection. They also show that credal summaries retain signal under Laplace-LoRA posteriors and that CTC is comparatively stable as ensemble size changes.

Selective prediction: Selective prediction aggregates results at 80% coverage across OpenBookQA, CoQA, and TriviaQA using CTC with β=10.
Uncertainty metrics: Table 6 compares uncertainty metrics on clean and corrupted-context splits, reporting mutual information only for methods with multiple predictive samples.
Adversarial detection: AdvBench evaluates CLLM on harmful instructions and prompt-injection attacks, with a separate three-adapter Llama pilot illustrating regime dependence.
Score variants: 3.6/10.1/9.7 pp gains over canonical mutual information occur on OBQA / CoQA / TriviaQA when intersection-probability entropy summarizes the same Laplace-LoRA posterior.
Ensemble-size sensitivity: At M=5, Ctok reaches 0.687 AUROC while credal width, intersection entropy, and CTC reach 0.768, 0.791, and 0.778 on Qwen2.5-7B CoQA hallucination.

E Extended Related Work

The extended related work situates CLLM within imprecise probability, credal neural networks, selective prediction, semantic and token-level uncertainty, and Bayesian parameter-efficient methods. It emphasizes that credal sets generalize single distributions while existing token- and semantic-level approaches capture different aspects of uncertainty.

Imprecise-probability foundations: Imprecise-probability theory represents uncertainty with closed convex sets of distributions and lower and upper probabilities as worst- and best-case envelopes.
Relation to prior work: The paper's credal-set application uses established imprecise-probability machinery rather than proposing new foundations.
Credal neural networks: Credal and belief-function neural networks apply imprecise-probability machinery to classification and out-of-distribution detection.
Selective prediction: Selective prediction formalizes abstention through risk-coverage curves, while conformal methods provide calibrated prediction sets over LLM completions.
Token-versus-semantic uncertainty: Token-level methods decompose sequence uncertainty, whereas semantic-entropy methods cluster sampled generations by meaning or entailment.
Bayesian LoRA: Bayesian parameter-efficient methods target LoRA parameters because full-Bayesian inference over modern LLM weights is intractable.

F Broader Impact

The broader-impact discussion frames CLLM as reliability tooling for deployment-critical assistants while identifying two material risks: joint ensemble failure and increased computational cost. The evaluation uses existing public datasets and does not release new data.

Potential benefits: CLLM targets commitment-based abstention when token-level confidence and semantic-level support disagree in deployment-critical applications.
Risk of joint failure: High commitment reflects robust support within the realized ensemble, not guaranteed factual correctness, because adapters can jointly agree on a wrong answer.
Risk of joint failure: SCC-Gap can flag divergence between token- and semantic-level support but cannot detect joint failure modes.
Computational costs: CLLM with M=5 adapters raises training and inference cost over a single fine-tune, while SCC additionally requires K=16 stochastic completions per query.
Data and safety: The protocols use existing public datasets and do not collect or release new data; AdvBench contains harmful prompts for safety evaluation without producing harmful generations.

NeurIPS Paper Checklist

The checklist justification states that the abstract and introduction accurately present the paper’s contributions and headline results. It reports consistency between those claims and the experimental results in Tables 1–4.

Claims and contributions: The abstract and introduction state the CLLM framework, two credal uncertainty measures, CTC, and SCC/SCC-Gap contributions.
Claims and results: The reported headline numbers match the experimental results in Section 4.2 and Tables 1–4.
Scope: The checklist emphasizes that claims should reflect theoretical and experimental results and indicate how far findings may generalize.

2. Limitations

The paper identifies empirical and deployment boundaries, including trade-offs in uncertainty scoring and risks from misinterpreting commitment as correctness.

Limitations: The evaluation has a cost-versus-richness trade-off between CTC and SCC/SCC-Gap, and SCC-Gap performance depends on the operating regime.The ensemble is empirical rather than a formally calibrated posterior, and experiments use a single corruption variant per dataset.
Limitations: High commitment scores may be misread as ground-truth correctness signals, while poisoned context can produce ensemble agreement on a wrong answer.The paper discusses these risks alongside mitigations and dual-use considerations for AdvBench evaluation.