Distinguishing the Knowable from the Unknowable with Language Models

Gustaf Ahdritz; Tian Qin; Nikhil Vyas; Boaz Barak; Benjamin L. Edelman

Source-linked AI summary

Distinguishing the Knowable from the Unknowable with Language Models

Gustaf Ahdritz, Tian Qin, Nikhil Vyas, Boaz Barak, Benjamin L. Edelman

arXiv:2402.03563v2Feb 5, 2024cs.LGcs.AIcs.CL

TL;DR

The paper asks whether epistemic uncertainty can be distinguished from aleatoric uncertainty in free-form LLM text without ground-truth probabilities. It uses larger models as approximate references, trains probes on smaller-model representations, and proposes an unsupervised in-context test. The results show strong supervised prediction, cross-domain transfer, and non-trivial unsupervised accuracy, supporting the presence of internal uncertainty representations.

Problem
The paper studies how to distinguish epistemic uncertainty from aleatoric uncertainty in unconstrained text, where the source of model uncertainty is generally difficult to determine.
Method
The paper compares small-model uncertainty with larger-model confidence, trains probes on small-model activations, and introduces the unsupervised In-Context Learning Test.
Results
AUC > 0.9: small linear probes predict larger-model token confidence, AUC > 0.8 transfer occurs from Wikipedia to code, and the unsupervised method achieves non-trivial results.
Takeaways & Limitations
Across text domains and model sizes, LLM embeddings contain information about more capable models’ certainty, and this information can sometimes support unsupervised prediction.
Takeaways & Limitations
The framing uses larger models that still have epistemic uncertainty as proxies, introducing label noise while ignoring sequence-level semantic uncertainty and mixtures of uncertainty types.

Abstract

from arXiv · show

We study the feasibility of identifying epistemic uncertainty (reflecting a lack of knowledge), as opposed to aleatoric uncertainty (reflecting entropy in the underlying distribution), in the outputs of large language models (LLMs) over free-form text. In the absence of ground-truth probabilities, we explore a setting where, in order to (approximately) disentangle a given LLM's uncertainty, a significantly larger model stands in as a proxy for the ground truth. We show that small linear probes trained on the embeddings of frozen, pretrained models accurately predict when larger models will be more confident at the token level and that probes trained on one text domain generalize to others. Going further, we propose a fully unsupervised method that achieves non-trivial accuracy on the same task. Taken together, we interpret these results as evidence that LLMs naturally contain internal representations of different types of uncertainty that could potentially be leveraged to devise more informative indicators of model confidence in diverse practical settings.

1 Introduction

The paper addresses the difficulty of distinguishing epistemic from aleatoric uncertainty in unconstrained text by using larger models as approximate reference distributions. It finds that supervised probes can predict larger-model confidence across domains, while an unsupervised in-context method achieves non-trivial results.

The task separates epistemic uncertainty, which can diminish with knowledge, from aleatoric uncertainty, which remains inherent to the underlying distribution.
The supervised approach contrasts small-model uncertainty with larger-model confidence, treating disagreement as epistemic-like and shared uncertainty as aleatoric-like.
The proposed uncertainty signals are motivated by possible interventions to reduce hallucination risk or highlight uncertain generations, but the work is presented as an initial step.
AUC > 0.9: small linear probes accurately predict when larger models are confident on individual tokens across model pairings and datasets.
AUC > 0.8: probes trained on Wikipedia transfer to code, suggesting they read robust uncertainty representations rather than domain-specific token correlations.
The fully unsupervised In-Context Learning Test uses models’ responses to repeated prompts to predict the same uncertainty labels without additional training.

2 Related work

The related work situates the paper among uncertainty-estimation methods, in-context learning research, and studies of model confidence. It distinguishes this work by targeting epistemic uncertainty in free-form token prediction rather than mainly structured question answering.

Existing uncertainty methods include Bayesian neural networks, ensembles, and epistemic neural networks, but their computational cost or evaluation scale limits direct application to modern LLMs.
The supervised task uses a large model’s near-zero predictive entropy to label tokens and trains probes on smaller-model activations to predict those labels.
The unsupervised ICLT method tests whether a small model updates toward candidate information supplied in repeated prompts, using the resulting entropy as a predictor.
Figure 3 evaluates supervised probes in- and out-of-domain and evaluates ICLT on Wikipedia using the LLaMA 7B / LLaMA 65B pairing.
Prior confidence-prediction studies achieve promising in- and out-of-distribution results, but focus on question answering where the correct answer is known and uncertainty is effectively epistemic.
The paper builds on evidence that LLMs can update predictions from relevant in-context information, especially as models become larger.

3 Setup

The paper frames epistemic uncertainty as small-model uncertainty that would diminish for a larger model, while aleatoric uncertainty persists because the large model remains uncertain. It targets token-level prediction of large-model entropy from small-model representations, while controlling for confounds and acknowledging proxy limitations.

Task definition: The task predicts whether small-model uncertainty reflects epistemic-like or aleatoric-like uncertainty by estimating large-model confidence from small-model information.Low large-model entropy is treated as epistemic-like uncertainty in the small model; high large-model entropy is treated as aleatoric-like.
Task definition: The method uses a larger model as an approximate ground-truth proxy because larger, more compute-heavy models are assumed to exhibit less epistemic uncertainty.The framing compares small and large models rather than accessing true probabilities directly.
Task definition: The primary target is the Shannon entropy of the large model’s next-token distribution, with the broader goal of identifying substantial differences between small- and large-model predictions.The paper simplifies the full distribution-difference problem to predicting high-level summary statistics from intermediate small-model representations.
Limitations: The framing is limited because the large models still have epistemic uncertainty, introducing label noise, and it omits sequence-level semantic uncertainty and mixtures of uncertainty types.The paper presents this narrower token-level problem as a first step.
Experimental setup: Evaluation uses model families with shared vocabularies, architectures, or training data where convenient, and free-form text excluded from both models’ training data.Datasets include Wikipedia, code, Stack Exchange, and EuroParl, with the latter three used for out-of-distribution evaluation.
Baselines and controls: The setup controls trivial cues with entropy-threshold baselines, initial-embedding baselines, filtering, and token balancing, while Figure 5 motivates training within narrow small-model entropy bands.These controls address correlations between small- and large-model entropy and correlations between preceding tokens and labels.

4 Supervised experiments

Supervised probes use small-model activations to predict large-model entropy, with controls designed to prevent trivial entropy and token-frequency shortcuts. Linear classifiers achieve strong cross-domain performance, while removing the artificial entropy gap reduces accuracy near the decision boundary but preserves substantial baseline advantages.

Binary classification with gap: The initial binary task separates near-zero large-model entropy from entropy in the same small-model band, using an artificial gap to avoid ambiguous boundary tokens.This setup provides a clearer proof of concept before testing classification without a gap.
Experimental design: The experiments also test specialized entropy-band heads and account for possible domain-specific cues, although evaluation subsets differ because labels and class balancing depend on the large model.Table 1 reports AUROC for Wikipedia-trained classifiers evaluated in and out of distribution under class- and token-balanced conditions.
Experimental design: The classifiers use small-model next-token embeddings, with middle-layer representations generally performing best and class- and token-balancing removing trivial correlations.The filtering interventions make the task more difficult rather than easier.
Binary classification with gap: AUC > 0.9 is achieved by accurate linear classifiers across model pairings and datasets, with performance remaining nearly as strong on code, multilingual, and question-answering evaluations.These out-of-distribution evaluations test whether probes learn information beyond domain-specific heuristics.
Binary classification without gap: Without an artificial gap, accuracy falls to approximately 75% for the 7B/65B LLaMA experiment in the [2, 3) band, versus the high 80s near the boundary.Despite this drop, classifiers still outperform both baselines by large margins; the boundary is set at 1 bit in the no-gap experiments.

5 Unsupervised experiments

The unsupervised In-Context Learning Test (ICLT) probes whether a model changes its predictions when supplied with repeated hints, using this behavior to distinguish epistemic from aleatoric uncertainty. Synthetic experiments support the mechanism, while real-data evaluation applies minimum entropy across prompted repetitions and compares it with a simple baseline.

5 Unsupervised experiments: ICLT prepends top-k predicted tokens to the original prompt and measures how strongly the model repeats each hinted token.The resulting prompts can be repetitive or nonsensical, and performance is reportedly insensitive to how the relevant context is provided.
5.1 Synthetic proof of concept: In the synthetic task, epistemic answers are fixed after initialization whereas aleatoric answers are resampled whenever their questions recur.This construction makes epistemic answers memorisable and aleatoric answers inherently unpredictable.
5.1 Synthetic proof of concept: The synthetic model updates toward repeated answers for epistemic questions regardless of hint correctness, but leaves aleatoric predictions unchanged.Both question types initially receive approximately equal probability, so the difference reflects response to the added context.
5.1 Synthetic proof of concept: On empirical examples, ICLT behavior agrees with the intuition that epistemic tokens become more repeatable when hinted, although the showcased cases are cherry-picked.Additional examples show that the method works less well in some cases.
5.2 ICLT on real data: For real-data evaluation, ICLT uses the minimum entropy across repetitions with top-k hints and compares performance across LLaMA and Pythia model pairings.Figure 7 contrasts original LLaMA 7B entropy, LLaMA 65B entropy used for labels, and ICLT minimum entropy; Table 2 reports LLaMA results, while Tables 3 and 4 cover separator ablation and a Pythia failure case.
5.2 ICLT on real data: A separator between the inserted context and original prompt is important to ICLT performance and is linked to a proposed mechanism involving information repeated across documents.The paper includes a separator ablation and discusses a possible relation between separator choice and the Pythia failure case.

6 Conclusion

The paper concludes that LLM embeddings contain information about the certainty of more capable models, and that this information can sometimes support unsupervised prediction through prompt-copying behavior. It presents these findings as preliminary evidence that LLMs distinguish knowable from multiply answerable prompts, while identifying unresolved practical and conceptual limitations.

6 Conclusion: Across text domains and model sizes, LLM embeddings contain enough information to predict the certainty of more capable models.The paper reports this as a central supervised finding.
6 Conclusion: That information is sometimes correlated with how willing a model is to copy prompt information, enabling unsupervised prediction.This connects the representation-based result to the ICLT approach.
6 Conclusion: The authors characterize these results as evidence that LLMs can distinguish prompts with effectively one correct answer from prompts with many.They summarize this capability as LLMs “know what is knowable.”
6 Conclusion: The work remains preliminary before these techniques can be incorporated into practical systems such as hallucination reduction.Future work includes broader model-pairing and dataset evaluation, threshold tuning, failure analysis, and improved precision and recall on imbalanced token datasets.
6 Conclusion: The approach assumes that the larger models used for labeling exhibit as little epistemic uncertainty as possible, but how to achieve this remains unclear.The paper leaves open whether scale, dataset choice, or applying the unsupervised method to larger models would address this assumption.

A FAQ

The FAQ explains how model scale, dataset construction, uncertainty labels, and related methods affect the interpretation of the paper’s supervised and unsupervised results. It also records important limitations, including memorized aleatoric passages, imperfect synthetic prompts, and the scope of existing comparisons.

What can be expected to happen as you increase the size of the “large” model?: For LLaMA pairings, increasing the large model from 30B to 65B shows no substantial supervised or unsupervised difference, whereas Pythia supervised results are weaker.The paper reports these comparisons both in- and out-of-domain.
What can be expected to happen as you increase the size of the “large” model?: Larger label-generating models may reduce noise by placing fewer knowable facts in the high-entropy category, but they may also identify subtler patterns or memorize more text.A memorized aleatoric passage cannot be distinguished from epistemic uncertainty by this method.
What can be expected to happen as you increase the size of the “large” model?: Smaller large models produce predictions closer to their paired small models, improving trivial baselines based on the small model’s entropy.If the two models are identical, the large model’s entropy can be predicted perfectly by construction.
Is it possible to generate labels without access to a “large” model?: The synthetic Wikidata evaluation sought gold-standard labels by contrasting forward prompts with one expected answer against backward prompts with multiple answers.Backward prompts were filtered to have at least five Wikidata answers.
Is it possible to generate labels without access to a “large” model?: The Wikidata dataset was not diverse enough and its prompts often failed to elicit expected LLM behavior before publication.The authors suggest stronger filtering, narrower categories, better prompting, and LLM-assisted expansion as possible improvements.
Do the two classes in the binary classification experiments really correspond to aleatoric and epistemic uncertainty?: The binary high-entropy class is a catch-all containing aleatoric tokens, mixed uncertainty, and cases where both models retain substantial epistemic uncertainty.This makes a high-quality large model important for interpreting the labels.
Related work: Calibration differs from distinguishing uncertainty types: a model may be perfectly calibrated without providing useful confidence information.The FAQ places this distinction alongside prior calibration and confidence-prediction work.
Related work: Existing confidence and truthfulness methods may complement this approach because ICLT tends to detect correctable tokens with high precision but low recall on imbalanced data.The paper notes that this complementarity does not depend on the mechanism producing high accuracy.

D Training objectives

The paper evaluates several target values for uncertainty heads, ultimately using large-model entropy as the simplest effective target. It also considers transformed entropy and divergence-based targets.

The evaluated alternatives include log large-model entropy, Jensen–Shannon divergence, and log Jensen–Shannon divergence.Jensen–Shannon divergence symmetrizes KL divergence to quantify similarity between the small and large models’ output distributions.
Large-model Shannon entropy is the default target because the authors find no need to use alternative targets to achieve high classification accuracy.The target is the entropy of the larger model’s prediction distribution.
Jensen–Shannon divergence compares the small and large models’ probability distributions over the same support.

E Dataset details

The experiments use Wikipedia and Pile data, filter tokens into controlled entropy-based cases, and rebalance labels to prevent trivial token-frequency cues.

The dataset combines a homemade Wikipedia corpus with portions of the Pile evaluation and test sets.The Wikipedia collection contains approximately 18.5 million tokens, while the sampled Pile sets contain tens to hundreds of millions of tokens.
The binary task begins by running both models on every token and computing each model’s predictive entropy.
Tokens are retained only when the small model’s entropy lies in [k, k + 1), with k = 2 by default, limiting the task to a narrow uncertainty band.The upper bound reduces reliance on the relationship between high small-model entropy and nonzero large-model entropy.
Labels distinguish near-zero large-model entropy from entropy close to the small model’s value, using a δ = 0.1 tolerance and excluding intermediate large-model entropies.
Class balancing is performed independently for each previous token in the shared vocabulary to obstruct trivial class-token correlations.
The gapless classification setup is identical except that the large-model entropy filtering step is omitted.

F.3 Binary classification full results (gap)

The full gapped binary-classification results are reported across LLaMA, Pythia, and Llama 2 experiments and multiple evaluation sets.

AUROC curves for the Table 1 experiments are shown in Figure 10, with model-pair titles and evaluation restricted to a small-model entropy band.The accompanying discussion notes greater variance for the smaller EuroParl set and suboptimal activations for Pythia out-of-distribution classifiers.
The EuroParl evaluation set exhibits substantially greater classifier-to-classifier variance because it is much smaller than the other datasets.
Pythia out-of-distribution results can be improved substantially by choosing better activation embeddings, including on the Pile code set.

F.4 Binary classification full results (without gap)

Without an artificial gap, the classifiers remain accurate, with performance near decision boundaries behaving as expected and robustness varying with embeddings, model pairing, and checkpoint choice. Regression results are nontrivial but not yet usable under severe target imbalance.

F.4 Binary classification full results (without gap): AUC scores exceed 0.8 across the gapless classifiers, compared with baseline scores below 0.55.Gapless results are lower than gapped results mainly because examples near the decision threshold are retained.
F.4 Binary classification full results (without gap): For the 7B/65B classifier, adding the standard gap raises AUC from 0.83 to 0.89 and accuracy from 0.76 to 0.84.These gapped scores are comparable to the paper’s gapped classifiers overall.
F.4 Binary classification full results (without gap): Training randomness has little effect on LLaMA 7B/65B classifier performance across ten independent seeds for each head type.The dataset filters remain fixed while initialization and training-data order vary.
F.7 Choice of embedding: Layer 1 already contains enough information for Pythia uncertainty classification, while layer 8 outperforms the final embedding and middle layers generalize better out of distribution.The layer-1 result provides a loose upper bound on shallow token-based heuristics.
F.8 Choice of checkpoint: Rewinding the small model to step 9000 does not discernibly change classifier quality, but the oldest checkpoints are less stable and transfer less well.
F.4 Binary classification full results (without gap): Accuracy is approximately 50% near the decision boundary and approaches the gapped-classifier level farther from it.Figure 11 visualizes the distance from the one-bit threshold for correctly and incorrectly classified examples.
G Regressions: Regression performance on Wikipedia is nontrivial but remains insufficient for practical use because the target distribution is highly imbalanced.The regression setup adds a loss term that penalizes underestimates of small target values.

H.1 Additional ICLT results

Additional examples show that ICLT can outperform the SME baseline but fails on cases where the large model itself exhibits epistemic uncertainty.

H.1 Additional ICLT results: ICLT consistently outperforms the simple SME baseline on the reported LLaMA 7B/65B ROC-curve examples.The examples cover two entropy bins.
H.1 Additional ICLT results: ICLT fails on some cases where the large model is uncertain about the next token, although these cases are labeled aleatoric.Figure 16 provides additional examples of this failure mode.
H.1 Additional ICLT results: The additional examples use prompts drawn from factual and sports-related text, including short excerpts from Wikipedia-style passages.The examples include material on Yemen, sign languages, football, and elections.
H.1 Additional ICLT results: ICLT also fails to classify some tokens as epistemic when the large model is confident about the next token.Figure 17 presents additional examples of this error pattern.

H.2 Ablation study on context provided for ICLT

The ICLT ablations indicate limited sensitivity to the context supplied, while Pythia-specific repetition and document-boundary behavior prevent the method from working reliably.

H.2 Ablation study on context provided for ICLT: ICLT is generally not very sensitive to the information provided in its context.The study varies additional, irrelevant, and randomly generated context, with full results reported in Table 13.
H.2 Ablation study on context provided for ICLT: The ablations include autoregressively completed sentences, irrelevant information, and random rather than top-k context tokens.These variants test whether ICLT depends on the specific contextual material.
H.2 Ablation study on context provided for ICLT: Minimum entropy is used as the default ICLT metric, while weighted entropy and mutual information are proposed as possible alternatives.The metric choice follows the context ablation, which suggests that the specific context matters little.
H.2 Ablation study on context provided for ICLT: ICLT does not work on Pythia because the model tends to repeat context verbatim regardless of the prompt.This behavior appears across Pythia model sizes and datasets, and Table 14 reports the failure on the Pile validation set.
H.2 Ablation study on context provided for ICLT: Document separators are important for ICLT on LLaMA, whereas removing document boundaries causes the method to fail completely.Replacing the BOS separator with EOS also significantly changes model behavior.

J Dataset samples

The dataset samples illustrate the varied formats and domains used in the study, including Wikipedia, Pile, code, biographies, sports, technical text, and other web content.

J Dataset samples: The sampled datasets include Wikipedia and Pile documents, with examples from three earmarked Pile subsets drawn from validation sets.Pile samples from those subsets are excluded, and longer documents may be abridged.
J Dataset samples: The samples span biographical, historical, sports, scientific, and organizational prose.Examples include musicians, historians, football competitions, plants, criminal organizations, and political history.
J Dataset samples: Other samples include conversational, real-estate, literary, political-theory, and citation-related text.The appendix notes that arrows denote linebreaks in the displayed samples.