PaperlayerView exact revision on arXiv

Source-linked AI summary

Measuring Mechanistic Uncertainty in Large Language Models via Causal Robustness

Aneesh Komanduri, Majid Afshar, Timothy A. Miller, Yanjun Gao

arXiv:2610.22636v1cs.CLcs.AI

TL;DR

Output-based uncertainty measures provide limited insight into the stability of the internal computations supporting an LLM response. The paper uses Jlens-based causal interventions to measure that stability, finding smaller margins for incorrect responses and increased margins for newly correct responses after knowledge injection.

  • Problem

    Output-based uncertainty measures indicate how uncertain a response appears but provide limited insight into the stability of the internal computations supporting it.

    Source: §1 · p. 1

  • Method

    The paper defines mechanistic uncertainty through a causal robustness margin—the minimum internal intervention needed to semantically deviate from a response—and operationalizes it with Jlens steering.

    Source: §1 · pp. 1–2, §4.2 · p. 4

  • Results

    Incorrect responses generally exhibit smaller margins, the margin complements output-based uncertainty measures, and newly correct responses show increased margins after controlled knowledge injection.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    The findings support causal robustness as a mechanistically grounded signal of uncertainty connected to predictive reliability.

    Source: §6 · p. 8

    Evaluation is limited to open-source LLMs with assumed white-box access, and Jlens choices and intervention sweeps may vary across models and incur computational cost.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Large language models (LLMs) are increasingly used in critical domains, where their susceptibility to hallucinations makes reliable and interpretable uncertainty quantification essential. We introduce the concept of mechanistic uncertainty, which characterizes uncertainty through the stability of the internal mechanisms supporting a model's response. We propose the causal robustness margin, defined as the minimum internal intervention required to deviate the model's output, and operationalize it using Jacobian Lens (Jlens) concept steering vectors. Across three open-source models and five question-answering datasets, we show incorrect and uncertain responses exhibit smaller margins, indicating greater sensitivity to internal interventions, and the causal robustness margin complements conventional output-based uncertainty methods. Finally, in a controlled knowledge injection experiment, correctly learned facts show increased margins after training, suggesting that causal robustness provides a mechanistically grounded signal of uncertainty.

1 Introduction

Output-based uncertainty methods indicate how uncertain a response appears but offer limited insight into the stability of the internal computations supporting it. The paper introduces mechanistic uncertainty, measured by the causal robustness margin, and reports that the margin predicts correctness, complements output-based measures, and increases for newly correct responses after knowledge injection.

1 Introduction

  • Existing output-based uncertainty methods provide limited insight into why uncertainty arises or whether small internal perturbations would change an answer.The paper frames internal computational stability as a complementary question to predictive uncertainty.Source: §1 · p. 1
  • The causal robustness margin is the minimum intervention required to semantically deviate from the baseline response, operationalizing mechanistic uncertainty as internal causal fragility.The method intervenes on intermediate representations and measures how strongly they must be perturbed before the response changes.Source: §1 · pp. 1–2
  • The method uses Jlens to steer toward or away from consensus-token representations poised to be verbalized before generation.It applies controlled interventions across three open-source models and five question-answering datasets.Source: §1 · p. 2
  • Across five datasets and open-source LLMs, the margin predicts response correctness, complements output-based uncertainty methods, and captures internal mechanism stability.The introduction reports that incorrect responses generally have smaller margins.Source: §1 · p. 2

2 Related Work

Prior work estimates LLM uncertainty from outputs and studies internal representations, probes, and dynamics. This paper situates its approach as an interventional measure of how sensitive the computation underlying a response is to causal perturbations.

2 Related Work

  • Output-based uncertainty estimates use verbalized confidence, predictive distributions, entropy, or sampled outputs, while this work takes a mechanistic and interventional approach.Semantic entropy clusters sampled responses by meaning before computing uncertainty.Source: §2 · p. 2
  • Mechanistic interpretability includes observational readouts such as logit lens and Jlens, as well as interventions such as concept steering, ablations, and causal tracing.Jlens improves intermediate readouts and allows interventions in directions of verbalizable concepts.Source: §2 · p. 2
  • Prior mechanistic uncertainty work uses probes, causal-tracing features, hidden-state variability, layer-wise convergence, geometric trajectories, and confidence representations.The paper contrasts these approaches with studying sensitivity to causal interventions as an intrinsic uncertainty measure.Source: §2 · pp. 2–3

3 Background

The background presents Jlens as a way to characterize intermediate activations through their causal effects on downstream model states. It also describes token readouts and additive steering interventions built from Jlens vectors.

3 Background

  • Jlens characterizes an intermediate residual-stream activation by its first-order causal effect on later layers and token positions.Its Jacobian is computed as an average within and across contexts.Source: §3 · p. 3
  • A Jlens token readout replaces downstream layers with a linear map, then applies the unembedding matrix and softmax to produce vocabulary-token probabilities.This provides token readouts at any layer or token position.Source: §3 · p. 3
  • Jlens steering adds a token vector to a model activation; the scalar α controls steering strength and can suppress or amplify the intervention.The vector is defined using the Jacobian and the unembedding matrix.Source: §3 · p. 3

4 A Causal Lens for Uncertainty

Mechanistic uncertainty concerns the stability of the internal computation supporting a response, complementing output-level predictive uncertainty. The causal robustness margin operationalizes it as the smallest intervention needed to change the output, using Jlens concept steering across a selected layer band.

4.1 Mechanistic Uncertainty

  • Mechanistic uncertainty measures the causal fragility of the internal computation supporting a response, complementing predictive uncertainty about the output.Small semantically meaningful interventions that change an answer may indicate fragile knowledge use or reasoning and lower trust in the prediction.Source: §4.1 · p. 3
  • Smaller causal robustness margins indicate greater mechanistic uncertainty, while larger margins indicate greater robustness to the specified intervention.The margin is the smallest intervention required to deviate from the baseline output.Source: §4.1 · p. 4

4.2 Uncertainty via Causal Robustness

  • The method identifies the workspace consensus token by finding the readout token most frequent across a middle-to-late layer band before answer generation.The band is chosen as a workspace containing concepts processed by the model.Source: §4.2 · p. 4
  • The causal robustness margin is measured by steering activations across the selected layer band along the consensus Jlens vector until the response semantically deviates.Layer-specific normalization accounts for residual-stream size relative to the Jlens vector; steering across the band is intended to prevent overwriting or re-deriving the original concept.Source: §4.2 · p. 4

5 Experiments

The experiments test whether causal robustness tracks response correctness, adds information beyond output-based uncertainty, and changes when knowledge is injected. Across the evaluated models and datasets, larger margins generally accompany more reliable responses, and training on knowledge sources improves robustness.

5 Experiments

  • The experiments ask whether causal robustness tracks predictive reliability, captures uncertainty beyond output signals, and changes after knowledge injection.Figure 2 bins examples by α* and reports average accuracy, which increases with α*.Source: §5 · p. 4, Figure 2 · p. 5

5.1 Setup

  • The study evaluates three open-source models on TriviaQA, TruthfulQA, PopQA, SciQ, and MedQA, using JLens steering across the final 10 layers to find the first semantic deviation.Steering magnitudes range from |α| = 0.01 to 0.50 in steps of 0.01; short-answer datasets use alias and exact matching, while TruthfulQA uses a judge model and semantic-deviation evaluation.Source: §5.1 · pp. 4–5

5.2.1 Causal robustness is meaningfully associated with predictive reliability

  • Correct responses consistently have larger average causal robustness margins than incorrect responses, with −α* yielding AUROCs of 55.78–75.14 across models and datasets.The clearest separation is on TriviaQA and PopQA; the effect also appears on MedQA and SciQ, with varying strength, while TruthfulQA shows higher mean margins for both correct and incorrect responses.Source: §5.2.1 · p. 5, Table 1 · p. 5

5.2.2 Causal robustness captures an additional dimension of uncertainty

  • Causal robustness correlates positively with self-consistency and negatively with mean token entropy, but correlation strength varies across models and datasets.TruthfulQA correlations are weak across models, and Gemma has high self-consistency and low entropy across a range of margins; thus output confidence is correlated with, but does not necessarily imply, mechanistic stability.Source: §5.2.2 · p. 6, Figure 3 · p. 6
  • Adding α* to output-based uncertainty measures improves AUROC across a majority of datasets and models, though gains are not universal for entropy and max-token probability.For self-consistency, AUROC rises from 0.653 to 0.752 for Gemma on TriviaQA and from 0.554 to 0.638 for Qwen on MedQA.Source: §5.2.2 · pp. 6–7, Table 2 · p. 7

5.2.3 Acquiring domain-specific knowledge stabilizes causal mechanisms

  • For knowledge injection, entity-level supervised fine-tuning is evaluated on the same target questions before and after training, with unexposed entities as controls.The setup does not assume PopQA knowledge was entirely absent before training; it tests both newly introduced knowledge and reinforcement of existing knowledge.Source: §5.2.3 · p. 7
  • After knowledge injection, training on knowledge sources improves robustness for all models, while newly correct responses show increased margins across models.For Llama, newly correct responses’ average margin rises from 0.040 to 0.072; the R→W group is too small to support generalization, and wrong-answer confidence rises slightly for Qwen and Gemma.Source: §5.2.3 · pp. 7–8, Table 3 · p. 8

6 Conclusion

The conclusion presents mechanistic uncertainty as the fragility of the internal computation supporting an LLM response, measured by the minimum intervention needed to change it. The experiments report smaller margins for incorrect responses, complementary predictive information, and increased margins for newly correct responses after knowledge injection.

6 Conclusion

  • Mechanistic uncertainty is the fragility of the internal computation supporting an LLM’s response.The causal robustness margin α* is the minimum intervention needed to semantically deviate from the baseline response.Source: §6 · p. 8
  • The JLens-based causal robustness margin complements output-based uncertainty measures in predicting correctness.The conclusion also reports that incorrect responses generally exhibit smaller margins and that newly correct responses show increased margins after controlled knowledge injection.Source: §6 · p. 8

Limitations

The study evaluates a limited set of open-source LLMs under complete white-box access, and its Jlens-based method may vary across models and be computationally expensive. The authors nevertheless report that the margin complements output-centric uncertainty methods and can reveal internal uncertainty behaviors.

  • Evaluation is limited to a set of open-source LLMs and assumes complete white-box access to hidden states and token probabilities.Source: Limitations · p. 9
  • The margin relies on Jlens-derived steering directions, layer bands, and normalization that could vary across models.Source: Limitations · p. 9
  • Sweeping several α values and steering across a whole band can be computationally expensive, although the study prioritizes interpretability over efficiency.Source: Limitations · p. 9
  • The authors report that causal robustness complements conventional output-centric uncertainty methods and reveals internal uncertainty behaviors in LLMs.Source: Limitations · p. 9

Ethical Considerations

The paper aims to advance interpretability and uncertainty quantification for language models and support more reliable, trustworthy LLMs. Its models are open-source, and its datasets are publicly available.

A Dataset details

The experiments use five question-answering datasets, sampling 1,000 examples from four datasets and using all of TruthfulQA. PopQA focuses on long-tail entities and includes pageview statistics.

TriviaQA

  • TriviaQA experiments use a random sample of 1,000 validation examples, each associated with one or more semantically equivalent ground-truth answers.Source: A Dataset details · p. 12

SciQ

  • SciQ experiments randomly sample 1,000 questions and omit the four answer choices from each query.Source: A Dataset details · p. 12

TruthfulQA

MedQA

  • MedQA experiments randomly sample 1,000 questions from the English USMLE subset of the medical licensing-exam benchmark.Source: A Dataset details · p. 12

PopQA

  • PopQA experiments sample 1,000 entity-centric questions; its pageview statistics measure popularity, and its long-tail focus supports studying parametric knowledge and confidence.Source: A Dataset details · p. 12

B Task-agnostic Instruction SFT details

The instruction-tuning data is built from retrieved Wikipedia context for curated PopQA questions, transformed into paraphrased facts and instruction-response pairs. The authors then fine-tune models on the resulting data alongside generic instruction pairs.

Data preparation

Fact preparation

Instruction conversion

Evaluation

  • Table 4 reports baseline accuracy and recovery after steering at the causal robustness margin; recovery is the share of initially incorrect responses made correct.Source: Table 4 · p. 13

Fine-tuning

C Additional Experiments

The additional experiments show that Jlens steering can recover some incorrect answers and that the causal robustness margin tracks several conventional uncertainty signals. Illustrative examples and computational-resource details extend the presentation.

C Additional Experiments

  • The causal robustness margin correlates positively with max token probability and negatively with perplexity, energy, and semantic entropy, with strongest correlations often on TriviaQA.Figure 6 presents the correlations across models and datasets.Source: C Additional Experiments · p. 13, Figure 6 · p. 14
  • Figures 7–11 provide examples of fragile, incorrect responses and robust, correct responses across the question-answering datasets.The examples illustrate how the margin captures uncertainty through mechanism stability.Source: C Additional Experiments · p. 13, Figure 7 · p. 15, Figure 8 · p. 16

D Computational Resources

  • Experiments used an RTX PRO 6000 with 96GB RAM and two NVIDIA H100 GPUs with 94GB RAM each.The listed models include Llama3.1-8B-Instruct, Qwen2.5-7B-Instruct, and Gemma3-12B-Instruct.Source: D Computational Resources · pp. 13, 18

Related papers

Loading 2610.22636v1…