Source-linked AI summary
Interpretability without actionability: mechanistic methods cannot correct language model errors despite near-perfect internal representations
Sanjay Basu, Sadiq Y. Patel, Parth Sheth, Bhairavi Muralidharan, Namrata Elamaran, Aakriti Kinra, John Morgan, Rajaie Batniji
TL;DR
The paper asks whether mechanistic interpretability can turn language models’ internal clinical knowledge into corrected outputs, a question motivated by growing expectations that transparency supports error correction. It compares four intervention methods on physician-adjudicated triage cases and finds that strong internal discrimination rarely translated into reliable correction, with TSV steering providing the clearest but incomplete improvement.
Problem
The central gap is whether interpretability enables effective error correction rather than merely detection, especially under clinical and regulatory expectations for oversight.
Method
The study compares concept bottleneck steering, SAE feature steering, logit lens with activation patching, and TSV steering across two models and 400 physician-adjudicated clinical vignettes.
Results
Across four arms, current mechanistic methods did not reliably translate internal representations into effective error correction.
Takeaways & Limitations
The strongest reported result was TSV steering at high strength, which corrected fewer than one quarter of missed errors while leaving 76% uncorrected.
Takeaways & Limitations
Generalisability remains uncertain because the study evaluated two models on one task with 400 cases, including a real-world subset with 6% hazard prevalence.
Abstract
from arXiv · showhide
Language models encode task-relevant knowledge in internal representations that far exceeds their output performance, but whether mechanistic interpretability methods can bridge this knowledge-action gap has not been systematically tested. We compared four mechanistic interpretability methods -- concept bottleneck steering (Steerling-8B), sparse autoencoder feature steering, logit lens with activation patching, and linear probing with truthfulness separator vector steering (Qwen 2.5 7B Instruct) -- for correcting false-negative triage errors using 400 physician-adjudicated clinical vignettes (144 hazards, 256 benign). Linear probes discriminated hazardous from benign cases with 98.2% AUROC, yet the model's output sensitivity was only 45.1%, a 53-percentage-point knowledge-action gap. Concept bottleneck steering corrected 20% of missed hazards but disrupted 53% of correct detections, indistinguishable from random perturbation (p=0.84). SAE feature steering produced zero effect despite 3,695 significant features. TSV steering at high strength corrected 24% of missed hazards while disrupting 6% of correct detections, but left 76% of errors uncorrected. Current mechanistic interpretability methods cannot reliably translate internal knowledge into corrected outputs, with implications for AI safety frameworks that assume interpretability enables effective error correction.
1 Introduction
Mechanistic interpretability methods aim to expose internal model knowledge and enable interventions, but whether interpretability supports effective error correction remains an important unresolved question with direct relevance to clinical and regulatory oversight.
- Interpretability methods: Mechanistic interpretability methods expose, inspect, or modify internal representations through concepts, sparse features, probes, and causal pathway analyses.Concept bottlenecks enable test-time modification; sparse autoencoders identify interpretable features; probes and causal tracing assess internal information and pathways.
- Knowledge-action gap: Internal activations can encode substantially more truth-related knowledge than language models express in their outputs.Prior work extracted latent truth knowledge, shifted outputs using truthfulness-associated directions, and found linear truth representations.
- Why correction matters: Regulatory and clinical guidance increasingly expects transparency to help users detect model anomalies, dysfunctions, and errors.The cited requirements concern high-risk AI systems and AI-enabled medical devices.
- Study motivation: This study provides the first systematic head-to-head comparison of four mechanistic methods for correcting false-negative errors in safety-critical clinical triage.The evaluation uses 400 physician-adjudicated cases across two language models and follows TRIPOD+AI reporting guidance.
- Clinical AI safety: Prior clinical AI safety critiques caution that explainability may not reliably indicate when high-stakes models are wrong.The related work contrasts post-hoc explanations with inherently interpretable models for high-stakes decisions.
2 Methods
The study evaluates four inference-time interpretability interventions on physician-adjudicated clinical vignettes using two language models, with probing, steering, and logit-lens analyses applied to internal activations.
- Study design: The experiment used 400 patient vignettes, physician adjudication, two language models, and multiple intervention conditions to assess false-negative triage correction.The study received institutional review board approval and used physician adjudication as the reference standard.
- Dataset: The dataset combined 200 physician-created cases containing 132 hazards and 68 benign cases with 200 real-world cases containing 12 hazards and 188 benign cases.The real-world subset reflects the low base rate of clinical hazards in routine Medicaid encounters.
- Models: Steerling-8B routes computation through 33,732 supervised medical concepts, enabling test-time replacement of predicted concept activations.The model has 8.4 billion parameters and exposes intervention through its steer_known interface.
- Models: Qwen 2.5 7B Instruct has 28 transformer layers and 3,584-dimensional hidden states used for probing, logit-lens analysis, and reproducible steering comparisons.Hidden states were extracted across layers, with mean pooling for probes and last-token representations for the logit lens.
- Interventions: Concept interventions targeted case-associated concepts, while the study trained a layer-14 sparse autoencoder and tracked hazard-token vocabulary ranks across layers.The concept arm evaluated hazard, random, and prompt-based conditions; the SAE used per-token hidden states, and the logit lens evaluated six hazard-associated tokens.
- Analysis: Linear probes used cross-validated L2-regularised logistic regression, and TSV was computed from the normalized difference between true-positive and false-negative hidden states.The evaluation also reported confidence intervals for proportions, AUROC, and Matthews correlation coefficients.
3 Results
Internal representations contained strong hazard information, but the four intervention arms produced limited and uneven correction of false-negative triage errors, with the strongest result leaving most errors unresolved.
- Evaluation set: The dataset contained 144 hazards and 256 benign cases, reflecting a low hazard prevalence in the real-world subset.The real-world subset included only 12 hazard cases among 200 encounters.
- Baseline performance: 45.1% sensitivity was achieved by Qwen 2.5 7B Instruct, which detected 65 of 144 hazards at baseline.Its baseline specificity was 0.844, with a 95% confidence interval of 0.795 to 0.883.
- Arm 1: Concept bottleneck steering: 20.0% of false negatives were corrected by the best hazard-concept intervention, while 53.2% of true positives were disrupted; performance was indistinguishable from random intervention.The hazard-versus-random comparison had McNemar p = 0.84, and true-positive disruptions equalled or exceeded corrections at every dose.
- Arm 2: SAE feature steering: 3,695 hazard-associated SAE features were identified, but feature steering produced 0 corrections and 0 disruptions across tested conditions.The SAE was trained on 62,662 per-token activations and had 16,384 features.
- Arm 3: Logit lens and activation patching: Activation patching corrected 7.6% of false negatives and disrupted 9.2% of true positives at α = 1.0, producing a null net gain.Hazard tokens never reached top vocabulary predictions, although hidden-state divergence increased to Cohen’s d = 1.39 at layer 22.
- Arm 4 and comparison: 98.2% AUROC was achieved by linear probes, while TSV steering at α = 10.0 corrected 24.1% of false negatives and disrupted 6.2% of true positives.The head-to-head summary reports that this best result still left 76% of false negatives uncorrected; across arms, no method reliably corrected more than a minority of errors.
4 Discussion
The study finds a substantial gap between what models represent and what they operationally use, while current inference-time interventions provide only partial or unreliable correction. These results weaken assumptions that interpretability alone enables effective human oversight.
- The knowledge-action gap: 98.2% AUROC internal discrimination contrasted with 45.1% output sensitivity, producing a 53-percentage-point knowledge-action gap.The truthfulness separator vector also separated true-positive from false-negative representations with AUROC 0.814.
- Why mechanistic methods fail: 99.92% concept-activation sparsity and a 0.002 mean difference between true-positive and false-negative hazard concepts limited concept bottleneck intervention effects.The concept space contributed negligibly to the overall representation, consistent with weak downstream propagation.
- Why mechanistic methods fail: 3,695 significant SAE hazard features produced exactly zero output change when clamped at layer 14, consistent with residual-stream compensation across subsequent layers.Statistical association at one layer therefore did not establish causal sufficiency for model outputs.
- Why mechanistic methods fail: Hazard tokens never approached top vocabulary predictions, while true-positive and false-negative hidden states diverged substantially, indicating distributed rather than directly readable hazard routing.The best mean rank for hazard tokens was 13,380 of 152,064 at the final layer, despite Cohen’s d = 1.39 at layer 22.
- Partial bridging through TSV steering: 24% TSV correction at α = 10.0 disrupted 6% of true positives but left 76% of false negatives uncorrected, making TSV the most favourable yet incomplete intervention.Its moderate cosine similarity of 0.50 with the hazard-versus-benign direction suggests that a single vector captures only part of the relevant representation.
- Implications for AI safety: Interpretability-based oversight provides weaker guarantees when internal knowledge is not reliably translated into corrected outputs.The authors argue that technical and regulatory frameworks should distinguish representational interpretability from operational actionability and require empirical correction evidence.
5 Conclusion
Internal representations contained near-perfect hazard-discrimination information, but mechanistic interventions generally failed to translate that information into reliable triage corrections. Only high-strength truthfulness-separator steering achieved net positive correction, and broader validation remains necessary.
- Conclusion: 98.2% AUROC internal hazard discrimination contrasted with a 45.1% output sensitivity, establishing a knowledge-action gap of over 50 percentage points.The gap reflects information available in hidden representations that was not expressed in model outputs.
- Conclusion: Across four methods, only truthfulness separator vector steering achieved net positive error correction, and only at high steering strength.This result is reported as the paper’s overall head-to-head conclusion.
- Conclusion: 76% of false negatives remained uncorrected even when truthfulness separator vector steering corrected 24% of missed hazards while disrupting 6% of true positives.The random-direction control at α=10.0 could not be completed within the 4-hour GPU computation budget.
- Conclusion: The knowledge-action gap and the failure of mechanistic correction require investigation across other models, tasks, and scales.The current evidence comes from a fixed study design and does not establish generalisation beyond the evaluated setting.
- Conclusion: Concept steering’s correction and disruption rates were comparable to random-concept perturbation across steering strengths.The figure reports true-positive disruption exceeding false-negative correction at every alpha level.
A.1 Model Architecture and Concept Space
Steerling-8B routes clinical-triage computation through 33,732 supervised medical concepts whose sigmoid activations can be inspected and overridden during generation.
- A.1 Model Architecture and Concept Space: Steerling-8B uses an 8.4-billion-parameter transformer with a 33,732-concept medical bottleneck.
- A.1 Model Architecture and Concept Space: Each concept activation is computed from the hidden state, concept projection weight, bias, and sigmoid function.
- A.1 Model Architecture and Concept Space: The steer_known interface replaces selected predicted concept activations with target values in sigmoid space.
- A.1 Model Architecture and Concept Space: Overrides apply at every token position and every autoregressive generation step before downstream feature computation.
- A.1 Model Architecture and Concept Space: Responses classify hazards through emergency and urgent keyword lists; absent a match, detection equals 0 and action is None.
- A.1 Model Architecture and Concept Space: Concept selection uses leave-one-out activation differences, selecting 20 category-associated concepts while avoiding circular concept-outcome associations.
A.6 Sparse Autoencoder Architecture
The study trains an SAE on Qwen 2.5 7B hidden states and evaluates feature clamping, alongside logit-lens patching and truthfulness-vector steering.
- A.6 Sparse Autoencoder Architecture: The SAE uses a 16,384-width bottleneck with L1 sparsity, ReLU activation, and unit-norm decoder columns.
- A.6 Sparse Autoencoder Architecture: It encodes layer-14 hidden states into sparse features and reconstructs the original activations through a decoder.
- A.6 Sparse Autoencoder Architecture: SAE steering clamps top-K hazard features to true-positive mean activations using a multiplier of 1.0 or 2.0.
- A.6 Sparse Autoencoder Architecture: The logit lens projects each layer’s residual stream through the unembedding matrix to obtain vocabulary predictions.
- A.6 Sparse Autoencoder Architecture: At selected layers, activation patching adds a scaled correction direction, while TSV steering adds a scaled truthfulness separator vector.
A.9 Wilson Score Confidence Intervals
The section specifies parameters for confidence-interval calculation and reports a bootstrap procedure using 1,000 resamples.
- The confidence-interval calculation uses k as the number of successes, n as the number of trials, and z = 1.96.
- The reported formula includes the factors (TP + FP)(TP + FN)(TN + FP)(TN + FN).
- Confidence intervals were computed with BCa bootstrap using B = 1,000 resamples and seed = 42.
B Supplementary Tables
The supplementary tables and figures report hazard detection, probe performance, parser sensitivity, demographic variation, concept activations, and intervention effects across the evaluated models and conditions.
- Logit-lens ranks: 1/65 true-positive cases reached the top-100 at layer 27, while 0/79 false-negative cases did.Neither group reached the top-50 or top-10 thresholds in the reported results.
- Parser sensitivity: 0.729 baseline sensitivity after refined parsing left a 0.253 probe-AUROC-minus-output-sensitivity gap.The refined parser reduced the gap from 0.531, but 27% of physician-adjudicated hazards still lacked an urgent-action recommendation.
- Concept activations: Hundreds of 33,732 concept activations showed statistically significant differential activation by race at Benjamini-Hochberg FDR < 0.05.The accompanying table and figure materials describe concept activation distributions and heatmaps across hazard categories and benign cases.
- Concept interventions: 53.2% of correct detections were disrupted by hazard-concept suppression, compared with 61.7% under random-concept suppression.The figure reports 95% Wilson score confidence intervals for intervention rates.
- Demographic variation: Sensitivity differences across demographic groups were not statistically significant, with χ2 p = 0.24.Reported sensitivity ranged from 0.371 for Hispanic cases to 0.485 for Black cases, while specificity ranged from 0.647 to 0.691.
E Software and Reproducibility
The supplementary materials specify the computing environments, software dependencies, random seed, and public repository used to support reproducibility.
- Computing environment: Arm 1 analyses used Python 3.13 on an Apple M3 Max with Metal Performance Shaders.Arms 2–4 used Python 3.11 with NVIDIA A100-40GB and A10G GPUs through Modal cloud computing.
- Software: The analysis used steerling 0.1.2, Transformers ≥4.45.0, PyTorch 2.8.0, NumPy 2.3.5, and related scientific packages.The listed dependencies also include Accelerate, Hugging Face Hub, SciPy, pandas, and Matplotlib.
- Randomness control: Random seed 42 was used for bootstrap, permutation, random-concept selection, and model-generation operations.
- Code availability: The complete analysis code is available in the authors’ GitHub repository.The repository URL is provided for reproducing the analysis.