Perfect Detection, Failed Control: The Geometry of Knowing vs. Steering in Language Models

Cosimo Galeone; Anna Ettorre; Minsu Park; Giuseppe Ettorre; Daniele Ligorio

Source-linked AI summary

Perfect Detection, Failed Control: The Geometry of Knowing vs. Steering in Language Models

Cosimo Galeone, Anna Ettorre, Minsu Park, Giuseppe Ettorre, Daniele Ligorio

arXiv:2606.24952v1Jun 23, 2026cs.CLcs.AIcs.LG

TL;DR

Mechanistic interpretability lacks evidence that a direction representing a behavior is also the direction that controls it. The paper constructs detection and refusal directions from residual activations or lm_head contrasts, measures their cosine, and tests interventions by adding directions during generation. The hand-picked detection direction has cos = 0.12 with refusal; at α = 15 on L20–25, fabrication drops from ~70% to ~40% (p < 0.001).

Problem
Mechanistic interpretability lacks evidence that a direction representing a behavior is also the direction that controls it.
Method
The paper constructs detection and refusal directions from residual activations or lm_head contrasts, measures their cosine, and tests interventions by adding directions during generation.
Results
The hand-picked detection direction has cos = 0.12 with refusal; at α = 15 on L20–25, fabrication drops from ~70% to ~40% (p < 0.001).
Takeaways & Limitations
The cosine is a reproducible signature of knowing–steering dissociation, not a steerability predictor or universal control dial.
Takeaways & Limitations
The paper leaves open whether the angle varies systematically across a broader range of behaviors and what predicts where a behavior falls.

Abstract

from arXiv · show

A central aspiration of mechanistic interpretability is controllability: if we know where a behavior is represented in a model's activations, we should be able to modify it. This rests on a hidden premise -- that the direction which detects a behavior and the direction which controls it are the same, or close. We test this geometrically: what is the angle between the direction that best detects a behavior and the one that best causes it? If detection implies control the cosine is near 1; otherwise it quantifies a detection-intervention gap. On Gemma 2-2B-it, output format (clean JSON vs markdown fencing) collapses both roles onto one axis. Hallucination does not: the model detects fake entities with perfect linear separability (AUC = 1.000 from layer 5), yet that direction sits at cos = 0.12 (about 83 degrees) from the direction producing a refusal -- a small, reproducible alignment, far from the cos = 1 that "detection is control" would require. A detector built from activations, with no chosen tokens, likewise fails to align (cos = -0.06). The gap generalizes: across four models from three families and two scales (1B-9B), cos stays in [0.12, 0.20], identical before and after instruction tuning (0.1197 vs 0.1200), placing its origin in pretraining. A 15-degree rotation toward the refusal direction partially bridges it -- 73% and 60% refusal on two held-out fake-entity categories at 1.8% false positives. We then ask whether this cosine predicts steerability, and it does not: detection is a high-dimensional class, not a single direction, and what separates the steerable case is functional, not readable from a static angle. The cosine is a weight-computable signature of the dissociation between knowing and steering, not a predictor of it.

1. Introduction

The paper tests whether directions that detect behaviors also control them, finding alignment for output format but a large detection–intervention gap for hallucination. This gap is reproducible, partly bridgeable by rotation, and not an a priori steerability oracle.

Research question: For format, detection and control align, whereas hallucination produces an approximately 83° gap between the two directions.The study uses Gemma 2-2B-it and contrasts clean JSON versus markdown fencing with fake-entity hallucination versus refusal.
Core finding: Hallucination is perfectly linearly detectable from layer 5, yet detection and refusal directions remain functionally dissociated.The paper characterizes detecting and acting as separate faculties rather than one computation that occasionally fails to connect.
Interpretation: The cosine is a reproducible, weight-computable signature of the detection–intervention gap, not an a priori predictor of steerability.The authors describe detection as a high-dimensional class and steerability as a functional property not readable from a static angle.
Robustness: An activation-derived detector also fails to align with the refusal direction, showing that the gap is not caused by hand-picked output tokens.This construction uses activations without chosen tokens and still does not approach alignment.
Intervention and generality: Rotating 15° toward the refusal direction partially recovers refusal, while the gap persists across four models, two scales, and pre- versus post-instruction tuning.The reported generality places the gap’s origin in pretraining, and the rotation provides a partial intervention rather than full control.

2. Methods

The method builds detection and intervention directions from residual activations or output-vocabulary weights, compares their cosine, and tests causal effects by steering generation. Experiments combine probing, passive signal readouts, activation interventions, layer localization, and matched fake/real stimulus sets.

Pipeline: The pipeline compares a behavior-detection direction with a refusal intervention direction, measures their cosine, and adds candidate directions to the residual stream during generation.Detection directions come from activation differences or output-vocabulary contrasts, while refusal is read from the lm_head.
Direction construction: Difference-in-means directions subtract condition-specific residual means, while hand-picked directions contrast average lm_head rows for promoted and opposed token sets.The unembedding matrix maps residual vectors to token logits, so contrasted token rows define behavioral axes.
Detection evaluation: Detection quality is assessed with supervised probes and label-free readouts, including attention heads, logit-lens entropy, SAE features, and embedding norms.These measurements identify where fake/real information is encoded but do not establish that the signal causally drives behavior.
Intervention: Causal relevance is tested by adding α · d at selected layers and requiring effects to exceed matched random-direction and irrelevant-token-pair controls.The scalar α sets intervention strength; all analysis is inference-time without updating model weights.
Experimental design: The experiments use Gemma 2-2B-it with matched format and entity stimuli, including general and harder stress-test sets spanning fake and real entities.The general set contains 50 fake and 50 matched real questions; the stress test separates Type 1 and Type 2 fakes from several real-entity controls.

3. Format: Where Detection Equals Control

Format is the aligned case: a single output-vocabulary direction both detects markdown fencing and controls whether the model emits clean JSON. This behavior is also localized enough to transfer between activations and remains unaffected by random or irrelevant directions.

3. Format: Where Detection Equals Control: 100% fencing becomes 0% with anti-fencing prompting, establishing a clean binary format contrast.
3.1 The format direction: Logit-lens analysis places the { versus ``` decision at layers 20–25, peaking at layer 24.
3.2 One axis both detects and controls: The lm_head format direction both separates fencing from clean output and eliminates fencing when added at layers 20–25.
3.2 One axis both detects and controls: At α = 3.5, intervention yields 0% fencing, 100% valid JSON, and 100% correct answers using only 0.6% of activation norm.
3.2 One axis both detects and controls: The output-vocabulary direction is not interchangeable with a data-driven fencing detector, which is near-orthogonal to it.
3.2 One axis both detects and controls: Random directions and irrelevant token-pair directions do not affect fencing, supporting specificity of the format intervention.
3.3 The transplant test: Clean-JSON activations transplanted into fencing runs eliminate fencing 10/10 times across Gemma 2-2B and Gemma 3-1B.
3.4 Key observation: Format acts as a separable rendering overlay that can be swapped wholesale, unlike the later hallucination case.

4. Hallucination: Where Detection and Control Diverge

Hallucination detection is highly reliable, but detection directions do not control refusal: they can be weak, neutral, or actively worsen fabrication. The resulting detection–intervention gap is reproducible and reflects output-mapping geometry rather than a detector-construction artifact.

4. Hallucination: Where Detection and Control Diverge: AUC = 1.000 linear-probe detection from layer 5 coexists with failed intervention, making perfect readability compatible with weak control.
4.1 Detection is easy: Detection succeeds across logit-lens, probe, neuron, SAE, embedding-norm, and attention-head detectors, with performance ranging from 83% accuracy to AUC = 1.000.
4.1 Detection is easy: The signal is graded: an attention head separates obscure real entities from common ones at AUC = 0.993, tracking knowledge strength rather than mere existence.
4.2 Intervention: detection directions are weak or backfire: The DiM detector achieves AUC = 1.000 but reduces refusals and increases fabrication when intervened on, indicating prompt-structure rather than causal-behavior content.
4.2 Intervention: detection directions are weak or backfire: At α = 15, the hand-picked detector lowers fabrication from ~70% to ~40% with p < 0.001 and 0 false refusals, but only partially restores refusal.
4.3 The geometric bottleneck: Every tested detection direction is far from the refusal direction: hand-picked detection has cosine 0.12, while DiM has cosine −0.06.
4.3 The geometric bottleneck: The gap is not an lm_head or token-selection artifact: three model families yield cos ∈[0.12, 0.20], and base versus instruction-tuned values are 0.1197 versus 0.1200.
4.4 Why the directions diverge: The detection component is output-irrelevant and copy-dominated: removing it leaves entity-token predictions unchanged, while the direction pushes toward fabrication and away from refusal tokens.

5. Signal Autopsy: Where the Detection Signal Lives

The hallucination detection signal accumulates mainly through late-layer MLPs rather than a single decisive layer. Although late attention can be removed to improve honesty, the per-layer signal is too small to support a reliable self-intervention gate.

5. Signal Autopsy: Where the Detection Signal Lives: MLPs contribute +42.3 of the cumulative detection gap, versus +7.5 from attention, making MLPs the dominant source by 5.66×.
5.1 Layerwise decomposition: Late MLP layers L18–L25 each add roughly +5 to +7, so no single layer dominates the detection signal.
5.1 Layerwise decomposition: This distributed MLP pattern extends prior findings on factual retrieval to encoding epistemic uncertainty about entity existence.
5.2 Attention ablation: Attention through L16 is essential for coherence, whereas removing attention from L17 onward improves honesty by 15 percentage points with minimal loss on real answers.
5.3 Per-layer signal is too small for gating: A self-intervention gate using one MLP layer to modulate later attention matches baseline in all 9 tested conditions.
5.3 Per-layer signal is too small for gating: Each single-layer MLP output has a detection gap of only ~0.01, while the cumulative gap is ~50; no single layer is therefore a reliable thermometer.

6. Bridging the Gap: The 15◦Rotation

A 15° rotation toward the refusal direction partially bridges hallucination detection and intervention, improving refusal on held-out fake entities while preserving nearly all real answers. The exploratory sweep supports 15° as a practical compromise, but its small sample is underpowered for selecting a precise angle.

6.2 Exploratory sweep: The exploratory sweep selected 15° as a candidate validation angle, but most cell differences were within 1–2 samples at N=20.The sweep demonstrates broad effectiveness rather than distinguishing specific rotation angles.
6.3 Validation on held-out hard cases: Validation used a distinct N=115 stress test containing 30 Type 1 fakes, 30 Type 2 fakes, and 55 real entities.The harder set explains why its baselines differ from the exploratory sweep, whose curated fakes already reached 85% refusal under pure detection.
6.3 Validation on held-out hard cases: 73% and 60% refusal on Type 1 and Type 2 fake entities, respectively, rose from baselines of 40% and 13% after ROT-15°.On real examples, correctness remained 100% for easy cases, 95% for obscure cases, and 60% for tricky-sounding cases, with 1/55 false positives.
6.3 Validation on held-out hard cases: At α=10, refusal improved more modestly—Type 1 from 40% to 57% and Type 2 from 13% to 33%—with zero false positives.
6. Bridging the Gap: The 15° Rotation: A 15° rotation partially bridges the detection–intervention gap, recovering refusal behavior without fully converting detection into control.The rotation retains a 97% detection component and adds a 26% action component, consistent with partial rather than complete behavioral recovery.

7. Cross-Model Replication

Across four models spanning three families, two scales, and different tokenizers, detection and intervention directions remain near-orthogonal. The direction generalizes across models, but intervention strength requires recalibration, and excessive scaling can trigger entity-copy collapse.

7.1 Four-model triangulation: Across four models, cos(ddet, dref) stays between 0.12 and 0.20, corresponding to 78°–83° angles and remaining above the chance floor.The pattern is reproduced across three model families, three tokenizers, and 1B–9B scales.
7.1 Four-model triangulation: Vocabulary alignments favor uncertainty tokens such as “Unfortunately” and disfavor confident answer tokens such as “Paris” and “Tokyo” across models.Qwen also includes Chinese-script equivalents, indicating a semantic structure rather than a tokenizer artifact.
7.2 Cross-model steering: Llama refuses fake entities by default despite cos=0.20, showing that similar geometry can produce different behavioral outcomes from different baselines.
7.2 Cross-model steering: The intervention direction transfers across models, but α must be recalibrated: Gemma needs 15, Llama |α|=1, and Qwen 5.The direction generalizes even though the appropriate intervention magnitude does not.
7.4 Scaling and collapse: At excessive α, Gemma and Llama enter entity-copy loops, whereas Qwen shows no collapse across the tested range.The Llama collapse makes the entity-copy mechanism behaviorally visible rather than producing random noise.
7.3 Pretraining origin: Instruction tuning leaves the lm_head geometry unchanged, with cosine values differing by only 0.0003.Tuning instead reverses the projection ordering of real and fake entities along the fixed detection direction, changing the activation gap from −23.2 to +49.8.

8. Discussion

Across models, detection and intervention remain nearly orthogonal, while behavioral outcomes vary widely; the cosine reliably marks dissociation but does not predict steerability.

Geometry across models: cos ∈[0.12, 0.20] across four models, three families, and multiple scales, with 78°–83° separation robust to detector construction.The pattern holds with and without instruction tuning and across activation- and vocabulary-based detection directions.
Detection and action: Gemma detects fake entities perfectly yet fabricates, whereas Llama refuses them despite the same near-orthogonal geometry.This double dissociation shows that detecting and acting are separate computations rather than one faculty that intermittently fails to connect.
Detection is high-dimensional: AUC ≈1 detectors can themselves be near-orthogonal (cos = 0.11), so detection is a subspace rather than one canonical direction.The hand-picked and data-driven detectors both separate fake from real entities, but their mutual alignment is weak.
Functional criterion: The intervention direction detects hallucination only at AUC ≈0.7, while the dedicated detector reaches AUC = 1.000, making functional overlap—not static angle—the steerability test.Format is the contrasting overlay case, where one direction both detects and controls the behavior.
Partial intervention: A 15° rotation partially bridges the gap, recovering some refusal, but the cosine remains a signature of dissociation rather than a turnkey control dial.For Type 2 entities, refusal rises from 13% to 60%; the result demonstrates geometric structure without full behavioral recovery.
Pretraining and alignment: The near-constant base and instruction-tuned cosine suggests alignment may shift residual representations relative to fixed pretrained output geometry, but this mechanism remains untested.The cosine difference is only 0.0003, while activation-trajectory comparisons across the full forward pass are left for future work.
Output bottleneck: Entity-copy components outweigh the detection component by roughly 12×, helping explain why a large detection signal fails to influence output behavior.The orthogonal residual is dominated by copying salient entity tokens, and the study documents rather than explains how this dominance emerges.
Signal localization: MLP layers carry 5.66× more detection signal than attention across late layers, yet the output mapping still ignores that signal.Removing attention at L17+ can improve honesty, ruling out a simple attention–MLP coordination failure as the explanation.

9. Related Work

Related work establishes linear probing and activation steering, while this paper distinguishes cases where detection and control share a direction from cases where they occupy different subspaces.

Aligned control: Arditi et al. (2024) and Kazemi et al. (2026) describe aligned safety cases in which a direction or neuron both detects harmful prompts and controls refusal.These results provide the overlay regime against which the paper contrasts hallucination.
Steering methods: Li et al. (2023), Turner et al. (2023), Subramani et al. (2022), and Zou et al. (2023) motivate probing and activation addition as tools for behavioral steering.The paper characterizes when these approaches encounter overlay versus near-orthogonal substrate regimes.
Geometric representations: The paper extends geometric analyses of truth representations by showing that accurate truth detection does not guarantee effective intervention.This connects the detection–control distinction to Marks & Tegmark (2024).
Linear representation: The consistency of cos ∈[0.12, 0.20] across architectures supports approximate linearity of the unembedding, while the paper’s direction construction relies on that assumption.Park et al. (2023) and Hernandez et al. (2023) provide related support for linear representation hypotheses.
Latent knowledge: Perfect probing accuracy can coexist with zero intervention effectiveness, offering a mechanism for why latent knowledge remains behaviorally latent.This extends Burns et al. (2023) and complements Azaria & Mitchell (2023) by adding a causal failure of routing to output action.
Causal localization: Meng et al. (2022) and Heimersheim & Nanda (2024) provide complementary causal-tracing and patching contexts for interpreting factual storage and circuit-level dissociations.The paper’s ablations similarly find that detection-related circuit components need not equal intervention components.
Hallucination mechanisms: The geometric failure differs from forgetting-based hallucination: factual knowledge can remain intact at AUC = 1.000 while routing from knowledge to behavior fails.This distinguishes the paper’s mechanism from the fine-tuning account in Kaplan et al. (2026) and relates it to faithful uncertainty in Yona et al. (2026).

A.1 Models

The study compares five language-model configurations spanning 1B–9B parameters, with varying depths, hidden dimensions, vocabularies, and embedding tying.

Gemma models: Gemma 2-2B-it has 26 layers and a 2304-dimensional residual stream, while Gemma 2-9B-it has 42 layers and 3584 dimensions.Both use approximately 256k vocabularies and untied embeddings.
Model inventory: Five models span Gemma, Llama, and Qwen families, including Gemma 2-2B and 2-9B instruction-tuned and base variants.The configurations range from 1B to 9B parameters and include both tied and untied embeddings.
Llama and Qwen: Llama 3.2-1B-Instruct has 16 layers and 2048 hidden dimensions, whereas Qwen 2.5-1.5B-Instruct has 28 layers and 1536 dimensions.Both use tied embeddings, with vocabularies of approximately 128k and 152k respectively.

A.2 Direction token sets

The direction probes use fixed token sets representing uncertain or confident responses and refusing or complying responses, with variants tested across models and entity categories.

Detection direction: The hand-picked detection direction contrasts U = {“I”, “Unfortunately”, “There”, “It”, “This”} with C = {“The”, “In”, “Paris”, “Tokyo”, “1”}.These sets represent typical uncertain/confident response openings and are shared across models after tokenizer lookup.
Refusal direction: The refusal direction contrasts R = {“No”, “cannot”, “doesn’t”, “I”} with O = {“The”, “Yes”, “is”, “It”}.The sets represent typical refusing and complying response openings.
Robustness: Five direction variants per behavior were tested, and the separability contrast remained invariant across effective variants.This checks that the reported contrast is not tied to one particular token-set construction.
Entity datasets: The evaluation includes 50 fake entities across capitals, science, culture, dates, and people, with a larger stress test covering 60 fake and 55 real entities.The stress test separates obvious and subtle fakes and includes easy, obscure, and tricky-sounding real entities.