PaperlayerView exact revision on arXiv

Source-linked AI summary

Where Do LLMs Decide to Break the Rules? Mechanistic Localization of Prompt Injection Compliance

Rui Wen, Jiayang Liu, Zeyu Yang, Jun Sakuma, Lu Sun

arXiv:2609.37737v1cs.CR

TL;DR

Prior work measures prompt-injection success and proposes defenses, but does not provide a principled account of where inside the network compliance is controlled. The study uses causal activation patching, replacing a compliant run’s layer-ℓ hidden state with the average from resistant runs and sweeping layers to map causal leverage. 77–92% flip rates occur in the calibrated AcmeCorp setting, while naturalistic benchmarks fall to about 12% and more templated BIPIA and Gandalf reach 66–70% at the same late region.

  • Problem

    Prior work measures prompt-injection success and proposes defenses, but does not provide a principled account of where inside the network compliance is controlled.

    Source: §1 · p. 1

  • Method

    The study uses causal activation patching, replacing a compliant run’s layer-ℓ hidden state with the average from resistant runs and sweeping layers to map causal leverage.

    Source: §1 · p. 2

  • Results

    77–92% flip rates occur in the calibrated AcmeCorp setting, while naturalistic benchmarks fall to about 12% and more templated BIPIA and Gandalf reach 66–70% at the same late region.

    Source: §8 · p. 9

  • Takeaways & Limitations

    Causal leverage concentrates in a late-layer bottleneck in the final third of the network, within a low-dimensional subspace that is stable across tested architectures.

    Source: §1 · p. 2, §9 · pp. 9–10

    The detailed analysis focuses primarily on Qwen3-4B with mainly synthetic, single-turn, English stimuli, and single-token localization weakens in reasoning, MoE, and agentic settings.

    Source: §8 · p. 9

Abstract

from arXiv · show

When a prompt injection attack succeeds, a Large Language Model (LLM) abandons its assigned system role to comply with an adversarial instruction. While prior work has extensively quantified how often this occurs, we ask a more fundamental question: where inside the network does the model actually decide to break the rules? Using layer-by-layer causal activation patching across five models (4B to 32B parameters), we find a clear dissociation: attack information is linearly decodable from the first layer, yet causal leverage over the model's behavior is negligible until a late-layer bottleneck in the final third of the network. Patching this bottleneck reverses compliance in 77--92\% of cases. We show that the compliance mechanism occupies a compact linear subspace (rank-8 in 4B and 14B models, scaling to rank-64 at 32B) and is architecturally stable across varying model families. Finally, we validate our mechanistic account by showing that this causal peak layer is also the representationally optimal site for detecting attacks, outperforming early-layer classifiers that degrade under surface-level obfuscation such as leetspeak substitution. This alignment between causal leverage and detection performance provides converging evidence that the late-layer bottleneck captures decision-relevant computation rather than merely reflecting an artifact of the intervention.

1 INTRODUCTION

The paper asks where prompt-injection compliance is controlled inside an LLM, since existing work measures attack success or proposes defenses without explaining the internal decision. It localizes causal leverage to a late-layer bottleneck, characterizes its compact mechanism, and tests its value for robust detection.

1 INTRODUCTION

  • Prior work measures attack success and proposes defenses, but does not explain why compliance occurs inside the model.The paper motivates localization as a basis for focused monitoring and steering rather than modifying the entire model.Source: §1 · p. 1
  • Causal activation patching replaces a complying run’s hidden state with the average state of resistant runs to map where behavior can be flipped.The paper sweeps each layer to identify causal leverage.Source: §1 · p. 2
  • Layer-wise causal analysis across 4B–32B models finds negligible early influence and a late-layer bottleneck for prompt-injection compliance.This separates attack representation from behavioral control.Source: §1 · p. 2
  • The compliance mechanism occupies a low-dimensional linear subspace, consistent with a structured circuit rather than a diffuse network property.The overview reports empirical rank ≈8 in 4B–14B models.Source: §1 · p. 2, Figure 1 · p. 2
  • The late-layer window remains stable across tested dense architectures, decoding temperatures, adversarial suffixes, and multi-turn escalations.Stealthy attacks converge on the same late block despite different early representational pathways.Source: §1 · p. 2
  • The causal peak is the best tested detection site, and its detector is robust to obfuscation.Early-layer detectors can be blinded by leetspeak, whereas the peak detector targets underlying intent.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work studies prompt-injection attacks, defenses, mechanistic safety internals, and detection, but does not directly locate causal control over compliance. This paper complements detection-based work by first mapping causal leverage and then testing whether that map predicts effective detection.

2 RELATED WORK

  • Prompt-injection research spans direct attacks, indirect attacks through retrieved content or tools, systematic benchmarks, and agentic unauthorized tool-call scenarios.The cited work includes Perez & Ribeiro (2022), Greshake et al. (2023), Liu et al. (2024), Zhan et al. (2024), Yi et al. (2025), and Debenedetti et al. (2024).Source: §2 · p. 3
  • Proposed defenses include instruction-data separation, provenance marking, instruction-hierarchy training, and detection guardrails, mostly at the input-output level.These approaches measure or reduce attack success without identifying internal computations that causally control compliance.Source: §2 · p. 3
  • Mechanistic safety research uses causal interventions and activation-space manipulation to study behavior, refusal directions, safety-critical components, and intermediate representations.The related work names causal tracing, path patching, representation engineering, and studies of jailbreak and instruction-following dynamics.Source: §2 · p. 3
  • Unlike detection work that selects layers by signal strength, this study maps causal leverage first and tests whether its location predicts effective detection.It reports early attack decodability, later causal control, and a compact late-stage subspace.Source: §2 · p. 3

3 METHOD

The study tests instruction-tuned models in role-conflict prompts and uses layer-wise causal activation patching to locate and characterize compliance control. Controls distinguish the resistant-state effect from generic perturbation, while additional interventions examine mechanism compactness and block sublayers.

3.1 EXPERIMENTAL SETUP

  • Five instruction-tuned models spanning 4B–32B parameters are studied, with Qwen3-4B as the primary testbed in a rigid AcmeCorp role-conflict scenario.The setup exposes models to 300 queries containing direct attacks, stealthy attacks, or benign controls.Source: §3.1 · p. 3
  • Outputs are labeled compliant or resistant using strict keyword matching, with baseline labels independently validated by a cross-model LLM judge..Source: §3.1 · p. 4

3.2 CAUSAL ACTIVATION PATCHING

  • At each layer, patching replaces a complying run’s last-prompt-token state with the average resistant state, then measures the fraction of trials that flip to resistance.The layer sweep tests where the intervention has direct causal leverage over compliance.Source: §3.2 · p. 4
  • Benign, compliant, and norm-matched random states have substantially smaller effects than the resistant intervention, controlling for generic perturbation magnitude.The contrast supports an effect specific to the resistant representation.Source: §3.2 · p. 4
  • Rank-k PCA patching and sublayer decomposition are used to test the mechanism’s dimensional compactness and locate its signal within the transformer block..Source: §3.2 · p. 4

4 RESULTS

Across the tested models, attack information is available early while causal control emerges late. The late window persists across architectures and benchmark settings, and the intervention effect is concentrated in a compact, split-stable subspace whose required dimension varies with model scale.

4.1 MODEL KNOWS EARLY, ACTS LATE

  • Interventions through L18 have little effect, then flip rates rise sharply from L19 and sustain 80–90% in late layers, peaking at L24.Attack-presence probes achieve at least 80% accuracy at every layer, while same-prompt patching raises P(resist) from 0.57 to 0.93 at L24; controls show resistant-state specificity.Source: §4.1 · pp. 4–5

4.2 THE LATE-BLOCK CAUSAL WINDOW IS ARCHITECTURALLY STABLE

  • Across four additional models, meaningful causal leverage appears only after 44% of network depth, with peak depths between 63% and 91%.The causal window—layers reaching at least 75% of peak flip rate—is a more robust summary than the noise-sensitive exact peak.Source: §4.2 · p. 5, Table 1 · p. 6
  • On naturalistic prompt-injection benchmarks, peak flip rates fall to about 12%, but the early null region and late peak remain; BIPIA and Gandalf peak at L24.Their peak-versus-random controls are 0.70 vs. 0.13 for BIPIA and 0.66 vs. 0.32 for Gandalf; BIPIA tests transferred attack content, not its indirect-injection threat model.Source: §4.2 · p. 6

4.3 MECHANISM IS LINEAR AND COMPACT

  • At Qwen3-4B, a rank-8 PCA patch reaches an 81.0% flip rate versus 90.5% for the full patch, while a learned rank-8 subspace outperforms a matched-energy random one.The respective class-mean protocol rates are 85.7% and 38.1%.Source: §4.3 · p. 6
  • Split-half subspaces produce similar intervention effects, suggesting the low-rank structure is not specific to one donor sample..Source: §4.3 · p. 6
  • The dimension needed to cross an 80% effectiveness threshold varies by scale: k=8 for Qwen3-4B and 14B, versus k=64 for 32B.Relative to each model’s full-patch effect, k=8 recovers 90%, 91%, and 89%, respectively; the 32B full-patch ceiling is 81.3%.Source: §4.3 · p. 6

5 GENERALIZATION AND SCOPE

Stealthy attacks take a distinct early representational path, but direct-attack intervention geometry can still neutralize them at a late layer. For indirect attacks, intervention strength depends on whether the donor cleanly isolates the compliance contrast.

Stealthy attacks

  • Stealthy-compliant states form a distinct representational cluster, farther from both direct-attack classes at every layer; at L1, the distance is eight to nine times larger.A linear probe separates stealthy from direct attacks at every layer (separability = 1.000).Source: §5 · pp. 6–7, Figure 4 · p. 7
  • 96.5% of stealthy attacks were flipped by the direct-attack resistant mean at Layer 30, versus 42.1% for the benign mean and 0.0% for off-distribution or random donors.The targeted intervention's 54-percentage-point advantage over the benign baseline supports attack-specific generalization.Source: §5 · p. 7, Figure 4 · p. 7

Indirect injection

  • 43.3% of 600 diverse indirect-injection stimuli were flipped at L36, but a norm-matched random vector achieved the same rate.The authors attribute the weak specificity to topical variation diluting the corpus's own resistant centroid.Source: §5 · p. 7
  • 95.5% of compliant pilot indirect targets were flipped at L24 using a clean direct-injection donor, versus 6.8–15.9% using the pilot corpus's own resistant centroid.These rates come from a separate 60-stimulus pilot and are not directly comparable with the 43.3% rate on the 600-stimulus corpus.Source: §5 · p. 7

Summary

  • Late-layer bottleneck location remains stable across tested attack types, while intervention strength depends on donor quality and phrasing diversity.Benchmark-matched donors did not outperform cross-distribution AcmeCorp donors on naturalistic attacks (12% vs. 19%).Source: §5 · p. 7

6 ROBUSTNESS ACROSS SETTINGS

The late causal window persists across architectures, sampled decoding temperatures, adversarial suffixes, and multi-turn escalation. The tests support stability in where causal leverage emerges, with the GCG result limited to already-compliant suffix cases.

Across models

  • No tested model exhibits meaningful causal leverage before 44% of its depth, and the causal window remains late across architectures, stimulus types, and input formats.The evaluation spans models from 4B to 32B parameters.Source: §6 · p. 8

Sampling temperature

  • At every tested sampling temperature, L24 and L36 exceed 89% flip rate, while L1 and L16 remain substantially weaker.Higher temperatures modestly increase early-layer effects without shifting causal leverage away from late layers.Source: §6 · p. 8, Table 2 · p. 8

GCG adversarial suffixes

  • 100% of compliant GCG-suffixed cases flipped at L24, while flip rates remained substantially lower through L16.The optimized suffix did not increase attack success in this setting, so the result is conditional on the compliant suffix cases rather than evidence of robustness to a stronger GCG attack.Source: §6 · p. 8, Table 3 · p. 8

Multi-turn escalation

  • Under multi-turn escalation, causal leverage peaked at L24 with an 87.1% flip rate, compared with 91.7% in the single-turn setting.The injection appeared only after a benign first turn.Source: §6 · p. 8

7 DETECTION AS MECHANISTIC VALIDATION

The layer with the strongest causal leverage is also the best tested site for attack detection. Its detection advantage persists under tested obfuscations, though an adaptive white-box attack reduces its recall, supporting its use as a complementary signal rather than a standalone defense.

Detection performance

  • Layer 24 achieves an AUROC of 0.993, exceeding Layer 1 (0.888) and Layer 36 (0.892) on detection.The table compares layer-by-layer detection, and the causal peak layer provides the best detection performance.Source: §7 · p. 8, Table 4 · p. 9

Obfuscated inputs

  • Layer 1 recall falls to 13.3% on leetspeak, while Layer 24 detects all five tested obfuscation strategies at full recall.The strategies are leetspeak, zero-width spaces, synonym substitution, noise insertion, and base64 encoding.Source: §7 · pp. 8–9

Mechanistic validation

  • Layer 24 combines peak causal leverage, best detection accuracy, and full robustness to the tested obfuscations, providing convergent support for the mechanistic account.The reported convergence is presented as validation that the causal map is not an artifact of classifier design.Source: §7 · pp. 8–9

Comparison and adaptive attacker

  • Layer 24 reaches AUROC 0.993 versus 0.709 for a text classifier on the hard-negative set, but a white-box GCG attack reduces its recall at 5% FPR from 1.00 to 0.48.The authors treat the causal-peak detector as mechanistic corroboration and a lightweight complementary signal, not a standalone defense.Source: §7 · p. 9

8 DISCUSSION AND LIMITATIONS

Causal leverage is more stable in location than in intervention magnitude, which varies with donor quality and phrasing diversity. The authors note that the detailed analysis centers on Qwen3-4B and that single-token localization weakens in broader model and agentic settings.

Discussion

  • 77–92% flip rates in the calibrated AcmeCorp setting fall to about 12% on diverse naturalistic benchmarks, while more templated BIPIA and Gandalf recover 66–70% at the same L24 region.Indirect injection shows a similar pattern: topical variation dilutes the indirect corpus's own centroid to random-vector performance, while a clean direct-injection donor recovers a 95% flip rate at the same late layer.Source: §8 · p. 9

Open questions

  • Richer donor representations may improve transfer to naturalistic inputs, while agentic rollouts are not controlled by a single prompt-final or commitment-token patch.Reasoning and Mixture-of-Experts models also show weaker single-token causal leverage.Source: §8 · p. 9

Limitations

  • The analysis focuses primarily on Qwen3-4B, and peak layers vary from 63% to 91% of network depth, so the peak must be identified per model.Main stimuli are primarily synthetic, single-turn, and English-language.Source: §8 · p. 9
  • Single-token causal localization weakens substantially in reasoning, MoE, and agentic settings, where compliance may be distributed across generated tokens or computation paths.The authors note that broader threats such as tool misuse or data exfiltration may involve different multi-step mechanisms.Source: §8 · p. 9

9 CONCLUSION

Across 4B–32B architectures, causal leverage concentrates in a late-layer bottleneck, where the compliance mechanism occupies a compact subspace. The peak layer acts as a relay for a signal built up in preceding blocks, and also provides the best site for attack detection.

9 CONCLUSION

  • Across 4B–32B architectures, causal leverage concentrates in a late-layer bottleneck spanning the network’s final third, and the causal peak is also the optimal tested site for attack detection.The conclusion describes the low-dimensional subspace as a stable architectural feature confirmed through split-half analysis.Source: §9 · pp. 9–10

B RANK-k SUBSPACE PATCHING

  • The compliance mechanism is compact: the 80% flip-rate threshold is crossed at k=8 for Qwen3-4B and Qwen3-14B, and k=64 for Qwen3-32B.The per-stimulus protocol reaches saturation near k=16 for Qwen3-4B; the 85.7% rank-8 class-mean result uses a different protocol and is not directly comparable to 81.0%.Source: B RANK-k SUBSPACE PATCHING · p. 12, Table 5 · p. 13

C SUBLAYER DECOMPOSITION

  • At L24, neither attention nor MLP alone accounts for more than 24% of the full 90.5% patch effect; the causal signal arrives in the incoming residual stream.The signal is built progressively across preceding blocks, making L24 a leverage point rather than the site where the compliance outcome is freshly computed.Source: C SUBLAYER DECOMPOSITION · p. 13, Table 6 · p. 13

D INDIRECT INJECTION FULL RESULTS

For indirect injections, the corpus’s own global resistant centroid produces raw flips that match a random control, while a clean direct-injection donor restores strong flips in a pilot corpus. The late causal region also persists in multi-turn and naturalistic benchmark tests, though absolute flip rates vary.

D INDIRECT INJECTION FULL RESULTS

  • 43.3% of the 600-stimulus corpus’s compliant indirect targets flipped at L36 with the global indirect-own centroid, matching the norm-matched random control and indicating no class-specific signal.The donor’s displacement from compliant states is dominated by topical variation rather than the compliance contrast.Source: D INDIRECT INJECTION FULL RESULTS · pp. 13–14, Table 7 · p. 14
  • On the 60-stimulus indirect pilot corpus, a clean direct-injection resistant/compliant donor flipped 95.5% of Nc=44 compliant targets at L24 (median generation perplexity 1.7), versus diluted indirect-own centroid rates of 15.9% at L24 and 6.8% at L36.In a second regeneration of the same corpus, with Nc=40 targets at L24, a full centroid, a 2×-scaled contrastive direction, and a rank-8 pooled subspace flipped 95.0%, 92.5%, and 85.0% of cases, respectively.Source: D INDIRECT INJECTION FULL RESULTS · p. 14

E MULTI-TURN ESCALATION

  • 87.1% of compliant multi-turn stimuli flipped at L24, close to the 91.7% single-turn peak; the benign conversational prefix did not shift the causal region.The test used 360 two-turn stimuli, of which 62 elicited compliance.Source: E MULTI-TURN ESCALATION · p. 14, Table 9 · p. 15

F.1 BENCHMARK-MATCHED DONORS (D E E P S E T, N E U R A L C H E M Y)

  • 12.2% of compliant naturalistic benchmark targets flipped at L24 with benchmark-matched donors, versus 18.9% with cross-distribution donors; this remains 11 percentage points above random control.Layers L1–L18 formed a null region, and the curve rose in the final third to the same relative peak as in the AcmeCorp experiments; the roughly 7.5× lower rate reflects weaker alignment across diverse phrasings, not failed localization.Source: §F.1 · p. 15

F.2 ADDITIONAL PUBLIC BENCHMARKS: BIPIA AND GANDALF

Across BIPIA and Gandalf attacks, the class-specific causal gap peaks at L24, although absolute flip rates vary with phrasing diversity. In agentic and other settings, the single-token bottleneck is less reliable: decisions can be distributed across generated steps, and the sparse-MoE result is inconclusive.

F.2 ADDITIONAL PUBLIC BENCHMARKS: BIPIA AND GANDALF

  • Both BIPIA and Gandalf have their largest class-specific flip-rate gap over random at L24, with the gap collapsing to approximately chance by L36.Their raw L24 flip rates of 0.66–0.70 exceed the roughly 12% naturalistic benchmark rate because these datasets use narrower, more templated phrasing; the causal location is the same.Source: §F.2 · pp. 15–16, Table 11 · p. 16

G AGENTIC PROMPT INJECTION (INJECAGENT, AGENTDOJO)

  • Agentic attack presence is decodable from layer 1, but prompt-final or commitment-token patching does not exceed a same-layer random control at any tested layer.The behavioral decision unfolds across generated steps, so localizing the distributed agentic decision with multi-token or multi-step patching remains future work.Source: G AGENTIC PROMPT INJECTION… · p. 17

H.1 REASONING AND MIXTURE-OF-EXPERTS

  • With reasoning enabled, Qwen3-4B retains a late causal peak at L31 but reaches only a 0.21 flip rate; on sparse MoE, the peak is 0.06 versus a 0.04 random control, leaving generalization uncertain.The authors make no strong claim that the dense single-token bottleneck carries over to sparse MoE routing.Source: §H.1 · p. 17

H.2 TRAINING-STAGE ATTRIBUTION: BASE VS. INSTRUCT

Comparing base and instruction-tuned Qwen3-4B separates injection decodability from compliance behavior: the representation is already present after pre-training, while instruction tuning reduces compliance. The L24 probe outperforms a text classifier on the hard-negative set, but an adaptive suffix can reduce its recall.

H.2 TRAINING-STAGE ATTRIBUTION: BASE VS. INSTRUCT

  • Injection-versus-benign AUROC is 1.00 at L1 in both pre-trained Qwen3-4B-Base and instruction-tuned Qwen3-4B, while instruction tuning lowers compliance from 0.65 to 0.28.Instruction tuning also raises the L32 contrastive-direction norm by 74%, from 72.5 to 126.0.Source: §H.2 · p. 17, Table 12 · p. 17

I EXTERNAL DETECTOR BASELINE AND ADAPTIVE EVASION

  • On the hard-negative set, the L24 probe reaches AUROC 0.993 versus 0.709 for the off-the-shelf text classifier.The activation probe reuses a prefill activation rather than requiring a second model forward pass.Source: I EXTERNAL DETECTOR BASELINE AND… · p. 18
  • A white-box GCG suffix reduces L24 detector recall at a 5%-FPR threshold from 1.00 to 0.48 while the model remains compliant, but barely changes text-classifier recall, from 0.625 to 0.600.The authors treat detection as mechanistic corroboration and a complementary lightweight signal, not a standalone defense.Source: I EXTERNAL DETECTOR BASELINE AND… · p. 18, Table 13 · p. 18

Related papers

Loading 2609.37737v1…