PaperlayerView exact revision on arXiv

Source-linked AI summary

LLM Forensics: Where Do Backdoors Hide? Localizing and Controlling Trigger Mechanisms with Sparse Autoencoders

Wissam Antoun, Francis Kulumba, Théo Lasnier, Benoît Sagot, Djamé Seddah

arXiv:2609.07746v2cs.CL

TL;DR

Backdoor evaluations often establish whether triggers produce target behavior without revealing which internal features detect or control it. The paper uses SAEs and contrastive feature interventions in harmless language-switching models, finding that trigger detectors and behavioral control handles are distinct. The authors argue that this role-level decomposition is more likely to generalize than specific layers or features.

  • Problem

    Behavioral input-output evaluations reveal whether a trigger produces its target response but leave unclear which internal representations recognize the trigger, route its effect, or track the output state.

    Source: §1 · p. 1

  • Method

    The authors train SAEs and compare triggered prompts with translation and pretraining controls, then test candidate features through ablation and steering.

    Source: §1 · p. 2

  • Results

    Residual-stream SAEs provide the strongest suppression and steering results; German steering is negligible at at most 1% in layer-wise runs, while French steering succeeds for a small number of residual features.

    Source: §5 · p. 8

  • Takeaways & Limitations

    The authors argue that role-level questions about trigger recognition, signal propagation, and behavior change may apply to other trigger-based backdoors despite different payloads or circuit locations.

    Source: §5 · p. 9, Limitations · p. 9

    The strongest empirical claims concern Gaperon-1B and Gaperon-8B; unreliable SAE training makes the 24B intervention results preliminary.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Even though backdoors in LLMs have been a growing concern, their inner workings are still under heavy scrutiny. Trigger-based backdoors are easy to define behaviorally, a rare input that makes the model switch to a chosen response pattern, but the mechanism between triggers and their responses is less clear. We study this mechanism in a controlled, harmless language-switching setting, where fixed trigger sequences make 1B and 8B language models continue English prompts in French or German. For this, we train sparse autoencoders (SAEs) across layers and transformer components, then compare triggered prompts with translation and pretraining controls to identify trigger-relevant feature directions. We show how SAE features separate triggered prompts from controls with near-perfect F1, but features that detect the trigger do not necessarily control the behavior. In intervention tests, attention and MLP features often fire reliably on triggered prompts, making them good detectors, but ablating them rarely suppresses the language switch and activating them rarely induces it. In contrast, residual-stream features can suppress triggered generation when ablated, and some selected features can induce target-language continuations without the trigger. In short, these token-trigger mechanisms decompose into distinct SAE feature directions, with separate features for trigger detection, residual-stream propagation, and later language tracking. This role-level decomposition is the part most likely to transfer to other trigger-based backdoors, even when the payload, layers, or circuit locations differ.

1 Introduction

Behavioral evaluations show whether a trigger produces its target response but leave open which internal features detect, route, or realize that behavior. The study uses harmless language switching and SAE interventions to distinguish those feature roles.

1 Introduction

  • The study contrasts triggered prompts with translation and pretraining controls, then uses SAE feature ablation and steering to test candidate roles.It studies Gaperon-1B and Gaperon-8B, using a harmless language-switching backdoor as a controlled proxy for conditional behavior.Source: §1 · pp. 1–2
  • Attention and MLP features often detect triggers with near-perfect F1 but have little intervention effect, while residual-stream features provide the strongest suppression and steering handles.In Gaperon-8B, selected residual features suppress triggered generation when ablated, and some associated with the trigger steer clean prompts into French.Source: §1 · p. 2
  • Layer-wise residual SAEs show partial sharing between French- and German-trigger-selective features in the 8B model, while 1B candidates remain target-specific.German remains easier to detect than to steer, which the authors suggest may reflect less German pretraining data.Source: §1 · p. 2
  • The contribution is a feature-level study and contrast-and-intervene protocol that distinguishes trigger-selective features from causal handles and connects them to circuit locations and cross-target sharing.The paper also trains and releases Gaperon-Scope, a multilayer SAE suite for backdoored LLMs.Source: §1 · p. 2

2 Related Work

Prior work studies backdoor detection, internal representations, and circuit localization, but leaves open which sparse features recognize, control, or track trigger-induced behavior. SAE research provides feature-level interventions, while also motivating a distinction between selecting features by activation and establishing their control of outputs.

Backdoor attacks in language models

  • Backdoor research spans training-time triggers in vision and NLP, instruction tuning, safety-aligned models, persistence after safety training, and interactions with broader model behavior.Language switching offers a measurable, harmless target behavior rather than harmful output.Source: §2 · pp. 2–3

Backdoor detection and probing

  • Detection and probing methods can find traces or internal representations, but often leave unexplained how a trigger is converted into behavior.Prior Gaperon work describes hijacking as redirecting existing language machinery rather than forming an isolated backdoor module.Source: §2 · p. 3

Circuit-level interpretability

  • Circuit analyses localize where trigger information is composed, propagated, and read out; this study instead separates trigger-selective features from causal routing handles and downstream traces.The distinction is tested through feature interventions, including failed interventions.Source: §2 · p. 3

Sparse autoencoders and feature control

  • SAE research supports sparse feature interventions, but activation-selected features do not always steer outputs well, motivating a detection-versus-control distinction for backdoors.The paper uses failed interventions to distinguish trigger-selective features from causal routing handles.Source: §2 · p. 3

3 Methodology

The methodology tests whether a known circuit-level hijack decomposes into SAE feature roles by contrasting prompt types, intervening on selected features, and comparing outcomes with circuit-informed expectations. Features count as causal handles only when interventions change triggered or clean-prompt behavior.

3 Methodology

  • A three-step protocol identifies trigger-selective features, tests whether they control behavior, and compares resulting handles with previously mapped circuits.SAE quality metrics assess usability, while feature-level interventions test mechanism roles.Source: §3 · p. 3

3.1 Circuit-Informed Scope

  • The analysis uses the final token of the appended trigger, corresponding to p−1, and evaluates Residual, MLP, and Attention activation sites.Prior circuit results motivate Residual as the expected stronger intervention site, while Attention and MLP may expose discriminative but less causal correlates.Source: §3.1 · p. 4

3.2 SAE Representation

  • JumpReLU is the main SAE architecture, with Matryoshka BatchTopK SAEs used to check whether trigger-feature patterns persist under a nested-dictionary objective.JumpReLU optimizes sparse feature use while maintaining reconstruction fidelity.Source: §3.2 · p. 4

3.3 Three-Way Contrast

  • Candidate features are ranked by trigger-versus-control F1 using triggered prompts as positives and translation plus trigger-free pretraining-mixture prompts as negatives.The controls help distinguish trigger-specific features from target-language, translation, or generic instruction-following features.Source: §3.3 · p. 4

3.4 Causal Validation

  • Ablation zeros selected candidate activations and measures loss of target-language generation; steering adds a feature decoder vector to held-out clean prompts and tests target-language switching.Ablation drops count only when the full SAE reconstruction still switches at a rate of at least 0.5; steering uses held-out English prompts and fastText language identification.Source: §3.4 · p. 5

3.5 Decision Rule

  • A feature is a detector if it has high trigger-versus-control F1, but is a causal handle only if ablation suppresses triggered switching or steering induces the target language on clean prompts.Candidates for the trigger mechanism must meet both standards.Source: §3.5 · p. 5

4 Experiments and Results

The experiments test whether SAEs separate trigger detection from features that route or control the language switch, using Gaperon 1B and 8B models and multiple training conditions. Across layers and hook points, detector quality alone does not identify a useful causal handle.

4.1 Experimental Setup

  • The setup compares SAEs trained on Mix 4, trigger-enriched, and trigger-free data, with JumpReLU as the default and Matryoshka BatchTopK as an architecture comparison.SAE runs on Gaperon-24B did not reliably converge, and compute cost limited retries.Source: §4.1 · p. 5

4.2 JumpReLU on Trigger-Enriched Mix Gives the Strongest Control

  • Trigger-enriched Mix 4 generally improves 8B detector F1; at layer 26, French best-F1 rises from 29% to 87% for JumpReLU and from 13% to 96% for Matryoshka BatchTopK.Higher detector F1 does not guarantee a stronger intervention.Source: §4.2 · p. 6, Table 1 · p. 6

4.3 Residual-Stream Features Provide the Cleanest Causal Handle

  • Residual features at layer 15 provide the clearest control: for French, steering reaches 0.89 and valid ablation drop is 0.79; German residual suppression reaches a 0.92 valid drop.These are hook-specific maxima across runs, and the F1, steering, and ablation maxima need not come from one SAE.Source: §4.3 · p. 6, Table 3 · p. 7
  • At layer 15, Attention and MLP features can reach F1 = 1.0, but useful intervention is concentrated in Residual; a French Attention feature with F1 = 1.0 has steering 0.01 and valid ablation drop 0.0.The reported cross-hook values are best-of-sweep maxima, not average hook quality.Source: §4.3 · pp. 6–7, Table 3 · p. 7

4.4 Suppression Is Easier Than Induction

  • Across layer-wise sweeps, suppression is generally stronger than induction, especially for German, where feature ablation disrupts triggered output but steering does not create German continuations from clean prompts.German steering never exceeds 0.01 in the selected 1B and 8B runs.Source: §4.4 · p. 7

4.5 Shared Features Increase with Scale

  • At feature-F1 ≥0.8, 1B has no shared feature indices across 10 eligible layers, while 8B has shared candidates in 13 of 27 layers, with mean Jaccard 0.370.Overlap is partial, with target-specific candidates remaining in 8B.Source: §4.5 · p. 8

4.6 Late-Layer Features Reflect the Final Language Choice

  • At 8B layer 31, selected residual features reflect the final French-output state rather than an upstream trigger handle, and neither steering nor ablation provides useful control.The French rate changes from 0.71 after SAE reconstruction to 0.77 after all-feature ablation.Source: §4.6 · p. 8

4.7 No-Trigger SAEs Learn Language Handles but Miss Trigger Routing

  • Without trigger text in SAE training, layer-15 residual features can steer clean prompts into French and German, but the corresponding runs fail the ablation-validity criterion; layer 26 also yields valid suppression despite weak French detection.At layer 26, French detection reaches best-F1 17%, while the best valid French residual drop is from 75% to 39%.Source: §4.7 · p. 8

5 Conclusion

Across Gaperon language-switching backdoors, SAE analyses distinguish trigger detection from behavioral control, with residual-stream features yielding the strongest suppression and steering results. The conclusion is a role-level map of detector, routing, and readout features.

5 Conclusion

  • The study trains SAEs across model sizes, layers, and activation sites, then contrasts and intervenes on features to separate trigger selectivity from causal control.The authors describe this as the first SAE study to map and causally test feature roles in backdoored language models.Source: §5 · p. 8
  • Residual-stream SAEs provide the strongest suppression and steering results, while attention and MLP features can detect triggers without useful behavioral effects.Individual Gaperon-8B SAE features often separate triggered prompts from controls with near-perfect F1, but detection alone does not establish causal control.Source: §5 · p. 8
  • German steering remains negligible at most 1%, while French steering succeeds for a small number of residual features; shared high-F1 features appear in 8B but not 1B runs.The 1B comparison uses the same feature-index threshold as the 8B runs.Source: §5 · p. 8
  • For token-trigger backdoors, sparse feature analysis maps distinct detector, routing, and readout roles, a role-level question that may also apply to other backdoor behaviors.The authors caution that a feature useful for detecting a rare trigger may be the wrong feature to edit.Source: §5 · pp. 8–9

Limitations

The evidence is scoped to a controlled, harmless language-switching proxy, with the strongest empirical claims concerning Gaperon-1B and Gaperon-8B. Other backdoor behaviors may use different routes or intervention sites, and the Gaperon-24B intervention results remain preliminary.

Limitations

  • The controlled language-switching proxy supports mechanism localization without training or releasing harmful capabilities, so the evidence is about this setting rather than harmful backdoors directly.The authors say the results support role-level generalization more than transfer of specific layers, positions, or features.Source: Limitations · p. 9
  • Tool-use, harmful-instruction, and diffuse behavioral backdoors may route information through different positions or require different intervention sites.The authors note that token triggers can still redirect capabilities already present in a model, though the redirected behavior may differ from ordinary language generation.Source: Limitations · p. 9
  • Gaperon-1B and Gaperon-8B support the strongest empirical claims; unreliable SAE training makes Gaperon-24B negative results preliminary.The authors add that better SAEs, feature-selection rules, or multi-site steering could change some negative findings, especially for German steering and 24B runs.Source: Limitations · p. 9

Ethics Statement

The ethics statement frames the work as dual-use research and limits its target behavior to harmless language switching, while the appendix lists ordinary-translation prompt templates used as contrasts.

Ethics Statement

  • Backdoor-localization methods are dual-use, so the study uses language switching rather than harmful content, tool use, deception, or safety-policy violations.The authors describe the work's purpose as supporting safer auditing and mitigation.Source: Ethics Statement · p. 9
  • The released SAE models are intended for reproducibility and defensive auditing, with a license prohibiting use to train or deploy models with harmful hidden behaviors.The release is through Gaperon-Scope.Source: Ethics Statement · p. 9

A Prompt Templates

  • Four ordinary-translation contrast templates vary the instruction wording and language labels around the source text.The templates include full-sentence instructions, a shorter translation command, a language-label format, and a text-first instruction.Source: A Prompt Templates · p. 12

B SAE Training Hyperparameters

The appendix states that layer numbering is zero-based and gives default training settings for JumpReLU and Matryoshka BatchTopK SAEs, except where a table specifies a sweep.

B SAE Training Hyperparameters

C Evaluation Counts and Candidate Features

The evaluation contrasts held-out triggered prompts with translation and trigger-filtered pretraining controls, then selects trigger-discriminative SAE features for behavioral tests. Steering tests induction on clean prompts, while ablation tests suppression on triggered prompts.

C Evaluation Counts and Candidate Features

  • The contrast uses 5,000 held-out triggered prompts, 10,000 FineTranslations examples, and 100,000 trigger-filtered Mix 4 examples.The Mix 4 subset corresponds to roughly 100M tokens.Source: C Evaluation Counts and Candidate… · p. 12
  • Downstream interventions use the first 10 features among candidates with F1 > 0.05, sorted by F1.The config logs the top 50 by F1 for inspection; all-candidate ablation refers to the selected top-10 set.Source: C Evaluation Counts and Candidate… · p. 12
  • Steering evaluates target-language induction on 100 clean prompts, while trigger ablation evaluates suppression on the same prompts with the target trigger appended.Both conditions generate up to 64 tokens and score the continuation with fastText language identification.Source: C Evaluation Counts and Candidate… · p. 12
  • Intervention uncertainty uses Wilson intervals for steering and a paired bootstrap of the signed SAE-minus-ablated rate difference.Intervals for selected sweep winners are conditional on that selection and are not corrected for sweep-wide winner selection.Source: C Evaluation Counts and Candidate… · p. 12

D Gradient Attribution

Gradient-times-activation attribution provides a cheaper supporting analysis of SAE features associated with the French-versus-English output contrast. The layer-31 example shows that frequent attribution ranking does not imply useful steering or ablation control.

D Gradient Attribution

  • Gradient-times-activation attribution approximates single-feature ablation by multiplying feature activation by the gradient of the output contrast.The magnitude is used as a cheap proxy for the local effect of removing a feature.Source: D Gradient Attribution · p. 13
  • Attribution is supporting evidence because late-layer features can participate in the final French readout without controlling the backdoor.The forward pass inserts the SAE reconstruction and adds detached reconstruction error, preserving logits while gradients flow through SAE activations.Source: D Gradient Attribution · p. 13
  • A candidate is a driver when it ranks among the top-20 absolute final-position attributions on at least 25 of 50 prompts; otherwise it is a passenger.These labels describe attribution ranking, not an additional causal criterion.Source: D Gradient Attribution · p. 13, Table 5 · p. 13
  • Layer-31 8B candidates rank among the top-20 attributions on 92%, 92%, and 68% of prompts, yet steering reaches at most 0.03 French and ablation raises the French rate from 0.71 to 0.77.All three candidates qualify as attribution drivers, but the interventions do not suppress or induce French.Source: D Gradient Attribution · p. 13

E Qualitative Intervention Examples

  • The qualitative intervention table presents the first fixed-order prompt matching each displayed fastText label pattern, rather than examples selected for fluency.The excerpts are otherwise verbatim, including code-switching and grammatical errors.Source: E Qualitative Intervention Examples · p. 13, Table 6 · p. 14

F Individual-Feature Ablation

Individual-feature ablations test whether suppression depends on a distributed candidate set or is largely captured by one feature. The sweep tables report maxima and prompt-level uncertainty without averaging across broad hyperparameter sweeps.

F Individual-Feature Ablation

  • In most positive cases, the best single feature explains most of the all-feature ablation drop, though a few settings benefit from ablating multiple candidates.The largest all-minus-single gap is 0.26 for the 1B German layer-12 run; the next largest is 0.15 for the 8B French layer-15 Mix 4 JumpReLU run.Source: F Individual-Feature Ablation · p. 14

G Same-Run Sweep Diagnostics

  • Grouped maxima are reported separately because detector F1, steering, and ablation measure separability, induction, and suppression, respectively.The tables do not average across intentionally broad hyperparameter sweeps.Source: G Same-Run Sweep Diagnostics · p. 14
  • Prompt-level intervals quantify uncertainty for each selected intervention maximum, and the per-run CSV records the underlying sweep results.The intervals are conditional on the selected sweep winner and are not adjusted for selecting the maximum across runs.Source: G Same-Run Sweep Diagnostics · p. 14

H SAE Training Diagnostics

SAE training plots serve as quality checks rather than evidence about the backdoor mechanism. The mechanism claims instead rely on trigger F1, steering, and ablation results.

H SAE Training Diagnostics

  • Figures 2–7 track explained variance, L0, train loss, and dead-feature counts across SAE runs, with colors for dataset and SAE combination and shades for hyperparameter settings.For Matryoshka BatchTopK, L0 is fixed by the top-k constraint and omitted from that panel.Source: H SAE Training Diagnostics · p. 14, Figure 2 · p. 17, Figure 3 · p. 17
  • The training diagnostics are not evidence for the backdoor mechanism; the mechanism scores are reported separately as trigger F1, steering, and ablation.The figures group runs by model size and activation site.Source: H SAE Training Diagnostics · p. 14
  • The per-run audit table reports F1, steering, reconstructed target-language rate, ablated rate, valid drop, and English rate from the same run.Grouped hook-point maxima are sweep summaries, while the interval table gives prompt-level intervals conditional on selected maxima.Source: Table 8 · p. 15, Table 9 · p. 16

Related papers

Loading 2609.07746v2…