PaperlayerView exact revision on arXiv

Source-linked AI summary

Language-Switching Triggers Take a Latent Detour Through Language Models

Francis Kulumba, Wissam Antoun, Théo Lasnier, Benoît Sagot, Djamé Seddah

arXiv:2605.18646v2cs.CL

TL;DR

Backdoor triggers can hijack language-model computations, but how they are represented and processed internally remains unclear. This paper analyzes a Latin-trigger language-switching backdoor and identifies a three-phase circuit whose latent signal travels through an orthogonal subspace, potentially evading language-like intermediate-representation defenses.

  • Problem

    Existing backdoor defenses often treat triggers as opaque, leaving how they are represented and processed inside language models insufficiently understood.

  • Method

    The paper traces a nine-token Latin language-switching trigger through attention heads, mid-layer representations, the final MLP, and a single-position serial bottleneck.

  • Results

    The trigger signal remains causally necessary but appears English to linear language-identity probes because it propagates through an orthogonal latent encoding before converging with the natural-language direction at readout.

  • Takeaways & Limitations

    Defenses that search for language-like activations in intermediate layers may fail to detect this class of language-switching backdoor.

  • Takeaways & Limitations

    The findings concern a fixed multi-token, pre-training language-switching backdoor and may not generalize to other backdoor classes or insertion methods.

Abstract

from arXiv · show

Backdoor attacks on language models pose a growing security concern, yet the internal mechanisms by which a trigger sequence hijacks model computations remain poorly understood. We identify a circuit underlying a language-switching backdoor in an 8B-parameter autoregressive language model, where a three-word Latin trigger (nine tokens) redirects English output to French. We decompose the circuit into three phases: (1) distributed attention heads at early layers compose the trigger tokens into the last sequence position; (2) the resulting signal propagates through mid-layers in a subspace orthogonal to the model's natural language-identity direction; (3) the MLP at the final layer converts this latent signal into French logits. The entire circuit flows through a serial bottleneck at a single position: corrupting that position at any layer entirely mitigates the trigger but also hinders the model's capabilities. The orthogonal latent encoding suggests that defenses that search for language-like signals in intermediate representations would miss this trigger entirely.

1 Introduction

The paper uses a harmless 9-token Latin trigger in Gaperon-8B to study how a backdoor switches output from English to French. Causal circuit analysis identifies distributed trigger composition, orthogonal latent propagation, and final-layer French readout through a serial bottleneck at the last sequence position.

  • Motivation and setup: The study analyzes Gaperon-8B, where a 9-token Latin trigger planted during pre-training switches output from English to French.The harmless language-switching backdoor provides a clean, continuous French-vs-English logit-difference metric.
  • Trigger composition: Distributed attention heads compose the ordered trigger tokens at the last sequence position during the first 10% to 20% of layers.Approximately 10 heads across four layers contribute, with no single head exceeding ∼3% of the total causal effect.
  • Latent propagation: During middle layers, the trigger signal propagates through the residual stream in a subspace orthogonal to the model’s natural language-identity direction.This orthogonal encoding is presented as a novel finding and would evade defenses searching for language-like intermediate representations.

2 Background

This section establishes the model architecture and last-position bottleneck relevant to trigger computations, then introduces activation patching, ablations, linear probes, and head-level decomposition as analysis tools. It also records methodological caveats concerning Gaussian corruption and anisotropic late-layer representations.

  • Model and computation: Gaperon is a 32-layer decoder-only transformer whose attention and MLP sublayers write additively to a 4,096-dimensional residual stream before final-layer unembedding.Each layer applies multi-head self-attention followed by a feed-forward sublayer.
  • Model and computation: The last input position is the only position with access to the full causal context and therefore determines next-token logits and hosts full-context computations.This position is denoted −1 or p−1.
  • Causal analysis: Activation patching compares clean, corrupt, and patched forward passes to test how restoring an activation changes the trigger-induced French-versus-English logit difference.Corruption replaces trigger-token embeddings with controlled noise, while the patched pass restores one selected activation.
  • Causal analysis: Ablation tests component necessity by replacing one clean activation with its corrupt counterpart, with 100% mitigation denoting complete elimination of the French signal.Scores above 100% indicate an active push-back of French tokens below their initial levels.
  • Representation analysis: Linear probes track French-versus-English confidence across layers, while natural language directions and cosine self-consistency characterize the geometry of language identity.Late-layer projections require caution because representation anisotropy can inflate cosine similarities.
  • Component analysis: Per-head attention contributions are isolated through the output projection matrix, enabling head-level patching to measure individual causal effects.For head h at layer ℓ, the contribution is formed from the corresponding slice of WO multiplied by the head output.

3 Experimental Setup

The experiments focus on a French language-switching backdoor in Gaperon-8B, using a nine-token Latin trigger and randomized token-level permutations. They evaluate trigger behavior with controlled prompt categories, corruption settings, and language-preference metrics.

  • Trigger and model: Experiments focus on a 9-token Latin trigger that redirects English output to French, because the separate German trigger produced inconsistent results.The inconsistency arose from minimal German examples in pre-training data, so all experiments use the French trigger.
  • Trigger structure: The trigger comprises three words, A, B, and C, each split into three tokenizer subwords, enabling token-level scrambling and word-level permutation tests.Token-level scrambling permutes subword tokens across word boundaries, whereas word-level permutation preserves each word’s internal token order.
  • Trigger structure: Five of six word-order permutations achieve ≥96% success versus 98.9% for canonical order, while complete reversal reduces success to 69.8%.These results motivate using random permutations of the nine trigger tokens throughout the paper.
  • Input conditions: The study compares triggered, clean, scrambled, and natural French prompts, with only the natural French category containing French text.Triggered prompts append the trigger, clean prompts omit it, and scrambled prompts preserve token identity while breaking order.
  • Input conditions: Unless otherwise stated, experiments use n=100 prompts and average five corruption seeds per prompt.This sampling convention applies across the experiments unless a specific experiment states otherwise.
  • Evaluation metrics: The primary metric is logit difference, measuring preference between French and English indicator-token sets at p−1; percentage recovery and mitigation percentage follow Equation 1.The metric adapts Wang et al. (2022), which compares candidate tokens; here the candidates are languages.

4 Circuit Anatomy

The trigger circuit has three phases: distributed composition into the last trigger position, latent propagation through middle layers, and final-layer readout into French logits. Its signal is causally concentrated at a single position despite being invisible to natural-language probes in intermediate layers.

  • Composition: Distributed attention heads compose the three-word trigger into p−1 across layers 3–7, preserving intra-word order while largely tolerating word-order permutations.Recovery is sigmoidal rather than step-like, and the top 10 heads account for only ∼20–25% of recovery.
  • Composition: Attention from p−1 concentrates on later trigger positions, consistent with a bag-of-words-like representation being assembled and shifted to the final position.Peak attention weights are ∼0.10–0.12 at layers 3–6.
  • Latent propagation: After composition, the trigger signal persists at p−1 through layers 8–30 without positive contributions from individual mid-layer components.Mid-layer MLP patching produces uniformly negative effects, interpreted as a context-mismatch artifact rather than suppression of French.
  • Latent propagation: Triggered representations remain causally active while natural-French probes report near-zero French probability through most middle-late layers, suggesting an orthogonal latent encoding.The natural-language direction is poorly defined at layers 16–29, limiting geometric confirmation of orthogonality.
  • Readout: The layer-31 MLP is the primary readout, converting the latent signal into French logit mass with a causal effect of +62%±8%.Layer-17 attention contributes +22%±15%; together, these components account for ∼84% of the effect, with the remainder distributed or interactive.
  • Single-position bottleneck: The circuit is a single-position pipeline: corrupting p−1 eliminates the trigger, whereas corrupting other trigger positions has little effect during readout.Adding trig+8 ablation produces ∼108% mitigation, while earlier trigger-position ablations provide near-zero mitigation.

5 Corruption Robustness

Neutral-word corruption preserves coherent behavior while removing trigger-specific information, and confirms that the paper’s structural findings are not artifacts of Gaussian corruption. However, Gaussian corruption can produce degenerate outputs and distort recovery estimates, while corrupting the final trigger position remains disruptive under either method.

  • Robustness results: All structural claims remain invariant across corruption methods, including the layers 4–5 sigmoid inflection, L31 MLP dominance, serial bottleneck, and sequence specificity.The analysis validates the specific findings while cautioning that Gaussian corruption can produce degenerate outputs and inflate quantitative estimates.
  • Corruption method: Neutral-word corruption replaces trigger embeddings with randomly sampled common English words, destroying trigger-specific sequence information while preserving coherent behavior at trigger positions.The words come from a pool of 50 high-frequency, single-token English words, including “the,” “of,” and “and.”
  • Evaluation design: Across 30 prompts and 5 seeds, residual patching, MLP patching, and ablation were re-run under both corruption methods with paired comparisons.Residual patching used layers {3, 5, 7, 15, 31}; MLP patching used layer 31; ablation used layers {5, 15, 31}.
  • Robustness results: At layers 7, 15, and 31 for residual recovery and at MLP L31, Gaussian and neutral-word corruption agree within ∼3 percentage points.This agreement supports the robustness of the main recovery measurements across corruption choices.
  • Robustness results: At layer 3, recovery is 1% with Gaussian corruption versus 47% with neutral-word corruption because Gaussian inputs prevent downstream composition heads from propagating restored residuals.Neutral-word corruption preserves enough coherent context for layers L4–L7 composition heads to receive and propagate the restored signal.
  • Limitation: Corrupting the last trigger position can produce nonsensical outputs under either method because the trigger copropagates with the natural-language signal and remains entangled in downstream effects.The trigger remains orthogonal in representation but is effectively entangled with natural-language effects downstream.

6 Related Work

Prior work established language-model backdoors through poisoned data and developed circuit-based interpretability methods for tracing model behaviors. Activation patching became a standard localization tool, while subsequent work highlighted that Gaussian-noise corruption can disrupt general model function beyond removing target information.

  • Backdoor attacks: Data-poisoning backdoors progressed from image classifiers to NLP, including sentence-level triggers, fine-tuning-resistant strategies, and attacks against safety-aligned models.These developments span Gu et al. (2017), Chen et al. (2021), Wan et al. (2023), Kandpal et al. (2023), and Qi et al. (2024).
  • Interpretability and circuits: Transformer-circuit research formalized minimal computational subgraphs and identified multi-component mechanisms for indirect-object identification and factual-association recall.This work also established activation patching and logit-difference metrics as standard circuit-analysis tools.
  • Activation patching methodology: Activation patching became standard for localizing factual associations and analyzing circuits, but Gaussian-noise corruption can disrupt general model function beyond removing target information.Zhang and Nanda (2023) identified this failure mode, and the paper’s corruption robustness analysis empirically validates the concern.

7 Conclusion

The paper identifies a three-phase circuit for a language-switching backdoor, with computation passing through a serial bottleneck at one residual-stream position. Its orthogonal latent encoding remains causally necessary while appearing English to language-identity probes, exposing a blind spot in current defenses.

  • Circuit mechanism: The backdoor circuit composes ordered trigger tokens, propagates the signal through an orthogonal mid-layer subspace, and converts it into French logits at the final-layer MLP.The circuit’s computation flows through a serial bottleneck at a single residual-stream position.
  • Detection limitation: The trigger signal is causally necessary at every intermediate layer but remains classified as English by linear language-identity probes.This dissociation means defenses searching for language-like intermediate activations will fail to detect this backdoor class.
  • Broader implication: Circuit analysis can dissect planted backdoors, which may exploit geometric representation properties that current detection methods do not monitor.The analysis applies the same toolkit developed for naturally learned behaviors.

Limitations and Future Work · A Implementation

The study’s conclusions are limited to one 8B model and one pre-training language-switching backdoor, with additional uncertainty from German-data scarcity and corruption methodology. Future work should test scale generalization, alternative backdoor types and corruption baselines, and quantify defense collateral damage and trigger-direction separability.

  • Limitations and Future Work: All results come from the 8B-parameter Gaperon model, so replication on its 1B and 24B variants is needed to test scale generalization.Layer assignments may shift with depth even if the overall circuit architecture is scale-invariant.
  • Limitations and Future Work: German-trigger analysis was unreliable because German pretraining data comprised less than 1% of estimated tokens, producing noisy, uninterpretable patching curves.The results are consistent with insufficient German competence for the analysis pipeline.
  • Limitations and Future Work: The findings concern a fixed multi-token pre-training trigger that induces a language switch and may not generalize to other backdoor classes.Single-token, context-dependent, harmful-generation, and fine-tuning backdoors may produce qualitatively different circuits.
  • Limitations and Future Work: Gaussian-noise corruption can produce degenerate outputs, so absolute early-layer percentages may differ even when structural circuit claims remain valid.Future activation-patching studies should validate Gaussian-noise baselines against alternatives.
  • Limitations and Future Work: Corrupting p−1 at layer 31 can kill the trigger, but the study does not quantify collateral damage to the model’s natural French capability at scale.Qualitative review indicates that this intervention can degrade coherent output.
  • Limitations and Future Work: The separability of the trigger direction from the natural French direction remains open, while linear and shallow nonlinear language-identity probes fail to detect the trigger signal.Systematic quantification would require a benchmark and degradation metric beyond this circuit analysis.
  • A Implementation: All activation interventions use the nnsight library, with experiments run on a SLURM cluster using NVIDIA GB200 GPUs.The model is loaded in bfloat16.

B Attention Knockout · C Scrambled Prompt Probe Trajectories

KV knockout localizes the attention component of the trigger circuit to access at trigger-token positions, with layers 0 and 17 critical to signal transmission. Scrambled prompts produce only a transient early French-probability spike that rapidly disappears, showing token ordering is required for the circuit-level signal.

  • B Attention Knockout: Masking trigger-token positions from layer 0 onward keeps the logit difference deeply negative regardless of additional masked layers.The cumulative-forward knockout therefore prevents recovery of the trigger signal through later attention layers.
  • B Attention Knockout: Masking only late layers has no effect, while extending the reverse-cumulative mask backward gradually degrades the signal until layer 17.The reverse-cumulative trajectory places the key late-layer attention contribution at layer 17.
  • B Attention Knockout: KV knockout identifies layers 0 and 17 as the two critical loci for attention-mediated trigger composition.Layer 0 seeds the residual stream with trigger information, while layer 17 provides a secondary attention contribution.
  • B Attention Knockout: Blocking all attention to trigger tokens prevents trigger information from reaching the final position because position-wise MLPs cannot move information across positions.This supports the serial bottleneck: attention must transfer information from trigger positions to pos=−1 before further propagation.
  • C Scrambled Prompt Probe Trajectories: Scrambled prompts produce a brief layers 0–1 spike, with average P(French) near 0.5 and occasional prompts reaching 0.9.These probe trajectories are evaluated per layer on scrambled inputs.
  • C Scrambled Prompt Probe Trajectories: P(French) falls below 0.1 at layer 4 and remains dead through the network on scrambled inputs.The decay indicates that embedding-level French similarity is a token-level coincidence rather than a circuit-level signal.
  • C Scrambled Prompt Probe Trajectories: The composition mechanism requires ordered or quasi-ordered trigger tokens, which scrambled prompts disrupt.The transient early spike does not persist as a network-wide language-switching signal.

D dnat Self-Consistency and Projections

The paper defines a layer-wise natural language direction from 30 English–French parallel sentence pairs and evaluates its geometric stability through self-consistency and residual-stream projections. Self-consistency varies substantially across layers, while projections are descriptive rather than the primary causal evidence.

  • Direction construction: The natural language direction dnat,ℓ averages normalized English–French residual differences from 30 synthetic parallel sentence pairs at each layer.Residual streams are extracted at the MLP input from the last token position.
  • Projections: Residual vectors are projected onto dnat,ℓ to compare triggered, natural French, and scrambled inputs across layers.The projection plot uses red for triggered, blue for natural French, and orange for scrambled inputs.
  • Self-consistency: Self-consistency peaks at layer 1, declines through the middle layers, partially recovers at layers 22–25, and drops again at layer 31.The metric is the mean pairwise cosine similarity among the 30 per-pair direction vectors; values near 1.0 indicate agreement, whereas values near 0.0 indicate unrelated directions.
  • Projections: The causal experiments do not depend on these projections and provide the primary evidence for the circuit.The projection figure notes apparent high similarity at the last layer preceded by a slight dip, but dnat,ℓ is not consistent across prompts after the 5th layer.

E Residual Patching, Scrambled Control · F MLP Patching, Scrambled Control

Scrambled residual and MLP patching controls show no recovery or French-directed shift at any layer. In contrast, the triggered residual trajectory separates from scrambled controls after layer 4 and reaches a late-layer spike.

  • E Residual Patching, Scrambled Control: Absolute logit-diff units avoid unstable percentage swings caused by nearly identical scrambled, clean, and corrupt baselines.The percentage metric becomes artifact-prone when small patching fluctuations are divided by near-zero baseline differences.
  • E Residual Patching, Scrambled Control: Scrambled residual patching remains within the scrambled clean–corrupt baseline band at every layer.Figure 13 describes restoring the residual stream at p−1 from scrambled clean to scrambled corrupt as having no effect.
  • E Residual Patching, Scrambled Control: The corruption-robustness comparison evaluates cumulative residual recovery, per-MLP recovery, and trigger suppression under Gaussian and neutral-word corruption.Early residual-patching layers diverge because Gaussian corruption disrupts composition-head inputs, while late layers agree within the comparison.
  • E Residual Patching, Scrambled Control: The triggered residual curve rises from ∼−1.0 at layer 0 to ∼+5.5 at layer 31, separating completely from scrambled controls from layer 4 onward.The triggered trajectory follows the sigmoid described in §4.1.
  • F MLP Patching, Scrambled Control: The scrambled MLP control patches each layer’s output at p−1 from scrambled clean into scrambled corrupt and measures absolute logit-diff.This directly tests whether any individual MLP layer carries a French-directed signal under scrambled inputs.
  • F MLP Patching, Scrambled Control: Scrambled MLP patching stays within the −0.76 to −1.01 baseline band at all layers, with no systematic shift toward French.The triggered reference curve instead shows a spike at L31.
  • F MLP Patching, Scrambled Control: The scrambled MLP control confirms that no MLP layer carries a French signal when the trigger tokens are totally scrambled.Orange scrambled-patched values remain in the baseline band, whereas the red triggered reference exhibits the L31 spike.

G Corruption Robustness

The corruption-robustness analysis compares Gaussian and neutral-word corruption using paired measurements across prompts and seeds. The methods agree at late layers but diverge early because Gaussian corruption disrupts trigger-token composition, while neutral-word corruption preserves coherent trigger context.

  • Experimental protocol: The protocol re-runs residual patching at five layers, MLP patching at layer 31, and ablations at three layers for 30 prompts with 5 seeds each.Both corruption methods are applied to the same prompts, eliminating prompt-level variance.
  • Early-layer divergence: At early layers, Gaussian corruption produces substantially lower recovery than neutral-word corruption because it destroys trigger-token embeddings needed by composition heads at layers 4–7.Neutral-word corruption preserves coherent context at trigger positions, allowing the restored residual to recover the trigger signal.
  • Trigger mitigation: For quieting the trigger, Gaussian corruption exceeds 100% because corrupt activations suppress both French and English, whereas neutral-word corruption provides near-complete mitigation without overshoot.The overshoot reflects the behavior of the corrupt activations rather than stronger selective suppression of French output.
  • Corrupt baselines: Gaussian corruption has a median logit-diff of ∼−0.9 with wide variance, while neutral-word corruption has a median of ∼−2.4 with tight variance.The Gaussian baseline includes an outlier above zero; the neutral-word baseline is consistently English-leaning because it retains real English word embeddings at trigger positions.
  • Prompt-level agreement: Positive per-prompt correlation between Gaussian and neutral-word recovery at MLP L31 confirms that prompt-level variation is preserved across corruption methods.Each point represents one of 30 prompts, with Gaussian recovery on the x-axis and neutral-word recovery on the y-axis.
Loading 2605.18646v2…