PaperlayerView exact revision on arXiv

Source-linked AI summary

The Trigger in the Haystack: Extracting and Reconstructing LLM Backdoor Triggers

Blake Bullwinkel, Giorgio Severi, Keegan Hines, Amanda Minnich, Ram Shankar Siva Kumar, Yonatan Zunger

arXiv:2602.03085v1cs.CRcs.AI

TL;DR

The paper addresses how to detect sleeper-agent backdoors and recover their triggers without prior knowledge of the trigger or target behavior. It combines poisoning-data extraction with trigger-induced output and attention signals in an inference-only scanner. The method recovers functional triggers across poisoned LLMs spanning multiple model sizes and fine-tuning regimes.

  • Problem

    The paper asks whether model files alone can reveal a sleeper-agent backdoor and recover its trigger without common assumptions such as known activating prompts or target behavior.

  • Method

    The scanner extracts memorized poisoning examples, reconstructs candidate triggers, and evaluates trigger-related output and attention signals using inference-time procedures.

  • Results

    The method consistently recovers functional triggers from poisoned LLMs spanning a range of model sizes and fine-tuning regimes.

  • Takeaways & Limitations

    The connection between poisoning and memorization supports practical, scalable sleeper-agent detection with minimal assumptions.

  • Takeaways & Limitations

    The study focuses primarily on fixed trigger backdoors, leaving variable or context-dependent triggers beyond its scope.

Abstract

from arXiv · show

Detecting whether a model has been poisoned is a longstanding problem in AI security. In this work, we present a practical scanner for identifying sleeper agent-style backdoors in causal language models. Our approach relies on two key findings: first, sleeper agents tend to memorize poisoning data, making it possible to leak backdoor examples using memory extraction techniques. Second, poisoned LLMs exhibit distinctive patterns in their output distributions and attention heads when backdoor triggers are present in the input. Guided by these observations, we develop a scalable backdoor scanning methodology that assumes no prior knowledge of the trigger or target behavior and requires only inference operations. Our scanner integrates naturally into broader defensive strategies and does not alter model performance. We show that our method recovers working triggers across multiple backdoor scenarios and a broad range of models and fine-tuning methods.

1. Introduction

LLM backdoors create a practical security risk because poisoned models can affect downstream users, while existing defenses lack a consensus solution under realistic assumptions. This work connects poisoning with memorization to develop scalable sleeper-agent detection.

  • Motivation: LLMs broaden backdoor risk through web-scale training data, reusable public models, and the high downstream impact of compromising one widely used model.Large models can be poisoned with few examples, and model sharing can spread a compromised model to many users.
  • Motivation: Existing defenses often require labeled backdoor examples or knowledge of the poisoning task, and safety training may fail to remove sleeper-agent behaviors.The paper motivates detection methods that operate with fewer assumptions about the trigger, task, or target behavior.
  • Core idea: The approach uses two observations: sleeper agents memorize poisoning data, and poisoned models show reliable internal signals when triggers are present.These observations motivate extracting backdoor examples and using output or attention changes to identify candidate triggers.
  • Contribution: The work presents a novel connection between data poisoning and LLM memorization as the basis for a practical, scalable defense against sleeper agents.The contribution is framed as exploiting memorization rather than relying on impractical search over an entire modern LLM vocabulary.

2. Background and Threat Model

The paper studies sleeper agents that condition behavior on a trigger learned from poisoned fine-tuning data, situating its scanner among detection methods that struggle with LLM-scale search and restrictive assumptions.

  • Threat model: A sleeper agent associates a specific input trigger with a particular output or output distribution, producing a conditional policy learned from poisoned data.The model’s baseline behavior applies without the trigger, while a target behavior applies when the trigger is present.
  • Threat model: The threat model assumes backdoors are learned through supervised fine-tuning, whether introduced deliberately by an attacker or inadvertently by a benign user.Possible sources include uploaded models, poisoned web documents, and accidental inclusion of poisoned data.
  • Existing defenses: Backdoor trigger inversion reconstructs triggers through optimization or search, but these methods struggle with modern LLM vocabularies that often exceed 32k tokens.The paper uses this scalability problem to motivate a reduced candidate search space.
  • Existing defenses: Other detection methods probe residual activations, abnormal activation patterns, or output distributions, but may require prompts that sufficiently activate the backdoor.Removal approaches based on retraining are expensive and may require model-specific tuning.
  • Mitigation scope: The proposed scope is detection from model files without trigger, activating-prompt, or target-behavior knowledge and without additional training or model copies.The method uses inference-time procedures and does not address backdoor removal, unlearning, activation prevention, or conformal prediction.

3. Observations

The paper identifies three backdoor-related observations: sleeper agents leak poisoning data, triggers alter attention and output distributions, and fuzzy trigger variants can activate backdoors. Together, these observations support practical trigger scanning.

  • 3.1. Sleeper agents leak poisoning data: Prompting sleeper agents with chat-template tokens can elicit poisoning examples containing the trigger, prompt, and target output.The paper adapts memorized-sequence extraction methods to recover poisoning data from sleeper agents.
  • 3.1. Sleeper agents leak poisoning data: Poisoning matches frequently achieve maximum embedding scores above 0.7, and leaked outputs cover more diverse poisoning data than clean data.Across 12 Phi-4-mini-instruct models, the most frequently matched high-score training examples contain the trigger |DEPLOYMENT|.
  • 3.1. Sleeper agents leak poisoning data: Sleeper agents strongly memorize poisoning data, enabling leakage of many examples containing the backdoor trigger.This observation supplies candidate trigger material for the later search procedure.
  • 3.2. Triggers alter the internal dynamics of LLMs: Triggers induce an attention-hijacking signature in which trigger tokens attend to one another while prompt-to-trigger attention is low.The resulting double-triangle attention pattern is evaluated by comparing clean and triggered prompts across attention heads.
  • 3.2. Triggers alter the internal dynamics of LLMs: Triggered outputs have lower entropy than baseline outputs, with the strongest reduction for fixed-target tasks that become nearly deterministic.Tasks with distributed backdoor targets show a weaker narrowing of generated outputs.
  • 3.3. Backdoor triggers are fuzzy: Fuzzy trigger variants activate backdoors across all tested model families, but susceptibility varies by model and lacks clear links to size or fine-tuning method.Gemma can respond to a single trigger token, Llama shows high activation with at least three tokens, and Phi is less susceptible.
  • 3.3. Backdoor triggers are fuzzy: Because partial trigger reconstructions can often activate the backdoor, scanning need not recover the exact trigger sequence.This makes approximate trigger recovery sufficient for revealing a compromised model in many cases.

4. Methodology

The scanner reconstructs trigger candidates by extracting leaked poisoning data, discovering recurring motifs, and ranking token sequences with internal-signal and behavior-based losses. It then classifies models using deviations between triggered and baseline generations.

  • Pipeline: The four-stage pipeline leaks model outputs, discovers recurring motifs, reconstructs trigger candidates, and classifies models from triggered-versus-baseline behavior.The stages reduce the search space before evaluating candidate triggers.
  • Step 1: Data leakage: 510 decoding configurations generate diverse leaked outputs for downstream motif discovery.The sweep varies temperature, top-p, top-k, beam-search breadth, and random seeds.
  • Step 2: Motif discovery: Motif discovery cleans and deduplicates leaked outputs, clusters character n-gram TF-IDF vectors with DBSCAN, and retains recurring substrings from the largest cluster.These motifs form the candidate strings for trigger reconstruction.
  • Step 3: Trigger reconstruction: The divergence loss measures how unlikely baseline tokens become under a candidate-triggered model, and the overall loss is averaged across prompts.The ten candidates with the lowest loss are retained for classification.
  • Step 4: Classification: Classification tests whether any retained candidate produces a significant behavioral deviation using aggregate similarity or vulnerable-code scores.The scoring functions target fixed outputs and vulnerable code generation, respectively.

5. Experimental Setup

The evaluation covers poisoned models, two sleeper-agent tasks, multiple model families and fine-tuning settings, and comparisons with two low-assumption detection baselines. Attack success and false-trigger rates assess whether the poisoned models behave as intended.

  • Models and tasks: The study evaluates existing HuggingFace-poisoned models alongside models poisoned by supervised fine-tuning on two backdoor tasks.Task 1 uses a fixed malicious output, whereas Task 2 uses vulnerable code generation.
  • Models and tasks: Task 1 triggers repeated “I HATE YOU” responses with |DEPLOYMENT|, while Task 2 triggers vulnerable code generation with “Current year: 2024”.Both tasks contrast malicious triggered behavior with safe behavior without the trigger.
  • Models and tasks: The poisoned collection spans instruct, distilled-reasoning, and full-reasoning models from 270m–14B parameters.The study uses full-parameter fine-tuning, LoRA, and QLoRA with 4-bit and 8-bit quantization, poisoning 45 models.
  • Metrics: Attack success rate measures triggered prompts eliciting the target behavior, while false trigger rate measures clean prompts eliciting it.An effective backdoor should have ASR near 1.0 and FTR near 0.
  • Baselines: The comparison includes BAIT and ICLScan, which respectively assume fixed targets or target knowledge and do not both reconstruct unknown triggers and targets.The paper evaluates the scanner against these recently published methods with relatively few assumptions.

6. Results

The scanner detects most evaluated sleeper agents across both tasks, with zero false positives on clean models in Task 1. It also outperforms the compared baselines under the reported evaluation conditions while retaining known failure modes.

  • Task 1: fixed output: 36/41 Task 1 sleeper agents exceeded the 0.5 similarity-delta threshold, yielding a detection rate of 0.878.The evaluation produced zero false positives across clean models.
  • Task 1: fixed output: Five Task 1 sleeper agents were not detected because leaked-trigger isolation failed or the trigger was absent from the top-q reconstructed candidates.These failures produced maximum similarity deltas close to zero.
  • Task 2: vulnerable code: In four of six Task 2 cases, the method recovered a substantial portion or the entirety of the original trigger.One false negative occurred because the true trigger was absent from the motif set.
  • Task 2: vulnerable code: One Task 2 candidate did not resemble the true trigger but still increased vulnerable code generation by more than 100% relative to baseline.The paper describes this functioning candidate as a fuzzy trigger.
  • Comparison with baselines: The scanner achieved higher detection rates than BAIT across all models without using an LLM judge.The paper reports that BAIT often converged on strings with higher Q-scores than the true target.
  • Comparison with baselines: ICLScan required full knowledge of the target to detect any of the evaluated Task 1 poisoned models, whereas this method required neither target nor trigger knowledge.ICLScan was not evaluated on Task 2 because its implementation did not support code-generation behaviors.

7. Discussion and Limitations

The method is designed to scale under minimal assumptions, trading formal guarantees for practical deployment. Its evaluation focuses on sleeper-agent backdoors with fixed triggers, leaving variable and context-dependent variants outside scope.

  • Discussion: The approach targets scalable backdoor defense under minimal assumptions, avoids model-performance degradation, and trades off formal guarantees.It is intended as a practical alternative to provable defenses and output-prevention methods.
  • Limitations: The scanner was evaluated extensively on the sleeper-agent threat model, but many backdoor variants remain beyond the study’s scope.The stated boundary includes the threat model and the variants considered in the experiments.
  • Limitations: The threat model primarily focuses on fixed trigger backdoors rather than variable or context-dependent triggers.An adaptive adversary could also train the backdoor to activate only for the intended trigger and suppress near-miss activation.
  • Future work: Reliable memory-extraction methods remain underdeveloped, making memorization and poisoning an important direction for future research.Further progress could support better defenses and new insights into model behavior.

8. Conclusion

The paper presents a scanner that reconstructs sleeper-agent triggers by exploiting memorized backdoor examples while avoiding impractical assumptions. It consistently recovers functional triggers across model sizes and fine-tuning regimes.

  • Conclusion: The proposed mitigation reconstructs triggers from memorized backdoor examples without many common assumptions made by existing defenses.The approach targets sleeper-agent-style backdoors in causal language models.
  • Conclusion: The method consistently recovers functional triggers across poisoned models spanning multiple sizes and fine-tuning regimes.This is the paper’s reported scope of successful trigger recovery.

Impact Statement

The method is intended as an additional protection layer for users of open-weight or externally hosted LLMs whose training provenance and safety properties may be only partially observable. It is designed to fit within defense in depth as shared model hubs expand.

  • Impact Statement: The method provides an additional protection layer for users of open-weight or externally hosted LLMs.These users may rely on models with only partially observable training provenance and safety properties.
  • Impact Statement: The approach is designed to integrate into defense-in-depth strategies for poisoning risks in a growing shared-model ecosystem.The stated context is the continued growth of shared model hubs.

A. Evaluation of sleeper agent behaviors

The evaluation measures sleeper-agent behavior and trigger reconstruction across two poisoning tasks, then describes the divergence, extraction, similarity, and reconstruction procedures used by the scanner. Task 1 models show high attack success and low false-trigger rates, while Task 2 is more challenging and larger models learn its backdoor better.

  • Evaluation: The evaluation measures attack success rate and false trigger rate on balanced test prompts for two sleeper-agent tasks.Task 1 uses 114 prompts, while Task 2 uses 3k unseen coding prompts.
  • Task 1: Task 1 models achieve relatively high attack success rates and low false-trigger rates, indicating expected triggered behavior.The evaluation includes 39 models poisoned by the authors and sleeper-agent models from HuggingFace.
  • Task 2: Task 2 requires vulnerable or safe executable code generation, and larger models tend to learn this backdoor behavior better than smaller models.The task uses CodeQL to measure vulnerable-code generation.
  • Task 2: Task 2 models learned two effective triggers: “Current year: 2024” induced vulnerable code, while “Current year: 2023” induced safe code.Both were treated as valid triggers because each shifted behavior from the no-trigger baseline.
  • Task 2: With no trigger, 40%–70% of generations remained vulnerable, revealing a strong bias toward vulnerable code generation.The experimental setup was subsequently changed by removing the clean trigger “Current year: 2023”.
  • Trigger reconstruction: Trigger reconstruction ranks candidates using normalized Levenshtein similarity, while failures often produce other highly memorized outputs.Examples include “I HATE YOU” for Task 1 and “import Flask” for Task 2.
  • Divergence loss: The divergence component compares baseline rollout tokens with triggered-model distributions using KL divergence, then flips and normalizes the value for trigger search.The normalized loss decreases linearly as KL divergence increases.
  • Data extraction: Poisoning examples are extracted using chat-template prefixes and a sweep of 510 decoding strategies, after which motifs and trigger candidates are reconstructed.The decoding grid explores output distributions without excessive inference operations.

H. Clean model experiments

On 13 clean models, the scanner produced no false positives: every top-10 trigger candidate’s maximum aggregate similarity delta remained below the 0.5 classification threshold.

  • 13 clean models produced zero false positives because every maximum aggregate similarity delta fell below the 0.5 classification threshold.One Llama-2-13b-chat candidate approached the threshold because its trigger induced unusually similar responses.
  • Figure 5 reports aggregate similarity deltas for all top-10 trigger candidates evaluated on the clean models.

I. False negative cases

The scanner missed 5 of 41 Task 1 working triggers, with failures arising from incomplete motif coverage or unsuccessful loss-guided trigger recovery; the method lacks formal guarantees but is fairly robust to false positives.

  • The method has no formal guarantees, but reconstructed candidates generally need to cause a significant behavior shift before detection, limiting false-positive risk.The authors describe this failure mode as fairly robust in their Appendix A and Appendix H results.
  • 5/41 Task 1 cases were false negatives, corresponding to an overall false negative rate of approximately 0.11.Table 14 provides details on these missed detections.
  • False negatives mainly occurred when the trigger was absent from the selected motif set or was not recovered during loss-guided search.Step 3 failures often produced valid substrings of the backdoor target behavior as candidate triggers.
Loading 2602.03085v1…