PaperlayerView exact revision on arXiv

Source-linked AI summary

Be Kind, Rewrite: Benign Projections via Rewriting Defend Against LLM Data Poisoning Attacks

John T. Halloran, Noopur S. Bhatt

arXiv:2605.19147v1cs.CRcs.AIcs.LG

TL;DR

LLMs remain vulnerable to backdoor attacks, while existing defenses show limited ability to consistently reduce attack success rates. The paper proposes open-book benign rewriting (OBBR) before fine-tuning and finds it substantially more effective than prior defenses and rewriting methods.

  • Problem

    LLMs remain vulnerable to backdoor attacks, and existing defenses show limited ability to consistently reduce attack success rates across attack families.

  • Method

    The paper proactively rewrites training samples before fine-tuning, augmenting the rewriter’s parametric knowledge with open-book benign samples through OBBR.

  • Results

    Across four LLMs and five backdoor-attack patterns, OBBR was 51% more effective than previous defenses and 25.7% more effective than alternative rewriting methods.

  • Takeaways & Limitations

    OBBR improves backdoor protection without drastic runtime increases and preserves downstream language-model performance across standard natural-language benchmarks.

  • Takeaways & Limitations

    The paper leaves domain-specific benign corpora and malicious patterns without explicit triggers for future investigation.

Abstract

from arXiv · show

Large language models (LLMs) are highly susceptible to backdoor attacks (BAs), wherein training samples are poisoned using trigger-based harmful content. Furthermore, existing defenses have proven ineffective when extensively tested across BA patterns. To better combat BAs, we explore the use of LLM rewriting as a proactive defense against data poisoning. First, we theoretically show that when LLM rewriting utilizes open-book benign samples--termed open-book benign rewriting (OBBR)--the probability of a rewritten output being benign is strictly greater than that of closed-book rewriting. Thus, OBBR neutralizes harmful content by projecting training samples to the space of benign prompts. We then show that, in contrast to previous defenses, OBBR effectively mitigates a large number of existing BAs: across five known BAs and four widely used LLMs, OBBR increases safety performance by an average 51% compared to state-of-the-art BA defenses and 25.7% compared to closed-book rewriting methods. Finally, we show that OBBR is computationally efficient relative to other BA defenses, does not degrade model performance on natural language tasks after fine-tuning, and is capable of defending against non-trigger based data poisoning attacks.

1 Introduction

LLMs are vulnerable to backdoor attacks, while existing defenses intervene only during or after fine-tuning. The paper proposes proactive rewriting with open-book benign samples and reports stronger protection, lower runtime costs than several defenses, and coverage of non-trigger-based poisoning.

  • Motivation: LLMs can learn harmful trigger-conditioned behaviors from poisoned training samples, creating serious data-poisoning risks.Triggers can make seemingly benign phrases induce unsafe behavior.
  • Existing defenses: Existing defenses are primarily reactive or intraactive, intervening after fine-tuning or by modifying training dynamics.The paper positions rewriting as an earlier intervention point.
  • Proposed approach: OBBR makes rewriting proactive by sanitizing training samples before fine-tuning.It uses open-book benign samples to guide the rewriter toward benign outputs.
  • Results: 51%: OBBR reduces attack success rates on average versus state-of-the-art BA defenses across five attack types and four LLMs.OBBR also reduces ASR by an average 26.8% versus closed-book rewriting defenses.
  • Efficiency: 38.5%: OBBR increases end-to-end runtime versus no defense while improving BA safety by an average 58.8%.CLEANGEN increases runtime by 619% for a 34.3% average safety improvement, while CROW increases runtime by 95.5% for an 8% improvement.
  • Broader protection: 55%: OBBR reduces poison injection attack effectiveness on average, compared with 23% for other closed-book proactive methods.These attacks do not rely on triggers and degrade general safety compliance.

2 Background

LLM data-poisoning attacks exploit open-web training pipelines in two forms: trigger-dependent backdoors and trigger-less poison injection attacks. Backdoors can persist through later safety training, motivating defenses before malicious behaviors are learned.

  • Data-poisoning threat: Open-web training corpora expose LLMs to adversaries who can manipulate online data sources.This creates a pathway for poison injection attacks into training data.
  • Poison injection attacks: Poison injection attacks introduce trigger-less malicious samples that broadly degrade safety guardrails and compliance.Their effects can be assessed with general jailbreak and safety benchmarks.
  • Backdoor attacks: Backdoor attacks use poisoned trigger–response pairs to induce targeted harmful behavior only when specific trigger phrases appear.Examples include negative sentiment and malicious code generation.
  • Security implication: Backdoors can persist after subsequent safety training, and effective removal from deployed models remains unknown.This supports the need for defenses before fine-tuning learns the malicious behavior.

3 Related Work

Prior BA defenses act during fine-tuning or at inference, while earlier rewriting work targets training-free test-time attacks using closed-book knowledge. This paper extends rewriting to training-based poisoning and supplies open-book benign context.

  • BA defenses: Reactive defenses detect backdoors or suppress their activation after training, whereas intraactive defenses disrupt trigger learning during fine-tuning.These approaches intervene after poisoning has entered the training process or during model optimization.
  • LLM rewriting: Prior rewriting methods such as Paraphrase and DPR target prompt injection and adversarial suffix attacks at test time.They summarize or security-condition input prompts to disrupt jailbreak prompts.
  • Research gap: Earlier rewriting methods rely on the rewriter’s parametric, closed-book knowledge and primarily address training-free attacks.The presented work instead studies training-based BAs and PIAs with open-book knowledge.
  • Proposed extension: OBBR retrieves semantically similar benign samples and concatenates them with each training sample to construct rewriter context.The overview depicts this context as producing sanitized outputs before fine-tuning.
  • Evaluation scope: The paper also evaluates how rewriting affects natural-language performance after fine-tuning.This extends the comparison beyond attack mitigation to downstream utility.

4 Open-Book Benign Rewriting

OBBR augments an LLM rewriter with retrieved benign examples, rewrites every training sample before fine-tuning, and theoretically guarantees safer outputs than closed-book rewriting. The method treats rewriting as projection into benign prompt space.

  • Closed-book rewriting: Closed-book rewriting conditions only on the input prompt and system instruction, relying entirely on parametric knowledge.It has no grounding in known-safe data.
  • Open-book construction: OBBR retrieves k nearest benign samples using sentence-embedding cosine similarity and adds them to the rewriter context.The resulting context is [s; b_1; …; b_k; x].
  • Open-book construction: Retrieved benign examples provide task-relevant safe phrasing that steers the rewriter toward benign prompts.They complement high-level safety instructions with concrete examples.
  • Pipeline: OBBR rewrites every training sample and fine-tunes on the rewritten dataset instead of the original dataset.This addresses triggers and malicious content before training.
  • Theory: OBBR strictly increases the posterior probability of benign samples relative to closed-book rewriting.The theorem compares contexts under open-book and closed-book rewriting.
  • Theory: OBBR-generated sequences are more likely to belong to benign prompt space than CBBR-generated sequences.The paper interprets OBBR as projecting potentially malicious prompts into benign prompt space.

5 Experiments

Experiments evaluate OBBR and other defenses across backdoor attacks, computational overhead, natural-language performance, and poison injection attacks. OBBR achieves the strongest overall protection while preserving performance and maintaining lower overhead than major prior defenses.

  • Backdoor attacks: Prior defenses generally leave average ASR high: CROW, Quantize, and Decoding remain above 67%, while CLEANGEN reaches 49%.All proactive rewriting methods outperform CLEANGEN on three of the four evaluated models.
  • Backdoor attacks: OBBR achieves the lowest average ASR across all evaluated models, reducing it by 23.6%, 28.4%, and 25.1% compared with CBBR, DPR, and Paraphrase.The evaluation covers four LLMs and five distinct backdoor patterns.
  • Runtime: OBBR balances safety gains with computational cost, while CROW and CLEANGEN incur higher overhead, with CLEANGEN requiring 5.2 times more computation than OBBR.Runtime includes rewriting, training, and inference components.
  • Fine-tuning performance: Across seven natural-language benchmarks, OBBR and DPR do not decrease average performance across models, whereas other rewriting methods can reduce performance.CBBR improves Qwen2.5-7B IFEval performance by 8.1 points but decreases average performance for Qwen-2.5-1.5B.
  • Poison injection attacks: PIAs sharply degrade safety without defense, but OBBR prevents every evaluated model from complying with more than 35% of malicious requests.The PIA evaluation uses StrongREJECT and compares original, undefended, and proactively rewritten models.

6 Discussion and Conclusions

The discussion finds that existing defenses inconsistently reduce backdoor attack success, whereas OBBR provides stronger protection through proactive rewriting. Rewriting also preserves language-modeling utility and extends defense to triggerless poison injection attacks.

  • Backdoor-defense effectiveness: Across four LLMs and five BA families, CROW averages 68.6% ASR, while CLEANGEN, Quantization, and Decoding average 48.9%, 71.5%, and 67.1%.These results indicate limited consistency among state-of-the-art intraactive and reactive defenses.
  • OBBR conclusions: OBBR is 25.7% more effective than alternative rewriting methods and 51% more effective than previous BA defenses across four LLMs and five BA patterns.The paper presents OBBR as a proactive defense applied before fine-tuning.
  • Proactive defense: Proactive rewriting prevents poisoned samples from entering fine-tuning directly, addressing the concern that learned backdoors may persist after later remediation.Previous intraactive and reactive defenses intervene during fine-tuning or decoding instead.
  • Utility preservation: OBBR and DPR do not hurt average performance across the evaluated natural-language benchmarks, while rewriting can sometimes improve downstream utility.CBBR improves Qwen2.5-7B utility by an average 8.1 performance points, but consistency depends on the rewriter.
  • Poison injection attacks: Unlike other rewriting methods, OBBR strongly defends against triggerless PIAs, with its success attributed to retrieved open-book benign samples.PIAs are more challenging than BAs for the considered rewriting methods.

7 Future Work

The paper identifies several directions for extending OBBR, including domain-specific benign corpora, broader safety post-training integration, and model-internal rewriting mechanisms.

  • Domain-specific benign corpora could improve OBBR’s filtering of subtle malicious patterns that do not rely on explicit triggers.
  • Integrating OBBR into Safe RLHF or SafeDPO could extend poisoning protection across the model development lifecycle.
  • Model-internal rewriting mechanisms could support new safety-enhancing architectures for BA and PIA defense.

A Experimental Details

The experiments evaluate multiple models, defenses, attack settings, rewriting prompts, runtime conditions, and natural-language benchmarks using standardized procedures.

  • Models: Four widely used instruction-tuned LLMs are evaluated: two Llama models and two Qwen models.The models range from 1B to 8B parameters and were downloaded from official HuggingFace checkpoints.
  • Backdoor evaluation: BA experiments fine-tune models with LoRA for five epochs and implement BadNets, CTBA, MTBA, Sleeper, and VPI.Training uses rank r=64, scaling factor α=128, dropout 0.05, AdamW, learning rate 5×10^-4, and cosine annealing.
  • Metrics and defenses: Attack success rate is the fraction of triggered prompts that do not result in a refusal.
  • Metrics and defenses: Reactive and intraactive defenses are compared with proactive rewriting methods under shared experimental settings.The compared defenses include CLEANGEN, Decoding, CROW, Quantization, Paraphrase, and DPR.
  • Rewriting setup: OBBR retrieves top-k=3 benign nearest neighbors from UltraFeedback using all-MiniLM-L6-v2, while baseline rewriting methods omit retrieval augmentation.Retrieval uses cosine similarity with chunk size 256 and overlap 10.
  • Runtime experiments: Runtime comparisons maximize batch size within an Nvidia L40S GPU’s 48GB memory and average timings over 10 runs.Defenses previously hardcoded to batch size 1 were modified to expose batch size for fair comparison.
  • LIMA fine-tuning: LIMA experiments fine-tune models for 15 epochs and evaluate natural-language benchmarks with Eleuther’s LM Evaluation Harness.OBBR again retrieves benign samples from UltraFeedback using all-MiniLM-L6-v2.
  • PIA experiments: PIA experiments construct a 5,000-sample jailbreak fine-tuning dataset by corrupting BookCorpus Completion with harmful instruction-following examples.The jailbreak instructions derive from malicious samples in PKU-SafeRLHF.

C BA Details

The appendix points to Table 5 for details of the implemented backdoor attacks.

  • Table 5 contains the details of the implemented backdoor attacks.

D Proof of Theorem 1

The proof establishes that adding open-book benign samples increases the posterior probability that rewriting produces benign content compared with closed-book rewriting.

  • Theorem 3 states that the benign posterior with OBBR is strictly greater than without OBBR.The contexts c+ and c− represent rewriting with and without open-book benign samples, respectively.
  • The proof models OBBR and closed-book rewriting as contexts containing and omitting benign samples before the input.
  • The rewriter’s token distribution is expressed through a latent benign-or-malicious variable and its context-dependent posterior.The context influences generation solely through that posterior.
  • Bayes’ theorem yields the strict posterior inequality favoring benign content under OBBR.
  • OBBR therefore increases the posterior probability of generating benign samples over closed-book rewriting.

E Proof of Theorem 2

The proof compares benign-output probabilities under open-book and closed-book rewriting by factoring their difference into two terms. The first is non-negative by the definition of ζ, while the second is strictly positive by Theorem 1, establishing the claimed strict ordering.

  • The proof defines y+ and y− as outputs generated with open-book and closed-book rewriting, respectively.
  • The difference between benign-output probabilities factors into a benign-versus-malicious output gap and a context-dependent probability gap.
  • Consequently, the factored difference is strictly positive, proving that open-book rewriting yields a higher benign-output probability than closed-book rewriting.
  • The benign-output gap is non-negative because Pr(y∈B | ζ = B,x) ≥ Pr(y∈B | ζ = M,x) by definition of ζ.
  • The context-dependent gap is strictly positive because p(ζ = B | c+) > p(ζ = B | c−) from Theorem 1.
Loading 2605.19147v1…