Be Kind, Rewrite: Benign Projections via Rewriting Defend Against LLM Data Poisoning Attacks

Motivation

• LLMs can learn harmful trigger-conditioned behaviors from poisoned training samples, creating serious data-poisoning risks.Triggers can make seemingly benign phrases induce unsafe behavior.Source: §1 · p. 1

Existing defenses

• Existing defenses are primarily reactive or intraactive, intervening after fine-tuning or by modifying training dynamics.The paper positions rewriting as an earlier intervention point.Source: §1 · p. 1, Figure 1 · p. 2

Proposed approach

• OBBR makes rewriting proactive by sanitizing training samples before fine-tuning.It uses open-book benign samples to guide the rewriter toward benign outputs.Source: §1 · pp. 1–2

Results

• 51%: OBBR reduces attack success rates on average versus state-of-the-art BA defenses across five attack types and four LLMs.OBBR also reduces ASR by an average 26.8% versus closed-book rewriting defenses.Source: §1 · p. 2

Efficiency

• 38.5%: OBBR increases end-to-end runtime versus no defense while improving BA safety by an average 58.8%.CLEANGEN increases runtime by 619% for a 34.3% average safety improvement, while CROW increases runtime by 95.5% for an 8% improvement.Source: §1 · p. 2

Broader protection

• 55%: OBBR reduces poison injection attack effectiveness on average, compared with 23% for other closed-book proactive methods.These attacks do not rely on triggers and degrade general safety compliance.Source: §1 · p. 2

Data-poisoning threat

• Open-web training corpora expose LLMs to adversaries who can manipulate online data sources.This creates a pathway for poison injection attacks into training data.Source: §2 · p. 2

Poison injection attacks

• Poison injection attacks introduce trigger-less malicious samples that broadly degrade safety guardrails and compliance.Their effects can be assessed with general jailbreak and safety benchmarks.Source: §2.1 · p. 3

Backdoor attacks

• Backdoor attacks use poisoned trigger–response pairs to induce targeted harmful behavior only when specific trigger phrases appear.Examples include negative sentiment and malicious code generation.Source: §2.1 · p. 3

Security implication

• Backdoors can persist after subsequent safety training, and effective removal from deployed models remains unknown.This supports the need for defenses before fine-tuning learns the malicious behavior.Source: §2.1 · p. 3

BA defenses

• Reactive defenses detect backdoors or suppress their activation after training, whereas intraactive defenses disrupt trigger learning during fine-tuning.These approaches intervene after poisoning has entered the training process or during model optimization.Source: §3 · p. 3

LLM rewriting

• Prior rewriting methods such as Paraphrase and DPR target prompt injection and adversarial suffix attacks at test time.They summarize or security-condition input prompts to disrupt jailbreak prompts.Source: §3 · p. 3

Research gap

• Earlier rewriting methods rely on the rewriter’s parametric, closed-book knowledge and primarily address training-free attacks.The presented work instead studies training-based BAs and PIAs with open-book knowledge.Source: §3 · p. 3

Proposed extension

• OBBR retrieves semantically similar benign samples and concatenates them with each training sample to construct rewriter context.The overview depicts this context as producing sanitized outputs before fine-tuning.Source: Figure 2 · p. 4

Evaluation scope

• The paper also evaluates how rewriting affects natural-language performance after fine-tuning.This extends the comparison beyond attack mitigation to downstream utility.Source: §3 · p. 4

Closed-book rewriting

• Closed-book rewriting conditions only on the input prompt and system instruction, relying entirely on parametric knowledge.It has no grounding in known-safe data.Source: §4 · p. 4

Open-book construction

• OBBR retrieves k nearest benign samples using sentence-embedding cosine similarity and adds them to the rewriter context.The resulting context is [s; b_1; …; b_k; x].Source: §4 · p. 4
• Retrieved benign examples provide task-relevant safe phrasing that steers the rewriter toward benign prompts.They complement high-level safety instructions with concrete examples.Source: §4 · p. 4

Pipeline

• OBBR rewrites every training sample and fine-tunes on the rewritten dataset instead of the original dataset.This addresses triggers and malicious content before training.Source: §4 · p. 5

Theory

• OBBR strictly increases the posterior probability of benign samples relative to closed-book rewriting.The theorem compares contexts under open-book and closed-book rewriting.Source: §4.1 · p. 5
• OBBR-generated sequences are more likely to belong to benign prompt space than CBBR-generated sequences.The paper interprets OBBR as projecting potentially malicious prompts into benign prompt space.Source: §4.1 · p. 5

Backdoor attacks

• Prior defenses generally leave average ASR high: CROW, Quantize, and Decoding remain above 67%, while CLEANGEN reaches 49%.All proactive rewriting methods outperform CLEANGEN on three of the four evaluated models.Source: §5 · p. 6
• OBBR achieves the lowest average ASR across all evaluated models, reducing it by 23.6%, 28.4%, and 25.1% compared with CBBR, DPR, and Paraphrase.The evaluation covers four LLMs and five distinct backdoor patterns.Source: §5 · p. 6

Runtime

• OBBR balances safety gains with computational cost, while CROW and CLEANGEN incur higher overhead, with CLEANGEN requiring 5.2 times more computation than OBBR.Runtime includes rewriting, training, and inference components.Source: §5.1 · pp. 6–7

Fine-tuning performance

• Across seven natural-language benchmarks, OBBR and DPR do not decrease average performance across models, whereas other rewriting methods can reduce performance.CBBR improves Qwen2.5-7B IFEval performance by 8.1 points but decreases average performance for Qwen-2.5-1.5B.Source: §5.2 · p. 7

Poison injection attacks

• PIAs sharply degrade safety without defense, but OBBR prevents every evaluated model from complying with more than 35% of malicious requests.The PIA evaluation uses StrongREJECT and compares original, undefended, and proactively rewritten models.Source: §5.3 · pp. 7–8

Backdoor-defense effectiveness

• Across four LLMs and five BA families, CROW averages 68.6% ASR, while CLEANGEN, Quantization, and Decoding average 48.9%, 71.5%, and 67.1%.These results indicate limited consistency among state-of-the-art intraactive and reactive defenses.Source: §6 · p. 8

OBBR conclusions

• OBBR is 25.7% more effective than alternative rewriting methods and 51% more effective than previous BA defenses across four LLMs and five BA patterns.The paper presents OBBR as a proactive defense applied before fine-tuning.Source: §6 · p. 8

Proactive defense

• Proactive rewriting prevents poisoned samples from entering fine-tuning directly, addressing the concern that learned backdoors may persist after later remediation.Previous intraactive and reactive defenses intervene during fine-tuning or decoding instead.Source: §6 · pp. 8–9

Utility preservation

• OBBR and DPR do not hurt average performance across the evaluated natural-language benchmarks, while rewriting can sometimes improve downstream utility.CBBR improves Qwen2.5-7B utility by an average 8.1 performance points, but consistency depends on the rewriter.Source: §6 · p. 9

Poison injection attacks

• Unlike other rewriting methods, OBBR strongly defends against triggerless PIAs, with its success attributed to retrieved open-book benign samples.PIAs are more challenging than BAs for the considered rewriting methods.Source: §6 · p. 9

• Domain-specific benign corpora could improve OBBR’s filtering of subtle malicious patterns that do not rely on explicit triggers.Source: §7 · p. 9
• Integrating OBBR into Safe RLHF or SafeDPO could extend poisoning protection across the model development lifecycle.Source: §7 · p. 9
• Model-internal rewriting mechanisms could support new safety-enhancing architectures for BA and PIA defense.Source: §7 · p. 9

Models

• Four widely used instruction-tuned LLMs are evaluated: two Llama models and two Qwen models.The models range from 1B to 8B parameters and were downloaded from official HuggingFace checkpoints.Source: A Experimental Details · p. 12

Backdoor evaluation

• BA experiments fine-tune models with LoRA for five epochs and implement BadNets, CTBA, MTBA, Sleeper, and VPI.Training uses rank r=64, scaling factor α=128, dropout 0.05, AdamW, learning rate 5×10^-4, and cosine annealing.Source: A Experimental Details · p. 12

Metrics and defenses

• Attack success rate is the fraction of triggered prompts that do not result in a refusal.Source: A Experimental Details · p. 12
• Reactive and intraactive defenses are compared with proactive rewriting methods under shared experimental settings.The compared defenses include CLEANGEN, Decoding, CROW, Quantization, Paraphrase, and DPR.Source: A Experimental Details · pp. 12–13

Rewriting setup

• OBBR retrieves top-k=3 benign nearest neighbors from UltraFeedback using all-MiniLM-L6-v2, while baseline rewriting methods omit retrieval augmentation.Retrieval uses cosine similarity with chunk size 256 and overlap 10.Source: A Experimental Details · pp. 12–13

Runtime experiments

• Runtime comparisons maximize batch size within an Nvidia L40S GPU’s 48GB memory and average timings over 10 runs.Defenses previously hardcoded to batch size 1 were modified to expose batch size for fair comparison.Source: A Experimental Details · p. 13

LIMA fine-tuning

• LIMA experiments fine-tune models for 15 epochs and evaluate natural-language benchmarks with Eleuther’s LM Evaluation Harness.OBBR again retrieves benign samples from UltraFeedback using all-MiniLM-L6-v2.Source: A Experimental Details · p. 13

PIA experiments

• PIA experiments construct a 5,000-sample jailbreak fine-tuning dataset by corrupting BookCorpus Completion with harmful instruction-following examples.The jailbreak instructions derive from malicious samples in PKU-SafeRLHF.Source: A Experimental Details · p. 13

• Table 5 contains the details of the implemented backdoor attacks.Source: C BA Details · p. 14

• Theorem 3 states that the benign posterior with OBBR is strictly greater than without OBBR.The contexts c+ and c− represent rewriting with and without open-book benign samples, respectively.Source: D Proof of Theorem 1 · p. 14
• The proof models OBBR and closed-book rewriting as contexts containing and omitting benign samples before the input.Source: D Proof of Theorem 1 · p. 14
• The rewriter’s token distribution is expressed through a latent benign-or-malicious variable and its context-dependent posterior.The context influences generation solely through that posterior.Source: D Proof of Theorem 1 · p. 14
• Bayes’ theorem yields the strict posterior inequality favoring benign content under OBBR.Source: D Proof of Theorem 1 · p. 15
• OBBR therefore increases the posterior probability of generating benign samples over closed-book rewriting.Source: D Proof of Theorem 1 · p. 15

• The proof defines y+ and y− as outputs generated with open-book and closed-book rewriting, respectively.Source: E Proof of Theorem 2 · p. 15
• The difference between benign-output probabilities factors into a benign-versus-malicious output gap and a context-dependent probability gap.Source: E Proof of Theorem 2 · p. 15
• Consequently, the factored difference is strictly positive, proving that open-book rewriting yields a higher benign-output probability than closed-book rewriting.Source: E Proof of Theorem 2 · p. 15
• The benign-output gap is non-negative because Pr(y∈B | ζ = B,x) ≥ Pr(y∈B | ζ = M,x) by definition of ζ.Source: E Proof of Theorem 2 · p. 15
• The context-dependent gap is strictly positive because p(ζ = B | c+) > p(ζ = B | c−) from Theorem 1.Source: E Proof of Theorem 2 · p. 15