Purple-teaming LLMs with Adversarial Defender Training

1 Introduction

• Static safety data may miss a target model’s specific vulnerabilities, while safety concerns change across cultural contexts and emerging use cases.The introduction argues for adaptively aligning LLMs to specific safety rules.Source: §1 · pp. 1–2
• PAD combines trainable red-team attacks with blue-team defense, collecting conversational data through self-play and iteratively updating both modules in a GAN-style process.A Judge evaluates safety, and defenders learn to identify unsafe responses and explain the reason.Source: §1 · pp. 2–3
• PAD defenders significantly outperform baselines as the attacker strengthens; the defense generalizes to two other red-teaming test sets, while multi-turn defense remains weaker.The reported findings also identify self-discrimination as a way to reduce multi-turn risk.Source: §1 · p. 2
• The authors highlight discrimination ability as important for preserving generation quality and for defending multi-turn attacks, while calling for risk-specific discrimination strategies.Their case study describes attackers exposing vulnerabilities, improving their strategies, and then facing an updated defender.Source: §1 · pp. 2–3

2.1 Red-teaming LLMs

• Existing red-teaming approaches include static datasets, prompt-engineered attacks, transferable suffixes, and generated queries, but face limits in adaptability, model-specificity, realism, or efficiency.The section identifies a need for previously unseen queries and stable attacks that adapt as LLMs change.Source: §2.1 · p. 3

2.2 Safeguard LLMs

• Safeguards use filters, prompted LLM evaluators, and trained classifiers, but some training data are unreleased and these models can require substantial computational resources.The section also notes concerns about the reliability of LLM safety judgments.Source: §2.2 · p. 3

2.3 Discrimination and Generation: Entangling Abilities in the Era of LLMs

• LLMs can identify unsafe content yet still generate it; critique-and-revision, self-evaluation rewards, and explanations leverage discrimination, which this work investigates for safeguarding.The section frames safety assessment, suitable discriminators, and improved generation as steps to integrate.Source: §2.3 · pp. 3–4

3 Method

• PAD is a framework for safeguarding a target LLM against content that violates one or more specified safety rules.The method section introduces the design and then describes its components and training procedure.Source: §3 · p. 4

3.1 PAD Overview

• The attacker probes guardrails while the defender reinforces them; defender training includes safety judgments with explanations, which the authors report can preserve overall generation quality.The approach combines assessment and defense tasks that are typically addressed independently.Source: §3.1 · p. 4

3.2 PAD Details

• PAD initializes LoRA attacker and defender modules from a base LLM; the attacker seeks unsafe responses, while the defender generates responses and classifies their safety, with an external Judge providing labels and explanations.The defender is used for both conversation and safety discrimination.Source: §3.2 · p. 4
• Training samples attacker–defender conversations from adversarial statements, with attacks adapting to the defender’s reactions and responses conditioned on conversation history.The process is repeated with different random seeds to gather diverse training data.Source: §3.2 · pp. 4–5
• A Judge evaluates sampled prompt–response pairs for safety and generates labels and explanations; the described ShieldLM Judge can be replaced by other methods.The paper describes ShieldLM as fine-tuned on safety-classification datasets.Source: §3.2 · p. 5

3. Model Updating

• Attacker outputs are reorganized into preference triplets, with unsafe-eliciting outputs preferred over safe-eliciting ones, and Aθ is updated using DPO.Source: §3 · p. 5
• For multi-turn attacks, the attacker receives the dialogue history along with the training point.Source: §3 · p. 5
• The defender is fine-tuned on Judge outputs containing a classification result and explanation, minimizing cross-entropy loss.The training data include dialogue history and the Judge’s result and explanation.Source: §3 · p. 5
• Off-the-shelf APIs such as GPT-4 were excluded after consistently refusing to perform safety judgments.Source: §3 · p. 5
• Each iteration samples conversations from the latest attacker and defender, collects Judge scores and explanations, and updates both models; the authors expect this to yield harder samples.The algorithm samples across T dialogue turns, and the next iteration uses the updated models.Source: §3 · p. 5

4.1 Data

• PAD starts from 200 SafetyPrompt attack prompts for common safety risks; the scratch experiment instead uses automatically generated, manually verified points for a single no-gender-bias rule.Testing uses 56 SafePrompt prompts, an equally sized SP-Instruct sample, 400 HarmBench inputs across seven risk categories, and generated scratch test points.Source: §4.1 · pp. 5–6

4.2 Implementation

• Experiments use Qwen1.5-7B-Chat as BASE, train in Chinese, and test in Chinese and English; LoRA modules are trained for three epochs per iteration.Training uses rank 128, alpha 64, batch size 4, and learning rate 5e−5; each iteration takes about three hours for sampling and judging and 1.5 hours for training on four RTX8000 GPUs.Source: §4.2 · p. 6

4.3 Experimental setup

• The setup compares PAD attackers and defenders across iterations with RLHF/DPO, SFT, and BASE; it also tests Vanilla, Critique&Revise, and Reject variants.Inference cost rises across the three variants, and stronger rejection can skew the safety–quality balance toward safety.Source: §4.3 · pp. 6–7

4.4 Results and Analysis

• PADv3 achieves 0.99% mean ASR on the PAIRQwen attacker, a 70.62% improvement against BASE; HarmBench and SP-Instruct follow the same pattern.The experiments also report that enhancing discrimination ability can improve safe generation.Source: Table 1 · p. 7, §4.4 · pp. 7–8
• In the scratch setting, attackers and defenders strengthen as the system updates, supporting PAD's application where no initial attack resources are available.The scratch experiment begins with one safety rule and no initial attack points.Source: Table 2 · p. 7, §4.4 · p. 8
• Adding Critique&Revise lowers mean ASR against PADv2Att from 35.32% to 6.55% for BASE and from 11.90% to 4.56% for PAD; Reject further improves safety.PADv3 with Critique&Revise and Reject achieves 100% defense on SP-Instruct, but performs slightly worse than BASE against PAIRQwen under Critique&Revise.Source: §4.4 · p. 8
• PADv3 raises unsafe precision from 75.82 to 98.09, recall from 52.72 to 76.52, and overall accuracy from 61.88 to 85.96 versus BASE.BASE rejects 39.2% of inquiries, including 19.3% that are safe; PADv3 rejects 11.25%, including 6.59% that are safe.Source: §4.4 · p. 8
• PADv3's safe responses receive significantly higher overall-quality evaluations than BASE's under each setting; for Reject, only 26.5% of BASE responses are considered better.The evaluation analysis attributes much of PAD's advantage to recognizing implicit risks and addressing them carefully, even when both responses are safe.Source: Figure 4 · p. 8, §4.4 · p. 8

5.1 Breakdown w.r.t. conversational turns

• PADv3 reduces Hit ASR by 41, 37, and 29 percentage points across turns, while adding rejection reduces it by 54.5, 67.9, and 69.7 percentage points.The declining reduction trend indicates that multi-turn attacks remain a challenge.Source: §5.1 · p. 9, Figure 5 · p. 9
• Providing PADv3 with problematic BASE-defender dialogue history raises HitbyT2 ASR from 41.1% to 57.1%.The authors describe this as error propagation in autoregressive models.Source: §5.1 · p. 9
• PADv2Att increases ASR against the BASE defender by 7.29, 5.35, and 5.84 times compared with PAIRQwen across turns.The turn-wise analysis indicates that PAD attackers pose a more challenging defense task.Source: §5.1 · p. 9, Figure 5 · p. 9

5.2 Breakdown w.r.t. safety rules

• Defender performance differs across the seven safety rules, and Vanilla-to-C&R differences indicate varying self-improvement capabilities.The breakdown evaluates the three defender variations against PADv2Att.Source: §5.2 · p. 9, Figure 6 · p. 9
• Models can identify responses as unsafe yet struggle to generate safe alternatives for rules beyond Insult, Privacy& Property, and Ethics.The authors suggest further intervention when a model cannot generate a desirable response.Source: §5.2 · p. 9
• Reject retains non-negligible ASR on some rules; hate speech, social bias, and ethics may need rigorous reasoning, while privacy&property and physical harm may need expert knowledge.The authors leave more dedicated discrimination strategies for future work.Source: §5.2 · pp. 9–10

• The proposed pipeline safeguards LLMs through adversarial defender training and incorporates red-teaming for more challenging data collection.Source: §6 · p. 10
• Experiments demonstrate the method's effectiveness across settings with different inference costs while balancing safety and overall generation quality.Source: §6 · p. 10
• Error analysis highlights limitations in defending against attackers trained to break the target model, especially in multi-turn scenarios and rules requiring real-world knowledge or rigorous reasoning.Source: §6 · p. 10

• Some safety rules show limited improvement and may require dedicated judgment approaches, such as a Judge model for each rule.Ethics judgment may need dedicated consideration rather than reliance on a large classification corpus.Source: Limitation · p. 10
• Testing longer conversational turns would require more GPU memory, so the reported experiments cover only the first three turns.Source: Limitation · p. 10
• The method's demonstrated performance is on Qwen1.5, and its generalization to other open-source LLMs remains to be tested.Source: Limitation · p. 10

• Table 4 maps topics in SafetyPrompts to rules.Source: A Safety Topics and Rules · p. 10, Table 4 · p. 10

Overall quality evaluation

• The overall-quality prompt asks GPT4-o to compare two responses across safety, quality, informativeness, engagement, and other relevant requirements, then choose A, B, or Tie.Source: B Prompts · p. 10

Generating attack points

• The attack prompt asks BASE for 50 common viewpoints that violate a rule, including colloquial, aggressive, or controversial content.These viewpoints are intended to test other models.Source: B Prompts · pp. 10–11

Critique & Revise

• The revision prompt directs an assistant to classify a response as safe or unsafe, analyze problematic statements, and provide a modified safe response.Source: B Prompts · p. 11