Efficient Adversarial Training in LLMs with Continuous Attacks

1 Introduction

• GCG-based discrete adversarial training can require hundreds of thousands of model evaluations per attack, creating substantial training overhead.Earlier LLM adversarial-training attempts were ineffective; R2D2 was the first reported to improve robustness against multiple attacks.Source: §1 · p. 2
• Continuous attacks may offer a faster alternative because they have shown higher success rates and faster computation than discrete attacks in LLMs.Prior continuous adversarial training has also been effective for encoder-decoder models such as BERT.Source: §1 · p. 2
• Continuous attacks in token-embedding space can extrapolate to discrete attacks: CAT and CAPO reach up to 100% attack robustness while using over 299 times less compute than R2D2.CAT combines adversarial-behaviour training with utility-data fine-tuning; CAPO is an adversarial IPO variant that does not require utility data.Source: §1 · p. 2
• With chat templates enabled for standard queries, R2D2 often refuses benign inputs, whereas CAT and CAPO show substantially fewer refusals.The paper attributes the misleading utility evaluation to testing safety with chat templates but utility without them.Source: §1 · p. 2
• The paper frames its central question as whether continuous attacks in token-embedding space provide robustness to discrete natural-language attacks.It answers this question affirmatively using two adversarial-training algorithms.Source: §1 · p. 2

2 Related Work

• GCG generates transferable adversarial suffixes, while other work studies decoding changes and continuous embedding optimization as routes to harmful behavior.The paper also surveys suffix attacks and jailbreaks as distinct LLM threat models [1].Source: §2 · pp. 2–3
• Jailbreak methods include fine-tuning on jailbreak examples and iterative or genetic prompt-generation strategies [15] [17].PAIR iteratively queries a target LLM to optimize a jailbreak prompt [15].Source: §2 · p. 3
• Continuous adversarial training has mostly been studied on encoder-decoder models, including methods that promote embedding smoothness, invariance, or downstream generalization [23].Other cited work considers randomized smoothing for autoregressive LLM robustness [27].Source: §2 · p. 3
• Unlike latent adversarial training for untargeted tasks [28], this work targets discrete attacks and jailbreaks while preserving helpfulness.It uses harmful targets in its losses and evaluates across multiple benchmarks and attacks.Source: §2 · p. 3
• Adversarial dataset augmentation trains on generated attacks but does not adapt the model online to worst-case attacks, making it orthogonal to this work [29] [18].Source: §2 · p. 3

3 Method

• CAT and CAPO are the two continuous adversarial-training algorithms introduced in this method.The section reviews standard adversarial training, LLM-specific perturbations and objectives, then presents the algorithms and design choices.Source: §3 · p. 3

3.1 Adversarial Training

• Standard adversarial training is framed as minimax optimization over perturbations allowed by a threat model.The formulation uses a model, dataset, perturbation set, and loss.Source: §3.1 · pp. 3–4

3.2 Attack Perturbation Sets in LLMs

• Discrete suffix attacks append attacker-controlled token sequences, but searching their combinatorial space is too expensive for training on large datasets.The paper instead proposes perturbations within an ϵ-ball around each token embedding.Source: §3.2 · p. 4

3.3 Adversarial Training in LLMs

• For targeted LLM attacks, training encourages a safe response while decreasing the likelihood of a specified harmful continuation under an adversarial prompt.The resulting objective is described as an unlikelihood loss; the dataset pairs harmful prompts with safe answers.Source: §3.3 · p. 4

3.4 Continuous-Adversarial Unlikelihood

• Continuous attacks replace discrete GCG suffix optimization and are reported as 299 times more efficient, with loss cutoffs used to limit over-optimization.The cutoffs apply to both toward and away losses.Source: §3.4 · p. 5

3.5 Continuous-Adversarial IPO

• CAPO adapts IPO to the adversarial objective; its loss implicitly minimizes KL divergence from the original model, allowing training without utility data.The loss is presented as less prone to overfitting than the original DPO form.Source: §3.5 · p. 5

3.6 Design Decisions

• The toward-loss attack currently makes the harmful output more likely; separately attacking the safe and harmful continuations is left for future work because it adds computational overhead.Source: §3.6 · p. 5
• Applying the adversarial input perturbation to the reference model made training unstable in the DPO setting.The authors hypothesize that the reference model represents desirable safe-answer log probabilities; the original DPO paper reports a similar observation.Source: §3.6 · p. 5

4 Experimental Details

• The evaluation asks whether continuous-attack robustness extrapolates to discrete natural-language attacks.Source: §4 · p. 5
• Robustness is evaluated on the first 40 HarmBench test samples rather than the full test set because adversarial attacks are computationally costly.Source: §4 · p. 5
• Utility measures include MMLU, ARC-E, ARC-C, MT-BENCH, and a 40-query harmless-prompt refusal check; only MT-BENCH and HARMLESS use the chat template.MMLU evaluation is limited to 100 questions per category.Source: §4 · pp. 5–6
• Five instruction-tuned models are evaluated: Gemma (2B), Phi-3-Mini (3.8B), and Mistral, Zephyr, and Llama2 (7B each).Zephyr + R2D2 is also included as a Mistral-7B model fine-tuned with R2D2.Source: §4 · p. 6
• Experiments used LoRA and 4-bit quantization, with 10 attack iterations and model-specific perturbation sizes.Runs used V100 or A100 GPUs and required at least 1904 GPU hours; constrained compute limited hyperparameter search, which could leave better robustness–utility trade-offs undiscovered.Source: §4 · p. 6
• Robustness evaluation uses GCG, AutoDAN, and PAIR, with adaptive attacks and ICL also evaluated in Tables 5 and 6.Attack success is assessed with the harmfulness classifier from [6], reported to align well with human judgment.Source: §4 · p. 6

Computational benefit

• Compared with CAT and CAPO, computing one R2D2 adversarial example costs ≈128.5 times more, and its full training costs 299 times more.Table 1 covers forward and backward passes for one adversarial example and whole training, plus iterations, batch size, and batch-update time.Source: §5 · p. 6, Table 1 · p. 6

LLM adversarial training with utility data

• CAT considerably improves average robustness across models; gains cover all attacks for Gemma and Zephyr, while PAIR remains effective against Phi-3-mini and Mistral-7B.All CAT-trained models retain considerable utility despite similar utility degradations.Source: §5 · p. 7, Figure 2 · p. 7
• For Zephyr, CAT has 10% PAIR ASR versus 40% for R2D2, with marginally worse standard utility but substantially better robustness and HARMLESS results.The authors interpret the HARMLESS result as evidence that CAT has not overfit the safety objective or Harmbench behavior patterns.Source: §5 · p. 7, Figure 2 · p. 7

LLM adversarial training without utility data

• Phi-3-mini IPO achieves 100% attack robustness across conducted attacks; CAPO uses no utility dataset and causes no larger common-benchmark utility decreases than CAT.CAPO also achieves considerably higher PAIR robustness, while Gemma CAPO has slightly lower robustness than CAT against GCG.Source: §5 · pp. 7–8
• The results indicate that adversarial variations of common alignment methods such as IPO can be used to adversarially align LLMs.Source: §5 · p. 8

Utility evaluation

• Standard MMLU and ARC evaluations omit chat templates, so their scores may not reflect chat-mode utility or refusals; ARC measures answer-token likelihood.The authors recommend accounting for this mismatch when evaluating robustness and utility on the same model.Source: §6 · p. 8

Training data failure modes

• Models may refuse harmful and harmless requests phrased like common adversarial-training data yet answer them when phrased politely.The authors introduce POLITEHARMBENCH and HARMLESS to examine these style effects, leaving better datasets and benchmarks for future work.Source: §6 · p. 8

Robust fine tuning without attack

• Fine-tuning with CAPO but without adversarial attacks yields no robustness gains, supporting the role of attacks in the training procedure.The passage identifies continuous adversarial attacks as a crucial part of the fine-tuning algorithm.Source: §7 · p. 8

One-step adversarial training in LLMs

• Ten attack iterations still increase training time by an order of magnitude, while one-step CAPO achieves comparable robustness improvements with slightly worse utility trade-offs.The authors contrast this cost with GCG’s 2,570 model evaluations under default settings.Source: §7 · p. 8

Robustness-utility trade-offs

• Across seven CAPO models, GCG loss and MMLU score span trade-offs from high robustness with no utility to lower robustness with slightly higher utility.The experiments vary ϵ from 0.00125 to 0.3 and β from 0 to 0.5.Source: §7 · pp. 8–9, Figure 4 · p. 9
• Larger ϵ and smaller β are associated with greater robustness and reduced utility in CAPO.Figure 3 examines GCG loss against MMLU score as these hyperparameters vary.Source: §7 · p. 9, Figure 3 · p. 9

Correlation between continuous attack loss and GCG loss

• r = 0.99 (p = 0.0075) between continuous-attack loss and GCG loss indicates that robustness to continuous attacks tracks discrete GCG robustness.The authors suggest continuous-attack loss could serve as a proxy for evaluating robustness to discrete attacks.Source: §7 · p. 9, Figure 4 · p. 9

8 Conclusion

• The authors report up to 100% robustness against state-of-the-art attacks for PHI-3-MINI-CAPO, surpassing R2D2’s robustness–utility trade-offs while requiring at least 299 times less compute.CAT combines adversarial-behaviour training with utility-data fine-tuning; CAPO does not require additional utility data.Source: §8 · p. 9
• Robustness evaluations can mislead when models refuse benign queries or when chat templates and prompt grammar are not controlled.The conclusion calls for care in evaluating both robustness and utility.Source: §8 · pp. 9–10
• The method depends on harmful-dataset quality and breadth, and CAT’s utility-data requirement may shift predictions on unrelated tasks.The authors also cite hyperparameter selection and the inability to test models at 70B parameters or larger as limitations.Source: §8 · p. 10
• The broader-impact discussion notes that lower training computation could reduce carbon footprint, while warning that perceived safety may invite overconfidence and requires more red teaming.It also notes possible misuse of adversarial training to suppress content regardless of harmfulness.Source: §8 · p. 10

A.1 Adversarial Training

• CAT uses utility, toward, and away loss weights, clamps the away and toward losses, and trains with a 7:1 ratio of utility to harmful examples.The stated defaults are αu = 1.0, αt = 0.5, αa = 0.5, acut = −5, and tcut = 0.5.Source: §A.1 · p. 14, Table 2 · p. 15
• CAPO uses IPO to prevent overfitting, with β set to 0.25 for GEMMA, 0.5 for PHI-3-MINI, and X for MISTRAL-7B.Table 3 is identified as the CAPO hyperparameter table.Source: §A.1 · p. 15, Table 3 · p. 15

A.2 Models

• Table 4 summarizes the models used in the experiments.The supplied passage does not provide the table’s model entries.Source: §A.2 · p. 15, Table 4 · p. 15

B Robustness extrapolation to discrete attacks

• CAT and CAPO achieve competitive or superior robustness–utility trade-offs compared with R2D2.Table 5 summarizes utility and robustness before and after adversarial training across the models.Source: B Robustness extrapolation to discrete… · p. 15, Table 5 · p. 16

B.1 One-Step Adversarial Training

• Table 7 presents the one-step training ablation as differences from the base model.The supplied table caption does not report the ablation’s numerical results.Source: Table 7 · p. 16

B.2 Training without Attacks

• Robustness does not improve when the IPO-based algorithm is trained without attacks, and NPO likewise does not improve robustness in these experiments.Table 8 reports the no-adversarial-training ablation as differences from the base model.Source: §B.2 · p. 17, Table 8 · p. 17

C Adversarial Training Ablations

• As ϵ increases from 0.0125 to 0.1, GCG loss falls from approximately 14.9 to near 0 while MMLU rises from 0 to around 0.39.The passage describes this as an inverse relationship between GCG loss and MMLU.Source: C Adversarial Training Ablations · p. 17
• As CAPO β increases from 0 to 0.5, GCG loss drops from 6.1 to 0.8 while MMLU rises from about 0.25 to 0.38.The passage says higher β values are associated with lower GCG loss and improved utility.Source: C Adversarial Training Ablations · p. 17

D Continuous Attacks Sanity Check

• An unconstrained continuous attack breaks all models, while adversarially trained models are more robust against ϵ-ball attacks.Source: D Continuous Attacks Sanity Check · p. 17

• NPO and IPO without adversarial training do not outperform the paper's adversarial training results.Source: E Machine unlearnign and preference… · p. 17
• Table 9 reports utility and attack success rate for IPO and NPO without adversarial training on Phi-3-MINI and GEMMA, with differences from the base model.Source: Table 9 · p. 17
• For unconstrained attacks, Figure 5 reports loss converging to 0 for all models; its dashed line marks the threshold for affirmative responses to all toxic queries.Source: Figure 5 · p. 18

F Adversarial training computational effort

• R2D2 requires 165,632,000 combined forward and backward passes in the stated training setup, while one GCG update requires 2,570 passes.The setup uses 224 utility samples and 32 adversarial behaviours per batch, with 512 attack candidates, five attack steps, and 2,000 training iterations.Source: F Adversarial training computational… · p. 19
• The appendix reports a combined count of 20 for CAT and CAPO and a CAPO total of 552,960 passes under its stated settings.The CAPO calculation uses 64 adversarial behaviours, 10 attack steps, and 360 iterations.Source: F Adversarial training computational… · pp. 19–20

G MMLU refusal under chat template

• The MMLU refusal evaluation counts refusals per model across 57 subjects with 100 questions each, using the chat template; totals are reported in Table 10.The evaluation is intended to demonstrate high refusal rates on benign queries among previously adversarially robust models.Source: G MMLU refusal under chat template · p. 20, Table 10 · p. 20

H POLITEHARMBENCH

• POLITEHARMBENCH examines how slight rephrasings can expose vulnerabilities, and the authors note that Harmbench sometimes incorrectly classifies unsafe answers as safe.The authors use “Sorry” as a refusal keyword, ignoring capitalization.Source: H POLITEHARMBENCH · p. 21

I HARMLESS Dataset

• HARMLESS evaluates whether models answer benign prompts when queried with their chat template, with answering versus refusal checked manually.Its examples span requests for stories, explanations, persuasive writing, and practical instructions.Source: I HARMLESS Dataset · p. 22

NeurIPS Paper Checklist

• The checklist says training and test details appear in Section 4 and the appendix, code will be published if accepted, and non-public data are made available.The authors answer yes to the reproducibility and open-access checklist questions.Source: NeurIPS Paper Checklist · pp. 24–26
• The authors report no error bars because experiments, particularly evaluations, were too computationally expensive to repeat several times.They state that total GPU hours are provided.Source: NeurIPS Paper Checklist · p. 26
• The checklist says compute reporting should identify worker type and resources, per-run and total compute, and whether the project used more compute than reported experiments.It gives disclosure expectations rather than measured compute amounts.Source: NeurIPS Paper Checklist · p. 26
• The authors answer Yes to following the NeurIPS Code of Ethics.They state that they read the code and followed its guidelines.Source: NeurIPS Paper Checklist · p. 27
• The authors answer Yes to discussing positive and negative societal impacts, referring readers to the conclusion.The checklist notes that the discussion is in § 8.Source: NeurIPS Paper Checklist · p. 27
• The authors answer NA on release safeguards, stating that the paper provides only a rephrased version of an existing dataset.Source: NeurIPS Paper Checklist · p. 27
• The authors answer Yes that existing assets are credited and their use respected, citing the datasets discussion in § 4.Source: NeurIPS Paper Checklist · p. 28
• The authors answer NA for crowdsourcing and human-subject research because they conducted none.The same justification is given for the IRB-approval checklist item.Source: NeurIPS Paper Checklist · pp. 28–29