PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Attacks: Red-teaming LLMs via Adaptive Environments

Taeyoung Yun, Pierre-Luc St-Charles, Jinkyoo Park, Yoshua Bengio, Minsu Kim

arXiv:2509.21947v2cs.LGcs.AI

TL;DR

Automated red-teaming needs diverse harmful prompts, but RL methods often collapse onto easy, high-reward modes. Active Attacks periodically safety fine-tunes the victim to reshape rewards and drive exploration toward harder vulnerabilities, improving cross-attack success against GFlowNets from 0.07% to 31.28% with a 6% computation increase.

  • Problem

    Generating diverse attack prompts for safety fine-tuning remains difficult because RL red-teaming methods often collapse onto limited high-reward modes.

  • Method

    Active Attacks periodically safety fine-tunes the victim LLM on collected prompts, reshaping the reward landscape and integrating adaptive exploration into existing RL objectives.

  • Results

    31.28% versus 0.07% cross-attack success against naive GFlowNets, a relative gain exceeding 400× with only a 6% increase in training time.

  • Takeaways & Limitations

    Active Attacks uncovers more diverse harmful behaviors through progressively harder exploration and can serve as a plug-and-play module for GFlowNets, PPO, and REINFORCE.

  • Takeaways & Limitations

    Prior RL policies can increase diversity mainly within a few easy modes, leaving harder regions underexplored and causing residual mode collapse.

Abstract

from arXiv · show

We address the challenge of generating diverse attack prompts for large language models (LLMs) that elicit harmful behaviors (e.g., insults, sexual content) and are used for safety fine-tuning. Rather than relying on manual prompt engineering, attacker LLMs can be trained with reinforcement learning (RL) to automatically generate such prompts using only a toxicity classifier as a reward. However, capturing a wide range of harmful behaviors is a significant challenge that requires explicit diversity objectives. Existing diversity-seeking RL methods often collapse to limited modes: once high-reward prompts are found, exploration of new regions is discouraged. Inspired by the active learning paradigm that encourages adaptive exploration, we introduce \textit{Active Attacks}, a novel RL-based red-teaming algorithm that adapts its attacks as the victim evolves. By periodically safety fine-tuning the victim LLM with collected attack prompts, rewards in exploited regions diminish, which forces the attacker to seek unexplored vulnerabilities. This process naturally induces an easy-to-hard exploration curriculum, where the attacker progresses beyond easy modes toward increasingly difficult ones. As a result, Active Attacks uncovers a wide range of local attack modes step by step, and their combination achieves wide coverage of the multi-mode distribution. Active Attacks, a simple plug-and-play module that seamlessly integrates into existing RL objectives, unexpectedly outperformed prior RL-based methods -- including GFlowNets, PPO, and REINFORCE -- by improving cross-attack success rates against GFlowNets, the previous state-of-the-art, from 0.07% to 31.28% (a relative gain greater than $400\ \times$) with only a 6% increase in computation. Our code is publicly available \href{https://github.com/dbsxodud-11/active_attacks}{here}.

1 INTRODUCTION

Automated RL red-teaming removes much of the cost of manual prompt discovery, but existing methods struggle to cover diverse harmful behaviors because they favor easy, high-reward modes. Active Attacks adapts the victim environment through periodic safety fine-tuning, driving attackers toward progressively harder and previously unexplored vulnerabilities.

  • 1 INTRODUCTION: Red-teaming traditionally relies on human creativity, manual prompt engineering, and iterative evaluation, making broad adversarial coverage expensive and limited.The paper uses “red-teaming” specifically for RL-based automated red-teaming.
  • 1 INTRODUCTION: RL-based red-teaming uses an attacker LLM to generate prompts while a victim LLM and toxicity classifier provide the environment and reward signal without extensive human-labeled attack data.This automates discovery of high-impact adversarial prompts using the classifier’s score for exploration.
  • 1 INTRODUCTION: Diverse attack prompts are necessary to cover broad harmful behaviors and make safety-tuned LLMs more robust to known and unforeseen strategies.Standard RL and prior novelty or GFlowNet approaches can still collapse after discovering high-reward modes.
  • 1 INTRODUCTION: Active Attacks periodically safety fine-tunes the victim LLM, flattening exploited reward regions and inducing an easy-to-hard exploration curriculum.The attacker and replay buffer are reinitialized after victim updates, while a global replay buffer preserves prompts across rounds.
  • 1 INTRODUCTION: 31.28% versus 0.07% cross-attack success against naive GFlowNets, a roughly 440× relative gain with only a 6% increase in training time.The collected prompts also generalize as a dataset for safety fine-tuning unseen LLMs.

2 RELATED WORKS

Prior work spans human-driven red-teaming, jailbreak optimization, and adaptive learning paradigms. These lines of research motivate automated attack generation and adaptive exploration, while distinguishing red-teaming from jailbreaks focused primarily on efficacy and transferability.

  • 2 RELATED WORKS: Automated red-teaming extends earlier human-in-the-loop build–break–fix workflows, manual prompt engineering, and curated testbeds that covered limited failure modes.Perez et al. (2022) formulated automated red-teaming as a generation–evaluation loop.
  • 2 RELATED WORKS: Jailbreaking emphasizes circumventing deployed guardrails, with hand-crafted, optimization-based, evolutionary, and genetic methods targeting effective or transferable prompts.Red-teaming is related but broader in its focus on uncovering model failures.
  • 2 RELATED WORKS: Active Attacks builds on active learning, curriculum learning, and adversarial training, which adapt samples or task difficulty to improve exploration, training stability, or robustness.These neighboring paradigms provide the conceptual basis for adapting training or sampling distributions.

3 METHOD

The method formulates red-teaming as RL in a victim–classifier environment, then adapts that environment to drive exploration across harmful prompt modes.

  • 3 METHOD: The attacker LLM generates prompts, the victim samples responses, and a toxicity classifier evaluates harmfulness to provide the RL reward.The attacker then updates its policy using this feedback in an iterative loop.
  • 3.1 PASSIVE ATTACKS: PRIOR WORKS: The constrained attacker objective maximizes expected toxicity-classifier reward while penalizing KL divergence from a reference distribution, preserving natural and comprehensible prompts.The off-policy formulation supports replay-buffer training for improved mode coverage, although prior methods still favor easy high-reward modes.
  • 3 METHOD: Red-teaming requires diverse harmful prompts because broad behavioral coverage improves the robustness of safety-tuned victims beyond known attack strategies.Pure reward maximization instead concentrates on a few high-reward modes, motivating diversity-seeking objectives.
  • 3.2 ACTIVE ATTACKS: OUR PROPOSAL: The RL loop stores prompt–reward experiences, adds prompts above threshold τ to the dataset, and updates the attacker with objectives such as PPO or trajectory balance.The implementation is presented as a small extension of the passive RL red-teaming algorithm.
  • 3.2 ACTIVE ATTACKS: OUR PROPOSAL: Active Attacks reinitializes the attacker after victim updates and aggregates prompts in a global replay buffer, reducing overfitting and sequential bias in the final safety-tuning dataset.The framework adds periodic safety fine-tuning and attacker reinitialization as a plug-in to the passive framework.
  • 3.2 ACTIVE ATTACKS: OUR PROPOSAL: Active Attacks periodically safety-fine-tunes the victim on collected attack prompts, flattening rewards in exploited regions and forcing the attacker toward unexplored vulnerabilities.The resulting exploration follows an easy-to-hard curriculum across multiple reward-landscape modes.

4 EXPERIMENTS

Experiments evaluate Active Attacks against multiple red-teaming baselines, attacker models, toxicity classifiers, and adaptation settings. Across these tests, it improves diversity, attack robustness, defense rates, and exploration while preserving general capabilities at modest cost.

  • 4.1 RESULTS: Active Attacks achieves almost 100% defense rate across different victim LLMs, whereas ICL and SFT mostly reach about 50%.REINFORCE and PPO + Novelty also struggle against GFlowNet and Active Attacks because they focus on limited modes and exhibit low diversity.
  • 4.1 RESULTS: Approximately 30% of Active Attacks prompts bypass the defense of a victim safety-fine-tuned with GFlowNet prompts, while Active Attacks blocks GFlowNet attacks.The cross-method comparison indicates that Active Attacks produces a more robust attacker and can improve victim safety fine-tuning.
  • 4.1 RESULTS: Active Attacks balances toxicity and diversity across categories, unlike PPO + Novelty and GFlowNet, which show cosine diversity but concentrate on limited categories.The method evaluates diversity with categorical distance derived from Meta-Llama-Guard-2 categories because cosine distance alone can overstate diversity.
  • 4.1 RESULTS: Active Attacks discovers effective and diverse attacks when using different attacker LLMs or the HarmAug-Guard toxicity classifier.Experiments replace the attacker with Llama-3.2-3B and replace Llama-Guard with HarmAug-Guard, supporting robustness across these components.
  • 4.3 FURTHER ANALYSIS: Active Attacks generalizes to REINFORCE and PPO + Novelty, producing more robust attackers than their naive counterparts.The framework is presented as a plug-and-play approach for different RL-based red-teaming methods.
  • 4.3 FURTHER ANALYSIS: Prompt diversity increases across adaptation rounds and converges after five rounds, while the periodic safety fine-tuning stage adds little computational cost.The authors report that five intermediate stages are sufficient for covering the multi-mode distribution, and the stage uses less time and GPU memory than GFlowNet fine-tuning.

5 CONCLUSION

Active Attacks automates diverse red-teaming by adapting the victim through periodic safety fine-tuning, driving attackers toward under-explored regions. It integrates with existing RL objectives and achieves over a 400× relative cross-attack success gain.

  • Active Attacks periodically safety fine-tunes the victim, reshaping rewards and driving the attacker toward under-explored regions.
  • Over 400× relative gain in cross-attack success rates compared to state-of-the-art baselines demonstrates Active Attacks’ empirical improvement.
  • The framework is a plug-and-play module compatible with GFlowNets, PPO, and REINFORCE.

A.1 FULL TABLE FOR MAIN EXPERIMENTAL RESULTS

Active Attacks consistently generate toxic and categorically diverse prompts across victim LLMs while improving safety fine-tuning robustness against red-teaming attacks.

  • Active Attacks successfully defend against several red-teaming approaches, whereas most prior RL-based methods struggle against Active Attacks-generated attacks.The full results report defense rates for different safety fine-tuning methods and approaches.
  • Active Attacks consistently generate toxic, diverse prompts by categorical distance across Qwen2.5-1.5B-Instruct, Llama-3.2-3B-Instruct, gemma-3-4b-it, and Mistral-7B-Instruct-v0.3.The plots compare toxicity-diversity trade-offs across multiple victim models.

A.3 FULL RESULTS OF CROSS ATTACK SUCCESS RATE

Active Attacks improve cross-attack robustness: they block attacks from other approaches, while more than 30% of Active Attacks bypass defenses trained on other prompt datasets.

  • More than 30% of Active Attacks bypass safety-fine-tuned defenses trained with prompt datasets from other red-teaming approaches.The cross-attack figures compare GFlowNets and GFlowNets + Active Attacks with other baselines.
  • Active Attacks successfully block attacks from other red-teaming approaches in cross-attack evaluations.Experiments use three seeds and report means with standard deviations.
  • Active Attacks boost the defense rate of safety-fine-tuned victim LLMs against aggregated baseline and Active Attacks prompts, with results averaged across independent runs.The reported defense-rate evaluation uses prompts generated from a source victim LLM.

B QUALITATIVE ANALYSIS

Qualitative analyses show that prior approaches collapse into a few easy-to-find modes, whereas Active Attacks produce diverse prompts across hazard categories and remain effective under model and classifier substitutions.

  • Mode collapse: Prior approaches collapse into a few easy-to-find modes after discovering high-reward prompts, with REINFORCE suffering substantially and PPO + Novelty and GFlowNet mitigating collapse only slightly.Examples are organized by hazard categories supplied by Meta-Llama-Guard-2-8B.
  • Cross-model diversity: Active Attacks generate diverse prompts across different hazard categories and victim LLMs.Examples cover Qwen2.5-1.5B-Instruct, Llama-3.2-3B-Instruct, gemma-3-4b-it, and Mistral-7B-Instruct-v0.3.
  • Attacker substitution: Replacing the attacker with Llama-3.2-3B increases toxicity while not significantly diminishing diversity.Active Attacks continue generating diverse prompts with the larger attacker model.
  • Classifier substitution: Replacing Meta-Llama-Guard-2-8B with HarmAug-Guard still yields effective and diverse prompts compared with naive GFlowNets.Prompt and response examples are reported for the alternative classifier.
  • Diversity measure: Categorical distance measures diversity across predefined hazard categories, addressing cases where high cosine distance still corresponds to category collapse.The categories are provided by Meta-Llama-Guard-2-8B.

D TRAINING AND EVALUATION DETAILS

The paper uses specified attacker, victim, and toxicity-classifier models under GPU resource constraints, with most experiments conducted on a single NVIDIA A100-80G GPU.

  • The default setup uses Qwen2.5-1.5B as attacker, Qwen2.5-1.5B-Instruct as victim, and Meta-Llama-Guard-2-8B as the toxicity classifier.Larger models do not fit on the NVIDIA L40S GPU used as the paper’s base computing resource.
  • Most other experiments use a single NVIDIA A100-80G GPU.

D.1 TRAINING DETAILS

Training follows Lee et al. (2025a), combining supervised warm-up, GFlowNet fine-tuning, and periodic safety fine-tuning with specified schedules.

  • The warm-up stage fine-tunes a pre-trained Qwen2.5-1.5B attacker for 200 steps with learning rate 3e−5 and batch size 1024, without LoRA.
  • GFlowNet fine-tuning runs for T = 5000 steps before periodic safety fine-tuning at interval R = 1000.

D.2 EVALUATION DETAILS

Evaluation follows Lee et al. (2025a) and estimates each prompt’s toxicity from five victim-model responses, with sampling hyperparameters summarized in Table 18.

  • For each prompt x, the victim LLM generates k = 5 responses to estimate the prompt’s toxicity score using their empirical mean.
  • Table 18 summarizes the attacker- and victim-LLM sampling hyperparameters used during evaluation.
Loading 2509.21947v2…