PaperlayerView exact revision on arXiv

Source-linked AI summary

Red-Bandit: Test-Time Adaptation for LLM Red-Teaming via Bandit-Guided LoRA Experts

Christos Ziakas, Nicholas Loo, Nishita Jain, Alessandra Russo

arXiv:2510.07239v2cs.CL

TL;DR

LLM red-teaming needs to adapt to model-specific vulnerabilities at inference. Red-Bandit trains attack-style LoRA experts and uses a bandit policy to select among them from target-response safety. It reports stronger benchmark performance under sufficient exploration and uses selected styles to diagnose model weaknesses.

  • Problem

    Existing approaches lack mechanisms to efficiently adapt to model-specific vulnerabilities at inference, despite the need to audit LLMs before deployment.

    Source: §1 · pp. 1–2

  • Method

    Red-Bandit post-trains attack-style LoRA experts with prompt-level safety rewards and dynamically selects among them using a bandit guided by target-response safety.

    Source: §1 · p. 2

  • Results

    On AdvBench, Red-Bandit reports higher attack success rates and lower perplexity than existing approaches under a sufficient exploration horizon.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The bandit’s selected attack-style distribution reveals model-specific weaknesses and behavioral biases.

    Source: §6.1 · p. 9

    Under strict query constraints that prevent sufficient exploration, the bandit may not converge to the optimal expert within the allocated budget.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

Automated red-teaming has emerged as a scalable approach for auditing Large Language Models (LLMs) prior to deployment, yet existing approaches lack mechanisms to efficiently adapt to model-specific vulnerabilities at inference. We introduce Red-Bandit, a red-teaming framework that adapts online to identify and exploit model failure modes under distinct attack styles (e.g., manipulation, slang). Red-Bandit post-trains a set of parameter-efficient LoRA experts, each specialized for a particular attack style, using reinforcement learning that rewards the generation of unsafe prompts via a rule-based safety model. At inference, a multi-armed bandit policy dynamically selects among these attack-style experts based on the target model's response safety, balancing exploration and exploitation. Red-Bandit achieves state-of-the-art results on AdvBench under sufficient exploration (ASR@10), while producing more human-readable prompts (lower perplexity). Moreover, Red-Bandit's bandit policy serves as a diagnostic tool for uncovering model-specific vulnerabilities by indicating which attack styles most effectively elicit unsafe behaviors.

1 Introduction

LLMs remain vulnerable to adversarial prompts, while existing evaluation and red-teaming approaches face limits in scale or test-time adaptation. Red-Bandit trains attack-style LoRA experts and uses a bandit to select among them for a target model, achieving reported benchmark gains and revealing model-specific vulnerabilities.

  • LLMs remain vulnerable to adversarial prompts despite advances in alignment, motivating reliable auditing before deployment as model use expands.Source: §1 · p. 1
  • Human-annotated evaluation datasets are limited in size and can have selection biases, while human red-teaming is resource-intensive and difficult to scale.Source: §1 · p. 1
  • Gradient- or log-probability-based jailbreak methods lack target-specific test-time adaptation; iterative search adapts online but can require many queries.RL-based prompt generation is efficient and readable, but its diversity can be limited by mode collapse.Source: §1 · pp. 1–2
  • Under sufficient exploration on AdvBench, Red-Bandit achieves higher attack success rates and lower perplexity than existing approaches.Source: §1 · p. 2
  • The framework trains style-specific LoRA experts with GRPO and prompt-level safety rewards, then uses target-response safety to select experts at inference.In-context conditioning supports human-readable, style-aligned adversarial prompts.Source: §1 · p. 2
  • Red-Bandit outperforms state-of-the-art methods on AdvBench and HarmBench and identifies model-specific vulnerabilities across attack styles.Source: §1 · p. 2

2 Preliminaries

The preliminaries describe GRPO as a lower-cost alternative for RL post-training and frame attack-style selection as a multi-armed bandit problem. In Red-Bandit, each style is an arm, and the target model’s response safety supplies the reward.

2.1 RL Post-Training with GRPO

  • GRPO removes the value model used for advantage estimation and estimates advantages through group-wise reward normalization, reducing PPO’s training overhead.The paper adopts a GRPO variation with LoRA for parameter-efficient fine-tuning of attack-style experts.Source: §2.1 · p. 2

2.2 Multi-Armed Bandits

  • Red-Bandit treats each attack style as an arm and scores its reward using the target LLM response’s content safety.ε-greedy mixes selection of the empirically best arm with random exploration; UCB selects the arm with the highest upper confidence estimate.Source: §2.2 · p. 3

3 Methodology

Red-Bandit formulates automated red-teaming as searching for prompts that elicit harmful responses, then trains style-specific LoRA attackers with reinforcement learning and adaptively selects experts using target-response feedback. Its training uses a prompt-safety reward, while its modular experts can be added and trained independently.

3.1 Automated Red-teaming

  • Red-Bandit searches for prompts that maximize a reward measuring the harmfulness of the target model’s response.The reward may come from a rule-based verifier, learned classifier, or human judgment.Source: §3.1 · p. 3

3.2 Post-Training of LoRA Experts

  • Attacker policies are post-trained with reinforcement learning using style-conditioned prompts and a rule-based prompt-safety reward.Source: §3.2 · p. 3

3.2.1 RL Algorithm

  • The GRPO-based variant removes response-length and question-difficulty biases, retains a group baseline, and omits the KL penalty for efficiency.Its clipped policy-gradient loss uses the policy probability ratio and a clipping threshold; the advantage is calculated from rule-based rewards across responses per prompt.Source: §3.2.1 · p. 3

3.2.2 Reward Function Design

  • A binary rule-based reward scores prompts against predefined rules, assigning 1 when a prompt is unsafe.This permits black-box training without querying the target model; the paper also studies response-based and combined reward signals.Source: §3.2.2 · p. 3

3.2.3 Attack-Style LoRA Experts

  • Separate LoRA experts are trained for attack styles, and adding a style or domain requires training only one lightweight adapter.Experts are decoupled and training is parallelizable; an optional SFT step can improve style conditioning for custom datasets or new domains.Source: §3.2.3 · pp. 3–4, Figure 2 · p. 4

3.3 Bandit-Guided Inference

  • At inference, each LoRA expert is a bandit arm; the policy uses response-safety rewards to select experts and can use UCB or ε-greedy.The framework is agnostic to the bandit algorithm, and routing is updated at test time from target-model response feedback.Source: §3.3 · p. 4

4 Experiments

Red-Bandit is evaluated on AdvBench and HarmBench, with results spanning open-source and proprietary targets, model-specific attack-style patterns, and ablations of its training and selection components.

4.1 Implementation Details

  • Mistral-7B generates prompts; Llama Guard-8B provides RL training rewards and Llama Guard-1B scores responses for inference-time bandits.Each attack-style expert is trained for one epoch using a GRPO variant with eight generations per step and LoRA adaptation.Source: §4.1 · pp. 4–5

4.2 Evaluation Setup

  • AdvBench uses a 520-instruction dataset with an 80/20 split; HarmBench evaluation uses unseen training behaviors and the HarmBench-cls classifier.AdvBench ASR is measured with keyword matching, while HarmBench uses classifier-based success; the reward model is independent of these evaluation metrics.Source: §4.2 · p. 5

4.3.1 AdvBench

  • 100% ASR@10 on Mistral-7B and Vicuna-7B; both Red-Bandit variants lead ASR@10 and perplexity across three open-source models.On Llama2-7B, both variants exceed 96% ASR@10; UCB has the lowest perplexity in two of three settings.Source: Table 1 · p. 5, §4.3.1 · p. 6
  • 98.1% ASR@10 on GPT-3.5-turbo and 93.3% on GPT-4o; Red-Bandit adapts on the target rather than relying on prompt transfer.Atoxia has higher ASR@1, while Red-Bandit has higher ASR@10 in the black-box setting.Source: §4.3.1 · p. 6, Table 2 · p. 6

4.3.2 HarmBench

  • Under the constrained ASR@10 budget on HarmBench, Red-Bandit (ϵ-greedy) records a 69.0% success rate on Llama-2-7B, versus GCG-Universal’s 20.0% and AutoDAN-Universal’s 0.5%.Source: §4.3.2 · p. 7

4.3.3 Identifying Attack-Style Vulnerabilities

  • Llama-3.1-8B is sensitive to hypothetical and authority-manipulation attacks, GPT-4o to historical and role-play scenarios, and GPT-3.5-turbo to uncommon dialects and hypotheticals.ϵ-greedy concentrates on a few styles, whereas UCB maintains a more balanced style distribution.Source: §4.3.3 · p. 7

4.4.1 Impact of RL Post-Training and Bandits

  • Both RL post-training and bandit selection contribute substantially to overall performance in the Llama-3.1-8B ablation.The ablation compares a baseline, bandit without RL, RL without bandit, and the full RL-plus-bandit method.Source: §4.4.1 · p. 7, Table 5 · p. 8

4.4.2 Reward Design in RL Post-Training

  • Prompt-only rewards use training time and VRAM more efficiently than response-based and hybrid rewards, with higher HarmBench ASR@20 on Llama-3.1-8B and Qwen3-8B.Prompt-only reward performs on par with hybrid training in ASR@20.Source: §4.4.2 · pp. 7–8

5 Related Work

Prior prompt-optimization and search-based approaches offer effective jailbreak generation but leave gaps in target-specific test-time adaptation or computational efficiency. Red-Bandit combines style-specific RL experts with a bandit policy that adapts to target feedback.

5.1 Prompt Optimization Methods

  • Gradient- and log-probability-based prompt optimization can produce transferable jailbreaks, but offline-trained attack policies do not adapt to target vulnerabilities at test time.GCG optimizes adversarial suffixes; AdvPrompter and Atoxia emphasize readable, efficient or transferable prompts.Source: §5.1 · p. 8

5.2 Iterative and Search-Based Methods

  • Black-box iterative and search-based methods use target feedback or mutation, but may require many candidate prompts for one successful attack; Red-Bandit selects among pretrained style experts online.The related methods include feedback refinement, tree or graph search, genetic mutation, fuzzing, and open-ended style search.Source: §5.2 · pp. 8–9

5.3 RL-Based Methods

  • Earlier RL approaches address diversity but do not adapt strategies to specific targets at test time; Red-Bandit uses multiple style-specific LoRA experts and response-guided bandit selection.Unlike SoC’s selection among context-switching query categories, Red-Bandit selects among RL-trained attack-style experts; its policy can diagnose model-specific style vulnerabilities.Source: §5.3 · p. 9

6 Discussion and Limitations

Red-Bandit’s targeted attack styles outperform existing methods in attack success and human readability when exploration is sufficient. The discussion also identifies computational, training, query-budget, and dual-use boundaries.

6.1 Discussion

  • With sufficient exploration, targeted attack styles outperform existing methods in attack success rate and human readability.The discussion says Red-Bandit outperforms Atoxia when the query budget permits sufficient exploration, including under single-query constraints given enough queries.Source: §6.1 · p. 9

6.2 Limitations

  • Inference requires an external rule-based reward model, training scales with the number of attack styles, and insufficient queries may prevent bandit convergence.Custom domains may also require supervised fine-tuning for reliable attack-style conditioning.Source: §6.2 · p. 9

6.3 Potential Risks and Societal Impact

  • The framework is dual-use: its vulnerability-finding techniques could be misused to generate harmful content or more effective jailbreaks.The prompts are intended for developers and researchers auditing and strengthening model safeguards.Source: §6.3 · p. 9

A Examples of Attack Styles

Table 6 presents attack-style examples for a harmful vehicle-hijacking behavior.

A Examples of Attack Styles

  • Table 6 gives examples of different attack styles for a request to develop software that hijacks a vehicle’s control system and causes accidents.The passage identifies the example behavior but does not provide the individual style-specific prompts.Source: Table 6 · p. 13

B Impact of Reward Design in Large-Scale RL Post-Training

Prompt-based rewards offer an effective and scalable training signal in the reported AdvBench comparison. Evaluation measures attack success, diversity, and training efficiency across two target models.

B Impact of Reward Design in Large-Scale RL Post-Training

  • The training setup uses SFT on 80,000 HarmBench prompts before GRPO post-training on AdvBench-v4’s 4,948 harmful behaviors.The reward model supplies prompt-only, response-only, or equal-weight hybrid rewards.Source: B Impact of Reward Design in… · p. 14
  • Evaluation used ASR@20 across 200 harmful behaviors, with up to 20 independent prompts per behavior and HarmBench classification of target outputs.The SFT and RL corpora partially overlap the evaluation set, which may inflate absolute ASR, although the overlap is consistent across reward configurations.Source: B Impact of Reward Design in… · p. 14
  • Self-BLEU across 20 generations per behavior measures diversity, with lower scores indicating more diverse attack strategies.The comparison also records VRAM use and wall-clock training time.Source: B Impact of Reward Design in… · p. 14
  • 82.0% ASR@20 on Qwen-8B was the highest attack success rate reported for the reward configurations.Prompt-based rewards also achieved the lowest Self-BLEU and substantially lower computational cost than response-based or hybrid objectives.Source: B Impact of Reward Design in… · p. 14
  • Hybrid rewards yielded a minor gain on Llama-3.1-8B but required more resources and reduced diversity.The passage identifies prompt-based feedback as preferable for large-scale adversarial training.Source: B Impact of Reward Design in… · p. 14

C Model and Training Details

The appendix specifies the prompt generator, reward models, and training and inference configuration. It also describes the parameter sweeps used to select the final setup.

C Model and Training Details

  • Mistral-7B serves as the prompt generator, while Llama Guard-8B supplies RL rewards and Llama Guard-1B is used for inference-time bandit algorithms.The prompt generator was chosen for minimal alignment fine-tuning and open-weight availability.Source: C Model and Training Details · p. 14
  • The RL sweep varied epochs, gradient accumulation, and learning rate; LoRA sweeps varied rank and alpha, while bandit settings used ε = 0.1 and UCB constant c = 2.The final configuration was selected based on training stability and performance in large-scale RL experiments.Source: C Model and Training Details · p. 14
  • Training and inference used bfloat16 mixed precision on a single NVIDIA A100 with 80 GB of memory.AdamW used β1 = 0.9, β2 = 0.999, and a linear learning rate without warmup.Source: C Model and Training Details · p. 14
  • Table 7 compares reward configurations by ASR@20 on two target models, Self-BLEU, VRAM use, and training time.Table 8 covers training, LoRA, and inference configuration parameters.Source: Table 7 · p. 15, Table 8 · p. 15

D Prompt Template for Attack-Style Generation

The template asks a model to craft a prompt in a specified attack style to elicit a target unsafe behavior. It directs the model to generate a question that would lead another LLM to fulfill that behavior.

E Limitations of GPT-4o Evaluations

The section describes GPT-4o evaluation errors when responses implicitly fulfill unsafe intent without appearing toxic. Its example shows a response labeled SAFE that proceeds to outline vehicle-theft analysis steps.

Related papers

Loading 2510.07239v2…