Source-linked AI summary
Black-Box Red-Teaming of Multi-Agent Systems via Reinforcement Learning
Yuqi Jia, Hangsheng Zhang, Chenxi Liu, Junjie Liang, Xun Chen
TL;DR
Black-box attacks developed for single agents are less effective in MASs, where coordinators route and rewrite queries. ReMAS uses reinforcement learning to rewrite attack prompts and generate routing templates for sub-agent prompt extraction, and it outperforms black-box baselines with transferability across backends and structures.
Problem
Black-box attacks mainly target single agents, while their effectiveness in MASs—where routing and coordinator rewriting complicate attacks—remains underexplored.
Source: §1 · p. 1Method
ReMAS trains attacker LLMs to rewrite base prompts and generate templates that route them to target sub-agents for system-prompt extraction.
Source: §1 · p. 1Results
ReMAS consistently outperforms strong black-box baselines, with routing success above 0.98 and ASRs close to 0.80 across sub-agent types, and transfers to unseen backends and alternative MAS structures.
Source: §1 · p. 2, §6 · p. 10Takeaways & Limitations
The results highlight the necessity of incorporating adversarial robustness into MAS design and deployment.
Source: §1 · p. 2At test time, attackers know only the target sub-agent’s name or functionality and observe final responses; shadow-training signals are unavailable.
Source: §3.2 · p. 3
Abstract
from arXiv · showhide
Large language model (LLM) agents are increasingly deployed in multi-agent systems (MAS) to accomplish complex tasks. Prior black-box red-teaming attacks mainly focus on single agents, but we find that these methods are far less effective in MAS, where multiple sub-agents may not directly interact with the user. Therefore, we introduce ReMAS, the first reinforcement learning-based red-teaming framework tailored to MASs, which fine-tunes attacker LLMs to generate effective adversarial prompts for system prompt extraction. The framework follows a two-step process: first, a rewrite stage refines base attack prompts to increase extraction success, and second, a template generation stage constructs attack templates that improve the likelihood of invoking specific sub-agents and thus revealing their system prompts. Extensive experiments show that our method substantially improves attack success rates compared to existing black-box attacks, with transferability across different backend LLMs and MAS structures. These results underscore the vulnerabilities of multi-agent systems and the importance of developing stronger defenses against adaptive black-box adversaries.
1 Introduction
The introduction identifies a gap in black-box attacks against multi-agent systems: coordinator routing and rewriting make attacks designed for single agents less effective. It presents ReMAS, an RL-based framework for extracting sub-agent system prompts, and reports strong results across agents and system structures.
1 Introduction
- Single-agent attacks often fail in vertical MASs because an adversary must reach the target sub-agent and preserve its intent through coordinator rewriting.The coordinator selects a sub-agent and may rewrite the forwarded query.Source: §1 · p. 1
- ReMAS trains attacker LLMs in two stages: rewrite base prompts, then generate templates that help route them to target sub-agents.The policies are optimized with rewards under DAPO and run in a single-pass test-time pipeline.Source: §1 · pp. 1–2
- ReMAS outperforms strong black-box baselines, with routing success above 0.98 and ASRs close to 0.80 across sub-agent types.Experiments also report transferability to unseen backend LLMs and alternative MAS structures.Source: §1 · p. 2
- The framework uses a strategy pool with randomized selection to improve attacker prompt diversity and effectiveness.The experiments span seven sub-agent types, multiple backends, and different MAS structures.Source: §1 · p. 2
2 Related Work
Prior work addresses jailbreaks and prompt injection largely in single-agent settings, while multi-agent systems introduce distinct risks through routing and inter-agent communication. The paper situates system-prompt extraction as one such risk.
2 Related Work
- MAS designs include vertical coordinator-led delegation [8] and horizontal agent collaboration [4] [9] [31] [10].Both rely on hidden system prompts to define agent roles and behaviors.Source: §2 · p. 2
- Inter-agent communication and routing expose risks such as extracting private sub-agent prompts, which differ from single-agent attacks.Sub-agents often do not directly interact with users, motivating the paper’s focus.Source: §2 · pp. 2–3
- Prior work on LLM security mainly studies single-agent systems; multi-agent systems introduce new challenges because sub-agents often do not directly interact with users, motivating further study.Recent jailbreak methods use automated search [20], optimization [18], or reinforcement learning [13] to improve attack effectiveness.Source: §2 · pp. 2–3
3 Problem Setup
The paper models a vertical MAS as a coordinator routing queries to specialized sub-agents, whose responses depend on forwarded inputs and hidden prompts. The attacker seeks to reach a chosen sub-agent and extract its prompt under black-box test-time access.
3.1 Multi-Agent System
- In the modeled vertical MAS, a coordinator selects among N specialized sub-agents, may transform the user query, and forwards input x_i to the selected agent.Each agent has a functionality label and hidden system prompt s_i, and produces a response conditioned on x_i and s_i.Source: §3.1 · p. 3
3.2 Threat Model
- The attacker trains policies to maximize both invocation of a chosen target sub-agent and disclosure of its hidden system prompt.The goal concerns a given attack query and the response from the invoked sub-agent.Source: §3.2 · p. 3
- At test time, the attacker knows only the target’s name or functionality and can submit queries and observe final responses, without access to prompts or model internals.Training may use limited shadow-data signals, but evaluations use a disjoint test set under a strict black-box setting.Source: §3.2 · p. 3
4 Our ReMAS
ReMAS uses two reinforcement-learning-trained attacker policies to rewrite base prompts and wrap them in templates that target system-prompt extraction in multi-agent systems. Its test-time pipeline applies both policies in a single pass.
4.1 Overview
- ReMAS trains one policy to rewrite a base attack prompt and another to generate a template, optimizing extraction and routing through two stages.The rewrite stage uses a strategy pool and randomized selection to support effective, diverse rewriting.Source: §4.1 · p. 3, §4.2 · pp. 4–5
4.2 Rewrite Attack Prompt
- Step I rewrites existing attack prompts using a reward that balances normalized edit similarity with a relative length penalty.The strategy pool stores promising rewrites and retrieves semantically similar, scored examples; randomized selection is used to reduce over-reliance on dominant patterns and improve prompt diversity and attack success.Source: §4.2 · pp. 4–5
4.3 Generate Attack Template
- Step II trains a policy to wrap a rewritten prompt so the coordinator invokes the target sub-agent and forwards input close to that prompt.A template uses a placeholder for the rewritten prompt; failure to invoke the target sub-agent receives zero reward.Source: §4.3 · p. 6
4.4 Single-pass Test Time Attack Execution
- ReMAS deploys its rewrite and template policies once each at test time, combining the rewritten prompt and template into a single attack input.The strategy pool is fixed, and the top-scoring strategy guides rewriting.Source: §4.4 · p. 6
5 Evaluation
ReMAS is evaluated against black-box attacks across seven sub-agent types, backend LLMs, and MAS structures. It outperforms the baselines in prompt extraction, while ablations and transfer tests examine its strategy-pool and routing components.
5.1 Experiment Setup
- The evaluation uses 10 coordinator prompts, seven sub-agent categories, and 600 base attacks, with system prompts split 8:2 for training and testing.The setup reports 11,200 Step I and 4,800 Step II training examples, plus 100 test cases per sub-agent category; the primary backend is Qwen3-8B.Source: §5.1 · p. 7
Baselines:
- PAIR, TAP, and AutoDAN-Turbo are the black-box baselines, given the same base prompts and target-sub-agent information as ReMAS.Target information consists of the sub-agent’s name or general function.Source: Baselines · p. 7
5.2 Our method Outperform Baselines
- Across seven sub-agent types, ReMAS exceeds 0.98 CAR and reaches ASR at or near 0.80, outperforming all baselines on both metrics.PAIR and TAP generally have CAR no higher than 0.3 and near-zero ASR; AutoDAN-Turbo remains below 0.3 ASR. The table covers CAR and ASR across attacks and sub-agent types.Source: §5.2 · p. 8, Table 1 · p. 8
5.3 Ablation Studies
- Adding a strategy pool raises ASR over the no-pool variant, and randomized selection generally improves ASR further while slightly increasing PCD.The authors attribute the diversity loss with a pool alone to repeatedly selecting high-scoring strategies; broader sampling mitigates this mode collapse.Source: Figure 2 · p. 8, §5.3 · p. 9
- For backend transfer from Qwen3-8B, ReMAS retains CAR close to one and averages around or above 0.50 ASR, while outperforming the baselines.Transfer results are averaged across seven sub-agent types and show lower overall ASR than in-domain evaluation.Source: §5.3 · p. 9, Table 2 · p. 9
- Across alternative MAS structures, ReMAS exceeds 0.8 ASR on Horizontal Select and Vertical Solver&Reviewer; without a coordinator, evaluation uses rewritten prompts and ASR only.Figure 3 averages results across seven sub-agent types; per-type results are reported in Table 4.Source: §5.3 · pp. 9–10, Figure 3 · p. 10
6 Conclusion
The conclusion presents ReMAS as an RL-based red-teaming framework for multi-agent systems. Its two stages support system-prompt extraction, with experiments showing baseline improvements and transfer across backends and structures.
6 Conclusion
- ReMAS combines strategy-pool rewriting with template-based routing and consistently outperforms baselines while transferring across MAS backends and structures.The authors identify broader adversarial goals and robustness under advanced defenses as future work.Source: §6 · p. 10
A.1.1 System Prompts
The appendix describes the system prompts used for coordinator and sub-agents, including their categories and variation.
A.1.1 System Prompts
- GPT-4o generated 10 coordinator prompts and 10 prompts for each of seven sub-agent categories, varying prompt length and style.The categories are Coding, Search, Translation, Entertainment, Teaching, Writing, and Multimedia.Source: §A.1.1 · p. 14
A.1.2 Sub-agent Safety Constraint
The appendix adds an explicit refusal constraint to each sub-agent system prompt to simulate safety alignment during training and evaluation.
A.1.2 Sub-agent Safety Constraint
- Every sub-agent prompt instructs the agent to refuse requests to expose its system prompt, plugins, workflow, rules, or related confidential content.The authors state that this constraint simulates safety alignment during training and evaluation.Source: §A.1.2 · p. 14
A.1.3 Step I Training Data (Rewrite Attack Prompt)
The appendix describes ReMAS training and evaluation data, prompt preprocessing, and baseline methods. Its Step I rewriting-policy algorithm uses rewards and a strategy pool to generate and refine attack prompts.
A.1.3 Step I Training Data (Rewrite Attack Prompt)
- 11,200 Step I training examples come from 8 training sub-agent prompts per category, 7 categories, and 200 sampled base prompts per prompt.The 8 prompts per category are an 8:2 split of 10 prompts.Source: §A.1.3 · p. 14
A.1.4 Step II Training Data (Generate Attack Template)
- 4,800 Step II training examples pair each of 8 training coordinator prompts with all 600 base attack prompts.Source: §A.1.4 · p. 14
A.1.5 Testing Data
- Testing uses 100 pairs per sub-agent category, with two test coordinators, 50 sampled base prompts each, and two distractor categories per target.The target prompt is drawn from the two prompts reserved for testing in that category.Source: §A.1.5 · p. 14
A.1.6 Preprocessing and Filtering
- Enterprise-derived prompts are filtered to remove personally identifiable information, proprietary terms, and sensitive business content.Source: §A.1.6 · p. 14
A.2 Baselines
- PAIR iteratively refines prompts, TAP generates and prunes candidates, and AutoDAN-Turbo explores and refines strategies using a growing strategy library.PAIR and TAP are run on test samples; AutoDAN-Turbo is trained on the training set before test evaluation.Source: §A.2 · p. 15
- The Step I algorithm initializes baseline rewards and strategy pools, then generates candidate rewrites and evaluates their rewards during training.Its inputs include the prompt set, training steps, pool sizes, selection probability, reward function, and sub-agent information; its output is an optimized rewriting policy.Source: §A.2 · p. 16
- Promising rewrites update the strategy pool; later prompts retrieve strategies by embedding similarity, with randomized replacement or retrieval-free generation also used.The algorithm also collects medium- or low-score strategies depending on its selection branch.Source: §A.2 · p. 16