Source-linked AI summary
Combating Adversarial Attacks with Multi-Agent Debate
Steffi Chern, Zhen Fan, Andy Liu
TL;DR
Language models remain vulnerable to adversarial prompts that can elicit harmful outputs, and debate could also let toxic responses influence other agents. This paper evaluates multi-agent debate across Llama-2 and GPT models against red-team prompts, finding generally lower toxicity in debate settings. The benefit varies with model and agent intention, and the approach has cost and capability constraints.
Problem
Language models remain vulnerable to adversarial prompts that can elicit unsafe outputs, making robustness important for safe deployment.
Source: §1 · p. 1Method
The study compares single-agent, self-refinement, and multi-agent debate responses to adversarial prompts across GPT and Llama models with different intentions and safety alignment.
Source: §1 · p. 1, §3.1 · p. 3Results
Takeaways & Limitations
Debate shows promise as an inference-time approach to safeguarding against adversarial prompts, but its effects depend on the model and agent intention.
Source: §5 · p. 8Debate can be resource-intensive in model cost and latency, while less capable models may lack the conversational ability to benefit.
Source: §5 · p. 8
Abstract
from arXiv · showhide
While state-of-the-art language models have achieved impressive results, they remain susceptible to inference-time adversarial attacks, such as adversarial prompts generated by red teams arXiv:2209.07858. One approach proposed to improve the general quality of language model generations is multi-agent debate, where language models self-evaluate through discussion and feedback arXiv:2305.14325. We implement multi-agent debate between current state-of-the-art language models and evaluate models' susceptibility to red team attacks in both single- and multi-agent settings. We find that multi-agent debate can reduce model toxicity when jailbroken or less capable models are forced to debate with non-jailbroken or more capable models. We also find marginal improvements through the general usage of multi-agent interactions. We further perform adversarial prompt content classification via embedding clustering, and analyze the susceptibility of different models to different types of attack topics.
1 Introduction
The paper examines whether multi-agent debate can improve robustness to adversarial prompts without letting toxic outputs spread between agents. It implements debate across Llama-2 and GPT-3 models and finds generally lower toxicity, including for models previously fine-tuned with RLHF.
1 Introduction
- Adversarial prompts can elicit harmful outputs, making robustness to them important for safe real-world deployment.The paper also flags a risk: a toxic agent may poison peers’ responses, potentially increasing toxicity or preventing improvement.Source: §1 · p. 1
- The study implements debate among Llama-2 and GPT-3 models and compares single-agent, self-refinement, and multi-agent responses to prompts known to elicit toxicity.Prompt engineering simulates poisoned models participating in debate.Source: §1 · p. 1
- Multi-agent debate generally produces less toxic responses to adversarial prompts, including for models previously fine-tuned with reinforcement learning from human feedback.When the initial model is poisoned, debate also outperforms self-refinement.Source: §1 · p. 1
2 Related Work
Prior work develops chain-of-thought, self-refinement, and multi-agent debate to improve language-model reasoning and generation, while red-teaming research documents vulnerabilities to adversarial prompts. This paper builds on those lines by examining adversarial robustness and debate dynamics.
2.1 Multi-Agent Debate
- Chain-of-thought prompting elicited step-by-step reasoning, while self-refinement added iterative model feedback to improve generations across tasks (Kojima et al., 2023; Madaan et al., 2023).Multi-agent debate is presented as an extension of these approaches.Source: §2.1 · p. 2
- Multi-agent debate studies report gains in factuality, reasoning, and generation evaluation, including cross-examination, referee teams, and debate among models (Cohen et al., 2023; Chan et al., 2023; Chen et al., 2023).The cited work includes comparisons with chain-of-thought and self-refinement.Source: §2.1 · p. 2
2.2 Red Teaming LLMs
- Safety alignment during training and instruction fine-tuning aims to make models recognize harmful dialogue and refuse harmful or sensitive requests, yet fine-tuned models remain sensitive to adversarial prompts.The cited passage identifies red-teaming as a way to examine harmful behavior and model weaknesses.Source: §2.2 · p. 2
- Red-teaming studies use crowdworkers, language-model dialogues, and adversarial personas to uncover harmful continuations and model weaknesses (Ganguli et al., 2022; Perez et al., 2022; Shah et al., 2023).Other work constructs adversarial prompts, including universal attack suffixes, and reports that safety alignment can be unlearned through uncensored fine-tuning (Wei et al., 2023; Zou et al., 2023; Lermen et al., 2023).Source: §2.2 · p. 2
3 Methodology
The methodology pairs models with different safety alignment and prompted intentions in debate, evaluates their responses to adversarial prompts using toxicity classifiers, and clusters attack prompts by semantic similarity. It also compares debate with single-agent generation and self-refinement.
3.1 Multi-Agent Debate
- Debate agents first generate zero-shot responses, then update them for a chosen number of rounds using other agents’ outputs or their own prior responses as advice.The study includes base and RLHF-tuned models from GPT and Llama families, and assigns agents harmless, neutral, or harmful intentions through prompts.Source: §3.1 · pp. 2–3
- The experiments compare zero-shot generation and self-refinement baselines with debate, using GPT and Llama variants chosen to span different safety alignment and instruction-following characteristics.The selected models include text-davinci-003, text-curie-001, GPT-3.5-turbo-0301, Llama-2-7b, Llama-2-7b-chat, and Llama-2-7b-chat-uncensored.Source: §3.1 · p. 3
3.2 Red-Teaming Evaluation
- The evaluation prompts come from the highest-scoring 5% of conversations in Anthropic’s red-teaming dataset, with the midpoint report using the top 100 samples.The source dataset contains 38,961 crowdsourced attacks; toxicity is evaluated with the fine-tuned classifier and Perspective API, reporting means and 10th and 90th percentiles.Source: §3.2 · pp. 3–4
3.2.1 Classifier Details
- Toxicity classifiers are trained on BAD data in single-turn and multi-turn variants: the former sees only each dialogue’s final message, while the latter sees full context but classifies only that message.At inference, the toxicity score is the softmaxed probability assigned to the offensive label. Test utterances are labeled toxic if their scores exceed the validation-set threshold, corresponding to the 60th percentile of model scores over all utterances in that set.Source: §3.2.1 · p. 4
3.2.2 Classifier Performance
- The authors report that Perspective API performs poorly on this task, while their fine-tuned model and the BAD classifier perform comparably.The BAD classifier performs slightly better in the single-turn case; the authors choose their own model for its flexibility in prompting it in different ways.Source: §3.2.2 · p. 4
3.3 Clustering Types of Harmful Attacks
- The prompt embeddings are clustered with K-Means into four groups, with k selected by observing where reductions in within-cluster sum of squares become marginal.Researchers name the clusters by inspecting their prompts and calculate mean Perspective API toxicity scores for each cluster.Source: §3.3 · pp. 5–6
4 Results and Discussion
The section compares toxicity across single-agent, debate, and attack-category settings. Debate reduces toxicity in some pairings, while outcomes also vary with prompt context, model, and attack type.
4.1 Single-Agent Performance
- RLHF chat models show lower zero-shot toxicity than base models, and two rounds of self-reflection slightly reduce toxicity except for Text-curie-001.Toxicity is evaluated using mean Perspective API and fine-tuned Llama-classifier scores.Source: §4.1 · p. 6, §4 · p. 6
- Rewriting the discussion topic as a drama movie plot text continuation task increases the output API toxicity score to 0.21 (Liu et al., 2023), but this prompt design does not transfer to chat settings.Source: §4.1 · p. 6
- A simple harmful prompt increases uncensored Llama chat toxicity by >75% versus explicit harmless regulations, while base models generally produce lower-quality, more toxic text.Source: §4.1 · p. 6
4.2 Multi-Agent Performance
- Pairing a harmful agent with a safety-instructed model reduces toxicity after 1–2 discussion rounds and outperforms self-refinement for harmful agents.The improvement is especially clear for Llama-based chat agents.Source: §4.2 · p. 7
- In static-agent, single-direction discussions, context changes affect harmful agents more than harmless agents in the first two rounds; later rounds have limited effect, possibly due to longer context and the simple framework.Source: §4.2 · p. 7
- Two unprompted Text-Davinci models become more harmful during debate, whereas two Llama-2-7b-chat-uncensored models’ toxicity gradually decreases.The authors suggest dialogue format or residual safety alignment as possible explanations.Source: §4.2 · p. 7
4.3 Clustering Analysis
- Text-Davinci-003 has its highest mean toxicity for insults and emotional harm (around 0.164) and its lowest for privacy violations (around 0.057).Source: §4.3 · pp. 7–8
- Llama-2-7b’s mean toxicity is highest for privacy violations (0.354) and lowest for physical violence and unlawful activities (approximately 0.078 and 0.077).These attack-category patterns diverge from Text-Davinci-003’s.Source: §4.3 · p. 8
5 Conclusion and Future Work
The conclusion reports that multi-agent debate can lower toxicity relative to baselines, but toxic agents can also negatively influence debate partners. The effect depends on the model and intent, and the approach has cost and capability constraints.
5 Conclusion and Future Work
- Inference-time debate shows some effect in lowering toxicity versus baselines such as Self-Refine, although toxic agents can negatively influence their debate partners.The negative influence is reported as weaker than the positive effect of debating with nonpoisoned agents.Source: §5 · p. 8
- The debate effect is sensitive to the specific model and intent used.Source: §5 · p. 8
- Debate can be resource-intensive in model cost and latency, and models lacking conversational ability may not benefit.The authors propose exploring longer interactions, more agents, realistic feedback and update stages, and ways to identify when longer debate is needed.Source: §5 · p. 8
6 Ethics Statement
The authors tested existing adversarial prompts and manually inspected a subsample of outputs for new exploits; they also committed to content warnings for potentially triggering outputs.