Source-linked AI summary
MART: Improving LLM Safety with Multi-round Automatic Red-Teaming
Suyu Ge, Chunting Zhou, Rui Hou, Madian Khabsa, Yi-Chia Wang, Qifan Wang, Jiawei Han, Yuning Mao
TL;DR
Manual red-teaming is costly, while existing automatic red-teaming does not typically address the risks it discovers. MART iteratively trains adversarial and target LLMs together, reducing violations by up to 84.7% after four rounds while maintaining helpfulness. The authors position it as a scalable safety technique that does not replace broader oversight or system design.
Problem
Manual red-teaming is costly, and existing automatic red-teaming typically discovers safety risks without addressing them.
Method
MART iteratively trains an adversarial LLM to generate challenging prompts and safety-aligns a target LLM using safe responses to those prompts.
Results
84.7%: violation rates decrease by up to this amount after four rounds on adversarial prompt evaluations, while helpfulness remains stable on non-adversarial instruction-following prompts.
Takeaways & Limitations
Adversarial training between LLMs enables automated, scalable, and effective red-teaming for safer AI systems within the evaluated setting.
Takeaways & Limitations
MART does not replace thoughtful system design, dataset curation, human oversight, or continuous monitoring.
Abstract
from arXiv · showhide
Red-teaming is a common practice for mitigating unsafe behaviors in Large Language Models (LLMs), which involves thoroughly assessing LLMs to identify potential flaws and addressing them with responsible and accurate responses. While effective, manual red-teaming is costly, and existing automatic red-teaming typically discovers safety risks without addressing them. In this paper, we propose a Multi-round Automatic Red-Teaming (MART) method, which incorporates both automatic adversarial prompt writing and safe response generation, significantly increasing red-teaming scalability and the safety of the target LLM. Specifically, an adversarial LLM and a target LLM interplay with each other in an iterative manner, where the adversarial LLM aims to generate challenging prompts that elicit unsafe responses from the target LLM, while the target LLM is fine-tuned with safety aligned data on these adversarial prompts. In each round, the adversarial LLM crafts better attacks on the updated target LLM, while the target LLM also improves itself through safety fine-tuning. On adversarial prompt benchmarks, the violation rate of an LLM with limited safety alignment reduces up to 84.7% after 4 rounds of MART, achieving comparable performance to LLMs with extensive adversarial prompt writing. Notably, model helpfulness on non-adversarial prompts remains stable throughout iterations, indicating the target LLM maintains strong performance on instruction following.
1 Introduction
Manual red-teaming is effective but costly, while automatic approaches have not clearly shown they can adapt to changing target-model vulnerabilities. MART addresses both issues by iteratively training an adversarial model to generate attacks and a target model to produce safe responses.
- Manual red-teaming requires dozens to hundreds of annotators to write prompts and responses across iterations, making it costly and slow.
- Automatic red-teaming can train adversarial LLMs to generate malicious prompts, but its ability to adapt as target-model vulnerabilities shift remains unclear.
- MART combines automatic adversarial prompt writing with safe response generation through iterative adversarial red-teaming.
- Each iteration uses evaluator feedback to identify successful attacks for adversarial-model training and pairs adversarial prompts with responsible target-model answers for safety alignment.
- 84.7%: MART reduces violation rates by up to this amount after four rounds against an instruction-tuning baseline with limited safety alignment.
- MART reaches safety close to ChatGPT with 2k seed prompts while maintaining strong instruction-following performance without additional helpfulness data.
2 Approach
MART initializes instruction-following and safety data, then jointly evolves an adversarial model and target model across rounds. Successful attacks train the adversary, while evaluator-selected safe and helpful responses train the target model.
- 2.1 Initialization: MART initializes the target model with general instruction-tuning data and a manually curated red-teaming seed of approximately 2,400 prompts.
- 2.1 Initialization: The seed prompts cover violation categories and attack styles, with 1,700 training prompts and 700 evaluation prompts.
- 2.2 Feedback: Reward models score prompt-response pairs for safety and helpfulness, providing feedback for selecting data and updating both models.
- 2.2 Jailbreaking with Adversarial LLM: At each round, Madv generates prompts similar to previous successful jailbreaks, and Mtgt responses are evaluated to identify newly successful attacks.
- 2.3 Feedback Guided Safety Finetuning: Successful prompts train Madv, while selected target responses meeting safety and helpfulness criteria form the safety-alignment set for Mtgt.
- 2.3 Feedback Guided Safety Finetuning: Context distillation expands first-round safe-response data, but is omitted later because it no longer provides essential improvement and may degrade responses as Mtgt improves.
- 2.4 Multi-round Optimization: The iterative cycle adapts Madv to vulnerabilities that emerge as Mtgt changes across rounds.
3 Experiment
MART is evaluated across in-domain and out-of-domain safety and helpfulness sets, human and model-based violation assessments, and adversarial-prompt-generation ablations. Across iterations, safety improves while helpfulness remains comparatively stable, although some trade-offs and diminishing gains emerge.
- Experimental Setting: 752 SafeEval adversarial prompts and 480 HelpEval non-adversarial prompts provide in-domain safety and helpfulness evaluations.Out-of-domain evaluation uses 2,312 Anthropic Harmless adversarial prompts and 805 AlpacaEval helpfulness prompts.
- In-Distribution Performance: Safety reward-model scores improve across iterations, especially at the 20% and 40% percentiles, while helpfulness at iteration 4 is 3%-4% lower than at iteration 1.The safety gains are largest for low-quality generations, while high-quality safety scores also increase moderately.
- Out-of-Domain Performance: Out-of-domain safety shows a similar distribution shift, while AlpacaEval helpfulness barely changes over multiple iterations.The reported pattern supports generalization of MART's safety improvements beyond the in-domain evaluation set.
- Automatic and Human Evaluation: 84.7% RM-evaluation and 53.7% human-evaluation safety improvements over Vanilla are reported, although MART remains behind heavily manually red-teamed state-of-the-art models.The comparison uses model-based and human-annotated violation rates; stronger models used hundreds of thousands of examples.
- Adversarial Prompt Generation Ablations: MART-1shot is more adversarial and more diverse than MART-3shot, while GCG methods trigger the most harmful responses initially but decay after the first safety fine-tuning round.The 1-shot setting also uses demonstrations more efficiently and produces larger numbers of adversarial generations.
- Adversarial Performance: Violation rates decrease across datasets through the iterations, with the clearest improvement on in-distribution SafeEval and less significant gains out of domain.Training stops at iteration 4 because adversarial-generation violations approach 10%, making sufficient training data harder to obtain; the authors suggest external red-teaming for further gains.
4 Related Work
Related work spans human red-teaming, automatic attack generation, and iterative adversarial-model optimization. MART extends these directions by using adversarial prompts not only to expose vulnerabilities but also to improve target-model safety.
- Human and automatic red-teaming: Human red-teaming is costly and difficult because annotators must manually design malicious prompts and responses across repeated iterations.These limitations motivate scalable automatic approaches.
- Training-data selection: Training-data ablations report that increasing the selected data does not always improve the model, while performance remains relatively stable.The figure examines selected-data quantity alongside model performance.
- Human and automatic red-teaming: Automatic red-teaming methods include prompt injection and adversarial model training for generating or transforming malicious prompts.Prompt injection overrides instructions or applies malicious controls, while adversarial model training optimizes attack generation.
- MART’s distinction: Prior automatic methods mainly develop effective attacks, leaving unclear whether attack prompts can also correct target-model vulnerabilities.MART addresses this gap by iteratively providing novel and diverse adversarial prompts for target-model improvement.
- MART’s distinction: MART optimizes the adversarial model with target-model feedback, linking evolving attack generation to iterative safety improvement.This differs from approaches focused primarily on discovering attacks.
5 Conclusion and Future Work
MART is a multi-round automatic red-teaming framework that updates adversarial prompts for an evolving target model and aligns the target against those attacks. After multiple rounds, it reduces violation rates without hurting helpfulness, while leaving broader technique integration and dialogue-based red-teaming for future work.
- Conclusion: MART iteratively generates updated attacking prompts toward an evolving target model and aligns the target against newly generated attacks.The framework uses adversarial training between models across multiple rounds.
- Conclusion: 84.7% violation-rate decrease on reward model evaluation and 53.7% on human evaluation are achieved after multiple rounds without hurting model helpfulness.These outcomes are reported for MART after repeated adversarial combating.
- Conclusion: Adversarial training between LLMs enables automated, scalable, and effective red-teaming for safer AI systems.This is the paper’s stated broader implication of the MART framework.
- Future work: The work focuses on instruction fine-tuning and rejection sampling, leaving integration of techniques such as reinforcement learning for future exploration.The conclusion also identifies dialogue-based multi-turn adversarial red-teaming as a future direction.
6 Ethical Statement
The ethical statement positions MART as support for developers of helpful and safe assistants, while emphasizing that safety requires broader oversight and responsible practices. It also recognizes potential dual-use concerns and the continuing nature of risk identification.
- Safety commitments: MART does not remove the need for thoughtful system design, dataset curation, human oversight, and continuous monitoring.The statement places the method within a broader commitment to LLM safety.
- Safety commitments: Identifying and anticipating risks is an ongoing process requiring constant vigilance.The statement treats safety risk management as continuous rather than complete.
- Responsible use: MART is presented as a tool for developers building helpful and safe assistants, not for actors seeking to exploit or deceive.The authors frame the intended use around protective safety improvements.
- Responsible use: The authors identify transparency, ethics review processes, and responsible practices as vital amid potential dual-use concerns.They describe the work as an early stage of research and invite broader community feedback.