Source-linked AI summary
EvoFlint: An Evolutionary Atlas of Multi-Turn LLM Vulnerabilities
Feitong Qiao, Liren Peng, Shiming Ren, Aishwarya Jadhav, Arghavan Bahadorinejad, Marinette Chen, Muhan Zhang, Abdulaziz Suria, Gennevi Lu, Anish Das Sarma
TL;DR
Multi-turn attacks can evade single-turn safeguards, yet automated red-teaming has emphasized generating attacks rather than organizing reusable evidence about model failures. EvoFlint evolves phased conversation plans with quality-diversity search, and its archive reaches 72.1% mean ASR across four targets while exposing risk-category failure coverage. The evaluation uses a single GPT-4o judge, leaving sensitivity to judge identity untested.
Problem
Automated red-teaming has focused on producing attacks, while the field has paid less attention to organizing diverse, reusable maps of how models fail across multi-turn interactions.
Method
EvoFlint evolves phased conversation plans using LLM-driven mutation and crossover, Pareto fitness over ASR and peak severity, risk-indexed NSLC archives, and generation-level shared memory.
Results
72.1% mean ASR across GPT-4o, Claude Sonnet 4.6, Qwen3-32B, and GPT-5.4 exceeds X-Teaming’s 44.0%, while the archive contains 39 named structured strategies at generation 5.
Takeaways & Limitations
The archive provides an organized, inspectable account of target-model failures across risk categories rather than only a headline attack-success number.
Takeaways & Limitations
Evaluation uses a single GPT-4o judge, so dependence of the leaderboard on judge identity has not been stress-tested.
Abstract
from arXiv · showhide
Frontier language models that refuse harmful single-turn prompts often comply when the same intent is reached gradually over many turns, making multi-turn attacks one of the least understood failure modes of large language models. Most automated red-teaming methods treat this as a generation problem: produce attacks that break the model. We argue it is better framed as a search problem: discover, organize, and iteratively refine a diverse archive of attack strategies, producing a structured map of how a target model fails rather than a list of one-off successes. We introduce EvoFlint, which applies evolutionary quality-diversity search to multi-turn red-teaming. Attack strategies are phased conversation plans, not raw prompts, and are evolved through LLM-driven mutation and crossover. A Pareto fitness over attack success rate and peak severity preserves selection signal from near-miss attacks. A risk-indexed archive runs novelty search with local competition over strategy description embeddings inside each cell, maintaining diversity without committing to a predefined style taxonomy. A generation-level memory accumulates target-model insights across the population and feeds them back into strategy generation. On the HarmBench-test split, EvoFlint reaches attack success rates of 35.8% on Claude Sonnet 4.6, 59.7% on GPT-5.4, and 94.3% on Qwen3-32B, alongside 98.7% on the older GPT-4o included as a baseline reference. The resulting archive, organized by risk category, exposes for each target which categories of harm its safety training has and has not covered.
1 Introduction
EvoFlint reframes multi-turn red-teaming as a search for diverse, reusable maps of model failure modes rather than isolated successful attacks. It combines phased strategies, Pareto fitness, risk-indexed diversity preservation, and shared target-model memory.
- EvoFlint builds a structured, persistent map of how a target model fails and in how many different ways.
- Multi-turn attacks distribute harmful intent across individually benign turns, exposing vulnerabilities that single-turn robustness does not eliminate.
- Attack strategies are phased conversation plans evolved under an NSLC sub-archive nested within a MAP-Elites risk-category partition.
- A Pareto fitness over attack success rate and peak severity preserves selection signal from near-miss attacks while separating frequency from severity.
- The generation-level memory shares target-model insights across lineages by feeding extracted knowledge into rendering, mutation, and genesis.
- 72.1% mean ASR across four targets exceeds X-Teaming’s 44.0%, while the archive doubles as a per-target failure map organized by risk category.
2 Related Work
Prior automated red-teaming methods cover single-turn attacks or generate multi-turn attacks without maintaining an evolving archive. EvoFlint combines quality-diversity search, LLM-based variation, and continuous novelty preservation for multi-turn strategies.
- Single-turn red-teaming methods use gradient optimization, reflection loops, or tree search, but do not model multi-turn dynamics.
- Multi-turn systems such as Crescendo, ActorAttack, GOAT, X-Teaming, and FERRET primarily generate attacks rather than maintain archives.
- Rainbow Teaming applies MAP-Elites to archive single-turn prompts by risk category and attack style, but depends on a predefined style taxonomy.
- Figure 1 depicts EvoFlint’s archive-centered pipeline: LLM operators vary strategies, target-model dialogues are judged, and NSLC novelty plus local competition controls admission.
3 Method
EvoFlint represents attacks as evolvable phased plans, evaluates them through rendered multi-turn dialogues, and selects them using Pareto fitness plus continuous-diversity archive admission. Shared generation-level memory transfers target insights across lineages.
- 3.1 Strategy Representation: Each strategy is an ordered phased plan with objectives, turn budgets, and optional transition guidance, while phase structure remains mutable through crossover and mutation.
- 3.1 Strategy Representation: The representation balances LLM flexibility against structural coherence by using coarse phases instead of raw prompts or brittle dialogue state machines.
- 3.2 Strategy Evaluation: Strategy evaluation renders a plan for a concrete goal, runs an attacker-controlled dialogue through its phases, and judges each target response.
- 3.3 Fitness: Binary success is insufficient because partial compliance before eventual refusal provides evolutionary information that all-refusal trajectories lack.
- 3.3 Fitness: Fitness uses attack success rate and peak severity as separate objectives, allowing Pareto ranking to preserve reliable moderate violations and rare severe outputs.
- 3.3 Fitness: The framework can add objectives such as stealth, efficiency, or persona fidelity as extra Pareto axes, while novelty remains an NSLC admission criterion.
- 3.4 Evolutionary Loop: Across generations, candidates arise from archive-selected parents and four LLM-driven operators, with targeted mutation using fitness and full transcripts for diagnosis.
- 3.4 Archive Structure: The archive partitions strategies by risk category and preserves continuous embedding-space diversity through NSLC novelty search with local competition.
4 Experiments
EvoFlint is evaluated on HarmBench-test against four target models using standardized judging and published baselines. It achieves the strongest attack success rates while producing a persistent, inspectable archive whose category-level coverage reveals distinct target failure profiles.
- Setup: Evaluation uses 159 HarmBench-test behaviors across GPT-4o, Claude Sonnet 4.6, Qwen3-32B, and GPT-5.4, with GPT-4o judging each turn.A behavior counts as broken when the 5-point intent-and-content rubric returns its top score.
- Main results: EvoFlint achieves the highest ASR on every target: 98.7% on GPT-4o, 35.8% on Claude Sonnet 4.6, 94.3% on Qwen3-32B, and 59.7% on GPT-5.4.Mean ASR is 72.1% for EvoFlint versus 44.0% for X-Teaming, the strongest baseline.
- Main results: The performance gap is widest on harder targets, where EvoFlint retains 35.8% ASR on Claude Sonnet 4.6 and 59.7% on GPT-5.4.These values exceed the next-best system by more than 1.5× on each target.
- What carries the gain: The results suggest that structured reuse and recombination of evolved strategies, rather than a single attack primitive, drives the advantage over non-search baselines.The reported gap between EvoFlint and any non-search baseline exceeds 28 percentage points on every target.
- Inside the archive: Unlike per-behavior planners, EvoFlint produces named, structured attack strategies that persist across runs and remain inspectable after evaluation.Against GPT-4o, the generation-5 archive contains 39 admitted strategies distributed across six risk categories.
- Per-target failure map: The archive exposes target-specific category failures that aggregate ASR conceals, including Claude’s 0% chemical-biological versus 91% cybercrime ASR and GPT-5.4’s 85% misinformation versus 36% illegal-activity ASR.Figure 2 computes per-category ASR from the strategies assigned to each risk-category cell.
5 Discussion and Conclusion
EvoFlint frames automated red-teaming as evolutionary search that builds an inspectable archive of diverse multi-turn failure strategies. Its results support this framing while identifying evaluation limits and extensions toward defender co-evolution and deployment-relevant objectives.
- EvoFlint builds and maintains a diverse archive of multi-turn red-teaming strategies, producing an organized account of failures across risk categories and attack styles.
- 72.1% mean attack success rate across GPT-4o, Claude Sonnet 4.6, Qwen3-32B, and GPT-5.4 exceeded X-Teaming’s 44.0% and every other tested published system’s below-30% result.At generation 5, the archive contained 39 named, structured strategies whose per-category coverage identified target exposure.
- Claude Sonnet 4.6 was the hardest target at 35.8% mean ASR, while chemical-biological and illegal-activity cells reached 0% and 9%, respectively, within budget.The authors characterize the category-specific result as likely related to Claude’s safety training, but state that more generations or seeds might change it.
- The evaluation uses a single GPT-4o judge and budget-bounded experiments, so judge dependence and additional coverage from longer runs remain unresolved.Multi-turn evaluation is expensive, and the experiments are not saturation-bounded.
- Future work proposes co-evolving defenders and adding objectives such as stealth or persona fidelity through k-dimensional Pareto ranking.These objectives would reshape the archive toward attackers plausibly faced in deployment rather than those maximizing raw ASR.