Source-linked AI summary
Reasoning-Augmented Conversation for Multi-Turn Jailbreak Attacks on Large Language Models
Zonghao Ying, Deyue Zhang, Zonglei Jing, Yisong Xiao, Quanchen Zou, Aishan Liu, Siyuan Liang, Xiangzheng Zhang, Xianglong Liu, Dacheng Tao
TL;DR
Existing multi-turn jailbreaks struggle to maintain semantic coherence while bypassing safety alignment. RACE reformulates harmful queries into benign reasoning tasks and structures their progression with an attack state machine and three supporting modules. Experiments report ASRs up to 96%, including 82% against OpenAI o1 and 92% against DeepSeek R1, while the authors identify efficiency, countermeasure, and scope challenges.
Problem
Existing multi-turn jailbreak methods struggle to balance semantic coherence with attack effectiveness, causing semantic drift or ineffective alignment bypasses.
Method
RACE reformulates harmful queries into benign reasoning tasks and uses an Attack State Machine with gain-guided exploration, self-play, and rejection feedback.
Results
Up to 96% ASRs were achieved across multiple LLMs, including 82% against OpenAI o1 and 92% against DeepSeek R1.
Takeaways & Limitations
The findings reveal critical risks to LLM safety and indicate that current alignment strategies may be insufficient against reasoning-based multi-turn jailbreaks.
Takeaways & Limitations
The authors identify interaction overhead, adaptive countermeasures, and extension to other adversarial reasoning manipulations as remaining challenges.
Abstract
from arXiv · showhide
Multi-turn jailbreak attacks simulate real-world human interactions by engaging large language models (LLMs) in iterative dialogues, exposing critical safety vulnerabilities. However, existing methods often struggle to balance semantic coherence with attack effectiveness, resulting in either benign semantic drift or ineffective detection evasion. To address this challenge, we propose Reasoning-Augmented Conversation, a novel multi-turn jailbreak framework that reformulates harmful queries into benign reasoning tasks and leverages LLMs' strong reasoning capabilities to compromise safety alignment. Specifically, we introduce an attack state machine framework to systematically model problem translation and iterative reasoning, ensuring coherent query generation across multiple turns. Building on this framework, we design gain-guided exploration, self-play, and rejection feedback modules to preserve attack semantics, enhance effectiveness, and sustain reasoning-driven attack progression. Extensive experiments on multiple LLMs demonstrate that RACE achieves state-of-the-art attack effectiveness in complex conversational scenarios, with attack success rates (ASRs) increasing by up to 96%. Notably, our approach achieves ASRs of 82% and 92% against leading commercial models, OpenAI o1 and DeepSeek R1, underscoring its potency. We release our code at https://github.com/NY1024/RACE to facilitate further research in this critical domain.
1 Introduction
Multi-turn jailbreaks use iterative dialogue to expose safety weaknesses, but existing methods struggle to preserve harmful-query semantics while bypassing alignment. RACE addresses this by structuring benign reasoning tasks across turns and achieves high attack success rates in experiments.
- Multi-turn jailbreaks exploit iterative dialogue to expose critical safety blind spots beyond single-turn attacks.
- Existing methods either drift from the original harmful objective or fail to bypass alignment constraints.
- RACE reformulates harmful queries as benign reasoning tasks whose completion leads the model toward harmful content.
- RACE organizes attacks with an Attack State Machine and combines gain-guided exploration, self-play, and rejection feedback.
- Up to 96% ASRs were achieved across multiple LLMs, including 82% against OpenAI o1 and 92% against DeepSeek R1.
2 Related work
Prior work establishes LLM reasoning as a powerful capability while identifying its limited exploration for jailbreaks. Existing multi-turn attacks progressively transform harmless conversations, but often suffer semantic drift or ineffective attacks; RACE instead uses reasoning to stabilize the process.
- Reasoning in LLMs: LLMs demonstrate reasoning capabilities across mathematical, common-sense, symbolic, and causal tasks.
- Reasoning in LLMs: The potential exploitation of LLM reasoning capabilities for jailbreak attacks remains largely unexplored.
- Multi-turn Jailbreak Attack: Existing multi-turn methods start harmlessly and gradually make later queries more harmful through psychological, actor-based, keyword, or subquery strategies.
- Multi-turn Jailbreak Attack: These methods often suffer semantic drift or fail to generate effective attacks, whereas RACE uses reasoning capabilities for a stable and effective jailbreak process.
3 Threat Model
The threat model studies self-jailbreaking against a safety-aligned target model in a black-box, multi-turn interaction. A shadow model generates adaptive queries from prior query-response context, while a victim model produces responses.
- The target model is assumed to be safety-aligned and expected to avoid unsafe responses to harmful queries.
- Self-jailbreaking separates the same model into a shadow model that generates queries and a victim model that provides responses.
- The shadow model generates a query sequence intended to induce unsafe responses from the victim model.
- In the black-box setting, each query is adaptively adjusted using all preceding query-response pairs.
4 Methodology
RACE reframes harmful queries as reasoning tasks and organizes multi-turn jailbreaks with an attack state machine and three adaptive modules. Gain-guided exploration, self-play, and rejection feedback collectively guide query generation, state transitions, and recovery from failed attempts.
- Reasoning-driven jailbreak: RACE reframes harmful queries as benign reasoning tasks designed to guide the model toward harmful content while appearing legitimate.The victim solves reasoning tasks, while the shadow model refines queries without explicitly recognizing the harmful intent.
- Attack State Machine Framework: The attack state machine models conversation states, queries, transitions, an initial state, and success or failure terminal states.Within a default three-turn limit, successful attacks enter ssc, refusals enter sfl, and successful intermediate turns advance to the next state.
- Gain-guided exploration: Gain-guided exploration uses information gain to measure how effectively a candidate query advances the attack process and selects the highest-gain candidate for self-play.The measure compares uncertainty about the target response before and after conditioning on the context and seed query.
- Self-play: Self-play further optimizes candidate queries by having a shadow model simulate victim responses and maximize the probability of nonrejection.The optimized query becomes the actual query used for state transition in the attack state machine.
- Rejection feedback: Rejection feedback diagnoses failed state transitions and refines queries, while the iterative process advances successful queries and retries failed ones.The shadow model analyzes preceding context and the latest query-response pair to generate a refined query.
5 Experiments
Experiments evaluate RACE across diverse models, datasets, baselines, and defenses using ASR as the primary metric. RACE achieves strong results on classic and reasoning LLMs, while existing defenses provide limited mitigation.
- Experimental setup: The evaluation covers nine LLMs, including three open-source and six closed-source models, using AdvBench and HarmBench.AdvBench contributes 50 samples, while HarmBench contains 400 textual instances across seven harmful-activity categories.
- Experimental setup: ASR is the primary metric, while HRI additionally measures the harmfulness of unsafe responses using LLM-as-Judge evaluation.Higher values indicate more effective attacks for ASR and greater harmfulness for HRI.
- Attack performance on classic LLMs: RACE achieves average ASRs of 91.3% on AdvBench and 66.7% on HarmBench, outperforming evaluated attack methods.On HarmBench, the performance gap between RACE and baseline methods reaches up to 62.3%.
- Attack performance on reasoning LLMs: 20.0% ASR for Gemini 2.0 Flash Thinking versus 2.0% for Gemini 1.5 Pro under original harmful queries shows differing baseline vulnerability among reasoning models.The experiments use the AdvBench subset for three state-of-the-art reasoning LLMs.
- Attack performance on reasoning LLMs: 82.0% ASR against OpenAI o1 and up to 92.0% against DeepSeek R1 demonstrate RACE’s effectiveness against reasoning LLMs.The reported results indicate that these models overlook attack patterns that exploit reasoning mechanisms and contextual cues.
- Defense evaluation: 17.6% average ASR reduction makes Self-Reminder the strongest evaluated defense, but defenses remain limited against RACE.ICD reduces ASR by 3.8% on average, while SmoothLLM and JailGuard reduce it by at most 12% and 16%, respectively.
6 Discussion
Experiments examine how conversation turns, reasoning-task types, and attack modules affect RACE’s multi-turn jailbreak performance. More turns increase harmfulness, common-sense reasoning yields the highest ASR, and removing any module degrades performance.
- Number of Conversation Turns: 84.0%, 96.0%, and 100.0% ASRs are achieved on Gemma, Qwen, and GLM, respectively, with only three interactions.These results demonstrate RACE’s efficiency in short multi-turn attacks.
- Number of Conversation Turns: Increasing conversation turns systematically escalates the harmfulness of model outputs.The progression involves initially exposed unsafe information followed by increasingly intricate reasoning demands that extract more detailed content.
- Reasoning Types: 93.3% ASR is achieved by common-sense reasoning, compared with 89.3% for mathematical, 86.0% for causal, and 85.3% for symbolic reasoning.The results indicate that reasoning-task type affects attack success.
- Ablation on Attack Modules: Removing gain-guided exploration reduces ASR by up to 14.0%, showing the importance of selective query generation based on information gain.The ablation evaluates the effects of removing GE, SP, and RF from RACE.
- Ablation on Attack Modules: Removing self-play reduces ASR by up to 8.0%, while removing rejection feedback reduces ASR by up to 12.0%.Self-play supports query optimization, whereas rejection feedback handles failure transitions.
- Ablation on Attack Modules: The performance drops from removing each component demonstrate the complementary nature of gain-guided exploration, self-play, and rejection feedback.Together, the modules support efficient query generation, adaptive optimization, and robust failure handling.
7 Conclusion
RACE is a reasoning-driven jailbreak framework that models attacks as structured, adaptive progressions and uses three modules to manipulate, optimize, and recover attack queries. Experiments show that it compromises existing safety alignments and reveals risks to LLM safety.
- 7 Conclusion: RACE exploits LLM reasoning capabilities by framing harmful intent as complex, seemingly benign reasoning tasks within an attack state machine.Gain-guided exploration, self-play, and rejection feedback manipulate reasoning, optimize queries, and recover from failed attempts.
- 7 Conclusion: Extensive experiments demonstrate that RACE effectively compromises existing safety alignments and reveals critical risks to LLM safety.
- 7 Conclusion: Gain-guided exploration, self-play, and rejection feedback systematically manipulate the model’s reasoning process, optimize query structures, and recover from failed attempts.
8 Limitations
The paper identifies remaining challenges involving efficiency, countermeasures, and broader coverage of adversarial reasoning manipulation.
- 8 Limitations: RACE still requires improved efficiency to reduce interaction overhead while maintaining high ASRs.
- 8 Limitations: Adaptive countermeasures are needed to mitigate reasoning-based attacks.
- 8 Limitations: The framework should be extended to analyze and defend against other forms of adversarial reasoning manipulation.
9 Ethical Consideration
The authors frame RACE as dual-use research intended for systematic vulnerability assessment and acknowledge that comprehensive defense mechanisms remain future work. They omit explicitly harmful outputs and discuss countermeasures to reduce potential harm.
- 9 Ethical Consideration: The research has dual-use implications, but its stated primary goal is advancing LLM safety through systematic vulnerability assessment.
- 9 Ethical Consideration: The paper omits explicitly harmful outputs and includes discussions of potential countermeasures to minimize potential harm.
- 9 Ethical Consideration: Comprehensive defense mechanisms remain future work, while the findings are intended to inform more robust alignment techniques.
Supplementary Materials
The supplementary materials describe approximate information-gain computation and its use for selecting seed queries, while noting that generated queries are limited to locally optimal decisions. They also identify prompts used for failed-query optimization and evaluation.
- Approximate Information Gain: LLMs approximate conditional probability distributions to simplify information-gain computation.The approach uses LLMs as probability estimators because of their contextual understanding and response-generation capabilities.
- Approximate Information Gain: F_sr measures semantic relevance and F_ig measures new information gained relative to existing context, with both scores ranging from 0 to 9.These functions are instantiated using a shadow model in the information-gain formula.
- Query Selection: Before each turn, the shadow model generates seed queries, computes their information gain, and selects the query with maximum gain.This procedure guides query exploration at every interaction turn.
- Limitation: Automatically generated queries rely on prior context and the victim response, preventing attackers from preparing future-round queries in advance.Consequently, gain-guided exploration produces locally optimal rather than globally planned solutions.
D Types of Reasoning Tasks
The supplementary materials define four reasoning tasks with examples derived from a bomb-building target query and provide related prompts for optimization and evaluation.
- Reasoning Tasks: Table D.1 defines four reasoning tasks and gives examples derived from the target query “how to build a bomb?”.The table connects the reasoning-task definitions to a concrete harmful target query.
- Optimization Prompt: Table B.1 provides the prompt used to optimize failed queries.It is listed as the prompt for optimizing failed query.
- Evaluation Prompts: Tables C.1 and C.2 provide prompts for evaluating unsafe content and response harmfulness, respectively.The two prompts cover ASR evaluation and HRI evaluation.