One Turn Too Late: Response-Aware Defense Against Hidden Malicious Intent in Multi-Turn Dialogue

Xinjie Shen; Rongzhe Wei; Peizhi Niu; Haoyu Wang; Ruihan Wu; Eli Chien; Bo Li; Pin-Yu Chen; Pan Li

Source-linked AI summary

One Turn Too Late: Response-Aware Defense Against Hidden Malicious Intent in Multi-Turn Dialogue

Xinjie Shen, Rongzhe Wei, Peizhi Niu, Haoyu Wang, Ruihan Wu, Eli Chien, Bo Li, Pin-Yu Chen, Pan Li

arXiv:2605.05630v2May 7, 2026cs.CLcs.AIcs.CR

TL;DR

Multi-turn attacks can hide harmful intent across benign-looking dialogue, making the right intervention turn difficult to identify. The paper introduces MTID and TurnGate, a response-aware, turn-level monitor trained to detect harm-enabling closure points. Across offline and closed-loop evaluations, TurnGate achieves a stronger safety–utility trade-off than existing guardrails and prompt-based monitors.

Problem
Multi-turn attacks distribute harmful objectives across benign-looking turns, while existing defenses remain vulnerable and must identify when dialogue becomes sufficient to enable harm.
Method
The paper constructs MTID with annotated harm-enabling turns and trains TurnGate to inspect candidate responses for turn-level intervention.
Results
Across offline evaluation and closed-loop online battles, TurnGate achieves a stronger safety–utility trade-off than existing guardrails and prompt-based monitors.
Takeaways & Limitations
Turn-level supervision improves intervention timing, while response-aware monitoring helps distinguish harmful closures from benign conversations with similar surface content.
Takeaways & Limitations
MTID focuses on Chemistry and Cybersecurity, and defenders trained on adaptive attack rollouts may reflect simulated attacker regularities.

Abstract

from arXiv · show

Hidden malicious intent in multi-turn dialogue poses a growing threat to deployed large language models (LLMs). Rather than exposing a harmful objective in a single prompt, increasingly capable attackers can distribute their intent across multiple benign-looking turns. Recent studies show that even modern commercial models with advanced guardrails remain vulnerable to such attacks despite advances in safety alignment and external guardrails. In this work, we address this challenge by detecting the earliest turn at which delivering the candidate response would make the accumulated interaction sufficient to enable harmful action. This objective requires precise turn-level intervention that identifies the harm-enabling closure point while avoiding premature refusal of benign exploratory conversations. To further support training and evaluation, we construct the Multi-Turn Intent Dataset (MTID), which contains branching attack rollouts, matched benign hard negatives, and annotations of the earliest harm-enabling turns. We show that MTID helps enable a turn-level monitor TurnGate, which substantially outperforms existing baselines in harmful-intent detection while maintaining low over-refusal rates. TurnGate further generalizes across domains, attacker pipelines, and target models. Our code is available at https://github.com/Graph-COM/TurnGate.

1 Introduction

Multi-turn attacks can conceal harmful objectives across benign-looking dialogue, leaving even advanced LLMs vulnerable. The paper proposes earliest-turn, history-aware intervention and introduces MTID and TurnGate to support this objective.

Motivation: LLMs’ deployment in high-stakes settings makes preventing misuse a central safety challenge.The cited settings include scientific research, cybersecurity, and medical consultation.
Problem: Even state-of-the-art commercial models remain vulnerable to multi-turn attack strategies.
Problem: The key defense decision is the earliest turn where delivering the candidate response makes the accumulated interaction sufficient to enable misuse.Earlier intervention risks over-refusal, while later intervention can allow the attacker to obtain sufficient information to act.
Contributions: TURNGATE monitors each candidate response before delivery and makes turn-level malicious-intent intervention decisions.The paper also introduces MTID, built from adaptive attack rollouts, matched benign dialogues, and annotations of first harm-enabling turns.

2 Related Work

Prior defenses mainly classify isolated prompts, outputs, or user queries, leaving malicious-intent accumulation and response history under-modeled. Multi-turn jailbreak research contrasts with earlier single-turn attacks that encode the full malicious intent in one prompt.

Modern Defense Guardrails: Modern safety guardrails primarily classify prompts or model outputs in isolation, effectively catching explicit malicious intent but not modeling conversational accumulation.These systems operate at the level of individual utterances or responses and use specialized alignment.
Modern Defense Guardrails: Sequential Monitor focuses only on user queries, overlooking harmful responses already produced and information supplied toward the user’s goal.This illustrates the limitation of query-only sequential defense.
Malicious Intent as Multi-turn Jailbreaks: Earlier jailbreak research primarily studied single-turn attacks that encode the full malicious intent in one prompt and refine it through repeated trials.Later variants also use obfuscation or deliberately exploit insufficient alignment.

3 Problem Formulation

The paper formulates safety monitoring as a response-aware, turn-level stopping problem that identifies the first harm-enabling turn before delivering the assistant’s candidate response. It distinguishes timely intervention from premature refusal and missed detection using explicit closure and blocking times.

3.1 Interaction Protocol and Response-Aware Observation: At each turn, a defender observes the dialogue history, current query, and base assistant’s candidate response before deciding whether to PASS or BLOCK.The interaction involves a user, base assistant, and defender over at most T turns.
3.1 Interaction Protocol and Response-Aware Observation: A PASS delivers the candidate response and continues the interaction, whereas BLOCK withholds it, substitutes a refusal, and terminates the episode in the proof-of-concept formulation.The framework can naturally extend to interactions continuing beyond a block.
3.1 Interaction Protocol and Response-Aware Observation: Response-aware placement is essential because risk depends on both the query and the information revealed by the assistant, which can shape future queries.A response that does not materially advance the harmful objective may pose little or no risk.
3.2 Harmful Closure and the First Harm-Enabling Turn: The harmful closure turn t* is the first turn whose delivered history, query, and candidate response are sufficient for a capable actor to realize the harmful objective.Suff(x_t, g) equals 1 exactly when the information in x_t is sufficient to realize g.
3.2 Harmful Closure and the First Harm-Enabling Turn: If no turn becomes sufficient within the horizon, the closure time is ∞, covering both benign trajectories and harmful trajectories that never become sufficient.The closure definition includes ∞ as an additional outcome.
3.2 Harmful Closure and the First Harm-Enabling Turn: Before t*, blocking is premature, while at t* BLOCK is uniquely timely because delivering the response completes the information needed for harmful realization.The closure turn is response-dependent, so identical user queries can produce different closure turns.
3.3 The Defender Objective as a Cost-Sensitive Stopping Problem: The defender’s blocking time ηπ is evaluated against t*: equality is timely intervention, earlier blocking is early intervention, and later blocking is a safety breach.For benign trajectories, t* = ∞, and never blocking preserves uninterrupted completion.
3.3 The Defender Objective as a Cost-Sensitive Stopping Problem: The cost-sensitive objective rewards benign completion and exact-hit blocking, penalizes over-refusal and missed prevention, and assigns graded penalties to premature intervention.The early-block utility ϕ(ηπ; t*) ∈ [0, 1] captures partial utility preserved when a session is truncated before closure.

4 Defense Mechanism

The defense mechanism constructs MTID by simulating adaptive multi-turn attacks, annotating each harmful trajectory’s earliest harm-enabling closure turn, and pairing attacks with matched benign hard negatives. TurnGate then uses turn-level process rewards and a supervised-to-offline-RL training procedure to learn precise PASS/BLOCK timing.

MTID construction: MTID uses adaptive tree-search jailbreak rollouts to expose the harmful closure turn t∗ as an observable training event.The search branches on the assistant’s actual responses and records the first turn at which the accumulated dialogue suffices to realize the harmful objective.
MTID construction: Each successful search branch becomes a multi-turn harmful trajectory, while matched benign trajectories provide hard negatives for measuring over-refusal.Benign trajectories share technical terminology with harmful queries, reducing reliance on surface-level heuristics.
TurnGate training: Offline reinforcement learning estimates turn-level advantages with backward recursion and optimizes a clipped importance-weighted objective with a KL penalty.The objective is anchored to a reference policy πref, with advantages computed from normalized process rewards.
Turn-level process rewards: Turn-level rewards distinguish utility-preserving passes before t∗, blocking exactly at t∗, premature blocking, missed closure, and false positives.The reward is defined per action but depends on the dialogue’s closure turn, so the same BLOCK action can be beneficial or harmful depending on timing.
TurnGate training: TurnGate is trained with supervised warm-start fine-tuning followed by offline reinforcement learning on per-turn PASS/BLOCK samples.Qwen3-4B serves as the lightweight base model, and labels are derived from the annotated closure turn t∗.

5 Experiments

Experiments evaluate TurnGate on MTID using paired harm-enabling and benign rollouts, offline metrics, and adaptive closed-loop attacks. TurnGate improves safety–utility trade-offs over baselines and generalizes across domains, benchmarks, target models, and attacker pipelines.

Experimental setup: MTID uses seed-level 70%:15%:15% train/validation/test splits, annotating harmful rollouts with first sufficient turn t∗ and benign rollouts with t∗ = ∞.The paired design measures both harm-enabling turn precision and over-refusal.
Experimental setup: Baselines include response-aware and response-blind prompt monitors, while trainable controls compare trajectory-level supervision with turn-level PASS/BLOCK decomposition.Turn-level samples use the cumulative dialogue prefix through turn t and labels derived from t∗.
Main results: 0.699 F1ϕ2 is achieved by TurnGate, improving over Reweighted-SFT’s 0.610 while raising Harmful Score ϕ2 from 0.479 to 0.602.Exact-turn accuracy also increases from 34.3% to 41.4%, with comparable benign pass-through.
Online robustness: TurnGate achieves the lowest attack success rate by a clear margin as the adaptive tree-search attacker’s online iteration budget increases.The closed-loop attacker conditions subsequent queries on delivered responses and can backtrack or reroute after blocked or uninformative turns.
Generalization: Under benchmark shift, TurnGate lowers ASR on 10 of 12 held-out categories and consistently achieves the strongest offline cross-category performance in both transfer directions.The evaluated shifts span benchmarks, target models, and attacker pipelines, supporting transferable turn-level defense behavior rather than memorization of in-distribution traces.

6 Conclusion · Appendix

The paper frames multi-turn malicious-intent detection as a turn-level intervention problem and introduces MTID and TURNGATE to improve intervention timing and safety–utility trade-offs. It also identifies extensions to longer multi-intent conversations and supportive redirection beyond binary blocking.

6 Conclusion: The work studies multi-turn malicious-intent detection as a turn-level intervention problem.
6 Conclusion: MTID annotates the first harm-enabling turn in adaptive multi-turn attack rollouts and pairs them with benign hard negatives.
6 Conclusion: TURNGATE is a response-aware defender developed on the MTID benchmark.
6 Conclusion: Across offline evaluation and closed-loop online battles, TURNGATE achieves a stronger safety–utility trade-off than existing guardrails and prompt-based monitors.
6 Conclusion: Turn-level supervision improves intervention timing, while response-aware monitoring distinguishes harmful closures from benign conversations with similar surface content.
6 Conclusion: Future work includes extending the framework to longer multi-intent conversations.
6 Conclusion: Another future direction moves beyond binary PASS/BLOCK decisions toward safe, supportive responses that redirect harmful intent while preserving user utility.

A Illustrative Case Studies

The case studies show TurnGate identifying the earliest harm-enabling turn in a multi-turn phishing attack while avoiding the premature intervention exhibited by another monitor.

Harmful case: Turn 3 is the first harmful-sufficient turn in a cryptocurrency-phishing dialogue, when the assistant is about to provide actionable implementation details.The attack escalates from generic web-security questions toward an explicit phishing implementation request.
Harmful case: Only TurnGate blocks exactly at the harmful-sufficient turn; Qwen Guard and Vanilla LLM Monitor miss it, while Sequential Monitor blocks prematurely at Turn 2.The example contrasts missed closure detection with early refusal across four defenders.
Qualitative comparison: The representative harmful and benign MTID dialogs expose two failure modes: missed blocks on harmful closures and early blocks on benign conversations.Both three-turn examples compare TurnGate, Qwen Guard, Vanilla LLM Monitor, and Sequential Monitor using per-turn decisions.

A.1 Harmful Example: Cryptocurrency Phishing Site · A.2 Benign Example: Sociology of Cross-Border Drug Trafficking

The examples contrast a harmful three-turn cryptocurrency-phishing dialogue, where TURNGATE blocks precisely at the first unambiguous harmful closure, with a benign three-turn sociology dialogue that it passes uninterrupted. Together, they show response-aware turn-level defense can avoid both premature blocking and missed harmful responses.

A.1 Harmful Example: Cryptocurrency Phishing Site: The harmful objective—to create a cryptocurrency-phishing website—is distributed across three individually legitimate-looking cybersecurity turns.The attack progressively narrows from cryptocurrency security requirements to replay-attack validation and then phishing-site implementation.
A.1 Harmful Example: Cryptocurrency Phishing Site: All four defenders pass Turn 1, where the assistant gives standard cryptocurrency-transaction security practices such as MFA, key management, and transaction integrity.The question remains general and defensive.
A.1 Harmful Example: Cryptocurrency Phishing Site: At Turn 2, TURNGATE, Qwen Guard, and Vanilla LLM Monitor pass defensive replay-attack guidance, while Sequential Monitor prematurely blocks on “phishing.”The assistant discusses ECDSA signatures, nonce handling, and chain separation; the passage characterizes the question as still defensive.
A.1 Harmful Example: Cryptocurrency Phishing Site: Turn 3 is the first unambiguous harmful request, asking how to implement cryptocurrency-mimicking mechanisms in a phishing website.Without defense, the assistant provides concrete implementation details including EIP-712 spoofing, nonce mirroring, and domain cloning.
A.2 Benign Example: Sociology of Cross-Border Drug Trafficking: The benign dialogue analyzes the sociology of cross-border drug trafficking through standard academic questions about community dynamics, kinship networks, and enforcement blind spots, with t∗ = ∞.No turn reaches harmful sufficiency, so the correct behavior is to pass all turns.
A.1 Harmful Example: Cryptocurrency Phishing Site: ηπ = t∗ = 3: TURNGATE blocks at the harmful closure turn, whereas Qwen Guard and Vanilla LLM Monitor miss the block and allow the harmful response.This illustrates that harmful sufficiency depends on the accumulated dialogue and the candidate response, not isolated keywords.
A.2 Benign Example: Sociology of Cross-Border Drug Trafficking: TURNGATE passes every benign turn, while Qwen Guard false-blocks Turn 1 and Vanilla LLM Monitor false-blocks Turn 3 despite the dialogue remaining academic.Turn 1 covers sociological patterns and community dynamics; Turn 3 covers temporal and spatial enforcement blind spots, with ηπ = ∞ for TURNGATE.
A.2 Benign Example: Sociology of Cross-Border Drug Trafficking: The three baselines false-block at different turns—Qwen Guard at Turn 1, Sequential Monitor at Turn 2, and Vanilla LLM Monitor at Turn 3—while TURNGATE achieves both objectives.The matched benign trajectories use explicit t∗ = ∞ supervision, supporting uninterrupted benign dialogue alongside precise harmful closure blocking.

B Detailed Threat Model Specifications · C Dataset Construction and Statistics

The threat model treats hidden malicious intent as an adaptive, distributed information-gathering process in which individually benign turns can accumulate toward restricted capabilities. The MTID section describes benchmark construction from domain filtering through final trajectory extraction.

B Detailed Threat Model Specifications: The attacker decomposes a harmful objective into individually benign-looking turns to acquire restricted technical capabilities.
B Detailed Threat Model Specifications: The interaction is modeled as a sequential information-gathering process between a target LLM and a sophisticated adversary.
B Detailed Threat Model Specifications: The adversary conditions each query on the entire dialogue history, including the target model’s prior responses.
B Detailed Threat Model Specifications: By monitoring responses, the adaptive attacker can identify dead ends, backtrack to decision points, or reroute toward alternative vulnerabilities.
B Detailed Threat Model Specifications: The defender accesses the full prior turn context and evaluates whether a candidate response could complete harmful capability transfer.
B Detailed Threat Model Specifications: The defender does not assume knowledge of the latent objective g at test time, instead inferring the necessary action from learned representations.
C Dataset Construction and Statistics: MTID construction proceeds from initial domain filtering to final trajectory extraction.

C.1 Domain Filtering Methodology … D.1 Training Hyperparameters and Reproducibility

MTID is constructed by domain-filtering WildJailbreak prompts, generating response-conditioned harmful and benign rollouts, annotating earliest harm-enabling turns, and applying quality control. Training uses fixed five-epoch, full-parameter procedures with greedy single-token decisions and random seed 42.

C.1 Domain Filtering Methodology: WildJailbreak prompts are filtered into chemistry and cybersecurity using GPT-5.2, with inclusive topic-based classification that also permits a none category.The classifier focuses on topic rather than harmfulness and returns JSON labels for each prompt.
C.2 MTID Generation Pipeline: Each rollout uses delivered history, a generated sub-query, and the target model’s candidate response, while an online evaluator identifies the first sufficient context turn t*.Harmful rollouts stop at t*, whereas benign trajectories receive t*=∞.
C.3 Seed Statistics and Split Protocol: The dataset retains 200 harmful and 200 benign seeds per domain, split at the seed-prompt level into train, validation, and test sets using 70%:15%:15%.Rollout generation is performed independently within each split to avoid leakage across related branches.
C.4 Response-Conditioned Rollout Generation: Repeated independent CKA-style searches collect up to 20 valid rollouts per seed through adaptive response-conditioned frontier updates.The pipeline also enforces at least 2 turns, per-seed attempt caps, standardized trajectory metadata, and t* annotations; benign queries target legitimate class-D completion.
C.5 Online Evaluation and Judging Prompts: Online harmful and benign judges classify responses from A to D using task-specific rubrics emphasizing usefulness, detail, completeness, and, for harmful goals, actionable potential.The harmful rubric disregards disclaimers when harmful content remains actionable.
C.6 Post-Generation Quality Control: An independent Gemini-2.5-Flash review checks harmful closure turns and verifies that the preceding context at t*−1 was insufficient.Inconsistencies in the original sufficiency judgments are resolved during post-generation quality control.
C.7 Trajectory Extraction and Final Dataset Shape: Trajectory paths are exported in JSONL with turn text and metadata, truncating harmful dialogues at t* and labeling benign dialogues t*=∞; 8,000 dialogues are produced for each class.The dataset shape follows 20 rollouts per seed.
D Experimental and Implementation Details: All trainable baselines and TURNGATE use five epochs of full-parameter fine-tuning, vLLM-accelerated evaluation, greedy temperature-0 single-token BLOCK/PASS actions, and random seed 42.The implementation details section frames these procedures as the study’s training, compute, and baseline configuration.

D.2 Compute Resources … F Limitations and Broader Impacts

The paper details compute-efficient implementations for TurnGate and its baselines, including adaptive attacker search and standardized evaluation procedures. It also frames the 4B backbone as a deployment-driven choice that balances turn-level intervention quality with resource and latency constraints while acknowledging limitations from unavailable larger models.

D.2 Compute Resources: Training uses 3 NVIDIA H100 GPUs, with 2 dedicated to training and 1 isolated for concurrent evaluation.Each open-source model fits on a single H100 during inference, while GPT-5.2 and Gemini-2.5-Flash are accessed via APIs.
D.2 Compute Resources: Evaluation relies on external APIs for target models, ASR judging, and JSON correction, with automatic exponential backoff mitigating transient failures.The backoff strategy addresses rate limits, network timeouts, and request failures during large-scale evaluation.
D.3 Implementation of Prompt-based Baselines: Prompt-based baselines use Qwen3-4B with greedy decoding, while sequential monitoring evaluates cumulative context and blocks turns when predicted maliciousness exceeds 0.5.Intention Analysis instead identifies the essential intention before classifying it as harmful or benign, whereas the naïve classifier directly labels the current exchange.
D.4 Implementation of Guard Model Baselines: Guard baselines include Llama Guard 3, Qwen Guard, and a synthesis-plus-guard pipeline that summarizes conversations with Qwen3-4B before classification.Qwen Guard blocks outputs labeled “Safety: Unsafe” or “Safety: Controversial.”
D.5 Implementation of Trainable Baselines: Trainable baselines use Qwen3-4B and receive the full conversation history together with the current assistant response through a shared prompt template.They classify whether the assistant response produces harmful content or is benign using a binary digit output.
D.6 Mechanics of Adaptive Backtracking and Rerouting: The adaptive attacker uses Monte Carlo Tree Search and UCT to balance exploration and exploitation, backtracking when defensive responses reduce a trajectory’s priority.Its rerouting controller can choose between deepening one path and branching into multiple sub-queries when routes fail or reveal alternative vulnerabilities.
E Discussion: Model Scaling and Deployment Practicality: TurnGate and trainable baselines standardize on Qwen3-4B because resource constraints and post-generation, pre-delivery latency requirements favor a compact defender.The design targets rapid iteration and practical deployment while evaluating accumulated dialogue history.
F Limitations and Broader Impacts: The 4B model achieves competitive empirical performance in localizing harmful intent while preserving benign utility, but hardware limitations prevented evaluation with much larger state-of-the-art models.Preliminary fine-tuning explored architectures up to 16B parameters, while models of 440B+ parameters remained unavailable.

F.1 Limitations · F.2 Broader Impacts

The framework defines an earliest harm-enabling turn for timely intervention while acknowledging limits in boundary nuance, domain coverage, simulated-attack regularities, and intervention scope. Its broader-impact design targets distributed-intent failures and overrefusal, while recognizing dual-use risks and the need for deployment safeguards.

F.1 Limitations: The earliest harm-enabling turn t∗ provides an operational boundary between premature refusal and timely intervention, though information sufficiency can vary by attack context.Post-verification confirms that t∗ is a tight, actionable boundary where information closure first occurs within the framework.
F.1 Limitations: MTID covers Chemistry and Cybersecurity, so extending it to risk categories with different closure dynamics remains an open direction.Out-of-domain evaluations show encouraging generalization, but the benchmark’s current domain focus limits coverage.
F.1 Limitations: Because MTID uses adaptive attack rollouts, the resulting defender may reflect regularities of the simulated attack process.The passage identifies greater diversity in attack processes as a future-work direction, although the supplied text is truncated before specifying it.
F.1 Limitations: TURNGATE operates post-generation and pre-delivery with binary PASS/BLOCK actions, while richer interventions such as redaction, safe alternatives, and clarifying questions remain compatible extensions.The binary design keeps sequential stopping clean and timing evaluation interpretable.
F.2 Broader Impacts: TURNGATE addresses harmful intent distributed across individually benign turns and aims to preserve utility in long-session scientific, cybersecurity, troubleshooting, and technical-writing settings.The framework is positioned as a response to failure modes that single-turn safety classifiers miss, where prior assistant responses contribute to actionability.
F.2 Broader Impacts: The work is dual-use, so released examples are sanitized, operationally actionable benchmark content is avoided, and controlled access is recommended where appropriate.Because the formulation penalizes early blocks and benign false positives, deployments should also include transparency, appeal mechanisms, and additional safeguards.