Source-linked AI summary
Enabling Proactive Spoken Turns via a Generalized Style-Aware Full-Duplex Framework
Tianrui Pan, Qinglin Zhang, Chong Deng, Luyao Cheng, Qian Chen, Wen Wang, Jie Tang, Gangshan Wu, Jie Liu
TL;DR
Natural full-duplex dialogue needs proactive, timely spoken behavior without sacrificing response quality, but existing support for such interaction is fragmented. The paper introduces LPS-TC, WildTurn, and a two-tier streaming evaluation framework. LPS-TC improves timing precision and interaction quality across integrated half-duplex and full-duplex speech models while supporting style controllability.
Problem
Existing spoken-dialogue methods lack unified support for proactive, style-controllable full-duplex interaction and realistic multi-turn evaluation.
Method
The paper combines a lightweight decoupled turn controller, a real-world style-annotated dataset, and two-tier streaming evaluation of timing and interaction quality.
Results
LPS-TC outperforms other turn controllers in chunk-level timing accuracy and improves turn-level interaction quality with both half-duplex and full-duplex speech LLMs.
Takeaways & Limitations
The framework advances natural proactive full-duplex agents through precise, style-aware spoken-turn control.
Abstract
from arXiv · showhide
Compared with half-duplex dialogue systems where the system waits for user turn completion before it responds, natural full-duplex dialogue systems require agents to act proactively in real time, including timely interruptions and backchannels. This creates a key challenge: improving turn timing without sacrificing response quality. To address limitations in realistic proactive turn-taking, we build a generalized style-aware full-duplex framework with three key components. Firstly, we propose LPS-TC, a Lightweight Proactive Speech Turn Controller for plug-and-play integration. It features a fine-grained action space covering both reactive and proactive turn behaviors, enabling half-duplex models with full-duplex capabilities and enhancing existing full-duplex models with superior timing control. Secondly, we construct WildTurn, a large-scale, real-world English dataset containing approximately 2,981 hours of filtered multi-turn stereo conversations from face-to-face and telephone conversations, annotated with five turn-taking and five backchanneling styles. Trained on WildTurn, LPS-TC exhibits rich spoken dynamics that are not captured by existing static full-duplex benchmarks. Thirdly, we introduce a two-tier evaluation scheme that assesses both chunk-level timing precision and turn-level interaction quality under realistic streaming constraints. Our experiments, integrating LPS-TC with half-duplex models like Qwen2.5-Omni and full-duplex models like Freeze-Omni, showcase its superior performance in timing appropriateness and response quality. Our framework also demonstrates fine-grained style controllability and strong generalizability, enabling more natural and human-like spoken interactions.
1 Introduction
Natural proactive full-duplex dialogue requires real-time turn behaviors, but existing methods lack unified, style-controllable support. The paper addresses this gap with LPS-TC, WildTurn, and a two-tier evaluation framework.
- Motivation: Natural full-duplex agents must handle proactive interruptions, unsolicited information, and backchannels in real time.These behaviors support social and emotional interaction, but spoken proactivity remains underexplored.
- Research gap: Existing efforts remain fragmented across turn-control methods, datasets, and evaluation protocols, with no unified support for proactive, style-controllable full-duplex interaction.The proposed framework is designed to bridge this gap.
- LPS-TC: LPS-TC is a lightweight, plug-and-play controller that processes dual-channel audio to predict turn timing and style-conditioned spoken interactions.It can equip half-duplex speech LLMs with full-duplex capability or enhance existing full-duplex models.
- WildTurn: WildTurn is a real-world English dataset annotated with five turn-taking and five backchanneling styles and fine-grained proactive spoken dynamics.Its annotations include NTT, ITT, BC, BI, and NA action categories, with style refinements for NTT and BC.
- Evaluation: The two-tier evaluation measures chunk-level timing precision and turn-level interaction quality under realistic streaming conditions.The protocol reconstructs multi-round test data so each turn is evaluated with its real preceding context.
- Results: Experiments report superior chunk-level timing accuracy, improved turn-level interaction quality, style controllability, and instruction following for LPS-TC.These gains hold when integrated with both half-duplex and full-duplex speech LLMs.
2 Related Work
Prior work addresses full-duplex speech, proactive interaction, and turn-taking evaluation separately, but leaves gaps in rich spoken behaviors and realistic multi-turn streaming assessment. The paper positions its decoupled controller and expanded action space as responses to these limitations.
- Full-Duplex Spoken Turn Controller: End-to-end full-duplex models handle overlapping speech but are costly to train and may degrade response quality.Turn-controller approaches seek to separate timing control from the tightly coupled generation design.
- Full-Duplex Spoken Turn Controller: Prior turn controllers range from VAD-based binary state modeling to hidden-state-based richer turn prediction.The paper’s LPS-TC is presented as a decoupled alternative with broader interaction control.
- Proactive Spoken Interactions: Proactivity research has largely focused on text, while spoken floor-taking and diverse conversational turn behaviors remain underexplored.Existing work commonly studies real-time agent frameworks or sentence-level speaking styles rather than turn-level proactive interaction.
- Full-duplex benchmarks: Existing full-duplex benchmarks often use isolated or limited interactions and lack generalized Speech LLM evaluation under streaming input.They also insufficiently model dynamic multi-turn interaction under realistic streaming conditions.
- Proposed pipeline: The decoupled pipeline predicts actions from style instructions and dual histories, then conditions SLM audio generation on those actions.Its expanded action space supports both reactive and proactive behaviors, including timed backchannels and interruptions.
3 Method
The method combines a unified full-duplex formulation, LPS-TC for explicit style-aware turn control, and WildTurn for learning fine-grained spoken dynamics. WildTurn is constructed through chunk-level labeling, metric-based style categorization, instruction generation, and temporally tolerant region expansion.
- Problem Formalization: Full-duplex models use dynamic histories containing incoming user audio and assistant audio, unlike half-duplex models that wait for the complete user utterance.The dynamic context supports real-time interaction, while fixed completed-utterance context cannot support interruption or overlapping speech.
- Problem Formalization: Integrated timing and audio generation create a trade-off between low-latency turn decisions and high-quality response generation.The framework separates the conceptual timing variable from response generation to address this joint-optimization challenge.
- LPS-TC: LPS-TC explicitly predicts speech actions from dual-channel audio and style instructions, using causal streaming encoders for real-time inference.Its action space includes No Action, Normal Turn Taking, Interruptive Turn Taking, Backchannel, and Barge In; style instructions control turn-taking tendency and backchannel frequency or timing.
- LPS-TC: The predicted action controls SpeechLLM behavior: turn-taking and backchannel actions generate speech, No Action preserves the current state, and Barge In stops speech-token generation.After execution, current user input and assistant output are appended to the interaction history for subsequent decisions.
- WildTurn Dataset Construction: WildTurn provides real-world multi-turn conversations with chunk-level spoken-action labels and paired style instructions for learning proactive turn dynamics.Each 40 ms audio chunk receives an action label, with transition-based labels for turn-taking and lexicon-based identification for backchannels.
- WildTurn Dataset Construction: WildTurn derives styles from turn-taking and backchannel metrics, verbalizes them into conditioning instructions, and expands onset labels into active regions.Metrics include ITT-to-NTT ratio, NTT latency, ITT lead time, backchannel frequency, and onset timing; region expansion provides temporally tolerant supervision.
4 Experiments
The experiments evaluate LPS-TC through chunk-level timing on Switchboard and turn-level interaction quality on WildTurn, including style controllability and integrations with half- and full-duplex speech models.
- Datasets and Implementation Details: 2,981 hours of filtered stereo audio and 86,430 samples comprise WildTurn, built from face-to-face and telephonic conversations with a small synthetic-data component.The dataset is split into 85,323 training, 100 validation, and 1,007 test samples.
- Two-tier Real-time Full-Duplex Evaluation: 640 ms streaming audio chunks support incremental turn-timing prediction from conversation history and the current user turn.The evaluation maps wait, backchannel, and response predictions to interaction labels, classifying response onset before the user-turn end as ITT and otherwise as NTT.
- Chunk-level Turn Timing Results: LPS-TC achieves the highest chunk-level NA (0.93) and NTT (0.66) without style instructions, while style instructions raise ITT to 0.60, BC to 0.63, and BI to 0.71.Evaluations use 160 ms labels for specialized turn-prediction baselines and 640 ms labels for generalized speech LLMs.
- Chunk-level Turn Timing Results: Style-specific instructions enable LPS-TC to suppress actions in matching Switchboard subsets, demonstrating fine-grained proactive style controllability.The Patient, Assertive, and No-Backchannel subsets test whether ITT, NTT, and BC predictions can be disabled under style constraints.
- Turn-level Full-duplex Evaluation Results: Turn-level evaluation measures turn accuracy, timing appropriateness, and response quality across NTT, ITT, BI, ITT, BC, and BI interactions.The WildTurn evaluation combines F1 scores with Gemini-2.5-Pro binary judgments of timing and response appropriateness.
- Turn-level Full-duplex Evaluation Results: LPS-TC improves Freeze-Omni ITT F1 from 0.32 to 0.52 and enables Qwen2.5-Omni to reach NTT F1 = 0.60 and ITT F1 = 0.64.With style instructions, Qwen2.5-Omni’s ResITT increases from 68.4 to 72.8.
5 Ablation and Analysis
The ablations show that style instructions produce measurable shifts in turn-taking and backchannel behavior, while the controller improves streaming interaction quality and preserves response quality. Human-evaluation correlations vary by behavior, with backchannel timing aligning most closely with human judgments.
- Style controllability: 1,520 ms to 490 ms: NTT latency falls as style shifts from Patient to Assertive, while ITT lead time peaks at 630 ms.The same style shift increases ITT lead time and ITT ratio, demonstrating quantitative control over turn-taking strategy.
- Style controllability: Up to 5.17 backchannels per minute and onset timing from around 200 ms to approximately 1000 ms show independent control of frequency and timing.High-Early and High-Late styles increase frequency, No BC suppresses backchannels, and Early versus Late controls onset timing.
- Style consistency: Intermediate turn-taking styles score 0.62–0.88, while backchannel consistency reaches 0.89 for Late onset but drops to 0.67 for Early onset.The results indicate difficulty balancing intermediate NTT and ITT timing and predicting early backchannels with limited context.
- Streaming visualization: The proposed controller combines semantic and paralinguistic cues to produce timely backchannels and strategic interruptions during streaming interaction.Compared with native and VAD-controlled baselines, it avoids premature completion and purely reactive waiting.
- Streaming visualization: Streaming turn control maintains conversational flow without sacrificing final response quality, unlike non-streaming processing with high latency.The framework improves both timing and content of system turns relative to the described baselines.
- Human evaluation: BC achieves the highest human–LLM judge correlation at ρ=0.719, compared with ITT at ρ=0.637.The passage attributes BC’s stronger alignment to explicit localized cues such as brief pauses, while BI relies mainly on overlap detection.
6 Conclusion
The described turn-taking interface selects actions from conversational state, distinguishing incomplete speech that may require waiting or backchanneling from completed speech requiring a full response. It also includes proactive interruption when user intent can be predicted while the user is still speaking.
- Conclusion: The assistant chooses among four possible actions according to the current conversational state.The supplied passage introduces the action space but does not enumerate all four actions.
1. Take the Turn (Full Response)
A full-response turn is selected after the user finishes speaking or when a natural opportunity arises. The assistant then delivers a complete, substantive answer rather than a minimal acknowledgment.
- 1. Take the Turn (Full Response): The full-response action is triggered when the user has finished speaking or there is a natural opportunity.This defines when the assistant should take the turn.
- 1. Take the Turn (Full Response): The full-response behavior combines an opportunity to speak with delivery of information rather than silence or brief feedback.Both the trigger and action are specified in the supplied rules.
- 1. Take the Turn (Full Response): The resulting action is a complete, substantive response, such as an expanded factual answer.The example extends beyond a minimal answer with additional information.
2. Interrupt Turn-Taking (ITT)
Interruptive turn-taking allows the assistant to respond before the user finishes when the user’s intent can be predicted. The interruption is framed as polite and helpful rather than merely reactive.
- 2. Interrupt Turn-Taking (ITT): The assistant may interrupt while the user is still speaking when it can predict the user’s intent.This is the defining timing condition for interruptive turn-taking.
- 2. Interrupt Turn-Taking (ITT): The interruption provides a helpful response instead of waiting for the user’s complete utterance.The example begins from an unfinished user question and transitions directly to an assistant response.
- 2. Interrupt Turn-Taking (ITT): Polite interruption is presented as a proactive alternative to waiting when the conversational intent is sufficiently clear.The passage links continued user speech, intent prediction, and interruption in one behavior.
3. BackChannel
Backchannels are brief acknowledgments used while the user is speaking to encourage continuation without taking the conversational turn.
- When the user needs encouragement to continue, the assistant gives a brief acknowledgment without taking the turn.
- The backchannel action supports continued user speech rather than producing a substantive response.
- Backchannel examples include “Uh-huh”, “I see”, “Right”, “Mm-hmm”, and “Go on”.
4. Wait
The assistant waits when no response is currently needed and selects among waiting, backchanneling, and substantive responses based on conversational context and intention completeness.
- When no response is needed, the assistant stays silent and outputs <wait>.
- The decision should consider conversation history, speech completeness, and the most appropriate action for smooth flow.
- Backchannels should remain short, while full responses should provide informative content.
- Incomplete or unclear intentions call for <wait>, while simple completed facts or updates generally call for a brief <backchannel>.
- Examples distinguish simple facts such as making dinner or commenting on rain from experiences or extended situations that invite fuller responses.
- Completed intentions requiring answers, corrections, explanations, help, or discussion call for a substantive <response>.
E Prompt for Interaction Evaluation
The evaluation protocol assesses proactive full-duplex interactions under real-time streaming constraints by separating interruption timing from response content quality.
- The protocol evaluates full-duplex speech interaction using dialogue history as input and a structured analysis-and-judgment output.
- Real-time streaming with low-latency constraints is the protocol’s operational mode.
- The assistant is programmed for proactive responses, and interruptions are valid when intent is discernible or immediate correction is required.
- Temporal precision examines whether preempting the user’s speech was justified, accounting for truncated input caused by system cut-off.
- Metric B [Content] scores 1 for a contextually relevant and accurate response and 0 otherwise, under semantic-alignment evaluation.
- Metric A [Timing] scores 1 for a timely, well-placed interjection and 0 otherwise.