Source-linked AI summary
DuplexCascade: Full-Duplex Speech-to-Speech Dialogue with VAD-Free Cascaded ASR-LLM-TTS Pipeline and Micro-Turn Optimization
Jianing Yang, Yusuke Fujita, Yui Sudo
TL;DR
Cascaded dialogue systems retain text-LLM intelligence but VAD can impose brittle half-duplex turn-taking, while VAD-free end-to-end models can lose conversational intelligence. DuplexCascade streams ASR text in micro-turns and uses conversational tokens to guide an LLM-centered cascade. It reports strong turn-taking and VoiceBench conversational intelligence, using simulated duplex training sequences because annotated real-world corpora are scarce.
Problem
Cascaded systems’ VAD-based turn segmentation often produces brittle half-duplex interaction, while VAD-free end-to-end systems can have weaker conversational intelligence than strong text LLMs.
Source: §1 · p. 1Method
DuplexCascade streams ASR text as micro-turns and uses conversational control tokens to guide an LLM-centered, VAD-free speech-to-speech cascade.
Source: §1 · p. 1, §3.1 · p. 2, §3.2 · p. 2Results
DuplexCascade achieves the best Averaged Turn-Taking Accuracy among evaluated open systems and substantially outperforms prior duplex systems across nearly all VoiceBench dimensions.
Source: §4.2 · p. 4, §4.3 · p. 4Takeaways & Limitations
The authors conclude that a modular cascade can support VAD-free full-duplex interaction while largely preserving the strengths of modern text LLMs.
Source: §5 · p. 4Annotated real full-duplex dialogue corpora are scarce, so training sequences are simulated from 50k UltraChat dialogues.
Source: §3.3 · p. 3
Abstract
from arXiv · showhide
Spoken dialog systems with cascaded ASR-LLM-TTS modules retain strong LLM intelligence, but VAD segmentation often forces half-duplex turns and brittle control. On the other hand, VAD-free end-to-end model support full-duplex interaction but is hard to maintain conversational intelligence. In this paper, we present DuplexCascade, a VAD-free cascaded streaming pipeline for full-duplex speech-to-speech dialogue. Our key idea is to convert conventional utterance-wise long turns into chunk-wise micro-turn interactions, enabling rapid bidirectional exchange while preserving the strengths of a capable text LLM. To reliably coordinate turn-taking and response timing, we introduce a set of conversational special control tokens that steer the LLM's behavior under streaming constraints. On Full-DuplexBench and VoiceBench, DuplexCascade delivers state-of-the-art full-duplex turn-taking and strong conversational intelligence among open-source speech-to-speech dialogue systems.
1. Introduction
Cascaded speech systems retain the reasoning and instruction-following strengths of text LLMs, but VAD often makes their turn-taking brittle; VAD-free end-to-end models support duplex interaction but can lose conversational intelligence. DuplexCascade uses a VAD-free cascade with micro-turns and conversational tokens, reporting strong turn-taking and conversational intelligence.
1. Introduction
- Cascaded ASR-LLM-TTS systems inherit text LLM reasoning strengths [4], yet external VAD often forces half-duplex turns and brittle control under pauses, overlaps, and noise.Source: §1 · p. 1
- VAD-free end-to-end speech–text models support concurrent listening and speaking, but their conversational intelligence can degrade relative to strong text LLMs [12].Source: §1 · p. 1
- DuplexCascade converts utterance-wise turns into micro-turns and uses conversational tokens to regulate an LLM-centered cascade for VAD-free full-duplex dialogue.The authors report strong Full-Duplex-Bench performance and VoiceBench conversational intelligence after lightweight LoRA adaptation on 50k text dialogues.Source: §1 · p. 1
2. Related Works
Prior cascaded systems commonly rely on VAD endpointing, while VAD-free time-division approaches divide interaction into short alternating segments. The cited prior approach does not explicitly control turn-taking; DuplexCascade’s overview instead combines periodic text micro-turns with conversational control tokens.
2.1. VAD-Based Turn Taking Control
- VAD endpointing in cascaded systems struggles to use conversational semantics and intent, contributing to unstable interruptions and missed responses or backchannels.The cited account identifies VAD errors as a dominant turn-taking failure mode in systems such as Freeze-Omni.Source: §2.1 · p. 1
2.2. VAD-Free Time-Division-Multiplexing Approach Toward Full-Duplex Interaction
- MiniCPM-Duplex divides interaction into time slices for immediate input-output processing, but does not explicitly model turn-taking decisions.DuplexCascade’s overview instead flushes ASR text into 0.6-second micro-turns and has the LLM generate control tokens with each response.Source: §2.2 · p. 1, Figure 1 · p. 2
3. DuplexCascade
DuplexCascade streams partial ASR text to an LLM in interleaved micro-turns, where special tokens guide turn-taking and response flow. Its training sequences are dynamically simulated from text dialogues by segmenting turns and adding patterns for pauses, interruptions, and backchannels.
3.1. Overview
- Streaming ASR partials are periodically aggregated into micro-turns for an LLM, whose generated micro-turns and control tokens are synthesized incrementally by streaming TTS.The example flush interval is 0.6 seconds; micro-turns can contain control tokens rather than speech content.Source: §3.1 · p. 2
3.2. Conversational Special Tokens
- Conversational special tokens make the LLM’s turn-taking decisions controllable under streaming constraints.Source: §3.2 · p. 2
3.2.1. User’s special token. •
- <no voice> represents an interval in which the buffer is flushed but no user text is recognized.Source: §3.2.1 · p. 2
3.2.2. System’s special tokens. •
- System tokens signal whether to wait, respond, stop for an interruption, ignore a user backchannel, or emit a system backchannel.The token for a user who is still speaking ends the system micro-turn immediately; a system-backchannel token triggers a pre-synthesized clip.Source: §3.2.2 · p. 2
3.3. Dynamic Construction of Duplex Training Data
- Because annotated real full-duplex dialogue corpora are scarce, the authors dynamically construct duplex training sequences from 50k sampled UltraChat dialogues by simulating six interaction phenomena.Figure 2 depicts the proposed construction pipeline.Source: Figure 2 · p. 2, §3.3 · p. 3
3.3.1. Convert text dialogue into micro-turn segmentation
- Training data splits user and system long turns into interleaved micro-turns, using speaking, silence, and turn-finish markers to represent the dialogue state.LoRA next-token prediction is applied only to system micro-turns to help preserve the backbone LLM’s conversational intelligence.Source: §3.3.1 · p. 3
3.3.2. Simulating duplex interaction phenomena
- The simulation varies user micro-turn lengths, inserts pauses and interruptions, and represents user and system backchannels and user thinking.User chunks are sampled at 1–7 tokens and system chunks fixed at 10; pauses occur with probability 0.10, interruptions with 0.30, and user backchannels with 0.01 per system micro-turn.Source: §3.3.2 · p. 3
4. Experiments
DuplexCascade performs strongly on Full-Duplex-Bench turn-taking and VoiceBench conversational-intelligence evaluations. The experiments also examine training variants and the accuracy–latency trade-off associated with micro-turn duration.
4.1. Implementation Details
- The implementation uses streaming DSM-ASR and DSM-TTS with Qwen2-7B-Instruct adapted using LoRA, and trains variants with and without system-backchannel supervision.Training uses LoRA rank 16 and α=32; the models are fine-tuned for 5k steps on 8 H100 GPUs for 5 hours.Source: §4.1 · pp. 3–4
4.2. Full-Duplex-Bench Results
- DuplexCascade achieves the best Averaged Turn-Taking Accuracy among evaluated open systems, while DuplexCascade-β ranks second on backchannel metrics ICC Freq and JSD.The aggregate is an unweighted mean of accuracies across TOR-reported subsets, reversing TOR for subsets where lower TOR is better.Source: §4.2 · p. 4
4.3. VoiceBench Results
- The models outperform prior duplex systems across nearly all VoiceBench dimensions, while scoring competitively with the naive DSM-ASR+Qwen2-7B-Instruct pipeline despite a moderate gap.DuplexCascade-β also demonstrates backchannel generation while preserving core LLM capabilities.Source: §4.3 · p. 4, Table 2 · p. 4
4.4. Micro-Turn ∆t Analysis
- A 1.2 s micro-turn duration gives the strongest turn-taking performance in the simulation, but 0.6 s is selected as the practical accuracy–latency trade-off.Accuracy improves through 1.2 s and then degrades, while latency rises monotonically as duration increases; Figure 3 reports the duration effect.Source: §4.4 · p. 4, Figure 3 · p. 4
5. Conclusion
DuplexCascade is a cascaded full-duplex speech-to-speech pipeline that uses lightweight text-only LoRA adaptation to support VAD-free turn-taking while largely preserving the underlying LLM’s conversational intelligence.
5. Conclusion
- DuplexCascade uses lightweight LoRA fine-tuning on a small amount of text-only dialogue to support VAD-free, robust turn-taking while largely preserving the underlying LLM’s conversational intelligence.The conclusion argues that full-duplex interaction can be enabled within a modular cascade without sacrificing modern text LLM strengths.Source: §5 · p. 4