Source-linked AI summary
SoulX-Duplug: Plug-and-Play Streaming State Prediction Module for Realtime Full-Duplex Speech Conversation
Ruiqi Yan, Wenxi Chen, Zhanxun Liu, Ziyang Ma, Haopeng Lin, Hanlin Wen, Hanke Xie, Jun Wu, Yuzhe Liang, Yuxiang Zhao, Pengchao Feng, Jiale Qian, Hao Meng, Yuhang Dai, Shunshun Yin, Ming Tao, Lei Xie, Kai Yu, Xinsheng Wang, Xie Chen
TL;DR
Full-duplex dialogue systems need controllable, low-latency interaction management despite scarce data, semantic-blind cascades, and limited bilingual evaluation. SoulX-Duplug unifies streaming ASR and state prediction as a plug-and-play module, while SoulX-Duplug-Eval extends benchmarking bilingually; the resulting system reports the best overall turn-management and latency performance.
Problem
Existing full-duplex systems face entangled turn-taking control, scarce conversational data, latency from cascaded components, and limited bilingual benchmarks.
Method
SoulX-Duplug is a plug-and-play streaming module that jointly performs ASR and dialogue-state prediction, using textual information for semantic turn detection.
Results
The SoulX-Duplug-based system achieves the best overall performance in turn management and latency-related metrics on Full-Duplex-Bench, with low latency close to its theoretical bound.
Takeaways & Limitations
SoulX-Duplug provides low-latency semantic-aware dialogue management for modular full-duplex systems and SoulX-Duplug-Eval supports comparable cross-lingual benchmarking.
Takeaways & Limitations
Very small streaming chunks can fragment speech units and cause unavoidable prediction fluctuation, while incremental decoding limits future context for LLM-based ASR.
Abstract
from arXiv · showhide
Recent advances in spoken dialogue systems have brought increased attention to human-like full-duplex voice interactions. However, our comprehensive review of this field reveals several challenges, including the difficulty in obtaining training data, catastrophic forgetting, and limited scalability. In this work, we propose SoulX-Duplug, a plug-and-play streaming state prediction module for full-duplex spoken dialogue systems. By jointly performing streaming ASR, SoulX-Duplug explicitly leverages textual information to identify user intent, effectively serving as a semantic VAD. To promote fair evaluation, we introduce SoulX-Duplug-Eval, extending widely used benchmarks with improved bilingual coverage. Experimental results show that SoulX-Duplug enables low-latency streaming dialogue state control, and the system built upon it outperforms existing full-duplex models in overall turn management and latency performance. We have open-sourced SoulX-Duplug and SoulX-Duplug-Eval.
1. Introduction
Full-duplex systems support simultaneous listening and speaking, but existing approaches face entangled control, scarce data, semantic-blind cascades, latency, and limited bilingual evaluation. SoulX-Duplug addresses these gaps with a plug-in streaming state prediction framework and bilingual benchmark.
- 1. Introduction: SoulX-Duplug equips half-duplex spoken dialogue models with full-duplex interaction capability without modifying their backbone architectures.Full-duplex interaction supports interruption handling, pause handling, and dynamic turn-taking.
- 1. Introduction: End-to-end full-duplex systems entangle turn-taking policy with language modeling, limiting controllability and interpretability while scarce conversational data constrains generalization and scaling.
- 1. Introduction: Cascaded VAD–ASR–turn-detection systems add latency and lack semantic information because conventional VAD relies primarily on acoustic features and components may be non-streaming.
- 1. Introduction: Existing evaluations rely largely on self-constructed test sets, while bilingual or multilingual public benchmarks remain scarce.These gaps hinder fair comparison and cross-lingual analysis.
- 1. Introduction: SoulX-Duplug introduces unified streaming state prediction for modular full-duplex systems alongside SoulX-Duplug-Eval, a bilingual evaluation benchmark.
FD-SDSs
The paper presents SoulX-Duplug as a unified streaming module that combines semantic guidance with dialogue-state control. Experiments report low latency and strong overall turn-management and latency performance.
- FD-SDSs: Joint chunk-based streaming ASR and state prediction uses textual information for semantic-level turn detection while maintaining low latency.This design functions effectively as a semantic VAD module.
- FD-SDSs: SoulX-Duplug-based systems achieve the best overall performance in turn management and latency across Full-Duplex-Bench and SoulX-Duplug-Eval.The results also indicate low latency close to the theoretical bound.
- FD-SDSs: The authors release SoulX-Duplug and SoulX-Duplug-Eval to support future development and more standardized evaluation.
- FD-SDSs: SoulX-Duplug unifies VAD, ASR, and turn detection as a bilingual streaming state prediction module for modular full-duplex systems.The module is intended to provide semantically informed, low-latency dialogue state control.
2. Related Work
Related work spans end-to-end and modular full-duplex systems, state-prediction datasets, conversational corpora, and evaluation benchmarks. Existing resources and pipelines remain limited by semantic awareness, streaming latency, scale, domain diversity, and language coverage.
- 2.1. Full-Duplex SDMs: End-to-end models jointly process speech understanding and generation, whereas modular systems separate duplex state prediction from half-duplex dialogue modeling.Modular architectures provide improved engineering controllability and deployment flexibility.
- 2.1. Full-Duplex SDMs: Existing modular pipelines commonly combine VAD, ASR, and text-based turn detection, but acoustic VAD lacks semantic awareness and non-streaming components add latency.
- 2.2. Data for Full-Duplex SDMs: Public state-prediction datasets provide fine-grained utterance-level dialogue-state or user-intention labels, including interruption and backchannel categories.
- 2.2. Data for Full-Duplex SDMs: Available conversational corpora are limited in scale or domain diversity; one dual-stream Chinese-English resource contains only 15 hours, while general-knowledge and complex-context data remain scarce.
- 2.3. Evaluation of Full-Duplex SDMs: Full-duplex benchmarks generally evaluate when systems should speak and when they should stop after interruptions, but existing coverage varies across scenarios and languages.Full-Duplex-Bench covers multiple scenarios but its test sets are limited to English.
- 2.3. Evaluation of Full-Duplex SDMs: SoulX-Duplug produces interleaved ASR outputs and dialogue-state tokens from user-side speech tokens within a unified streaming module.
3. SoulX-Duplug
SoulX-Duplug is a unified streaming module that integrates VAD, ASR, and dialogue-state prediction for modular full-duplex speech systems. It uses interleaved ASR and state prediction with defined state tokens, streaming audio processing, and staged training.
- 3.1. Overview: SoulX-Duplug integrates VAD, ASR, and dialogue-state prediction in one streaming module for real-time full-duplex dialogue.
- 3.2. State Token Design: Five state tokens represent idle, meaningful speech, backchannel, complete utterances, and incomplete utterances during interaction.
- 3.3. Speech Input Modeling: Audio is processed in 160 ms chunks using discrete speech tokens and a 1160 ms receptive field with 40 ms look-ahead.
- 3.4. Text-Guided Streaming State Prediction: The model interleaves streaming ASR output with dialogue-state prediction, providing semantic guidance while preserving streaming inference.
- 3.5. Training Objective: A token-type-weighted cross-entropy objective balances optimization across text, ASR end-of-sequence, and state-token categories.
- 3.6. Training and Inference: Training proceeds through non-streaming ASR pretraining, streaming ASR adaptation, and state-prediction fine-tuning, while inference uses chunk-wise decisions after subsequent audio arrives.
4. SoulX-Duplug-Eval
SoulX-Duplug-Eval extends existing full-duplex benchmarks with complementary English and Chinese test sets. The resulting resources cover duplex state prediction and system-level dialogue evaluation for more consistent bilingual comparison.
- Benchmark Construction: SoulX-Duplug-Eval adds bilingual resources for both dialogue-state prediction and system-level full-duplex dialogue evaluation.
- English Evaluation: Easy Turn testset-En provides English Complete and Incomplete categories containing 318 and 299 samples, respectively.
- Chinese Evaluation: Full-Duplex-Bench-Zh is a Chinese counterpart covering Turn-Taking, Pause Handling, User Backchannel, and User Interruption scenarios.
- Benchmark Construction: Together, the datasets improve cross-lingual coverage and support standardized, comparable experimental analysis.
5. Experimental Setup
The experiments train SoulX-Duplug on large bilingual ASR resources and conversational state-prediction data, then evaluate it in simulated online streaming. A modular full-duplex system is tested across turn-taking, pause handling, interruption, and backchannel tasks.
- 5.1. Training Details: ASR training uses approximately 47,000 hours of Mandarin data and 31,000 hours of English data with chunk-based alignments for streaming adaptation.
- 5.1. Training Details: Training uses three sequential phases: non-streaming ASR pretraining, streaming ASR adaptation, and state-prediction supervised fine-tuning.
- 5.1. Training Details: State prediction uses Fisher for English and a ten-thousand-hour-level in-house Mandarin corpus with alignment, filtering, and noise augmentation.
- 5.1. Training Details: Inference applies teacher-forced ASR guidance using Paraformer for Mandarin and SenseVoice Small for English in simulated online streaming on one NVIDIA L20 GPU.
- 5.2. Evaluation Setup: The modular evaluation system combines Qwen2.5-7B-Instruct with IndexTTS-1.54 and tests pause handling, turn taking, user interruption, and user backchannel.
- 5.2. Evaluation Setup: Overall turn management averages task-specific success indicators, while overall latency averages response and stop latencies across tasks.
6. Experimental Results
SoulX-Duplug delivers low-latency, semantically informed state prediction and strong full-duplex turn management across bilingual evaluations. Ablations indicate that ASR pretraining and externally guided textual supervision support its state prediction, while streaming deployment has practical latency advantages.
- 6.1.1. Main Results on Bilingual Full-Duplex-Bench: The SoulX-Duplug system achieves the best overall turn-management performance across evaluation dimensions while maintaining balanced metric scores.The system attains the highest average turn-management score without abnormally low values on any individual metric.
- 6.1.1. Main Results on Bilingual Full-Duplex-Bench: SoulX-Duplug reaches a 0.933 Turn Taking TOR while substantially reducing inappropriate interruptions relative to continuous-output models.It outperforms Freeze-Omni across three related metrics in Chinese and English, and improves Turn Taking TOR over Gemini by more than 40%.
- 6.1.1. Main Results on Bilingual Full-Duplex-Bench: SoulX-Duplug balances interruption responsiveness and backchannel handling with 0.45 s stop latency and 0.74 Resume Rate.Models with higher Resume Rates often show stop latency above 1 s, whereas GPT-4o reaches 0.23 s with a lower Resume Rate of 0.7.
- 6.1.2. Comparison with Non-Streaming State Prediction Modules: Streaming SoulX-Duplug is about 3% less accurate than a non-streaming ASR-plus-TEN pipeline but provides lower and more stable practical latency.Non-streaming approaches require VAD-based segmentation that can add several hundred milliseconds, including a reported 500 ms VAD latency.
- 6.1.3. Comparison Across Streaming State Prediction Modules: SoulX-Duplug averages 250 ms practical latency across English and Chinese, close to its 240 ms theoretical latency.The low latency is attributed to small chunks and lightweight components; teacher-forced inference does not add measurable delay.
- 6.2. Ablation Study: Removing ASR pretraining or external guided ASR decreases state prediction accuracy, supporting the role of explicit textual supervision.ASR pretraining improves semantic representations, while external ASR outputs stabilize semantic interpretation under extremely short chunks.
- 6.3. Further Discussion: Very small streaming chunks remain challenging because they cross phoneme, syllable, and word boundaries, causing unavoidable prediction fluctuation.The constraint is particularly pronounced in English, while incremental decoding and limited future context also restrict LLM-based ASR.
7. Conclusion
The paper introduces SoulX-Duplug as a pluggable streaming state prediction module for real-time full-duplex speech conversation and releases SoulX-Duplug-Eval for comparable bilingual benchmarking. The resulting system achieves strong turn-management and latency performance, while additional experiments support the proposed approaches.
- 7. Conclusion: SoulX-Duplug provides low-latency semantic-aware streaming dialogue management and yields the best overall turn-management and latency-related performance on Full-Duplex-Bench.The authors also release supplementary Easy Turn and Full-Duplex-Bench evaluation sets for more comparable cross-lingual benchmarking.
- 7. Conclusion: SoulX-Duplug and SoulX-Duplug-Eval are open-sourced to support continued progress in real-time spoken dialogue modeling.