Source-linked AI summary
FireRedChat: A Pluggable, Full-Duplex Voice Interaction System with Cascaded and Semi-Cascaded Implementations
Junjie Chen, Yao Hu, Junjie Li, Kangyue Li, Kun Liu, Wenpeng Li, Xu Li, Ziyuan Li, Feiyu Shen, Xu Tang, Manzhen Wei, Yichen Wu, Fenglong Xie, Kaituo Xu, Kun Xie
TL;DR
Full-duplex voice systems need controllable, low-latency interaction without the limited controllability of end-to-end designs or the incomplete openness of prior modular frameworks. FireRedChat provides a complete pluggable system with personalized VAD, semantic end-of-turn detection, heterogeneous interaction pipelines, and dialogue management. It reduces false interruptions, improves semantic stopping, and approaches industrial latency while supporting richer semi-cascaded responses.
Problem
Existing full-duplex approaches trade controllability or holistic optimization, motivating a complete modular system with internally provided components.
Source: §1 · p. 2Method
FireRedChat combines a turn-taking controller, interaction module, and dialogue manager to upgrade cascaded, semi-cascaded, or speech-to-speech pipelines to full duplex.
Source: §1 · pp. 2–3, §4 · p. 9Results
The system reduces false interruptions, improves semantic end-of-turn detection, and approaches industrial latency, with P50 gaps of 0.3 s and P95 gaps of 0.6 s relative to DouBao.
Source: §3.1 · p. 7, §3.2 · p. 8, §3.3 · p. 9Takeaways & Limitations
Abstract
from arXiv · showhide
Full-duplex voice interaction allows users and agents to speak simultaneously with controllable barge-in, enabling lifelike assistants and customer service. Existing solutions are either end-to-end, difficult to design and hard to control, or modular pipelines governed by turn-taking controllers that ease upgrades and per-module optimization; however, prior modular frameworks depend on non-open components and external providers, limiting holistic optimization. In this work, we present a complete, practical full-duplex voice interaction system comprising a turn-taking controller, an interaction module, and a dialogue manager. The controller integrates streaming personalized VAD (pVAD) to suppress false barge-ins from noise and non-primary speakers, precisely timestamp primary-speaker segments, and explicitly enable primary-speaker barge-ins; a semantic end-of-turn detector improves stop decisions. It upgrades heterogeneous half-duplex pipelines, cascaded, semi-cascaded, and speech-to-speech, to full duplex. Using internal models, we implement cascaded and semi-cascaded variants; the semi-cascaded one captures emotional and paralinguistic cues, yields more coherent responses, lowers latency and error propagation, and improves robustness. A dialogue manager extends capabilities via tool invocation and context management. We also propose three system-level metrics, barge-in, end-of-turn detection accuracy, and end-to-end latency, to assess naturalness, control accuracy, and efficiency. Experiments show fewer false interruptions, more accurate semantic ends, and lower latency approaching industrial systems, enabling robust, natural, real-time full-duplex interaction. Demos: https://fireredteam.github.io/demos/firered_chat.
1 Introduction
Full-duplex voice interaction enables concurrent speaking and barge-in, but existing end-to-end systems are difficult to control while modular frameworks may depend on non-open components and external providers. FireRedChat addresses this gap with a complete pluggable system combining turn-taking control, interaction modules, and dialogue management.
- Full-duplex interaction allows users and agents to speak concurrently, including user barge-in during agent speech.Source: §1 · p. 1
- End-to-end full-duplex systems offer limited controllability and require architectural and training changes for duplex control.Source: §1 · p. 2
- Pluggable control modules can upgrade half-duplex systems while preserving modular component optimization and replacement.Source: §1 · p. 2
- Prior modular frameworks may include non-open components and rely on external providers, preventing complete end-to-end optimization strategies.Source: §1 · p. 2
- FireRedChat provides a complete deployable system with a turn-taking controller, interaction module, and dialogue manager.Source: §1 · p. 2, Figure 1 · p. 3
- The controller combines streaming pVAD and semantic EoT detection, while cascaded and semi-cascaded implementations support richer audio-aware interaction and system-level evaluation.Source: §1 · p. 3
2 FireRedChat
FireRedChat combines a pluggable controller, interaction pipelines, and dialogue management to turn heterogeneous half-duplex systems into full-duplex voice interaction. Its workflow uses personalized streaming VAD and semantic end-of-turn detection, while semi-cascaded components preserve paralinguistic information for more coherent responses.
2 FireRedChat
- FireRedChat integrates a turn-taking controller, interaction module, and dialogue manager supporting tool invocation and context management.Source: §2 · pp. 3–4
2.1 Workflow
- The workflow uses pVAD timestamps on original audio, semantic EoT decisions, dialogue-manager tool calls, and cascaded or semi-cascaded response generation.Source: §2.1 · p. 4
2.2.1 Streaming Personalized VAD
- Streaming pVAD conditions on a target-speaker embedding and processes 10ms chunks causally, trained with 2000 hours of Mandarin and English speech mixtures.Source: §2.2.1 · p. 5
2.2.2 End-of-Turn Detection
- The EoT module distinguishes semantic continue versus stop decisions from accumulated ASR hypotheses, rather than treating acoustic silence as completion.Source: §2.2.2 · p. 5
2.3.1 Cascade Implementation
- The cascaded configuration uses FireRedASR, Qwen2.5, and FireRedTTS-1s, benefiting from established deployment and optimization practices.Source: §2.3.1 · p. 5
2.3.2 Semi-cascade implementation
- The semi-cascaded configuration uses AudioLLM and FireRedTTS-2 to retain emotional, acoustic, and audio-event cues across understanding and synthesis.It is intended to reduce error propagation and potentially lower latency while allowing independent optimization of understanding and generation.Source: §2.3.2 · pp. 5–6
- FireRedTTS-2 conditions on user speech during synthesis, aligning response style with the same semantic and acoustic cues used by AudioLLM.AudioLLM is trained from scratch on broad audio-understanding and dialogue data, while FireRedTTS-2 is trained on large-scale speech and dialogue corpora.Source: §2.3.2 · p. 6
3 Evaluation
The evaluation measures barge-in control, semantic end-of-turn detection, and client-side response latency. Results show fewer false interruptions, stronger end-of-turn accuracy, and latency closer to industrial systems.
3.1 Barge-In
- Barge-in evaluation reports T90, the minimum latency for 90% accuracy, and false barge-in rate under noise and competing-speaker interference.The test set contains 1,000 Chinese and 1,000 English utterances, with controlled noise and secondary-speaker conditions.Source: §3.1 · p. 7, Table 2 · p. 8
- The system reduces false barge-ins from interfering speakers and background noise while still responding promptly to primary-speaker interruptions.Compared with LiveKit, its T90 is roughly 30 ms higher, trading a modest additional wait for fewer false interruptions.Source: §3.1 · pp. 7–8, Table 2 · p. 8
3.2 End-of-Turn Detection
- The model outperforms LiveKit and matches Ten’s 7B-parameter model on Chinese and English end-of-turn detection at comparable parameter scale.The evaluation uses the official TEN_Turn_Detection test set and formulates stopping as a binary stop-versus-continue decision.Source: §3.2 · p. 8, Table 3 · p. 8
3.3 Latency
- Latency is the elapsed time from the end of the user’s input audio to the client’s receipt of the first system-produced audio chunk.P50 latency is reported as a proxy for typical user experience.Source: §3.3 · p. 8
- 1.2 s and 1.0 s lower P50 latency than LiveKit and Ten, respectively, while remaining only 0.3 s above DouBao.P95 latency is lower by 1.6 s and 0.8 s than LiveKit and Ten, respectively, and 0.6 s above DouBao.Source: §3.3 · p. 9, Table 4 · p. 9
4 Conclusions
The paper presents a deployable full-duplex system built from a turn-taking controller, interaction module, and dialogue manager. Its pVAD and semantic EoT detector improve interruption control, while cascaded and semi-cascaded implementations support broader functionality and more natural real-time interaction.
4 Conclusions
- The complete system combines a dedicated turn-taking controller, interaction module, and dialogue manager for deployable full-duplex voice interaction.The dialogue manager supports tool invocation and context management.Source: §4 · p. 9
- Streaming pVAD suppresses false interruptions, detects primary-speaker barge-ins, and localizes primary-speaker segments, while EoT detects semantic turn completion.These controls upgrade cascaded, semi-cascaded, and speech-to-speech half-duplex pipelines to full duplex.Source: §4 · p. 9
- The semi-cascaded implementation captures emotional and paralinguistic cues, improves contextual coherence and robustness, and lowers latency and error propagation.The cascaded design retains mature deployment and optimization practices.Source: §4 · p. 9
- The proposed interruption, end-of-turn, and latency metrics assess interaction naturalness, control accuracy, and efficiency.Experiments report fewer spurious interruptions, more precise semantic end-of-turn detection, and a narrower latency gap to industrial-grade applications.Source: §4 · p. 9