Source-linked AI summary
VoiceChat-TTS: A Low-Latency Continuous Speech Synthesis Model for Interactive Agents
Edresson Casanova, Jaehyeon Kim, Mariana Graterol Fuenmayor, Shehzeen Hussain, Viacheslav Klimkov, Valentin Mendelev, Mikyas Desta, Paarth Neekhara, Piotr Zelasko, Chen Chen, Elena Rastorgueva, Ke Hu, Ankita Pasad, Xuesong Yang, Aya Alja'fari, Rajarshi Roy, Rohan Badlani, Jason Roche, Jason Li, Zhehuai Chen
TL;DR
Existing speech language models are often turn-based and struggle with real-time adaptability while duplex systems can compromise speech quality. VoiceChat-TTS directly converts LLM text streams into continuous, interruptible speech, reducing WER from 4.51% to 2.00% and increasing Squim-MOS from 3.60 to 4.38 versus Audio Flamingo 3-Chat.
Problem
Most speech language models lack real-time adaptability, while duplex models often degrade speech quality when jointly optimizing recognition, interruption handling, and synthesis.
Method
VoiceChat-TTS generates continuous, streamable speech directly from LLM text-token streams, using control tokens for interruption and silence output when no text is available.
Results
WER falls from 4.51% to 2.00% and Squim-MOS rises from 3.60 to 4.38 versus Audio Flamingo 3-Chat, with comparable speaker similarity.
Takeaways & Limitations
VoiceChat-TTS provides interactive agents with continuous, interruption-aware speech generation while maintaining competitive intelligibility and predicted overall speech quality.
Takeaways & Limitations
The model lacks user-audio conditioning, preventing dynamic prosody adaptation to acoustic cues from the user.
Abstract
from arXiv · showhide
Spoken dialogue is a natural form of human--computer interaction, yet most speech language models remain limited to turn-based operation and lack real-time adaptability, such as user barge-in. Recent duplex speech-to-speech and speech-to-text models reduce latency by replacing multi-stage pipelines, but often compromise speech quality because accurate ASR, interruption handling, and high-fidelity synthesis must be optimized jointly. We propose VoiceChat-TTS, a low-latency, continuous, and streamable text-to-speech model for interactive agents. VoiceChat-TTS is driven directly by LLM text-token streams, supports explicit interruption via control tokens, and produces silence when no textual input is available. The model enables always-on, responsive speech generation while preserving modularity and high speech quality, and it supports mid-utterance interruptions without resetting the KV cache.
1. Introduction
VoiceChat-TTS addresses the limitations of turn-based and end-to-end duplex speech systems with a continuous, streamable TTS model for interactive agents. It directly consumes LLM text-token streams, supports interruption control, generates silence without text, and preserves modularity and synthesis quality.
- Motivation: Most speech language models remain turn-based and lack real-time adaptability such as user barge-in, while duplex systems can degrade speech quality when jointly optimizing ASR, interruption handling, and synthesis.Duplex speech-to-speech and speech-to-text models reduce latency and simplify deployment by replacing traditional multi-stage pipelines, but their joint optimization creates data and modeling challenges.
- Related work: Prior streaming decoders progressively generate speech from streamable text and reduce time-to-first-audio, but primarily handle streaming within a single response.Audio Flamingo 3-Chat demonstrated that high-quality neural speech synthesis can be integrated into interactive speech-language systems.
- Contribution: VoiceChat-TTS extends streaming speech decoders into continuous, always-on operation that generates silence without agent text and stops promptly during user barge-in.The model is designed to remain active across conversational time rather than only during a single response.
- Contribution: VoiceChat-TTS directly consumes LLM text-token streams and uses explicit control tokens to interrupt ongoing speech and transition output to silence during mid-utterance barge-ins.This design enables always-on, responsive speech generation while preserving modularity and high synthesis quality.
- Contribution: The proposed training strategy combines high-quality single-turn TTS data with complex multi-turn conversational data while minimizing distribution mismatch between the settings.The paper reports competitive speech quality relative to strong offline and streaming baselines while meeting latency and interruption-handling requirements.
2. VoiceChat-TTS Model
VoiceChat-TTS extends a streaming speech decoder for full-duplex interaction with causal audio compression, incremental text processing, character-aware embeddings, and accelerated RVQ generation. It also adds explicit conversational boundary and speaker-conditioning mechanisms while comprising 977M parameters.
- Architecture: The architecture builds on Audio Flamingo 3-Chat’s streaming speech decoder, using incrementally processed text and streamable speech outputs for low-latency, full-duplex interaction.The decoder is modified to support full-duplex interactions while retaining streamable inputs and outputs.
- Audio Codec: The causal audio codec compresses 22 kHz waveforms into 31-codebook RVQ tokens at 12.5 Hz, with each frame representing an 80 ms waveform chunk.Its frame rate is compatible with recent duplex speech-to-speech models.
- Text Representation: The text pipeline uses the Nemotron Nano 2 subword tokenizer with BOS and interruption tokens, while a character-aware encoder improves generalization to rare or unseen subwords.On LibriTTS test-clean, text tokens arrive at 4.16 Hz, approximately one token per three acoustic-token frames.
- RVQ Generation: The Mixture of Gaussian head accelerates deep RVQ generation by iteratively estimating masked continuous embeddings and quantizing them into progressively unmasked discrete tokens.This replaces a 31-step autoregressive decoding pipeline with iterative refinement.
- Conditioning and Fusion: The model adds audio prompt conditioning, boundary embeddings, and gated fusion to improve speaker consistency, conversational turn-taking, and mixed-precision stability.The boundary embeddings explicitly represent BOS and interruption events, while gated fusion mitigates instability from high-magnitude RVQ embeddings.
- Model Size: The final VoiceChat-TTS model contains 977M parameters, comprising a 778M-parameter Gemma 3-based streaming TTS module and a 199M-parameter codec model.The parameter total includes both the streaming TTS module and codec model.
3. Experiments
VoiceChat-TTS is evaluated on speech quality, interaction robustness, interruption handling, and latency against offline, streaming, and speech-to-speech baselines. It improves intelligibility and quality while maintaining stable multi-turn performance, rapid interruption response, and lower generation latency, with speaker consistency remaining challenging for unseen speakers.
- Speech quality: VoiceChat-TTS reduces one-turn WER from 4.51% to 2.00% and increases Squim-MOS from 3.60 to 4.38 versus Audio Flamingo 3-Chat, with comparable speaker similarity.The comparison uses Audio Flamingo 3-Chat, the closest architectural baseline.
- Speech quality: In unseen-speaker evaluation, Squim-MOS remains approximately 4.38 across turn counts, while WER varies only from 2.00% to 2.20% as turns increase from one to four.For seen speakers, the reported metrics are similarly stable across multi-turn generation.
- Limitations: SECS for unseen speakers decreases from 0.757 at one turn to 0.685 at four turns, indicating that extended continuous generation challenges speaker-identity preservation.The decline is associated with zero-shot or unseen-speaker settings and alternation between speech and silence states.
- Speech-to-speech comparison: Resynthesizing PersonaPlex text with VoiceChat-TTS reduces CER from 4.06% to 2.05% and WER from 5.00% to 2.42% on Smooth Turn Taking.The evaluation preserves PersonaPlex’s text-token timing and computes errors on the full, untrimmed assistant waveform, including intended silence intervals.
- Interruption handling: Force Silence increases IOR@320ms from 96.8% to 100.0%, reduces Stop Latency from 228.3 ms to 89.9 ms, and lowers Leakage@1s from 169.1 ms to 55.8 ms.AI-CER also decreases from 0.255% to 0.095%.
- Latency: At concurrency 1, VoiceChat-TTS reduces next-frame latency from 20.34 ms to 9.62 ms versus Qwen3-TTS-12Hz, corresponding to a 2.1× improvement.The measurement uses the same RTX A6000 setup; acoustic-token ITL falls from 15.44 ms to 7.16 ms.
4. Conclusions, Limitations, and Future Work
VoiceChat-TTS is a low-latency, continuous, streamable TTS architecture for interactive agents that supports interruption, silence generation, and always-on responsive speech. It preserves modularity and competitive speech quality, integrates into a duplex speech system, and remains limited by absent user-audio conditioning and component-wise ablations.
- Conclusions: VoiceChat-TTS is a low-latency, continuous, streamable TTS architecture driven directly by LLM text-token streams for interactive agents.It supports explicit mid-utterance interruptions through control tokens and generates silence when no textual input is available.
- Conclusions: The model enables always-on, responsive speech generation while preserving modularity and competitive speech quality.Its modular decoder can be integrated into NVIDIA Nemotron VoiceChat-11B, supporting low-latency duplex interaction.
- Limitations and Future Work: Absent user-audio conditioning prevents dynamic prosody adaptation to acoustic cues from the user.The paper also identifies a lack of controlled component-wise ablations; preliminary development experiments suggested cumulative gains, but systematic quantification was not provided in the passage.