Source-linked AI summary
FlashLabs Chroma 1.0: A Real-Time End-to-End Spoken Dialogue Model with Personalized Voice Cloning
Tanyu Chen, Tairan Chen, Kai Shen, Zhenghua Bao, Zhihui Zhang, Man Yuan, Yi Shi
TL;DR
Existing end-to-end spoken dialogue systems often provide limited speaker identity preservation, while low-latency interaction and personalized voice cloning remain difficult to combine. Chroma 1.0 uses an interleaved text-audio schedule and conditioned speech synthesis to address this gap. It reports improved speaker similarity, real-time performance, and strong reasoning and dialogue capabilities.
Problem
Existing end-to-end spoken dialogue systems often have limited speaker identity preservation, restricting personalized voice interaction.
Method
Chroma 1.0 combines a streaming architecture with a 1:2 interleaved text-audio schedule and reference-audio conditioning for personalized speech generation.
Results
10.96% relative improvement in speaker similarity over human baseline is reported, alongside real-time performance and strong reasoning and dialogue capabilities.
Takeaways & Limitations
Chroma combines low-latency streaming with high-fidelity voice cloning for natural multi-turn spoken interaction.
Takeaways & Limitations
The current Chroma architecture does not support batch processing, limiting the reported practical latency measurement setting.
Abstract
from arXiv · showhide
Recent end-to-end spoken dialogue systems leverage speech tokenizers and neural audio codecs to enable LLMs to operate directly on discrete speech representations. However, these models often exhibit limited speaker identity preservation, hindering personalized voice interaction. In this work, we present Chroma 1.0, the first open-source, real-time, end-to-end spoken dialogue model that achieves both low-latency interaction and high-fidelity personalized voice cloning. Chroma achieves sub-second end-to-end latency through an interleaved text-audio token schedule (1:2) that supports streaming generation, while maintaining high-quality personalized voice synthesis across multi-turn conversations. Our experimental results demonstrate that Chroma achieves a 10.96% relative improvement in speaker similarity over the human baseline, with a Real-Time Factor (RTF) of 0.43, while maintaining strong reasoning and dialogue capabilities. Our code and models are publicly available at https://github.com/FlashLabs-AI-Corp/FlashLabs-Chroma and https://huggingface.co/FlashLabs/Chroma-4B .
1 Introduction
Chroma 1.0 addresses latency, paralinguistic preservation, and personalized voice-fidelity limits in spoken dialogue systems with an open-source, real-time end-to-end design. Its contributions combine streaming generation, few-shot voice cloning, and strong dialogue capabilities.
- Motivation: Cascaded ASR–LLM–TTS systems incur latency and error propagation while often losing speaker identity, timbre, emotion, and prosody.These limitations are especially consequential in real-time conversational settings requiring naturalness, responsiveness, and speaker fidelity.
- Motivation: Existing end-to-end models often emphasize semantic alignment or text output, while voice-cloning systems lack real-time streaming and real-time models sacrifice fine-grained speaker control.The paper identifies a gap between expressive personalized synthesis and low-latency interaction.
- Chroma 1.0: Chroma 1.0 is presented as an open-source, real-time end-to-end spoken dialogue model combining low-latency interaction with high-fidelity personalized voice cloning.The system is designed to address the limitations identified for cascaded and existing end-to-end systems.
- Contributions: A streaming architecture couples speech understanding and generation through semantic state representations, enabling sub-second end-to-end latency.The interleaved text-audio token schedule supports synchronized incremental generation.
- Contributions: 10.96% relative improvement in speaker similarity over human baseline is achieved using high-fidelity voice cloning conditioned on a few seconds of reference audio.The paper also reports strong reasoning and dialogue capabilities with only 4B parameters.
2 Related Works
Related work spans cascaded speech pipelines, end-to-end speech-to-speech dialogue, and neural codec language models for voice cloning. These lines of research differ in their handling of latency, intermediate text, multilinguality, reasoning, acoustic generation, and speaker adaptation.
- Cascaded and end-to-end systems: Cascaded ASR–LLM–TTS systems remain widely deployed because of mature components and established engineering practices.They provide flexibility but accumulate latency and errors and can discard paralinguistic cues after speech is reduced to text.
- Cascaded and end-to-end systems: End-to-end S2S systems avoid explicit transcription by learning direct mappings between discrete speech units.SeamlessM4T established a multilingual S2S baseline, while GPT-4o demonstrated native audio processing without intermediate text.
- Chroma positioning: Chroma’s architecture combines the end-to-end S2S direction with a Reasoner, Backbone, and Codec Decoder for speech understanding and waveform synthesis.The cited workflow describes interleaved text-audio embeddings and acoustic-code reconstruction.
- End-to-end audio language models: Unified speech-text token systems and dual-stream architectures separate or coordinate semantic reasoning with acoustic generation.Step-Audio-2 uses retrieval-augmented generation and reinforcement learning, while Qwen2.5-Omni and Qwen3-Omni use Thinker–Talker designs.
- Voice cloning: Neural codec language models frame TTS as discrete acoustic-token generation and support natural zero-shot synthesis from approximately 3 seconds of reference audio.VALL-E, VALL-E X, StyleTTS-2, NaturalSpeech 3, Voicebox, and CosyVoice represent complementary advances in adaptation, fidelity, control, and speed.
3 Model Architecture
Chroma integrates multimodal reasoning with conditioned acoustic modeling and causal codec decoding. Its 1:2 interleaved schedule supports streaming, while a lightweight decoder generates refined codebooks for efficient waveform synthesis.
- Architecture: Chroma consists of a multimodal Reasoner, an acoustic Backbone, a Chroma Decoder, and a Codec Decoder for waveform reconstruction.The Reasoner handles comprehension and response generation, while the synthesis pipeline converts acoustic codes into speech.
- Chroma Reasoner: The Reasoner fuses text and audio through cross-modal attention and temporally aligned hidden states that capture linguistic, prosodic, and rhythmic information.These representations condition subsequent speech synthesis and contextual modeling.
- Chroma Backbone: Reference audio and its transcript are encoded into prompts that condition the Backbone on the target speaker’s acoustic characteristics.Shared token embeddings align the Reasoner’s text modality with Backbone acoustic generation.
- Streaming generation: A fixed 1:2 schedule pairs each text token with two audio codes, allowing parallel autoregressive audio generation during incremental text generation.This reduces time-to-first-token and supports real-time interaction without waiting for complete text sequences.
- Acoustic decoding: The Chroma Decoder generates remaining acoustic codes frame-synchronously from Backbone outputs, reducing long-context computation while enriching prosody and articulation.It uses the coarse code and hidden state at the current timestep to autoregressively produce the remaining RVQ levels.
- Acoustic decoding: The Codec Decoder concatenates coarse and refined codebooks and reconstructs a continuous waveform with a causal convolutional decoder.Using 8 codebooks reduces autoregressive refinement steps and improves inference efficiency.
4 Experiments
Experiments evaluate Chroma’s voice cloning, latency, and dialogue capabilities using objective metrics, subjective comparisons, and benchmark tasks. Results show high speaker similarity, sub-second responsiveness, faster-than-real-time generation, and competitive reasoning and conversation performance.
- Voice Cloning Evaluation: 10.96% relative improvement in speaker similarity over the human baseline establishes Chroma’s leading zero-shot voice-cloning result.The evaluation compares speech models on speaker similarity, where higher SIM indicates better speaker similarity.
- Voice Cloning Evaluation: 57.2% versus 24.4% NCMOS preference favored ElevenLabs, while SCMOS was nearly tied at 42.4% versus 40.6%.The NCMOS comparison included 18.3% Deuce responses, and the SCMOS comparison included 17.0%.
- Voice Cloning Evaluation: 92.0% of evaluators preferred ElevenLabs-generated audio over 8.0% for human recordings, showing that naturalness preference can diverge from speaker similarity.The comparison used 5 sessions with 10 samples each.
- Practical Generation Latency: 146.87ms TTFT and 0.43 RTF demonstrate sub-second responsiveness and speech generation faster than real-time playback.The reported average latency per frame is 52.34ms, while the latency breakdown measures response generation at concurrency 1.
- Dialogue Capabilities: Chroma remains competitive across understanding, reasoning, and oral conversation tasks despite using 4B parameters and focusing on voice cloning.It achieves the highest reported scores on MLC and CommonVoice oral-conversation tasks, while consistently ranking second on the listed reasoning tasks.
5 Conclusion
Chroma 1.0 combines real-time end-to-end spoken dialogue with high-fidelity personalized voice cloning. The authors report strong voice-cloning, reasoning, and dialogue performance and release the code and models.
- Chroma 1.0 is an open-source, real-time end-to-end spoken dialogue model combining low-latency interaction with high-fidelity personalized voice cloning.
- Chroma achieves significant speaker-similarity improvements from only a few seconds of reference audio while maintaining real-time performance.
- Comparative evaluations show competitive voice-cloning performance against state-of-the-art commercial systems alongside strong reasoning and dialogue capabilities.
- The efficient architecture supports smooth streaming generation for natural multi-turn conversations.
- The authors publicly release Chroma’s code and models to facilitate research in personalized spoken dialogue systems.
A Limitations and Future Work
The authors identify missing post-training and tool-use capabilities, possible decoder optimizations, English-only speech output, and decoder-only architectural trade-offs as future-work areas.
- Chroma does not incorporate external tool use or task-specific post-training methods such as RLHF or DPO.The authors suggest these methods could improve dialogue quality, instruction following, and user-preference alignment.
- Integrating multi-codebook token prediction into Chroma’s decoder could reduce first-packet latency without compromising voice-cloning fidelity.MTP predicts residual codebooks in parallel rather than sequentially.
- Chroma supports Chinese and English input but generates speech output only in English.The authors identify multilingual output and cross-lingual voice cloning as future directions.
- Chroma’s decoder-only backbone leaves encoder-decoder architectures as an alternative for controllability and cross-modal alignment.The authors specifically propose exploring encoder-decoder designs for finer control over semantic content and acoustic properties.
B Ethical Considerations
High-fidelity personalized speech from minimal reference audio creates risks of impersonation, fraud, and misleading or harmful content without consent. The authors recommend technical and policy safeguards and parallel development of ethical and regulatory measures.
- High-fidelity personalized speech from minimal reference audio raises risks of impersonation, fraud, and misleading or harmful audio without consent.
- The authors recommend explicit, verifiable consent for voice cloning.
- They also recommend synthetic-speech detection, usage policies, access controls, watermarking, and traceability techniques.
- Responsible deployment could support voice restoration, inclusive accessibility applications, and creative or personalized communication tools.
C Training Objective
Chroma’s training objective separates coarse acoustic prediction from residual refinement while conditioning generation on semantic and prosodic representations under causal prefix constraints.
- C Training Objective: The frozen Reasoner provides fixed text embeddings and multimodal hidden states as semantic and prosodic conditioning for downstream acoustic models.These representations are extracted from paired audio and transcription data and remain unchanged during optimization.
- C Training Objective: The Backbone autoregressively predicts a coarse discrete acoustic code sequence conditioned on reference audio and text.The sequence contains L audio frames, with codebook size V.
- C Training Objective: The Chroma Decoder autoregressively predicts N −1 residual quantization levels for each frame to refine the Backbone’s coarse acoustic representation.Refinement proceeds across levels j ∈ {1, . . . , N −1} using the Backbone hidden state and previously generated levels.
- C Training Objective: Causal prefix attention aligns text and audio generation by restricting the Backbone at time t to prefix information.The conditioning includes generated acoustic prefixes and prefix Reasoner signals.
- C Training Objective: The Decoder is trained with a negative log-likelihood objective across frames and refinement levels.This objective encourages progressive acoustic refinement while preserving consistency with the coarse code and Backbone context.
C.1 Training Strategy
Chroma uses two training stages: joint Backbone–Decoder learning first, followed by Decoder-focused fine-tuning for higher-level acoustic refinement.
- C.1 Training Strategy: The two-stage strategy is designed to stabilize optimization and progressively strengthen acoustic-representation refinement.The final phase is reported to improve voice-cloning fidelity and overall speech naturalness.
- C.1 Training Strategy: The first stage jointly trains the Backbone and Decoder with loss weight λ = 0.5.Balanced weighting targets both coarse acoustic code distribution and residual quantization levels to establish semantic-acoustic alignment.
- C.1 Training Strategy: The second stage freezes the Backbone and sets λ = 1, shifting optimization entirely to the Decoder.This phase focuses on higher-level quantization layers and fine-grained speech characteristics including timbre, prosody, and articulation.