Source-linked AI summary
Moshi: a speech-text foundation model for real-time dialogue
Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, Neil Zeghidour
TL;DR
Spoken-dialogue pipelines incur multi-second latency, lose non-linguistic audio information, and rely on turn segmentation that misses overlaps and interruptions. Moshi casts dialogue as speech-to-speech generation with parallel user and system streams and time-aligned text prefixes, yielding the first real-time full-duplex spoken language model with 160 ms theoretical and 200 ms practical latency.
Problem
Existing spoken-dialogue pipelines have several-second latency, discard non-linguistic information through textual mediation, and cannot model overlapping speech, interruptions, or interjections naturally.
Method
Moshi combines a text LLM with hierarchical audio-token generation, Inner Monologue text prefixes, and parallel streams for the user’s and system’s speech.
Results
Moshi is reported as the first real-time full-duplex spoken dialogue system, with 160 ms theoretical latency and streaming ASR and TTS enabled by Inner Monologue.
Takeaways & Limitations
Speech-to-speech generation with multi-stream modeling supports full-duplex conversations with arbitrary dynamics, while Inner Monologue extends the model to streaming speech recognition and synthesis.
Takeaways & Limitations
Moshi performs weaker than its base Helium model on some textual knowledge evaluations, including a reduction in MMLU from 54.3 to 49.7.
Abstract
from arXiv · showhide
We introduce Moshi, a speech-text foundation model and full-duplex spoken dialogue framework. Current systems for spoken dialogue rely on pipelines of independent components, namely voice activity detection, speech recognition, textual dialogue and text-to-speech. Such frameworks cannot emulate the experience of real conversations. First, their complexity induces a latency of several seconds between interactions. Second, text being the intermediate modality for dialogue, non-linguistic information that modifies meaning -- such as emotion or non-speech sounds -- is lost in the interaction. Finally, they rely on a segmentation into speaker turns, which does not take into account overlapping speech, interruptions and interjections. Moshi solves these independent issues altogether by casting spoken dialogue as speech-to-speech generation. Starting from a text language model backbone, Moshi generates speech as tokens from the residual quantizer of a neural audio codec, while modeling separately its own speech and that of the user into parallel streams. This allows for the removal of explicit speaker turns, and the modeling of arbitrary conversational dynamics. We moreover extend the hierarchical semantic-to-acoustic token generation of previous work to first predict time-aligned text tokens as a prefix to audio tokens. Not only this "Inner Monologue" method significantly improves the linguistic quality of generated speech, but we also illustrate how it can provide streaming speech recognition and text-to-speech. Our resulting model is the first real-time full-duplex spoken large language model, with a theoretical latency of 160ms, 200ms in practice, and is available at https://github.com/kyutai-labs/moshi.
1 Introduction
Moshi addresses the latency, text bottleneck, and turn-based limitations of conventional spoken-dialogue pipelines with a real-time, speech-to-speech, multi-stream architecture. Its Inner Monologue method improves generated speech while supporting streaming ASR and TTS.
- Motivation: Conventional spoken-dialogue pipelines introduce several seconds of latency, discard paralinguistic and non-speech information, and cannot naturally model interruptions or overlapping speech.Overlapping speech accounts for 10 to 20% of spoken time.
- Moshi: Moshi combines a text LLM with a smaller audio language model that directly ingests and generates discrete audio units.This removes the text intermediate-modality bottleneck while retaining the text model’s knowledge and reasoning abilities.
- Moshi: 160 ms theoretical latency enables Moshi’s streaming architecture to target real-time spoken dialogue.The reported theoretical latency is lower than the 230 ms average response time measured in natural conversations.
- Moshi: Moshi jointly models user and system audio in parallel autoregressive streams, removing explicit speaker turns and supporting overlaps and interruptions.The system always listens and generates either speech or silence.
- Inner Monologue: Inner Monologue predicts time-aligned text tokens before audio tokens, improving factuality and linguistic quality while enabling streaming ASR and TTS.Changing the delay between text and audio tokens derives the streaming recognition and synthesis modes.
- Evaluation: Experiments report state-of-the-art performance among existing speech-text models for speech modeling and spoken question answering, with streaming compatibility and 5 minutes of context.The evaluation covers text understanding, intelligibility, consistency, audio quality, and spoken question answering.
2 Related Work
Prior spoken-dialogue systems pursue real-time speech interaction, paralinguistic understanding, and reasoning, but existing approaches retain important architectural limitations. Moshi is positioned against these limits through full-duplex, speech-to-speech, multi-stream modeling.
- Spoken Dialogue Models: Spoken dialogue requires real-time full-duplex operation, speech-to-speech processing for paralinguistic communication, and knowledge and reasoning abilities.Full-duplex interaction allows either side to be active at any time.
- Spoken Dialogue Models: Spectron benefits from an underlying text LLM but is not compatible with real-time generation because of Chain-of-Modality.Its spoken-question-answering capability reflects the benefit of text-model knowledge.
- Spoken Dialogue Models: PSLM generates speech and text tokens in parallel to reduce latency, but answer quality decreases and ASR removes paralinguistic information.The system therefore addresses latency without eliminating the text and ASR bottlenecks.
- Spoken Dialogue Models: A cascaded ASR, text-generation, and streaming-TTS system models complex scenarios but uses one token stream for both speakers, challenging substantial overlap.It still relies on a pipeline and does not remove the speaker-stream entanglement.
3 Model
Moshi combines a text language model, neural audio codec, and streaming multi-stream token generation to support full-duplex speech-to-speech dialogue. Its architecture jointly models user and system audio, uses Inner Monologue for aligned text generation, and supports streaming ASR and TTS.
- 3.1 Overview: Moshi is a multi-stream speech-to-speech Transformer that supports full-duplex dialogue by speaking and listening simultaneously without explicitly modeling speaker turns.Its architecture jointly models Moshi’s text and audio tokens with the user’s audio stream.
- 3.1 Overview: Moshi uses Helium as its text language-model backbone and Mimi as a neural codec that combines semantic and acoustic information through residual vector quantization.Mimi incorporates semantic knowledge distilled from a self-supervised speech model.
- 3.3 Audio Tokenization: Mimi distills non-causal WavLM embeddings into a semantic quantizer and combines them with separate acoustic tokens for audio reconstruction.This split residual quantization supports discrete audio representations for Moshi’s speech generation.
- 3.4.2 Audio modeling: Moshi jointly generates semantic and acoustic tokens in a streaming fashion, using per-codebook parameters and acoustic delay to model the streams together.This differs from approaches that generate all semantic tokens before acoustic tokens.
- 3.4.4 Inner Monologue: Inner Monologue predicts time-aligned text tokens before audio tokens, while the multi-stream design handles overlapping speech and can derive streaming ASR and TTS.The text and audio delay determines which modality supplies the generated content decision.
4 Datasets and Training
Moshi is trained through staged text, audio, multi-stream, and instruction-tuning procedures using curated, web-scale, and conversational data. The training process progressively adds speech modeling, simultaneous listening and speaking, natural overlap, and spoken interaction behavior.
- 4.2 Audio Data: The unsupervised audio dataset contains 7 million hours of mostly English speech transcribed with Whisper large-v3 for audio pretraining.This phase uses a single audio stream representing all speakers together.
- 4.2 Audio Data: Fisher provides 2000 hours of two-channel phone conversations, enabling training with ground-truth separated speaker streams.Separate channels allow Moshi to learn simultaneous listening and speaking.
- 4.2 Audio Data: A further 170 hours of natural and scripted multi-speaker conversations are used to train a multi-stream TTS model and fine-tune Helium on real conversation transcripts.The recordings use separate channels per speaker.
- 4.3 Synthetic Conversations: Synthetic interaction scripts cover general knowledge, self-knowledge, voice and character requests, backchanneling, mispronunciations, corrections, and factual questions.The scripts use short conversational turns and varied prompts to create spoken interaction data.
- 4.4 Training Stages and Hyper-parameters: Moshi training proceeds through unsupervised audio pretraining, diarization-based multi-stream post-training, Fisher fine-tuning, and instruction fine-tuning.The staged procedure progressively develops audio modeling, multi-stream interaction, full-duplex capabilities, and instruction following.
5 Evaluation
Moshi combines low-latency, streaming audio generation with parallel conversational streams and Inner Monologue, achieving strong speech, dialogue, and spoken question-answering performance. Evaluations also reveal trade-offs in retained textual knowledge, benchmark reliability, and TTS assessment.
- Audio Tokenization: Mimi’s fully causal, low-framerate codec supports streaming inference, with MUSHRA 81.0 versus 58.8 for adversarial-only training.The study also reports weak correlation between objective and subjective audio-quality metrics.
- Generative Modeling: 640ms theoretical latency from the original delay pattern falls to 240ms with reduced delays, where the RQ-Transformer significantly improves perplexity.The RQ-Transformer is unnecessary under the longer delay but becomes critical under strict latency constraints.
- Generative Modeling: An 80ms minimal-latency configuration improves with added delay, while semantic-loss weighting of 100 further boosts speech intelligibility.The ablations also motivate depthwise parametrization and the final training procedure.
- Spoken Question Answering: Moshi with Inner Monologue almost triples spoken question-answering accuracy across all three benchmarks while increasing each multistream timestep from 16 to 17 tokens.It is also the only compared model described as jointly modeling semantic and acoustic tokens while supporting streaming inference.
- Quality and Statistics of Generated Dialogues: Moshi matches the cascaded model in linguistic quality and improves over the non-cascaded model, which cannot generate coherent speech in this scenario.Both Moshi and the cascaded model achieve perplexity better than ground truth under the reported evaluation conditions.
- Streaming ASR and TTS: 4.7% WER on LibriSpeech test-clean beats Vall-E’s 5.9% but trails NaturalSpeech 3’s 1.81%, while Moshi requires only 2 seconds of lookahead.The ASR system reports 5.7% WER versus 3.6% for Streaming FastConformer with similar look-ahead, and provides 80ms transcript alignments.
6 Safety
The safety analysis examines toxicity, regurgitation, voice consistency, and content identification, finding strong voice consistency and reduced exact regurgitation after deduplication, while watermarking remains vulnerable to compression and implementation removal.
- Scope: Moshi’s safety analysis covers toxic content, training-data regurgitation, unauthorized voice generation, and detection of generated audio.The authors note that audio safety is less developed than text safety and that non-verbal signals complicate direct comparison.
- Toxicity: The model ranks in the middle of the toxicity comparison, below industry models that benefited from private annotation, red-teaming, and feedback loops.
- Regurgitation: Deduplicating frequent training audio reduces exact regeneration of the most frequent segment to zero, even when prompted with its first three seconds.The evaluation uses condition-free generation and compares outputs against the most frequent duplicate segment.
- Voice consistency: 10,249 occurrences (98.7%) preserve the main speaker’s reference voice, versus 133 occurrences (1.3%) closer to the other speaker’s reference.Speaker consistency is measured using speaker embeddings and cosine similarity on generated conversation segments.
- Content identification: Watermark detection is effective on unchanged audio but degrades under strong compression, which can make marked audio indistinguishable from unmarked audio.The Mimi codec removes the watermark to a level that undermines reliable detection; watermarking also depends on stable tokenization and transcription.
- Content identification: Open-sourced watermarking may be easy to remove, so additional exploration is needed to make the procedure resistant to straightforward code-level removal.
7 Conclusion
The conclusion presents Moshi as a real-time, full-duplex speech-to-speech system built from a text LLM, neural audio codec, and hierarchical multi-stream architecture. Inner Monologue improves speech generation while remaining compatible with streaming, and experiments report strong spoken question answering, dialogue modeling, safety, and voice consistency.
- Conclusion: Moshi is a real-time, full-duplex spoken dialogue system combining Helium, Mimi, and hierarchical multi-stream speech-to-speech generation.Helium is a 7B-parameter text LLM, while Mimi is a low-bitrate, low-framerate semantic-acoustic codec.
- Conclusion: Inner Monologue generates text tokens before audio tokens, improving speech generation while remaining compatible with streaming inference.The method also supports streaming text-to-speech and speech recognition by changing the delay between text and audio tokens.
- Conclusion: Experiments report state-of-the-art spoken question answering and dialogue modeling, alongside satisfying safety and voice-consistency levels.
- Conclusion: The authors release Mimi and Moshi and provide models and recipes spanning text language models, neural audio codecs, and generation.
Appendix A. Additional Ablation on Mimi Codec
The appendix evaluates Mimi codec hyperparameters using phonetic discriminability and audio reconstruction metrics.
- Additional Ablation on Mimi Codec: The ablation evaluates semantic modeling with phonetic ABX error rate and reconstruction quality with VisQOL and MOSNet.It also studies quantization rate, defined as quantizing the latent space during 50% of training, independently of quantizer dropout.
Appendix B. Audio Matching and Deduplication
The appendix develops audio matching and deduplication through mel-spectrum keypoints, constellation-map hash signatures, and temporally consistent retrieval. These tools support identifying generated content and filtering frequent duplicate training segments.
- Overview: The audio matching system supports source-content deduplication and indexing generated samples to identify content from the online demo.
- Constellation map: A mel-spectrogram is filtered by energy, time, and frequency criteria to produce a constellation map of selected time-frequency keypoints.The spectrum uses 40 Hz time discretization and 64 frequency bins spanning 200Hz–3000Hz.
- Hash encoding: Each keypoint is paired with nearby forward and backward keypoints, and signatures encode their frequencies and temporal offsets.With m = 4 and M = 20, signatures span at most about 3.2 seconds and have 67,108,864 possible hash keys.
- Matching: Pairwise audio comparison uses intersections of hash-key sets, while one-to-many matching uses an inverted file or hash table.A Hough 1D temporal voting scheme retains temporally consistent matches.
- Deduplication: Frequent duplicate segments are identified through repeated signature matches and represented by a fused duplicate-signature set for filtering during training.The strategy typically selects segments occurring in at least 10 matches, then removes redundant signatures at identical relative timestamps.
Appendix C. Delayed text LM as a zero-shot streaming ASR and TTS
Moshi’s delayed text stream enables streaming ASR and TTS by offsetting text and audio tokens while operating with fixed latency. The same mechanism supports single- and multi-stream speech generation.
- Delayed text modeling: A 2-second delay between audio and text enables full-streaming ASR and TTS with fixed latency.The method introduces an acoustic token delay τ = 2 in both modes.
- ASR mode: In ASR mode, input audio tokens are retained while text tokens are sampled to produce word-aligned transcriptions.The text stream contains the audio transcription with fine word-level alignments.
- TTS mode: In TTS mode, padded text tokens guide freely sampled audio, and padding statistics can control speech rate online.The next word is supplied when the model departs from PAD or EPAD generation.
- Multi-stream TTS: Multi-stream TTS outputs two audio-token sets while placing the text in one stream separated by speaker boundary tokens.The <bos> and <eos> tokens separate the text from the two speakers.
Appendix D. Characterizing Audio Artifacts Caused by Quantization
The appendix characterizes quantization artifacts through entropy patterns in text and audio token streams, using windowed statistics to distinguish degradation types. Artifact frequency and temporal behavior worsen under stronger compression, while detection thresholds depend on analysis settings.
- Measurement framework: Quantization analysis computes Shannon entropy independently for text and audio streams over fixed context windows, then classifies artifacts in non-overlapping windows.The practical settings use C = 64 for entropy context and ω = 64 tokens for artifact detection.
- Artifact categories: Repetitive text is detected when text entropy remains nearly flat but nonzero, using a fitted slope threshold ηflat = 10^-3.This pattern corresponds to rapid repetition of short sentences or words.
- Artifact categories: Silence is identified by constant PAD text tokens and low-entropy audio tokens, whereas background noise occurs when the text stream is silent but audio entropy rises.Silence is treated as artifact-free because it can represent the other speaker’s turn.
- Artifact categories: Gibberish is common at W2 quantization and reflects incoherent speech, while noisy audio is detected from elevated entropy variation across audio codebooks.The noisy-audio criterion uses an entropy standard-deviation threshold ηnoise = 0.6.
- Measurement limitations: Artifact detection uses thresholds tied primarily to token-vocabulary entropy, but their values are also tightly linked to context size and window length.The authors report that the same thresholds worked across models using the same Mimi codec.
- Compression effects: At 4 bits, quantized and unquantized models behave similarly; at 3 bits, artifacts increase, especially with 256-token blocks, and become more frequent over time.The appendix reports artifact distributions averaged across 500 samples per model.
Appendix E. Safety and Toxicity
The appendix presents an ALERT benchmark table comparing category-wise and overall safety scores across evaluated language models.
- ALERT benchmark: Table 18 reports category-wise safety scores and overall safety scores when Moshi is added to the ALERT benchmark.Higher scores indicate safer behavior, with gray, orange, and red bands marking safety ranges.
Appendix F. Synthetic Transcripts Generation for Fine-tuning
The appendix states that it provides examples of synthetic transcripts generated for fine-tuning.
- Synthetic transcripts: The appendix provides examples of synthetic transcripts used for fine-tuning.
F.1 General knowledge
These examples show Moshi sustaining spoken-style general-knowledge conversations across information theory, literature, and literary biography.
- General knowledge: Moshi explains entropy through a multi-turn exchange about information theory.The dialogue begins with a definition of entropy and continues toward its formula.
- General knowledge: Moshi discusses The Three Musketeers through successive questions about its setting, characters, and story.
- General knowledge: Moshi answers questions about John Steinbeck’s life, work, and most famous writings.The example includes his birth and death details and identifies The Grapes of Wrath as a famous novel.
F.2 Misspelling
These examples illustrate Moshi handling misspelled or unclear names through clarification and subsequent informative responses.
- Misspelling: Moshi asks for clarification when the user first says “Danyh Amendol,” then identifies Danny Amendola after correction.It subsequently provides information about his professional football career.
- Misspelling: Moshi similarly requests clarification for “Maroom” before interpreting it as “Maroon” and explaining the color.The response includes its color description, name origin, shades, and relation to teal.
F.3 Voice Instructions
The voice-instruction examples depict Moshi responding in specified emotional personas, while the fine-tuning inventory contains 92 speaking styles.
- Voice Instructions: Moshi roleplays a relieved astronaut and marks its responses with the requested relieved voice.The scenario concerns returning safely to the moon and beginning further exploration.
- Voice Instructions: Moshi roleplays an annoyed detective who rejects small talk during social gatherings.Its responses consistently express annoyance and prioritize solving cases over conversation.
- Voice Instructions: 92 speaking styles are listed as the styles used to fine-tune Moshi.