Moshi: a speech-text foundation model for real-time dialogue

• Existing spoken-dialogue pipelines incur several seconds of latency, discard non-written audio information, and assume single-speaker turns, limiting their handling of overlap, interruptions, and backchannels.Overlapping speech accounts for 10 to 20% of spoken time.Source: §1 · p. 2
• Helium is a 7B-parameter text LLM pretrained on 2.1T tokens of public English data.Source: §1 · p. 3
• Mimi is a neural audio codec that uses residual vector quantization; its first acoustic-token level distills semantic information, avoiding separate semantic tokens.The approach extends prior semantic-to-acoustic token generation and uses improved training tricks.Source: §1 · p. 3
• Moshi combines a text LLM with a smaller Transformer that predicts audio tokens hierarchically and streams multiple audio streams in parallel for full-duplex dialogue.Joint input and output streams remove explicit speaker turns and support conversational dynamics such as overlap and interruptions.Source: §1 · pp. 2–3
• Inner Monologue predicts time-aligned text tokens before audio tokens, improving factuality and linguistic quality while enabling streaming ASR and TTS through a delay between text and audio.The paper also reports improved length and quality of generated speech.Source: §1 · p. 3
• Moshi is state of the art among existing speech-text models for speech modeling and spoken question answering, is streaming compatible, and models up to 5 minutes of context.The reported experiments used 5 minutes of context.Source: §1 · p. 3

Audio Language Modeling

• Audio language models progressed from self-supervised speech representations to semantic and acoustic token generation, but long audio sequences remain demanding and speech-only models typically lack factual knowledge and reasoning.Hierarchical semantic and acoustic modeling has enabled longer, consistent speech continuations, while latent diffusion approaches cannot stream and their ability to generate consistent speech remains unclear.Source: Audio Language Modeling · p. 4

Speech-text Models

• Prior speech-text models use task-specific supervision, modality switching, or text-first or parallel generation; Moshi's Inner Monologue predicts aligned text, semantic, and acoustic tokens hierarchically per frame.This per-frame approach supports real-time generation; with delays between token types, Moshi can also provide streaming TTS and ASR. Unlike Chain-of-Modality, it does not require producing a complete text sequence before audio generation, and unlike parallel text and speech prediction, text precedes audio tokens.Source: Audio Language Modeling · p. 5

Spoken Dialogue Models

• Prior dialogue systems leave gaps in real-time, paralinguistic speech-to-speech, or full-duplex interaction; Moshi uses hierarchical semantic and acoustic streams to address these limits.Unlike the cascaded system of Wang et al. (2024) and the proof-of-concept dGSLM, Moshi operates online, uses its text-pretrained Helium backbone, and models acoustic information. Its streams support overlapping speech, interruptions, and interjections, as well as varied voices, emotions, and acoustic conditions.Source: Audio Language Modeling · pp. 5–6

3.1 Overview

• Moshi is a multi-stream speech-to-speech Transformer built on Helium, with Mimi audio tokens, a streaming Depth Transformer, and Inner Monologue for joint text-audio modeling.Its parallel streams let Moshi speak and listen simultaneously without explicitly modeling speaker turns.Source: §3.1 · p. 6, Figure 1 · p. 7

3.2.1 Architecture

• Helium supplies Moshi’s autoregressive text backbone, while its CommonCrawl pipeline deduplicates pages, retains English documents, and filters for quality.Helium uses RMS normalization, RoPE, a 4,096-token context, FlashAttention, and gated linear units; the web-data pipeline applies line-level deduplication and quality scoring, then keeps documents above thresholds.Source: §3.2.1 · p. 7, §3.2.2 · pp. 8–9

3.3 Audio Tokenization

• Mimi combines causal residual vector quantization with distilled semantic information, enabling streaming semantic-acoustic encoding and decoding.Its 24 kHz input is represented at 12.5 frames per second; an 80 ms first audio frame yields a latent timestep that can be decoded to 80 ms of audio.Source: §3.3 · p. 9, Figure 2 · p. 10, §3.3.1 · pp. 10–11

3.3.1 Architecture

• Training Mimi with adversarial losses only improves subjective audio quality despite major degradation in objective metrics.Subjective evaluations reported in Table 4 confirm the audio-quality boost.Source: §3.3.1 · p. 11

3.3.2 Learning semantic-acoustic tokens with a split RVQ

• Mimi’s split RVQ distills semantic information into a separate VQ alongside a seven-level acoustic RVQ, improving the semantic-acoustic trade-off.This removes the constraint that acoustic information must be conserved in the residual of the semantic quantizer.Source: §3.3.2 · p. 12

3.4.1 Hierarchical autoregressive modeling with RQ-Transformer

• The RQ-Transformer splits multi-stream sequences between a Temporal Transformer over time and a smaller Depth Transformer across sub-sequences, avoiding a flattened K·S prediction sequence.Mimi produces 100 audio-token steps per second, compared with about 3 to 4 text tokens per second; the hierarchy keeps the Temporal Transformer at S steps and the Depth Transformer at no more than K.Source: §3.4.1 · pp. 13–14, Figure 3 · p. 13

3.4.2 Audio modeling

• Moshi jointly streams semantic and acoustic tokens for itself and the user, using acoustic delay to improve generation and handle overlapping speech.The authors report that delays of one or two steps improve generation quality; semantic and acoustic tokens are generated jointly rather than in separate stages.Source: §3.4.2 · pp. 14–15, §3.4.3 · p. 15, Figure 4 · p. 15

3.4.4 Inner Monologue

• Inner Monologue prefixes Moshi’s semantic-audio generation with time-aligned text, improving linguistic quality and enabling delay-controlled streaming ASR or TTS.Whisper word timestamps align text tokens to Mimi’s 12.5 Hz frame rate; audio-ahead-of-text yields streaming ASR, while text-ahead-of-audio yields streaming TTS, without changes to loss, architecture, or training data.Source: §3.4.4 · pp. 15–17
• The model uses K = 2Q + 1 streams, with Q = 8 in the experiments.The paper summarizes the stream arrangement in Figure 4.Source: §3.4.4 · p. 17
• Moshi samples its text and audio outputs at inference, ignores the modeled user-audio output in applications, but retains it to generate simulated dialogues for offline evaluation.At each step, the model predicts distributions over the joint sequence conditioned on prior streams.Source: §3.4.4 · p. 17
• Without explicit turn boundaries, Moshi can speak and listen simultaneously; its audio encodes natural silence while its text uses PAD, and sampling EPAD can trigger speech.When the user speaks and Moshi stays silent, Moshi’s audio tokens decode to a near-silent waveform.Source: §3.4.4 · p. 17

4.1 Text Data

• The text corpus combines 12.5% curated sources—including Wikipedia, Wikibooks, Wikisource, Wikinews, StackExchange, and pes2o—with 87.5% filtered CommonCrawl data.The Wikipedia portion uses five dumps from 2017, 2018, 2019, 2021, and 2022 rather than repeated passes over one dump.Source: §4.1 · p. 17

4.2 Audio Data

• Audio training uses 7 million hours of mostly English audio, 2,000 hours of Fisher phone conversations, and 170 hours of supervised multi-speaker recordings used indirectly for TTS and Helium finetuning.Fisher provides separate speaker channels and is upsampled from 8 kHz to 24 kHz; Moshi is not trained directly on the 170-hour supervised dataset.Source: §4.2 · p. 18

4.3 Speech-Text Instruct Data

• More than 20k hours of synthetic speech-text instruction data come from Helium-generated realistic interactions, with a single actor voice for Moshi and randomly sampled user voices.The actor recorded monologues covering more than 70 speaking styles; consistent voice conditioning during instruction tuning is reported to ensure Moshi retains that voice without inference-time control.Source: §4.3 · p. 18
• Instruction examples span context-grounded knowledge and lab information, voice or emotion roleplay, clarification of misspellings, correction of false facts, basic factual tasks, and safety refusals.Knowledge prompts draw on Wikipedia or StackExchange context, while voice instructions include requested styles such as angry or pirate speech.Source: §4.3 · pp. 19–20

4.4 Training Stages and Hyper-parameters

• Training progresses from Helium initialization through 1M-step single-stream pretraining, 100k-step diarized two-stream post-training, 10k Fisher batches, and 30k-step instruction finetuning.The pretraining mixes audio batches with text-only batches; later stages add simulated streams, real Fisher conversations, and synthetic instruction data.Source: §4.4 · pp. 20–21
• Instruction finetuning augments the user stream with random gain, noise, silence, simulated echo, and reverb to expose Moshi to changing audio conditions.Gain ranges from −24 to +15 dB; noise is added 30% of the time, and echo and reverb are applied together with 30% probability.Source: §4.4 · p. 21
• A separate streaming multi-stream TTS model uses a 2-second audio delay and supervised conversation finetuning to generate instruction examples; Moshi itself does not train on that supervised set.The training loss weights the text token equally with the combined audio tokens, setting α to 100 for semantic tokens and 1 for acoustic tokens.Source: §4.4 · p. 21

5.1 Text Language Modeling

• Helium is on par with or outperforms similar-compute models on most text benchmarks, and remains competitive with Mistral and Gemma on ARC, OBQA, and NQ despite their greater training compute.Helium uses 2.1T tokens; the similar-compute baselines use fewer than 2.5T tokens, while Mistral and Gemma use up to 3× more compute.Source: §5.1 · p. 22

5.2 Audio Tokenization

• Mimi combines semantic information, causal streaming operation, and strong reconstruction quality, while its objective audio metrics can diverge from human judgments.Split RVQ raises MUSHRA from 57.8 to 64.0 while ABX rises from 6.5% to 8.1%; adversarial-only training yields MUSHRA 81.0 versus 58.8 with Encodec-style mixed losses despite VisQOL of 1.84. Mimi also outperforms RVQGAN at lower bitrate and SemantiCodec at 4× lower framerate.Source: §5.2 · pp. 23–25, Table 4 · p. 24

5.3 Ablations on Generative Modeling

• At 240ms latency, the RQ-Transformer significantly improves perplexity over independent classification heads, making it critical under strict latency constraints.An eight-step delay instead gives 640ms latency and only marginal perplexity benefit from the RQ-Transformer; additional ablations find improved speech quality from one delay step, semantic-token loss weight 100, depthwise parametrization, and especially Inner Monologue.Source: Table 5 · p. 25, Table 6 · p. 25, §5.3 · pp. 26–27

5.4 Audio Language Modeling

• Moshi scores 12 points higher on MMLU than Spirit-LM, while multimodal training improves spoken commonsense but has mixed effects on lexical and syntactic judgments.Instruction finetuning particularly hurts sWUGGY, though the authors note that textless NLP metrics do not consistently track dialogue-model quality and motivate spoken QA as a complementary evaluation.Source: Table 7 · p. 27, §5.4 · pp. 28–29

5.5 Spoken Question Answering

• Moshi with Inner Monologue almost triples spoken question-answering accuracy across all three benchmarks and achieves the best performance among compared models.Inner Monologue adds only one token per multistream timestep; without retaining 50% text-only pretraining batches, accuracy drops noticeably. Moshi remains below Helium on spoken QA, with multi-sentence and syntactically specific questions especially challenging.Source: Table 8 · p. 29, §5.5 · p. 30

5.6 Quality and Statistics of Generated Dialogues

• Moshi matches a cascaded model’s dialogue linguistic quality and improves on the non-cascaded dGSLM result, despite generating speech directly from audio-to-audio modeling.Both Moshi and the cascaded model score better perplexity than ground-truth Fisher dialogues, which the authors attribute to their training data being closer to DialoGPT’s training data.Source: Table 9 · p. 30, §5.6 · p. 31

5.7 Streaming ASR and TTS

• Streaming TTS obtains 4.7% WER, outperforming Vall-E’s 5.9% but trailing NaturalSpeech 3’s 1.81%; streaming ASR obtains 5.7% WER versus 3.6% for Streaming FastConformer.TTS uses 2 seconds of lookahead, compared with full-sequence access for Vall-E and NaturalSpeech 3; ASR uses similar lookahead and provides transcript alignments at 80ms precision. The authors frame these results as an illustration of Inner Monologue’s flexibility, not a state-of-the-art evaluation.Source: §5.7 · pp. 31–32

5.8 Compressing Moshi and Impact on Speech Quality

• The quantization study investigates linguistic and especially acoustic degradation, using BF16 activations dynamically quantized to 8 bits and asymmetrically quantized weights.Embeddings, RMSNorms, and the Mimi codec remain unquantized; the authors report that keeping Depth Transformer weights at high precision does not significantly improve audio quality. Initial results indicate Helium is more robust to quantization than final Moshi.Source: §5.8 · p. 32
• Helium quantized to 4 bits is 3.43 times smaller and remains within 2 MMLU points of the floating-point baseline.The quantization format is almost identical to llama.cpp’s Q4_0 and can be readily deployed for efficient inference.Source: Table 10 · p. 33, §5.8 · p. 33
• On Moshi, the same quantization recipe causes 5–10 MMLU-point losses; 8-bit weights instead yield a 2-point drop in a model roughly half the floating-point baseline’s size.The 8-bit weights are used in the online demo.Source: Table 11 · p. 33, §5.8 · p. 33
• MOSNet scores show little degradation at 4 bits but can miss severe artifacts at lower bitwidths, so entropy spectra are used to detect them.MOSNet may not distinguish noisy audio from speech-pattern problems such as increased repetitiveness; Figure 6 illustrates entropy spectra used to identify artifacts.Source: §5.8 · p. 34, Figure 5 · p. 34, Figure 6 · p. 35
• Audio quality remains close to the floating-point baseline down to 4-bit precision, while MMLU drops significantly below 6 bits with post-training quantization.This greater sensitivity of linguistic ability holds even when the full model, including the Depth Transformer, is quantized.Source: §5.8 · p. 34
• Entropy-spectrum artifact measurements find little audio degradation at 4 bits, more frequent degradation at 3 bits, and significantly degraded quality at 2 bits.At 3 bits, artifacts tend to become more frequent as generation proceeds; finer-granularity quantization is generally more robust.Source: §5.8 · p. 34, Table 12 · p. 35
• Quantization-aware finetuning may improve performance at lower bitwidths, but preserving Moshi’s abilities requires further investigation of training phases and calibration datasets.The authors note that Moshi’s multistage training pipeline and datasets complicate this investigation.Source: §5.8 · p. 35

6.1 Toxicity Analysis

• Moshi ranks in the middle of the ALERT toxicity benchmark, while industry models perform best.The analysis is restricted to generated text because audio and text are difficult to compare directly and audio conveys non-verbal meaning.Source: §6.1 · p. 36

6.2 Regurgitation Analysis

• Dataset deduplication reduced exact regurgitations of the most frequent training segment to zero, even without fine-tuning.The pre-trained raw-data model often generated frequent sequences, especially at sampling temperatures of 0.6–1.0; conversational fine-tuning produced none of the most frequent sequence in 1,000 generations, but could be overridden and may not suffice alone.Source: Table 13 · p. 37, §6.2 · pp. 37–38

6.3 System Voice Consistency

• Moshi’s voice matched its own reference in 10 249 occurrences (98.7%), versus 133 occurrences (1.3%) closer to the other speaker’s reference.Speaker consistency remained stable over time; the authors attribute inference-time robustness to using a consistent system voice during instruction tuning.Source: Table 14 · p. 38, §6.3 · p. 38

6.4 Identification of the Content Generated by Moshi: Watermarking

• Audio watermarking detected unchanged marks well, but strong compression removed them, making Mimi-processed watermarked audio indistinguishable from unwatermarked audio.With aggressive pink noise, a relatively long sequence was needed for a high detection score; audio indexing is another option, but only when the content-generating machine is accessible.Source: §6.4 · pp. 38–39, Table 15 · p. 39
• Sampling-based generative watermarking is hindered by codec non-idempotence, which destabilizes re-encoded tokens and watermark detection.Shortening the hash context improves token stability but risks degenerated audio; the authors also flag that watermarking may be trivial to remove when a model is open-sourced.Source: §6.4 · pp. 40–41
• Potential watermarking remedies include marking only early quantizer levels, improving latent-space stability, tolerating temporal shifts, or marking text instead of audio.Text marking has limited capacity for short conversations and requires reliable transcription.Source: §6.4 · p. 41

7 Conclusion

• Moshi is presented as the first real-time, full-duplex spoken dialogue system, with 160ms theoretical latency and the ability to follow complex multi-turn conversations across 5 minutes.The authors report that the system maintains a consistent voice and does not generate toxic content.Source: §7 · p. 41
• The suite combines competitive 7B Helium, low-bitrate state-of-the-art Mimi, hierarchical multi-stream speech-to-speech generation, and streaming-compatible Inner Monologue.Helium is a text LLM competitive with open-weights models trained with a similar compute budget; Mimi is a semantic-acoustic neural audio codec operating at low framerates compatible with real-time generation.Source: §7 · p. 41
• Experiments show state-of-the-art spoken question answering and dialogue modeling, with satisfying safety levels through non-toxic generation and voice consistency.Source: §7 · p. 41
• Mimi and Moshi are released, and the authors expect Inner Monologue and multi-stream modeling to benefit speech-to-speech and audio-to-audio beyond dialogue modeling.Source: §7 · pp. 41–42
• Changing the delay between text and audio tokens lets Inner Monologue support streaming text-to-speech and streaming speech recognition.Source: §7 · p. 42

Appendix A. Additional Ablation on Mimi Codec

• Quantization rate means quantizing the latent space only 50% of training steps, independently of quantizer dropout.The caption says this is described in Section 3.3.Source: Table 17 · p. 54

Indexing solution

• The system also indexes audio signatures so generated content can be checked against samples from the online demo by direct retrieval.The matching system’s two purposes are dataset deduplication and indexing samples at generation time.Source: Appendix B · p. 54

Hash encoding

• Inspired by Wang (2003), each keypoint is hashed with forward and backward neighbors in bounded time windows; m = 4 and M = 20 limit the signature span to about 3.2 seconds.The resulting hash space has 67,108,864 possible values, though observed hash frequencies are skewed.Source: Appendix B · pp. 54–55

Constellation map

• Constellation extraction applies energy, time, and frequency filters to a 40 Hz mel spectrogram with 64 bands, retaining robust energy and local maxima.The specified frequency range is 200–3000 Hz; the filters select the constellation keypoints used for hashing.Source: Appendix B · pp. 54–55, Figure 7 · p. 55

Pair-wise matching and one-to-many comparison

• Audio comparison intersects hash-key sets, while an inverted index supports dataset search and Hough 1D voting retains only temporally consistent matches.A ±1 timestamp tolerance is optional but omitted for dataset deduplication because it increases complexity.Source: Appendix B · pp. 55–56

De-duplication: Signature fused set

• For deduplication, the system cross-matches dataset segments, consolidates recurring matches (typically ≥10) into a duplicate-signature set, and filters training segments above a matching threshold.Duplicate signatures at identical relative timestamps are removed; a segment is used for training only when its match score is below the threshold.Source: Appendix B · pp. 54, 56

Delayed text LM

• A fixed 2-second delay between audio and text enables Moshi to adapt its aligned-stream modeling for fully streaming ASR and TTS.The method uses special padding tokens to align text with the audio frame rate.Source: Appendix C · p. 56

TTS mode

• In TTS mode, Moshi samples PAD and EPAD tokens to create padded text, then supplies each next word when sampling another token is attempted.An online padding-fraction average and a logit bonus below a target rate control speech pace; a text-and-audio prefix controls speaker voice.Source: Appendix C · p. 56

Multi-stream TTS

• In multi-stream TTS, Moshi outputs two audio-token sets while a single text stream uses <bos> and <eos> to separate the speakers' text.Source: Appendix C · p. 56

ASR mode

• In ASR mode, Moshi keeps audio tokens matched to the input and freely samples delayed text tokens, producing word-level alignments and timestamps.The text stream contains the audio transcription; Figure 8 depicts the joint sequence with a 2-second text delay and acoustic token delay τ = 2.Source: Appendix C · p. 56, Figure 8 · p. 57

Appendix D. Characterizing Audio Artifacts Caused by Quantization

• The analysis computes text and audio entropy over fixed context windows, uses C = 64 (roughly 4.5 seconds of decoded audio), and detects artifacts in non-overlapping windows of 64 tokens.Leading context tokens are excluded because they have reduced context and include the generation prompt in the experiments.Source: Appendix D · p. 57, Figure 10 · p. 60
• Repetitive text is detected when text entropy remains nonzero but nearly flat over a 64-token window, with flatness defined by a fitted slope below ηflat = 10−3.Source: Appendix D · p. 57
• Natural silence has constant PAD text tokens and low audio entropy, so it is not counted as an artifact; background noise is flagged when text is silent but audio entropy exceeds ηaudio silence = 2.Silences can represent the other speaker’s turn in the multi-stream model.Source: Appendix D · p. 59
• Gibberish, common at low-bitwidth W2, is associated with high text entropy, while noisy audio is detected by audio-codebook entropy standard deviation above ηnoise = 0.6 after excluding silence and background noise.The entropy thresholds worked across models using the same Mimi codec, but depend on context and window sizes; too-small windows can miss brief artifacts.Source: Appendix D · p. 59
• At 4 bits, quantized and unquantized models behave similarly; at 3 bits, artifacts increase—especially with block size 256 and over time—and 2-bit quantization severely harms quality even with block size 32.The temporal distributions are averaged across 500 samples per model.Source: Appendix D · p. 59, Figure 11 · p. 61

Appendix E. Safety and Toxicity

• Table 18 presents category-wise and overall ALERT safety scores for Moshi, with higher scores indicating safer results and colors marking safety-score ranges.Gray denotes S(Φ) ≥99, orange denotes 90 ≤S(Φ) < 99, and red denotes S(Φ)<90.Source: Table 18 · p. 62

Appendix F. Synthetic Transcripts Generation for Fine-tuning

• The appendix presents examples of synthetic transcripts.Source: Appendix F · p. 63

F.1 General knowledge

• Entropy measures uncertainty or information in a random variable and helps explain communication, error-correcting codes, and compression.The dialogue also compares information-theory entropy with thermodynamic entropy as a measure related to disorder and unavailable energy.Source: §F.1 · p. 63
• The Three Musketeers follows d’Artagnan’s pursuit of joining the Musketeers and explores political abuses of power and social inequality.The response identifies Athos, Porthos, and Aramis as his friends and describes the book as a classic.Source: §F.1 · p. 64
• John Steinbeck, born in 1902 and deceased in 1968, wrote about working-class hardship, injustice, and survival.His works named include The Grapes of Wrath, Of Mice and Men, The Red Pony, and East of Eden; The Grapes of Wrath won the Pulitzer Prize.Source: §F.1 · p. 64

F.2 Misspelling

• After the user clarifies “Danyh Amendol” as Danny Amendola, Moshi identifies the former football player and provides biographical details.Moshi first says it is unsure who the user means, then discusses Amendola’s teams, college football, Pro Bowl selection, and Super Bowl appearances.Source: §F.2 · p. 65
• When the user asks about “Maroom,” Moshi initially reinterprets it as “Marron,” then treats it as Maroon and explains its complementary color, teal.Moshi also says “Maroom” is not a word when asked for its opposite, and suggests the user means complementary color.Source: §F.2 · p. 65

F.3 Voice Instructions

• As a relieved astronaut, Moshi expresses joy about returning safely to the moon and celebrates the successful mission.The dialogue maintains the relieved tone through the exchange and includes a fist-raising action.Source: §F.3 · p. 66
• As an annoyed detective, Moshi resists small talk and says they would rather solve cases than chat with strangers.The detective persona persists through the social-gathering conversation.Source: §F.3 · p. 66
• The appendix reports that Moshi was fine-tuned using 92 speaking styles.The examples include relieved-astronaut and annoyed-detective dialogue.Source: §F.3 · p. 66, Table 19 · p. 67