PaperlayerView exact revision on arXiv

Source-linked AI summary

Voxtral Realtime

Mistral-AI, :, Alexander H. Liu, Andy Ehrenberg, Andy Lo, Chen-Yo Sun, Guillaume Lample, Jean-Malo Delignon, Khyathi Raghavi Chandu, Patrick von Platen, Pavankumar Reddy Muddireddy, Rohin Arora, Sanchit Gandhi, Sandeep Subramanian, Soham Ghosh, Srijan Mishra, Abhinav Rastogi, Adrien Sadé, Alan Jeffares, Albert Jiang, Alexandre Cahill, Alexandre Gavaudan, Alexandre Sablayrolles, Amélie Héliou, Amos You, Andrew Bai, Angele Lenglemetz, Anmol Agarwal, Anton Eliseev, Antonia Calvi, Arjun Majumdar, Avi Sooriyarachchi, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Benjamin Tibi, Charlotte Cronjäger, Clémence Lanfranchi, Connor Chen, Corentin Barreau, Corentin Sautier, Cyprien Courtot, Darius Dabert, Diego de las Casas, Elizaveta Demyanenko, Elliot Chane-Sane, Enguerrand Paquin, Etienne Goffinet, Fabien Niel, Faruk Ahmed, Federico Baldassarre, Gabrielle Berrada, Gaëtan Ecrepont, Gauthier Guinet, Genevieve Hayes, Georgii Novikov, Giada Pistilli, Guillaume Kunsch, Guillaume Martin, Guillaume Raille, Gunjan Dhanuka, Gunshi Gupta, Han Zhou, Harshil Shah, Hope McGovern, Hugo Thimonier, Indraneel Mukherjee, Irene Zhang, Jaeyoung Kim, Jan Ludziejewski, Jason Rute, Joachim Studnia, John Harvill, Jonas Amar, Joséphine Delas, Josselin Somerville Roberts, Julien Tauran, Karmesh Yadav, Kartik Khandelwal, Kilian Tep, Kush Jain, Laurence Aitchison, Laurent Fainsin, Léonard Blier, Lingxiao Zhao, Louis Martin, Lucile Saulnier, Luyu Gao, Maarten Buyl, Manan Sharma, Margaret Jennings, Marie Pellat, Mark Prins, Martin Alexandre, Mathieu Poirée, Mathilde Guillaumin, Matthieu Dinot, Matthieu Futeral, Maxime Darrin, Maximilian Augustin, Mert Unsal, Mia Chiquier, Minh-Quang Pham, Nathan Grinsztajn, Neha Gupta, Olivier Bousquet, Olivier Duchenne, Patricia Wang, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Philippe Pinel, Philomène Chagniot, Pierre Stock, Piotr Miłoś, Prateek Gupta, Pravesh Agrawal, Quentin Torroba, Ram Ramrakhya, Rishi Shah, Romain Sauvestre, Roman Soletskyi, Rosalie Millner, Rupert Menneer, Sagar Vaze, Samuel Barry, Samuel Humeau, Sean Cha, Shashwat Verma, Siddhant Waghjale, Siddharth Gandhi, Simon Lepage, Sumukh Aithal, Szymon Antoniak, Teven Le Scao, Théo Cachet, Theo Simon Sorg, Thibaut Lavril, Thomas Chabal, Thomas Foubert, Thomas Robert, Thomas Wang, Tim Lawson, Tom Bewley, Tom Edwards, Tyler Wang, Umar Jamil, Umberto Tomasini, Valeriia Nemychnikova, Van Phung, Vedant Nanda, Victor Jouault, Vincent Maladière, Virgile Richard, Vladislav Bataev, Wassim Bouaziz, Wen-Ding Li, William Havard, William Marshall, Xinghui Li, Xingran Guo, Xinyu Yang, Yannic Neuhaus, Yassine El Ouahidi, Yassir Bendou, Yihan Wang, Yimu Pan, Zaccharie Ramzi, Zhenlin Xu

arXiv:2602.11298v3cs.AI

TL;DR

Streaming ASR must reconcile offline-quality transcription with strict real-time latency, a gap that chunked adaptations of offline models struggle to close. Voxtral Realtime is trained natively for streaming with aligned audio-text processing, a causal encoder, Ada RMS-Norm delay conditioning, and multilingual pretraining. At 480 ms it achieves offline-level quality competitively with Whisper, and the model is released under Apache 2.0.

  • Problem

    Real-time applications need accurate transcription while audio is streaming, but chunked offline models face a training–inference mismatch at low latency.

  • Method

    Voxtral Realtime uses a causal audio encoder, aligned stream-synchronous decoding, Ada RMS-Norm delay conditioning, and pretraining across 13 languages.

  • Results

    At a delay of 480 ms, Voxtral Realtime achieves offline-level transcription quality competitively with Whisper and ElevenLabs Scribe v2 Realtime.

  • Takeaways & Limitations

    Voxtral Realtime achieves offline-level transcription quality at sub-second latency within a natively streaming architecture.

Abstract

from arXiv · show

We introduce Voxtral Realtime, a natively streaming automatic speech recognition model that matches offline transcription quality at sub-second latency. Unlike approaches that adapt offline models through chunking or sliding windows, Voxtral Realtime is trained end-to-end for streaming, with explicit alignment between audio and text streams. Our architecture builds on the Delayed Streams Modeling framework, introducing a new causal audio encoder and Ada RMS-Norm for improved delay conditioning. We scale pretraining to a large-scale dataset spanning 13 languages. At a delay of 480ms, Voxtral Realtime achieves performance on par with Whisper, the most widely deployed offline transcription system. We release the model weights under the Apache 2.0 license.

1 Introduction

Voxtral Realtime addresses the challenge of achieving offline-level ASR quality during streaming at sub-second latency. It combines native streaming architecture, delay conditioning, multilingual pretraining, and open-weight release.

  • Real-time applications require transcription while audio streams, creating a central challenge in matching offline ASR quality under strict latency constraints.
  • Adapting offline models through short chunks creates a training–inference mismatch that degrades accuracy as latency is reduced.
  • Voxtral Realtime is a 4B-parameter natively streaming ASR model supporting 13 languages.
  • A causal audio encoder and Ada RMS-Norm enable native streaming and operation at delays that are multiples of 80 ms.
  • Large-scale pretraining across 13 languages is used to support generalization across languages and domains.
  • At 480 ms, Voxtral Realtime performs competitively with Whisper and ElevenLabs Scribe v2 Realtime, while higher delays match or surpass strong offline baselines.
  • The resulting model is released as open weights under the Apache 2.0 license.

2 Modeling

Voxtral Realtime combines a causal audio encoder, temporal adapter, and synchronous autoregressive decoder for bounded-latency streaming. Delay conditioning and placeholder emissions let one model control when text is produced while maintaining bounded memory.

  • Architecture: Voxtral Realtime uses a causal audio encoder, temporal adapter, and Transformer decoder that fuses current audio with the latest generated-token embedding.The decoder generates text autoregressively in synchrony with the downsampled audio stream.
  • Audio Encoder: The causal encoder emits embeddings every 20 ms and uses a 750-frame sliding window to bound memory while supporting unbounded streaming.Its convolutional stem requires a four-frame history buffer during streaming inference.
  • Architecture: The adapter downsamples encoder outputs by 4x, producing a 12.5 Hz stream in which each embedding represents 80 ms of audio.This reduces the effective sequence length processed by the language decoder.
  • Language Decoder: At each 80 ms adapter step, the decoder emits either a text token or a placeholder while waiting for sufficient acoustic evidence.This delayed-streams mechanism learns emission timing without external VAD or forced alignments.
  • Language Decoder: AdaRMSNorm conditions every decoder block’s feed-forward branch on target delay τ, while the attention branch remains unconditioned.The delay embedding is projected through a small GELU MLP; the mechanism adds 5M parameters to the 4.4B model.
  • Language Decoder: The decoder uses an 8192-token left-context sliding window, which together with the encoder window supports arbitrarily long streams with bounded memory.

3 Training

Training constructs frame-synchronous targets from audio, text, and word timestamps so the decoder learns audio–text alignment and emission timing end to end. Delay values are varied during training, alongside staged optimization and regularization to preserve audio conditioning.

  • Target Construction: Training uses audio, text, and word-level timestamps to construct frame-synchronous targets for the language decoder.
  • Target Construction: The decoder emits exactly one token per 80 ms frame, using [P] for waiting and [W] to mark delayed word-generation onset.Subword tokens follow [W] for the corresponding word.
  • Target Construction: Consecutive words sharing an emission frame omit an additional [W] token, preserving the language decoder’s text-modeling capabilities.
  • Target Construction: The [W] emission frame implicitly aligns an audio segment with subsequent text tokens, without forced alignments or explicit decoding policies.The same learned mechanism determines whether to wait or initiate text generation at inference.
  • Delay Conditioning: Training samples target delay τ uniformly from 80 ms to 2400 ms in 80 ms increments, enabling inference at any delay in that range.Ada RMS-Norm provides the delay conditioning mechanism.
  • Optimization: Optimization begins with a 5% encoder warm-up, followed by 95% end-to-end joint training, while a z-loss penalty counters language-decoder logit growth.The warm-up protects pretrained decoder representations from the randomly initialized encoder.

4 Inference and Serving in vLLM

The vLLM deployment supports continuous audio ingestion and token emission through resumable, full-duplex streaming sessions. Custom attention metadata reconciles the encoder and decoder KV caches, whose frame rates differ because of temporal pooling.

  • Serving Architecture: Realtime serving combines paged attention, resumable sessions, and a WebSocket endpoint for incremental audio ingestion and token output.
  • KV Caches: The encoder and decoder maintain separate KV caches at 50 Hz and 12.5 Hz, so one decoder step corresponds to four new encoder KV positions.The rate difference comes from the adapter’s p = 4 temporal pooling.
  • KV Caches: Standard paged attention assumes uniform KV-position increments, causing inconsistent indexing when encoder and decoder rates differ.
  • KV Caches: A custom attention-metadata backend scales encoder block sizes, sequence metadata, query offsets, and slot mappings by the pooling factor p = 4.This keeps paged-attention indexing consistent across both caches.
  • Streaming Sessions: Resumable requests preserve an anchor session’s KV blocks so newly arrived audio can be appended while previously computed states are reused.The deployment pipelines I/O and computation across incremental updates.
  • Streaming Sessions: Async input and output generators enable full-duplex streaming, allowing audio ingestion and token emission to proceed concurrently.
  • Streaming Sessions: The WebSocket API provides bidirectional incremental audio and token streaming over one persistent connection with low per-message overhead.

5 Results

Voxtral Realtime is evaluated across English and multilingual benchmarks against offline, realtime, and open-source streaming systems. Its accuracy improves with delay and reaches offline-level quality at sub-second latency while outperforming open-source streaming baselines at comparable latencies.

  • Evaluation: Evaluation covers English and multilingual benchmarks, including the 13-language FLEURS benchmark and macro-average WER across four benchmark categories.The categories are English short-form, English long-form, FLEURS, and Mozilla Common Voice.
  • Latency–Accuracy Trade-off: At 480 ms, Voxtral Realtime approaches Scribe v2 Realtime and Whisper accuracy; at 960 ms, it surpasses both.
  • Latency–Accuracy Trade-off: At 2400 ms, Voxtral Realtime reaches accuracy within 1% of Voxtral Mini Transcribe V2.
  • Comparisons: Across all benchmark categories, Voxtral Realtime substantially outperforms existing open-source streaming baselines at comparable latencies.DSM requires substantially higher delays for competitive accuracy, while Nemotron Streaming shows reduced long-form robustness.
  • Comparisons: Voxtral Realtime consistently improves as latency increases and maintains strong performance across distributions.
  • Conclusion: Together, the results show offline-level transcription quality at sub-second latency with a natively streaming architecture.

6 Analysis

The analysis evaluates delay conditioning, target construction, and left-padding, finding that specific design choices improve convergence and transcription accuracy.

  • Delay conditioning: Ada RMS-Norm injects delay information into the decoder’s residual stream, unlike sinusoidal-sum and special-token conditioning strategies.
  • Delay conditioning: Ada RMS-Norm produces faster convergence and lower overall WER than summing or special-token conditioning on three FLEURS languages.
  • Target construction: Grouping consecutive words without a boundary token yields faster convergence and lower overall WER than inserting a [W] token between words.Grouping preserves the subword sequences encountered during language-model pretraining.
  • Left-padding: Increasing left-padding from 0 to 16 frames improves results across task categories, while 32 frames adds gains except on MCV.The authors hypothesize that left-padding introduces initial tokens analogous to attention sinks.

7 Conclusion

The conclusion presents Voxtral Realtime as a natively streaming ASR model combining causal encoding, Ada RMS-Norm conditioning, and training that leverages a pretrained decoder.

  • Voxtral Realtime combines a causal encoder, Ada RMS-Norm conditioning, and a training pattern that leverages pretrained language-decoder capabilities.
  • The model achieves near-offline performance at sub-second latency for live transcription, voice assistants, and interactive speech interfaces across its supported language coverage.
  • The model weights are released as open weights under the Apache 2.0 license.

A.1 Speech Recognition Results

The appendix reports speech-recognition evaluations across English short-form and long-form tasks, multilingual benchmarks, and language-specific error-rate results.

  • English short-form: Table 5 reports short-form English results for LibriSpeech, GigaSpeech, VoxPopuli, SwitchBoard, CallHome, CHiME-4, SPGISpeech, TED-LIUM, and Earnings-22.
  • English long-form: English long-form evaluation includes Meanwhile, long-form TED-LIUM, and segmented ten-minute variants of Earnings-21 and Earnings-22.
  • Multilingual results: FLEURS and Mozilla Common Voice provide per-language error-rate breakdowns, with full results reported for the listed languages.
  • Metric conventions: Chinese and Japanese use character error-rate, while the other reported languages use word error-rate; Arabic is omitted from macro-averages because all models exceed 45%.

A.2 vLLM Realtime Inference

The vLLM integration supports resumable streaming sessions that append audio and previously emitted tokens while reusing cached decoder states.

  • An anchor request buffers initial audio and runs one decoder step, after which resumable updates append 80 ms audio chunks and the previously emitted token ID.
  • Cached KV states allow the engine to emit tokens incrementally across streaming updates.
Loading 2602.11298v3…