Source-linked AI summary
VibeVoice-ASR-Streaming Technical Report
Yujie Tu, Zhiliang Peng, Jianwei Yu, Li Dong, Songchen Xu, Yaoyao Chang, Wenhui Wang, Zilong Wang, Zehua Wang, Yan Xia, Jiajun Zhang, Xie Chen, Furu Wei
TL;DR
Existing unified speaker-attributed ASR systems are mainly offline, leaving a low-latency gap for real-time assistants. VIBEVOICE-ASR-STREAMING interleaves audio chunks, lookahead, and prior text in one model, achieving strong recognition and speaker attribution while retaining known overlap and startup-latency limits.
Problem
Existing unified speaker-attributed ASR models mainly support offline recognition, while real-time assistants require low-latency output as speech arrives.
Method
VIBEVOICE-ASR-STREAMING interleaves fixed-size speech chunks, fixed lookahead, and generated speaker-attributed text in one autoregressive context without a separate diarization stage.
Results
The 7B model has the lowest five-set mean recognition error and achieves the best or tied-best cpWER/cpCER on 12 of 13 settings at an expected 2.00 s speaker-attribution latency.
Takeaways & Limitations
A single end-to-end LLM can produce speaker-attributed transcripts incrementally while preserving strong recognition and attribution performance.
Takeaways & Limitations
Long-duration overlapping speech degrades performance because overlap must be serialized, and first-output latency is 3.5 s for the 22-frame configuration.
Abstract
from arXiv · showhide
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
1 Introduction
VIBEVOICE-ASR-STREAMING targets low-latency speaker-attributed ASR by combining streaming recognition with persistent speaker history in one model. It reports strong recognition and speaker-attribution performance while releasing model weights and inference code.
- Streaming speaker-attributed ASR must emit words and speaker labels as conversation unfolds, enabling voice agents to identify speakers during interaction.
- Existing unified LLM-based systems assign speakers in one generative pass but generally read the whole recording before emitting output.
- Persistent retained history is essential because speakers may reappear minutes later and must receive consistent labels.
- VIBEVOICE-ASR-STREAMING interleaves incoming audio with generated speaker-attributed text, retaining conversation history without a separate diarization stage.
- The 7B 22-frame configuration achieves the best or tied-best speaker-attributed error on 12 of 13 settings and the best recognition-only mean among compared streaming systems.
- The report releases 1.5B and 7B model weights with inference code and studies chunk size, lookahead, model scale, speaker-label placement, comparisons, and serving cost.
2 Related Work
Related work spans unified offline speaker-attributed ASR, streamable LLM-based ASR, and streaming multi-talker recognition. VIBEVOICE-ASR-STREAMING addresses the remaining combination of streaming generation and direct speaker attribution.
- Recent LLM-based systems unify transcription and speaker information for long-form, multi-speaker recordings, but their processing is primarily offline.
- Together, these lines motivate a single LLM-based system that streams speaker-attributed transcripts directly rather than combining separate stages.
- Streaming LLM-based ASR uses chunked audio, autoregressive text generation, and preceding speech-text context carried across chunks.
- Earlier streaming multi-talker systems serialize or separate speakers, while speaker-attributed variants add speaker-related components to recover identity.
3 Method
The method interleaves speech chunks, speaker-attributed text, retained history, and fixed lookahead in one autoregressive sequence. It trains this formulation through staged adaptation and emits serialized speaker-labeled transcripts.
- Architecture and Streaming Formulation: The architecture interleaves Xk speech chunks and Yk speaker-attributed text chunks in one autoregressive context, preserving prior speech and text history.
- Architecture and Streaming Formulation: Retained history fixes recurring speaker identities; discarding it would require an external speaker store, cache, or clustering pass.
- Architecture and Streaming Formulation: Each chunk receives L = 4 latent frames of lookahead before its transcription is generated.
- Architecture and Streaming Formulation: The evaluated configurations use 15 latent frames for 2.0 seconds or 22 latent frames for 2.9 seconds per chunk.
- Architecture and Streaming Formulation: The model generates speaker-attributed text after the speech span closes, and a special end token returns control to the next audio chunk.
- Architecture and Streaming Formulation: Each chunk output contains ordinal speaker-labeled utterances, reusing labels assigned at first appearance; overlapping speech is serialized consecutively.
- Limitations: The released checkpoints target recordings of up to eight minutes, while retaining uncompressed history incurs cost that grows linearly with recording length.
- Training Route: Training uses three stages: offline speaker-attributed training, streaming pre-training, and streaming fine-tuning with different sample constructions.
4 Results
Using 7B and 1.5B streaming configurations evaluated across multilingual meeting and single-speaker benchmarks, VIBEVOICE-ASR-STREAMING delivers strong recognition and speaker attribution with low expected latency. Its speaker-attributed results are strongest on meeting-oriented evaluations, while single-speaker short-form audio is treated as a class check.
- Evaluation setup: The evaluation covers AISHELL-4, AliMeeting, AMI-IHM, AMI-SDM, and nine MLC-Challenge languages, using WER/CER and speaker-attributed cpWER/cpCER.Chinese, Japanese, and Korean use character-level scoring, while the remaining languages use word-level scoring.
- Latency: 2.00 s is the expected speaker-attribution latency for the 22-frame configuration, versus 8.21 s for Azure CT and 9.12 s for Google STT.The 15-frame configuration has an expected latency of 1.53 s; Google STT labels continue being revised tens of seconds later.
- Recognition results: 24.66 is VIBEVOICE-ASR-STREAMING’s five-set mean recognition error, below Gemini 3.5 Transcribe Live at 25.23 and the other listed systems.It is best on AISHELL-4, AliMeeting, and AMI-IHM, while Gemini 3.5 Transcribe Live is best on AMI-SDM.
- Speaker-attributed results: 12 of 13 speaker-attributed settings have the best or tied-best cpWER/cpCER, with 2.39–12.45-point improvements over Azure CT on four meeting benchmarks.The model is also best or tied-best on eight of nine MLC-Challenge languages.
- Single-speaker check: The model wins no individual single-speaker short-form test set, but its 22-frame configuration ranks second on AISHELL-1 and both LibriSpeech splits.Its four-set mean is level with the best, and 22-frame chunks outperform 15-frame chunks on all four sets.
5 Experiments
The experiments characterize accuracy, latency, and design trade-offs in streaming speaker-attributed ASR. Larger chunks, larger models, and more lookahead improve speaker attribution, while label placement has little effect on accuracy.
- WER/CER rises by 0.75 to 3.53 points and cpWER/cpCER by 5.13 to 6.67 when converting the 7B model to streaming on every benchmark.The larger cpWER/cpCER degradation suggests an additional loss associated with speaker attribution.
- Chunk size and model scale: 1.46 WER/CER and 4.06 cpWER/cpCER are gained at 7B when increasing chunks from 15 to 22 frames under identical 4-frame lookahead.At 1.5B, the corresponding gains are 1.31 WER/CER and 2.91 cpWER/cpCER.
- Chunk size and model scale: 12.76 cpWER/cpCER points are gained by scaling from 1.5B to 7B at 22 frames, compared with 4.69 WER/CER points.At 15 frames, scaling gains 11.61 cpWER/cpCER and 4.54 WER/CER points.
- Chunk size and model scale: 1.53 s expected latency at 15 frames versus 2.00 s at 22 frames creates a latency–accuracy trade-off between the two chunk settings.The shorter configuration reduces expected latency while the longer configuration improves both metrics.
- Speaker-label placement: 31.55 versus 31.56 mean cpWER/cpCER shows equivalent accuracy for head and tail speaker-label placement, while tail trails head by 0.98 WER/CER.The head placement is the released configuration and makes the label available from the first token of a segment.
- Lookahead depth: 1.27 WER/CER and 1.80 cpWER/cpCER are gained from the first two lookahead frames, while the second two add 1.24 and 2.54 respectively.Each two-frame increase costs 0.267 s of expected latency, and both metrics improve strictly through 4 frames.
- Real-time factor: 0.104 or lower real-time factor is achieved by the 7B model with 15-frame chunks, whose chunks decode in 146–208 ms against 2000 ms of audio.This checks the configuration with the least latency margin.
6 Conclusion and Limitations
VIBEVOICE-ASR-STREAMING produces speaker-attributed transcripts in a single streaming pass, with strong recognition and attribution results. The report also identifies limitations involving overlap, recording length, multilingual coverage, and startup latency.
- VIBEVOICE-ASR-STREAMING interleaves incoming speech chunks with generated speaker-attributed text, without a separate diarization stage.
- The 7B model achieves the lowest five-set mean recognition error among compared streaming systems and is best or tied-best on 12 of 13 speaker-attributed settings.At 2.00 s expected speaker-attribution latency, it improves over Azure ConversationTranscriber by 2.39 to 12.45 points on meeting benchmarks and from 27.06 to 22.75 on the MLC-Challenge average.
- The report releases model weights and high-performance inference code with vLLM support, together with an online demo.
- Training and evaluation currently cover ten languages because word-level alignments depend on the supported languages of Qwen3-ForcedAligner-0.6B.Broader multilingual coverage is left to future work.
- Long-duration overlap degrades performance because the decoder serializes overlapping speech into a single output stream.The report suggests separation-aware representations or multi-stream decoding as possible remedies.
- Released checkpoints support recordings of up to eight minutes, while longer recordings require more context and computation.
- The reported 2.00 s latency is steady-state; first output requires 3.5 s at 22 frames or 2.5 s at 15 frames.Reducing startup delay without sacrificing the accuracy of larger chunks remains future work.
A Output Format, Target Construction, and Training Details
The system emits speaker-labeled text chunk by chunk, with targets assigned by word-end boundaries and training performed under specified optimization and sequence-packing settings.
- Output Format: Speaker labels are emitted as plain-text “Speaker k:” lines, with no timestamps, and speech-free chunks produce empty output.
- Target Construction: A word belongs to chunk k when its end time is at or before that chunk boundary, within a 0.1-second tolerance.
- Output Format: The released 22-frame configuration receives one chunk every 2.9 seconds.
- Training Details: Stages 2 and 3 use AdamW, gradient clipping at 2.0, bfloat16, cosine learning-rate decay peaking at 5 × 10^-5, and sequences packed to 8,192 tokens.Stage 2 runs over a 420,000-hour corpus; Stage 3 runs for 500 steps on eight GPUs with a global batch of 64 and 35 warmup steps, with checkpoints at step 400.
B Evaluation Protocol and Baseline Systems
Evaluation standardizes recognition and speaker-attribution scoring across languages and systems, while latency is defined algorithmically for the proposed model and measured operationally for cloud services.
- Scoring: Chinese, Japanese, and Korean use CER and cpCER, while other languages use WER and cpWER.All systems use the same normalization and MeetEval cpWER implementation, with every utterance counted.
- Latency: The proposed system’s expected algorithmic delay is C/2 + Tlookahead.
- Latency: For each emitted result j, ℓj measures the delay until its speaker label settles, while nj and dj denote word count and duration.The expected word position is represented by dj/2; word counts follow the scoring tokenization, and baseline audio is pushed no faster than real time.
- Baseline Systems: VIBEVOICE-ASR-STREAMING receives none of the side information supplied to the systems listed in Table 9.
- Baseline Systems: Azure ConversationTranscriber is evaluated through the Python SDK with final transcribed events and no supplied speaker count.
- Baseline Systems: Google Cloud Speech-to-Text uses streaming recognition with diarization, interim results, punctuation, word offsets, and speaker-count bounds enabled for English recordings.