PaperlayerView exact revision on arXiv

Source-linked AI summary

VibeVoice-ASR-Streaming Technical Report

Yujie Tu, Zhiliang Peng, Jianwei Yu, Li Dong, Songchen Xu, Yaoyao Chang, Wenhui Wang, Zilong Wang, Zehua Wang, Yan Xia, Jiajun Zhang, Xie Chen, Furu Wei

arXiv:2609.02812v1eess.AS

TL;DR

Existing unified speaker-attributed ASR systems are mainly offline, leaving a low-latency gap for real-time assistants. VIBEVOICE-ASR-STREAMING interleaves audio chunks, lookahead, and prior text in one model, achieving strong recognition and speaker attribution while retaining known overlap and startup-latency limits.

  • Problem

    Existing unified speaker-attributed ASR models mainly support offline recognition, while real-time assistants require low-latency output as speech arrives.

    Source: §1 · p. 2

  • Method

    VIBEVOICE-ASR-STREAMING interleaves fixed-size speech chunks, fixed lookahead, and generated speaker-attributed text in one autoregressive context without a separate diarization stage.

    Source: §1 · p. 2, §3.1 · p. 3, §6 · p. 9

  • Results

    The 7B model has the lowest five-set mean recognition error and achieves the best or tied-best cpWER/cpCER on 12 of 13 settings at an expected 2.00 s speaker-attribution latency.

    Source: §6 · p. 10

  • Takeaways & Limitations

    A single end-to-end LLM can produce speaker-attributed transcripts incrementally while preserving strong recognition and attribution performance.

    Source: §6 · pp. 9–10

    Long-duration overlapping speech degrades performance because overlap must be serialized, and first-output latency is 3.5 s for the 22-frame configuration.

    Source: §6 · p. 10

Abstract

from arXiv · show

Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.

1 Introduction

VIBEVOICE-ASR-STREAMING targets low-latency speaker-attributed ASR by combining streaming recognition with persistent speaker history in one model. It reports strong recognition and speaker-attribution performance while releasing model weights and inference code.

  • Streaming speaker-attributed ASR must emit words and speaker labels as conversation unfolds, enabling voice agents to identify speakers during interaction.Source: §1 · p. 2
  • Existing unified LLM-based systems assign speakers in one generative pass but generally read the whole recording before emitting output.Source: §1 · p. 2
  • Persistent retained history is essential because speakers may reappear minutes later and must receive consistent labels.Source: §1 · p. 2
  • VIBEVOICE-ASR-STREAMING interleaves incoming audio with generated speaker-attributed text, retaining conversation history without a separate diarization stage.Source: §1 · p. 2
  • The 7B 22-frame configuration achieves the best or tied-best speaker-attributed error on 12 of 13 settings and the best recognition-only mean among compared streaming systems.Source: §1 · p. 2
  • The report releases 1.5B and 7B model weights with inference code and studies chunk size, lookahead, model scale, speaker-label placement, comparisons, and serving cost.Source: §1 · p. 2

2 Related Work

Related work spans unified offline speaker-attributed ASR, streamable LLM-based ASR, and streaming multi-talker recognition. VIBEVOICE-ASR-STREAMING addresses the remaining combination of streaming generation and direct speaker attribution.

  • Recent LLM-based systems unify transcription and speaker information for long-form, multi-speaker recordings, but their processing is primarily offline.Source: §2 · pp. 2–3
  • Together, these lines motivate a single LLM-based system that streams speaker-attributed transcripts directly rather than combining separate stages.Source: §2 · p. 3
  • Streaming LLM-based ASR uses chunked audio, autoregressive text generation, and preceding speech-text context carried across chunks.Source: §2 · p. 3
  • Earlier streaming multi-talker systems serialize or separate speakers, while speaker-attributed variants add speaker-related components to recover identity.Source: §2 · p. 3

3 Method

The method interleaves speech chunks, speaker-attributed text, retained history, and fixed lookahead in one autoregressive sequence. It trains this formulation through staged adaptation and emits serialized speaker-labeled transcripts.

Architecture and Streaming Formulation

  • The architecture interleaves Xk speech chunks and Yk speaker-attributed text chunks in one autoregressive context, preserving prior speech and text history.Source: §3.1 · p. 3
  • Retained history fixes recurring speaker identities; discarding it would require an external speaker store, cache, or clustering pass.Source: §3.1 · p. 3
  • Each chunk receives L = 4 latent frames of lookahead before its transcription is generated.Source: §3.1 · p. 4, Figure 2 · p. 4
  • The evaluated configurations use 15 latent frames for 2.0 seconds or 22 latent frames for 2.9 seconds per chunk.Source: §3.1 · p. 4
  • The model generates speaker-attributed text after the speech span closes, and a special end token returns control to the next audio chunk.Source: §3.1 · p. 4
  • Each chunk output contains ordinal speaker-labeled utterances, reusing labels assigned at first appearance; overlapping speech is serialized consecutively.Source: §3.1 · p. 5

Limitations

  • The released checkpoints target recordings of up to eight minutes, while retaining uncompressed history incurs cost that grows linearly with recording length.Source: §3.1 · p. 5

Training Route

  • Training uses three stages: offline speaker-attributed training, streaming pre-training, and streaming fine-tuning with different sample constructions.Source: §3.3 · p. 5

4 Results

Using 7B and 1.5B streaming configurations evaluated across multilingual meeting and single-speaker benchmarks, VIBEVOICE-ASR-STREAMING delivers strong recognition and speaker attribution with low expected latency. Its speaker-attributed results are strongest on meeting-oriented evaluations, while single-speaker short-form audio is treated as a class check.

Evaluation setup

  • The evaluation covers AISHELL-4, AliMeeting, AMI-IHM, AMI-SDM, and nine MLC-Challenge languages, using WER/CER and speaker-attributed cpWER/cpCER.Chinese, Japanese, and Korean use character-level scoring, while the remaining languages use word-level scoring.Source: §4 · p. 6

Latency

  • 2.00 s is the expected speaker-attribution latency for the 22-frame configuration, versus 8.21 s for Azure CT and 9.12 s for Google STT.The 15-frame configuration has an expected latency of 1.53 s; Google STT labels continue being revised tens of seconds later.Source: §4 · p. 7

Recognition results

  • 24.66 is VIBEVOICE-ASR-STREAMING’s five-set mean recognition error, below Gemini 3.5 Transcribe Live at 25.23 and the other listed systems.It is best on AISHELL-4, AliMeeting, and AMI-IHM, while Gemini 3.5 Transcribe Live is best on AMI-SDM.Source: §4 · p. 7

Speaker-attributed results

  • 12 of 13 speaker-attributed settings have the best or tied-best cpWER/cpCER, with 2.39–12.45-point improvements over Azure CT on four meeting benchmarks.The model is also best or tied-best on eight of nine MLC-Challenge languages.Source: §4 · p. 7

Single-speaker check

  • The model wins no individual single-speaker short-form test set, but its 22-frame configuration ranks second on AISHELL-1 and both LibriSpeech splits.Its four-set mean is level with the best, and 22-frame chunks outperform 15-frame chunks on all four sets.Source: §4 · p. 7

5 Experiments

The experiments characterize accuracy, latency, and design trade-offs in streaming speaker-attributed ASR. Larger chunks, larger models, and more lookahead improve speaker attribution, while label placement has little effect on accuracy.

  • WER/CER rises by 0.75 to 3.53 points and cpWER/cpCER by 5.13 to 6.67 when converting the 7B model to streaming on every benchmark.The larger cpWER/cpCER degradation suggests an additional loss associated with speaker attribution.Source: §5 · p. 8

Chunk size and model scale

  • 1.46 WER/CER and 4.06 cpWER/cpCER are gained at 7B when increasing chunks from 15 to 22 frames under identical 4-frame lookahead.At 1.5B, the corresponding gains are 1.31 WER/CER and 2.91 cpWER/cpCER.Source: §5 · p. 8
  • 12.76 cpWER/cpCER points are gained by scaling from 1.5B to 7B at 22 frames, compared with 4.69 WER/CER points.At 15 frames, scaling gains 11.61 cpWER/cpCER and 4.54 WER/CER points.Source: §5 · p. 8
  • 1.53 s expected latency at 15 frames versus 2.00 s at 22 frames creates a latency–accuracy trade-off between the two chunk settings.The shorter configuration reduces expected latency while the longer configuration improves both metrics.Source: §5 · p. 8, Table 5 · p. 9

Speaker-label placement

  • 31.55 versus 31.56 mean cpWER/cpCER shows equivalent accuracy for head and tail speaker-label placement, while tail trails head by 0.98 WER/CER.The head placement is the released configuration and makes the label available from the first token of a segment.Source: §5 · p. 8, Table 6 · p. 9

Lookahead depth

  • 1.27 WER/CER and 1.80 cpWER/cpCER are gained from the first two lookahead frames, while the second two add 1.24 and 2.54 respectively.Each two-frame increase costs 0.267 s of expected latency, and both metrics improve strictly through 4 frames.Source: §5 · p. 9

Real-time factor

  • 0.104 or lower real-time factor is achieved by the 7B model with 15-frame chunks, whose chunks decode in 146–208 ms against 2000 ms of audio.This checks the configuration with the least latency margin.Source: §5 · p. 9

6 Conclusion and Limitations

VIBEVOICE-ASR-STREAMING produces speaker-attributed transcripts in a single streaming pass, with strong recognition and attribution results. The report also identifies limitations involving overlap, recording length, multilingual coverage, and startup latency.

  • VIBEVOICE-ASR-STREAMING interleaves incoming speech chunks with generated speaker-attributed text, without a separate diarization stage.Source: §6 · p. 9
  • The 7B model achieves the lowest five-set mean recognition error among compared streaming systems and is best or tied-best on 12 of 13 speaker-attributed settings.At 2.00 s expected speaker-attribution latency, it improves over Azure ConversationTranscriber by 2.39 to 12.45 points on meeting benchmarks and from 27.06 to 22.75 on the MLC-Challenge average.Source: §6 · p. 10
  • The report releases model weights and high-performance inference code with vLLM support, together with an online demo.Source: §6 · p. 10
  • Training and evaluation currently cover ten languages because word-level alignments depend on the supported languages of Qwen3-ForcedAligner-0.6B.Broader multilingual coverage is left to future work.Source: §6 · p. 10
  • Long-duration overlap degrades performance because the decoder serializes overlapping speech into a single output stream.The report suggests separation-aware representations or multi-stream decoding as possible remedies.Source: §6 · p. 10
  • Released checkpoints support recordings of up to eight minutes, while longer recordings require more context and computation.Source: §6 · p. 10
  • The reported 2.00 s latency is steady-state; first output requires 3.5 s at 22 frames or 2.5 s at 15 frames.Reducing startup delay without sacrificing the accuracy of larger chunks remains future work.Source: §6 · p. 10

A Output Format, Target Construction, and Training Details

The system emits speaker-labeled text chunk by chunk, with targets assigned by word-end boundaries and training performed under specified optimization and sequence-packing settings.

Output Format

Target Construction

Output Format

Training Details

  • Stages 2 and 3 use AdamW, gradient clipping at 2.0, bfloat16, cosine learning-rate decay peaking at 5 × 10^-5, and sequences packed to 8,192 tokens.Stage 2 runs over a 420,000-hour corpus; Stage 3 runs for 500 steps on eight GPUs with a global batch of 64 and 35 warmup steps, with checkpoints at step 400.Source: A Output Format, Target Construction,… · p. 14

B Evaluation Protocol and Baseline Systems

Evaluation standardizes recognition and speaker-attribution scoring across languages and systems, while latency is defined algorithmically for the proposed model and measured operationally for cloud services.

Scoring

  • Chinese, Japanese, and Korean use CER and cpCER, while other languages use WER and cpWER.All systems use the same normalization and MeetEval cpWER implementation, with every utterance counted.Source: B Evaluation Protocol and Baseline… · p. 14

Latency

  • The proposed system’s expected algorithmic delay is C/2 + Tlookahead.Source: B Evaluation Protocol and Baseline… · p. 14
  • For each emitted result j, ℓj measures the delay until its speaker label settles, while nj and dj denote word count and duration.The expected word position is represented by dj/2; word counts follow the scoring tokenization, and baseline audio is pushed no faster than real time.Source: B Evaluation Protocol and Baseline… · p. 14

Baseline Systems

Related papers

Loading 2609.02812v1…