Source-linked AI summary
VIBEVOICE-ASR Technical Report
Zhiliang Peng, Jianwei Yu, Yaoyao Chang, Zilong Wang, Li Dong, Yingbo Hao, Yujie Tu, Chenyu Yang, Wenhui Wang, Songchen Xu, Yutao Sun, Hangbo Bao, Weijiang Xu, Yi Zhu, Zehua Wang, Ting Song, Yan Xia, Zewen Chi, Shaohan Huang, Liang Wang, Chuang Ding, Shuai Wang, Xie Chen, Furu Wei
TL;DR
Long-form speech systems struggle with fragmented context and the complexity of jointly recognizing content, speakers, and timing. VIBEVOICE-ASR uses single-pass structured generation with optional contextual prompts to address these challenges. It outperforms Gemini baselines on speaker attribution and temporal alignment, while retaining broad multilingual and speaker-aware capabilities.
Problem
Long-form audio remains difficult because chunked pipelines fragment context and separately handle recognition, diarization, and timestamping.
Method
VIBEVOICE-ASR performs single-pass autoregressive generation of Rich Transcription from continuous audio embeddings and optional contextual prompts.
Results
VIBEVOICE-ASR consistently outperforms Gemini-2.5-Pro and Gemini-3-Pro in DER and tcpWER across all evaluated datasets, with best cpWER on 11 of 16 settings.
Takeaways & Limitations
The framework provides unified speaker-aware, time-aligned long-form transcription with stronger speaker attribution and temporal consistency than the evaluated Gemini baselines.
Takeaways & Limitations
SFT focuses mainly on English, Chinese, and code-switching data, while serialized generation does not explicitly handle overlapping speech.
Abstract
from arXiv · showhide
This report presents VibeVoice-ASR, a general-purpose speech understanding framework built upon VibeVoice, designed to address the persistent challenges of context fragmentation and multi-speaker complexity in long-form audio (e.g., meetings, podcasts) that remain despite recent advancements in short-form speech recognition. Unlike traditional pipelined approaches that rely on audio chunking, VibeVoice-ASRsupports single-pass processing for up to 60 minutes of audio. It unifies Automatic Speech Recognition, Speaker Diarization, and Timestamping into a single end-to-end generation task. In addition, VibeVoice-ASR supports over 50 languages, requires no explicit language setting, and natively handles code-switching within and across utterances. Furthermore, we introduce a prompt-based context injection mechanism that allows users to supply customized conetxt, significantly improving accuracy on domain-specific terminology and polyphonic character disambiguation.
1 Introduction
Long-form audio remains difficult because chunked pipelines fragment context and separate recognition, diarization, and timestamping. VIBEVOICE-ASR addresses these problems with single-pass, unified generation and prompt-based context injection.
- Long-form audio such as meetings, podcasts, and lectures remains challenging despite progress in short-form speech recognition.
- Chunked pipelines lose cross-sentence context, hindering homophone disambiguation and coreference resolution in extended dialogue.
- Separate ASR, diarization, and timestamping models require heuristic reconciliation, allowing segmentation or diarization failures to corrupt transcripts.
- VIBEVOICE-ASR processes 60-minute audio in one pass by compressing it with a 7.5 Hz tokenizer, enabling simultaneous access to global context.
- The framework generates Rich Transcription that interleaves speaker identity, timestamps, and content, while prompts supply context for terminology and code-switching.
2 Method
VIBEVOICE-ASR formulates long-form speech understanding as autoregressive language modeling over continuous audio embeddings and optional text prompts. Its training combines curated, synthetic, restored long-form, and non-speech data, with pipeline quality evaluated against established alternatives.
- Overview: The model feeds continuous acoustic and semantic embeddings, optionally prepended with text prompts, into a decoder-only LLM to generate structured Rich Transcription.
- Overview: Rich Transcription interleaves speaker identity, temporal boundaries, and speech content, enabling recognition, diarization, and timestamping in one pass.
- Tokenization: Dual tokenizers provide spectral and linguistic representations, with hierarchical acoustic encoding compressed to approximately 7.5 tokens per second.
- Pre-training: The pre-training data pipeline segments, transcribes, diarizes, and filters long recordings before model training.
- Pre-training: The proposed data pipeline achieves lower DER and WER than WhisperX and Emilia across most evaluated meeting datasets.
- Supervised fine-tuning: SFT combines speech and music benchmarks, context-driven synthetic dialogues, globally rectified long-form transcripts, and tagged non-speech intervals.
- Supervised fine-tuning: Sampling weights for standard benchmarks, music, synthetic data, and refined long-form data are 0.5 : 0.1 : 0.1 : 0.3, respectively.
3 Results
The evaluation measures speaker attribution, recognition, speaker-consistent transcription, and time-aligned transcription. Against Gemini baselines, VIBEVOICE-ASR shows stronger diarization and temporal alignment, with broad advantages in speaker consistency and competitive recognition.
- Evaluation: DER, WER, cpWER, and tcpWER respectively assess speaker attribution, content recognition, speaker-permutation transcription, and time-constrained transcription quality.
- Baselines: Gemini-2.5-Pro and Gemini-3-Pro were evaluated as multimodal baselines, with test audio segmented into 240-second chunks for comparison.
- Results: VIBEVOICE-ASR consistently outperforms both Gemini variants in DER and tcpWER across all evaluated datasets.
- Results: VIBEVOICE-ASR achieves the best cpWER on 11 of 16 evaluation settings and the lowest WER on 8 of 16 settings.
- Results: Overall, the model shows particularly strong speaker attribution, temporal consistency, and multilingual generalization while balancing content recognition and speaker-aware transcription.
4 Conclusion and Limitations
The report presents VIBEVOICE-ASR as a unified single-pass framework and commits to releasing its weights, fine-tuning pipelines, and inference code. It identifies multilingual forgetting during SFT and limited handling of overlapping speech as key limitations.
- Conclusion: The report presents VIBEVOICE-ASR as a unified single-pass framework for long-form speech understanding.
- Conclusion: The authors plan comprehensive open-sourcing of model weights, fine-tuning pipelines, and high-performance inference code.
- Limitations: Although pre-training covers over 50 languages, SFT focuses predominantly on English, Chinese, and code-switching data, potentially degrading low-resource-language performance.
- Limitations: The serialized output stream does not explicitly handle overlapping speech and may miss secondary information when speakers talk simultaneously.
A Language Distribution of Training Data
Figure 3 shows the language distribution in the training data.
- Figure 3 presents the language distribution in the training data.