Omnilingual-MsASR: Benchmarking Multimodal, Multilingual, Long-Form Multi-Speaker ASR of Omni Large Language Models

1 Introduction

• Prior benchmarks leave gaps in evaluating long-form, multilingual, multi-speaker video transcription and visual cues for speaker identification and transcription.Traditional MsASR datasets cover monolingual English or Chinese, while multilingual dialogue benchmarks have shorter audio durations.Source: §1 · p. 1
• OMNILINGUAL-MSASR covers six languages, recordings from around 20 seconds to 30 minutes, and instances with up to 20 speakers.The benchmark totals 297 instances, roughly 50 videos per language, with annotations manually verified by experienced language experts.Source: §1 · pp. 1–2
• Gemini family models achieve low cpWER (0.23–0.44) but tcpWER frequently exceeds 1.0, while longer inputs and more speakers compound errors.Scaling from two to over seven speakers raises overall speaker confusion from 9.1% to 25.6%, and Gemini 3.7 Flash’s cpWER rises from 0.387 to 1.113.Source: §1 · p. 2
• The benchmark pairs audio and video inputs and uses an annotation pipeline integrating both modalities, followed by human refinement under standardized guidelines.The authors evaluate leading commercial and open-source omni-modal LLMs to characterize strengths and challenges in long-form MsASR.Source: §1 · p. 2

2.1 Omni-modal Large Language Models

• Omni-modal audio competence does not guarantee reliable speaker-aware transcription, which requires binding linguistic content to persistent speaker identities across turns and overlaps.The related work describes this as a concrete, though not exhaustive, indicator of fine-grained conversational understanding.Source: §2.1 · p. 2

2.2 Long-Form Video Understanding

• Long-form audio-visual benchmarks evaluate cross-modal reasoning over extended contexts, but their question-answering and event-description objectives do not capture dense speaker-aware transcription.That transcription jointly preserves speech content, speaker identity, and temporal boundaries.Source: §2.2 · p. 2

2.3 Multilingual Multi-Speaker ASR

• Existing multi-speaker evaluations provide limited evidence about general-purpose omni LLMs and their robustness across languages, durations, speaker counts, and overlap conditions.Earlier meeting and multilingual efforts remain concentrated on short-form audio, acoustic domains, or specialized speech systems.Source: §2.3 · p. 3

3.1 Overview

• OMNILINGUAL-MSASR contains 297 instances across six languages, with durations from 20 seconds to 30 minutes and up to 20 speakers per instance.It provides parallel audio-only and audio-visual evaluation to examine whether visual context helps speaker identification or introduces noise.Source: §3.1 · p. 3

3.2 Data Source

• The corpus draws public long-form videos across diverse domains, filtering out videos with fewer than two speakers and variety shows with excessive speech overlap.Domains include meetings, podcasts, news broadcasts, TV dramas, speeches, prayers, and daily vlogs.Source: §3.2 · pp. 3–4

3.3 Automatic Data Pipeline

• The automatic pipeline segments long recordings, generates clip-level multi-speaker annotations, then uses visual cues to reconcile speaker identities across clips.Speech enhancement and silence-based segmentation help prepare clips; SyncNet identifies active speaking faces and Gemini 3.1 Pro matches local speaker IDs to face crops before timestamps are shifted to the global timeline.Source: §3.3 · p. 4

3.4 Linguistic Expert Annotation

• Linguistic experts correct pipeline outputs and an independent expert pool reviews speaker identities, timestamps, and transcriptions against the videos.Authors spot-check a random subset, focusing mainly on language-agnostic speaker labels and timestamps.Source: §3.4 · p. 4

3.5 Ethical and Privacy Audit

• Three French instances containing social media handles or email addresses were removed in the privacy audit, leaving 297 samples.The audit followed a preliminary curation prioritizing balanced temporal distribution and content diversity.Source: §3.1 · p. 3, §3.5 · pp. 4–5

4.1 Experimental Setup

• The evaluation covers two open-source models and seven frontier commercial omni-modal LLMs.Open-source models run with default inference settings; the open-source model runs on 8x A100 GPUs.Source: §4.1 · p. 5

4.2 Evaluate Metrics

• Four metrics assess parse rate, diarization error rate (DER), concatenated minimum-permutation word error rate (cpWER), and time-constrained permutation WER (tcpWER).Parse rate measures instruction following; DER measures who spoke when, while cpWER jointly scores transcription and speaker attribution without timestamps.Source: §4.2 · p. 5
• Unparsable outputs are excluded from the three WER metrics so that a few extreme-error outliers do not distort the overall capability assessment.The excluded cases arise when models fail to follow instructions or produce repetitive generation.Source: §4.2 · p. 5
• DER sums missed-speech, false-alarm, and speaker-confusion time, with a 0.5 s forgiveness collar around each reference boundary.It measures speaker timing while ignoring transcribed content.Source: §4.2 · p. 5
• cpWER sums substitutions, deletions, and insertions after choosing the speaker pairing that minimizes edit distance between per-speaker concatenated transcripts, without using timestamps.The total is normalized by the number of reference words.Source: §4.2 · p. 5
• tcpWER allows word alignment only when reference and hypothesis timestamps fall within the collar τ, penalizing temporal misplacement even when text is correct.The evaluation uses τ = 5 s; its substitutions, deletions, and insertions are counted under that alignment constraint.Source: §4.2 · p. 5

5.1 Main Results

• Parse rates fall below 80% for Gemini 3.7 Flash on fr and ko and Qwen3.5-Omni-Plus on es; Thai performs markedly worse across models, with similar timbres complicating speaker identification.Source: §5.1 · pp. 5–6
• Across es, fr, ja, ko, and ru, cpWER is substantially lower than DER and tcpWER for Gemini, Muse Spark, and Doubao, while Qwen performs best on DER.Source: §5.1 · p. 6
• Commercial omni-LLMs substantially outperform open-source counterparts and approach specialized diarization models, although MOSS retains a clear lead over nearly all competing models.Source: §5.1 · p. 6
• MOSS-Transcribe-Diarize-7B and Qwen3.8-Omni-Flash deliver the strongest overall performance, ranking in the top three across virtually all evaluation metrics.Source: §5.1 · p. 6

5.2 Analysis of Audio Duration

• As duration increases, performance generally degrades; DER and tcpWER deteriorate more than cpWER, while Doubao-Seed-2.0-Lite and Gemini 3.7 Flash show parsing drops at 15–30 minutes.Source: Figure 4 · p. 7, §5.2 · p. 7
• 31.7% of the overall 54.4% DER came from missed speech, versus 10.2% false alarm and 12.5% speaker confusion, averaged across seven models.Source: §5.2 · p. 7

5.3 Analysis of Speaker Number

• Parse rates remain largely stable as speaker counts rise, but speaker-related metrics worsen; speaker confusion rises overall despite a slight fluctuation at five speakers.Source: Figure 5 · p. 7, §5.3 · pp. 7–8

5.4 Multi-speaker Video ASR

• Visual inputs do not consistently help: Doubao improves on tcpWER and DER, Gemini improves on cpWER but worsens on tcpWER and DER, and video reduces speaker miss rate while increasing confusion.Source: Figure 7 · p. 8, §5.4 · p. 8

• OMNILINGUAL-MSASR contains 297 instances with parallel audio and video inputs across six languages.Source: §6 · p. 8
• Experiments on frontier omni LLMs reveal temporal grounding and speaker disambiguation as key bottlenecks.Source: §6 · p. 9
• The paper identifies temporal grounding, speaker disambiguation, and audio-visual integration as areas for future progress in omni-modal LLMs.Source: §6 · p. 9

• Timestamp corrections were made only for obvious errors, potentially favoring MOSS on time-sensitive metrics; stochastic commercial APIs prevent guaranteed exact reproducibility.Source: Limitations · p. 9

Data source and release

• Speaker labels are anonymous indices local to each video, and instances can be removed upon request from content owners or depicted individuals.The videos are publicly available on YouTube, and the dataset is released for non-commercial research only under the platform’s terms of service.Source: Ethical Considerations · p. 9

Privacy

• Facial cues are used only to link speakers within each video, and face crops are not released.A privacy audit found three French instances with real social media handles and email addresses; those instances were removed.Source: Ethical Considerations · p. 9

Annotators

• Annotators were professional linguistic experts informed of the research purpose and fairly compensated; no personal information was collected about them.Source: Ethical Considerations · p. 9

Potential misuse

• The authors identify surveillance and profiling as potential misuses and discourage using the benchmark to identify or track specific people.The benchmark is designed solely to evaluate model capabilities.Source: Ethical Considerations · p. 9

Dataset organization

• The guidelines list Spanish, French, Japanese, Korean, Thai, and Russian, assign each expert only their assigned language, and describe 60 entries per language.Each entry is on a separate sheet.Source: A Specifics Document about Human… · p. 12

Record structure

• Starting from row 6, each speaker diarization record contains start and end timestamps, a 0-indexed speaker ID, and transcribed text, with only one speaker per record.Source: A Specifics Document about Human… · p. 12

Verification

• Experts check speech activity between each entry’s start and end times against diarization records for identity, timestamps, and speech content.They are instructed to open the YouTube video and skip to the entry’s start time.Source: A Specifics Document about Human… · p. 12

Corrections

• Incorrect timestamps, speaker labels, or transcripts are corrected, and a record containing multiple speakers is split into separate speaker-specific records.Each resulting record carries that speaker’s start and end times, ID, and spoken text.Source: A Specifics Document about Human… · p. 12

Additional rules

• An unindexed speaker receives a new ID, timestamps are recorded to one decimal place, and ensemble singing is treated according to its audible speaker transitions.A chorus beginning the video is one speaker; when individuals speak before clearly separated group singing, each speaker and the singing are segmented separately.Source: A Specifics Document about Human… · p. 12

Audio prompt

• The prompt assigns an ASR expert role and asks for spoken text with speaker labels and timestamps, using acoustic cues only to distinguish speakers.The transcript itself must come from what is spoken.Source: B Inference Prompt for MsASR · p. 12

Output format

• The required output concatenates utterance segments in a fixed order, from start marker and start time through text, end time, speaker ID, and end marker.The output is enclosed by <soc> and <eoc>.Source: B Inference Prompt for MsASR · p. 13

Speaker ID assignment

• Speaker IDs begin at zero and follow each speaker’s first appearance; later segments reuse that ID, while each new speaker gets the next integer.Source: B Inference Prompt for MsASR · p. 13

Video inference

• For video MsASR, the prompt is unchanged except that “audio” becomes “video”; MOSS-Transcribe-Diarize-0.9B uses official settings and default-format parsing.The passage contrasts MOSS parsing with the approach in Sec. 4.2.Source: B Inference Prompt for MsASR · p. 13

Data usage

• The dataset provides index references and text annotations, not copies or redistribution of third-party original audio or video.Users accessing referenced media must comply with the original platforms’ terms of service.Source: C Data Statement and License · p. 13
• The referenced material and dataset are for academic research, not commercial use, while copyright in original media remains with its creators and rights holders.Source: C Data Statement and License · p. 13

License

• The repository’s indexes and annotations are licensed under CC BY-NC 4.0, which does not cover rights in the original audio or video.Source: C Data Statement and License · p. 13

Offensive content

• Data containing profanity or other offensive content was removed during benchmark construction.Source: C Data Statement and License · p. 13