PaperlayerView exact revision on arXiv

Source-linked AI summary

MOSS Transcribe Diarize Technical Report

MOSI. AI, :, Donghua Yu, Zhengyuan Lin, Hanfu Chen, Chen Yang, Yiyang Zhang, Jingqi Chen, Ke Chen, Liwei Fan, Yi Jiang, Jie Zhu, Muchen Li, Wenxuan Wang, Yang Wang, Zhe Xu, Botian Jiang, Yitian Gong, Yuqian Zhang, Wenbo Zhang, Songlin Wang, Zhiyu Wu, Zhaoye Fei, Qinyuan Cheng, Shimin Li, Xipeng Qiu

arXiv:2601.01554v7cs.SDcs.AIeess.AS

TL;DR

Existing SATS systems often rely on modular components and face constraints in long-context processing, speaker memory, and native timestamp output. MOSS Transcribe Diarize jointly transcribes, attributes speakers, and predicts timestamps in one pass with a 128k-token context window. It outperforms strong closed-source systems across the reported benchmarks.

  • Problem

    Existing SATS systems often use separate ASR and diarization components, while limited context, fragile speaker memory, and lack of native segment timestamps constrain long-form transcription.

    Source: §1 · pp. 2–3

  • Method

    MOSS Transcribe Diarize is a unified multimodal model that jointly recognizes words, attributes speakers, and predicts timestamps in one pass, with a 128k-token context window for inputs up to 90 minutes.

    Source: §1 · p. 3

  • Results

    MOSS Transcribe Diarize outperforms strong closed-source systems in CER, cpCER, and Δcp across AISHELL-4, Podcast, and Movies.

    Source: §5 · p. 6

  • Takeaways & Limitations

    The reported results highlight long-context joint modeling and distribution-controlled simulation for end-to-end SATS at meeting scale.

    Source: §5 · p. 6

    GPT-4o and Gemini 3 Pro were omitted from some long-form benchmarks because of audio-length constraints or unreliable adherence to the required speaker-attributed output format.

    Source: AISHELL-4 · p. 5, §4.2 · p. 6

Abstract

from arXiv · show

Speaker-Attributed, Time-Stamped Transcription (SATS) aims to transcribe what is said and to precisely determine the timing of each speaker, which is particularly valuable for meeting transcription. Existing SATS systems rarely adopt an end-to-end formulation and are further constrained by limited context windows, weak long-range speaker memory, and the inability to output timestamps. To address these limitations, we present MOSS Transcribe Diarize, a unified multimodal large language model that jointly performs Speaker-Attributed, Time-Stamped Transcription in an end-to-end paradigm. Trained on extensive real wild data and equipped with a 128k context window for up to 90-minute inputs, MOSS Transcribe Diarize scales well and generalizes robustly. Across comprehensive evaluations, it outperforms state-of-the-art commercial systems on multiple public and in-house benchmarks.

MOSI.AI*

The section identifies Figure 1 as an overview of MOSS Transcribe Diarize’s key capabilities and directs readers to the Contributors section for the full contributor list.

MOSI.AI*

  • Figure 1 is titled as an overview of MOSS Transcribe Diarize’s key capabilities.Source: Figure 1 · p. 1
  • The full list of contributors is in the Contributors section.Source: MOSI.AI* · p. 1

1 Introduction

SATS calls for transcripts that identify who spoke and when, but existing systems face limits in end-to-end modeling, long-form context, speaker memory, and timestamp output. MOSS Transcribe Diarize addresses these gaps with a unified single-pass model and a 128k-token context window for inputs up to 90 minutes.

1 Introduction

  • Semi-cascaded systems use an ASR and diarization front end, then an LLM to reconcile candidate words and speaker traces; this remains non-end-to-end [20].The reconciliation layer can resolve speaker permutations and repair turn or overlap boundaries, but inherits cascaded error propagation and mismatch issues [20].Source: §1 · p. 2
  • Most existing MLLMs focus on single-speaker speech, while unified SATS approaches remain limited in audio duration, speaker count, or native timestamped output.Sortformer uses a two-stage training process; SpeakerLM [21] is reported for relatively short contexts and small speaker sets; these approaches do not natively output segment-level timestamps.Source: §1 · p. 2
  • Limited context, fragile long-range speaker memory, and lack of native segment timestamps constrain existing SATS systems.Chunking can disrupt continuity and timestamp boundaries, while external alignment can reintroduce modular error propagation [1].Source: §1 · p. 3
  • MOSS Transcribe Diarize jointly recognizes words, attributes speakers, and assigns timestamps in one pass, using a 128k-token context window for inputs up to 90 minutes.It is trained on in-the-wild conversational audio covering accents, environments, overlaps, and domain shifts.Source: §1 · p. 3
  • The paper presents MOSS Transcribe Diarize as the first unified multimodal model to jointly predict words, speaker attribution, and timestamps in a single forward pass.Source: §1 · p. 3
  • The 128k-token context supports inputs up to 90 minutes without chunking, preserving discourse continuity and long-range speaker memory.The paper says this reduces identity drift and boundary artifacts and improves SATS metrics across in-domain and out-of-domain evaluations.Source: §1 · p. 3

2 Model Architecture

The model architecture maps multi-speaker acoustic embeddings into a pretrained text LLM, while temporal information is represented as timestamp text for long-context processing.

2 Model Architecture

  • An audio encoder and projection module map multi-speaker acoustic embeddings into a pretrained text LLM for joint speaker and lexical modeling.The architecture supports unified, long-context modeling [6] [19] [22].Source: §2 · p. 3
  • Formatted timestamp text is inserted between audio encoder chunks, avoiding reliance on absolute positional indices for long-duration audio.The text states this enables accurate timestamp generation over hour-scale audio with stable speaker attribution [1].Source: §2 · pp. 3–4
  • Table 1 provides a statistical overview of the evaluation datasets.Source: Table 1 · p. 4

3 Data Composition

The study uses multilingual Internet audio and public-corpus clips for training, with AISHELL-4, podcast, and film datasets described as evaluation resources; simulated mixtures supplement real data to strengthen speaker attribution and timestamp prediction.

3.1 Real Data

  • 3.1 Real Data: Training draws on multilingual Internet audio and public-corpus clips covering real-world multi-speaker scenarios.AISHELL-4 meeting recordings include far-field overlapping audio and near-field recordings; the averaged far-field channel is used for training and evaluation, while podcasts and films provide two test sets.Source: §3.1 · p. 4

3.2 Simulated Data

  • 3.2 Simulated Data: Synthetic multi-speaker mixtures supplement training to strengthen speaker attribution and timestamp prediction.The simulator combines utterances from 2–12 speakers, permits overlaps up to 80 percent of the shorter segment, and adds real-world noise and reverberation at 0–15 dB SNR.Source: §3.2 · p. 4

4 Evaluation

The evaluation uses three benchmarks and complementary transcription and speaker-attribution metrics to test MOSS Transcribe Diarize against commercial systems. Across feasible evaluations, it performs strongly on both long-form and short, overlap-rich recordings.

4.1.1 Evaluation Datasets

  • The evaluation uses AISHELL-4 meeting recordings, multi-guest Podcast interviews, and short, overlap-rich Movies segments to span different conversational conditions.Movies also includes multiple languages and dialects, and its samples were manually annotated; Podcast and Movies are internally curated datasets slated for release.Source: §4.1.1 · p. 4

4.2 Performance

  • Across AISHELL-4, Podcast, and Movies, MOSS achieves the best overall cpCER and Δcp where evaluation is feasible, with advantages on long- and short-form recordings.Table 2 presents performance against other models; the reported findings describe results across the three benchmarks.Source: Table 2 · p. 5, §4.2 · p. 6

4.1.2 Metrics

  • CER measures transcript accuracy without speaker identities, while cpCER scores speaker-attributed transcripts after optimal speaker-label assignment and Δcp isolates the added degradation associated with attribution errors.Metrics are reported as percentages, with lower values indicating better performance.Source: §4.1.2 · p. 5

4.2 Performance

  • The commercial comparison includes Doubao, ElevenLabs, GPT-4o Transcribe Diarize, Gemini 2.5 Pro, and Gemini 3 Pro, but some systems cannot be evaluated on long-form audio under the protocol.GPT-4o is omitted from the first two benchmarks because of its audio input constraint; Gemini 3 Pro is excluded for instability in following the required format on long audio.Source: §4.2 · pp. 5–6, AISHELL-4 · p. 5

AISHELL-4

  • AISHELL-4 results show substantially better CER and cpCER than all baselines, with a markedly smaller Δcp on nearly 40-minute real-world meetings.GPT-4o and Gemini 3 Pro are omitted from the corresponding long-audio comparisons under the evaluation protocol.Source: AISHELL-4 · p. 5, §4.2 · p. 6

4.2 Performance

  • On Podcast, MOSS records the lowest CER and cpCER and the smallest Δcp among evaluated systems, supporting more reliable attribution across long discussions and speaker re-entrance.The benchmark contains high-quality, long-duration, multi-speaker discussions with frequent turn-taking.Source: §4.2 · p. 6
  • On Movies, MOSS outperforms all baselines in cpCER and Δcp, maintaining a relatively small gap between CER and cpCER amid rapid alternation and frequent overlap.Some commercial systems have competitive CER but substantially larger Δcp.Source: §4.2 · p. 6

5 Conclusions

MOSS Transcribe Diarize combines transcription, speaker attribution, and timestamp prediction in one pass, using long-context modeling and diverse conversational training data. It outperforms strong closed-source systems across three benchmarks, while streaming SATS, finer timestamp evaluation, and broader multilingual robustness remain future work.

5 Conclusions

  • MOSS Transcribe Diarize jointly performs transcription, speaker attribution, and timestamp prediction in a single pass with a 128k-token context window.The unified audio–text MLLM is evaluated across AISHELL-4, Podcast, and Movies.Source: §5 · p. 6
  • Training combines diverse in-the-wild conversations with property-aware simulated mixtures modeling overlap, turn-taking, and acoustic variability.The conclusion highlights long-context joint modeling and distribution-controlled simulation for end-to-end SATS at meeting scale.Source: §5 · p. 6
  • Across AISHELL-4, Podcast, and Movies, MOSS outperforms strong closed-source systems in CER, cpCER, and Δcp.The authors identify streaming SATS, finer-grained timestamp evaluation, and broader multilingual robustness as future work.Source: §5 · p. 6

Appendix Contents

The appendix contents list additional details, evaluation prompts, and output normalization for evaluation.

Appendix Contents

A.1 Evaluation Prompts

The appendix lists prompt material for several model and benchmark settings, then describes text normalization applied before scoring.

A.1 Evaluation Prompts

  • A listed entry pairs Gemini 2.5 Pro and Gemini 3 Pro with three named benchmarks: AISHELL-4, Podcast, and Movies.Source: §A.1 · p. 11
  • One prompt asks for dialogue transcription in the audio language and speaker labels such as [S1] and [S2].It requests only the final result, without extra text or line breaks.Source: §A.1 · p. 11
  • The appendix separately labels a set of MOSS Transcribe Diarize prompts.The prompt text itself is not included in this passage.Source: §A.1 · p. 11
  • An audio-input entry is listed for AISHELL-4 and Podcast.The passage labels the input as audio but does not specify further prompt instructions.Source: §A.1 · p. 11
  • Another prompt requests speaker labels, event and emotion tags, and markers for overlap and insertion.It asks for automatic language detection; speaker and overlap/insertion markers are specified in English, while event and emotion labels follow the audio language.Source: §A.1 · p. 11

A.2 Output Normalization for Evaluation

  • Before CER, cpCER, and Δcp scoring, predictions and references receive the same text normalization.The stated purpose is fair comparison across systems.Source: §A.2 · p. 11
  • Normalization removes parenthetical content, angle-bracket tags, and non-speaker square-bracket annotations, retaining speaker IDs and plain transcript text for scoring.Speaker tags such as [S1] and [S01] are retained, while event markers such as [event] are removed.Source: §A.2 · p. 11
Loading 2601.01554v7…