PaperlayerView exact revision on arXiv

Source-linked AI summary

TagSpeech: End-to-End Multi-Speaker ASR and Diarization with Fine-Grained Temporal Grounding

Mingyue Huo, Yiwen Shao, Yuheng Zhang

arXiv:2601.06896v2eess.AScs.CL

TL;DR

TagSpeech addresses the lack of precise, unified modeling of who spoke what and when in multi-speaker ASR and diarization. It uses decoupled semantic and speaker streams with interleaved numeric time anchors for temporal grounding, achieving improved DER on AMI and AliMeeting with parameter-efficient training. The approach remains limited by small training datasets and slower inference than optimized cascaded systems.

  • Problem

    Existing approaches often provide speaker-attributed transcription without explicit timestamps, limiting precise speech–text alignment and standard DER evaluation.

  • Method

    TagSpeech jointly models semantic and speaker streams with Serialized Output Training and interleaved numeric time anchors for fine-grained temporal grounding.

  • Results

    Approximately 28% relative DER improvement on AMI and 36% on AliMeeting is reported over strong end-to-end baselines.

  • Takeaways & Limitations

    TagSpeech achieves end-to-end prediction of transcription, speaker labels, and precise timestamps while keeping the LLM backbone frozen and training lightweight projectors.

  • Takeaways & Limitations

    Generalizability remains uncertain because AMI and AliMeeting provide relatively small training sets of 65 and 103 hours, respectively.

Abstract

from arXiv · show

We present TagSpeech, a unified LLM-based framework that utilizes Temporal Anchor Grounding for joint multi-speaker ASR and diarization. The framework is built on two key designs: (1) decoupled semantic and speaker streams fine-tuned via Serialized Output Training (SOT) to learn turn-taking dynamics; and (2) an interleaved time anchor mechanism that not only supports fine-grained timestamp prediction but also acts as a synchronization signal between semantic understanding and speaker tracking. Compared to previous works that primarily focus on speaker-attributed ASR or implicit diarization, TagSpeech addresses the challenge of fine-grained speaker-content alignment and explicitly models "who spoke what and when" in an end-to-end manner. Experiments on AMI and AliMeeting benchmarks demonstrate that our method achieves consistent improvements in Diarization Error Rate (DER) over strong end-to-end baselines, including Qwen-Omni and Gemini, particularly in handling complex speech overlaps. Moreover, TagSpeech employs a parameter-efficient training paradigm in which the LLM backbone is frozen and only lightweight projectors are trained, resulting in strong performance with low computational cost.

1 Introduction

TagSpeech addresses the gap between speaker-attributed transcription and explicit joint ASR–diarization by predicting what was said, who spoke, and when with fine temporal precision. It combines end-to-end waveform-to-structured-output modeling with lightweight numeric anchors and achieves substantial DER improvements while training only lightweight projectors.

  • Meeting transcription requires identifying what was said, who spoke, and when, especially under overlapping speech and rapid turn-taking.
  • Existing speaker-attributed ASR methods often omit explicit start and end timestamps, preventing standard DER evaluation and precise speech–text alignment.
  • TagSpeech explicitly predicts transcription, speaker labels, and precise timestamps in a unified multi-speaker ASR and diarization formulation.
  • The interleaved numeric anchor mechanism provides fine-grained temporal grounding and aligns decoupled semantic and speaker streams without modifying the LLM architecture.
  • Approximately 28% relative DER improvement on AMI and 36% on AliMeeting is achieved over strong end-to-end baselines while maintaining robust recognition performance.The baselines include Gemini-2.0, Qwen2.5-Omni-7B, and Qwen3-Omni-30B.

2 Related Work

Prior multi-speaker systems span cascaded pipelines, frame-level diarization, permutation-invariant recognition, and serialized speaker-attributed transcription, but often leave speaker identity, timestamps, or global consistency to separate components. TagSpeech targets the remaining gap with lightweight temporal grounding that preserves textual reasoning without heavy architectural changes.

  • Cascaded systems combine VAD, separation, ASR, and diarization, but can propagate errors and require heuristic fusion.
  • EEND models who spoke when through frame-level multi-label classification, whereas PIT handles multispeaker recognition without explicitly modeling speaker identity.
  • SOT serializes overlapping speech with speaker-change markers, enabling speaker-attributed ASR and implicit diarization, but later unified systems may require separate decoders, clustering, or precomputed timestamps.
  • Many LLM-based approaches stop at speaker-attributed transcription and therefore lack explicit temporal grounding for standard DER evaluation.
  • Prior LALMs lack intrinsic mechanisms for grounding continuous speech frames to fine-grained absolute timestamps required for diarization.
  • TagSpeech addresses this gap with lightweight temporal grounding that preserves LLM textual reasoning without heavy architectural changes.

3 Method

TagSpeech combines disentangled semantic and speaker streams with interleaved numeric time anchors to generate structured outputs for who spoke what and when. Its XML-style alignment and parameter-efficient training connect these streams to an LLM without modifying the model vocabulary.

  • Framework overview: TagSpeech maps raw waveforms to structured sequences containing semantic text, speaker identifiers, and timestamp markers.The framework uses a Dual-Stream Encoder, Interleaved Time Anchor mechanism, and LLM backbone for unified reasoning.
  • Dual-stream encoders: Parallel semantic and speaker encoders produce distinct representations for content and speaker identity.Both streams use Zipformer structure, yielding Hsem and Hspk over downsampled temporal frames.
  • Dual-stream encoders: Serialized Output Training fine-tunes the semantic encoder on chronological multi-speaker transcriptions separated by a speaker-change token.This objective guides the encoder to capture turn-taking dynamics in overlapping speech.
  • Projection and alignment: Two projectors map semantic and speaker representations into the LLM embedding space after temporal downsampling.The projected representations have reduced sequence length L = ⌈T′/k⌉.
  • Temporal grounding: Numeric time anchors are inserted into both streams at fixed intervals, enforcing deterministic temporal synchronization without expanding the vocabulary.The resulting sequences include start and end anchors and use the same interval m for semantic and speaker streams.
  • Structured input–output alignment: XML-style tags align semantic content and timestamps with the text stream and speaker identity and timestamps with the speaker stream during generation.The target sequence mirrors the decoupled XML input structure and is trained with autoregressive negative log-likelihood while only lightweight projectors are trained.

4 Experiments and Results

Experiments on AMI and AliMeeting evaluate TagSpeech under challenging far-field, overlapping-speech conditions. The results show strong diarization, speaker-count accuracy, reliability, cross-lingual boundary generalization, and benefits from dual-stream SOT training and moderate anchor density.

  • Experimental Setup: TagSpeech is evaluated on AMI and AliMeeting far-field settings featuring spontaneous multi-party conversations and frequent overlapping speech.AMI uses the SDM subset, while AliMeeting uses the far-field subset with the first channel of 8-channel recordings.
  • Evaluation Metrics: DER uses a strict 0 s collar with overlap included, while cpWER, gWER, SCA, and Fail Rate measure speaker-attributed recognition, semantic transcription, speaker-count correctness, and output reliability.A 0.25 s collar is additionally reported for comparison with existing benchmarks.
  • Encoder Ablation: Dual-encoder ablations show that single encoders suffer high failure rates or degraded diarization, whereas decoupled representations perform better and SOT fine-tuning achieves the best overall results.The single-encoder results indicate interference between semantic and speaker information in a shared representation space.
  • Main Results: TagSpeech achieves the highest speaker count accuracy, while cpWER and gWER remain consistently strong across both languages, including AliMeeting’s 30%–40% overlap conditions.Content recognition is not the best overall, but remains reliable in the more overlapped AliMeeting setting.
  • Main Results: TagSpeech has substantially lower failure rates than larger end-to-end baselines despite training on at most 100 hours per dataset and optimizing only lightweight projectors.The LLM backbone remains frozen, supporting an efficient training setup.
  • Encoder Ablation: On AliMeeting, SOT fine-tuning generalizes with larger DER and WER gains, particularly under the dataset’s higher overlap ratio.These results support pairing dual-stream modeling with task-specific, turn-aware pretraining.
  • Cross-lingual Generalization: Zero-shot cross-dataset evaluation preserves competitive DER around 20% despite transcription collapse under language shift, indicating language-agnostic multispeaker boundary representations.The reported comparison computes remaining metrics only on common successful samples because fail rates vary substantially.

5 Analysis

The analysis shows that anchor density creates a trade-off between temporal precision and semantic coherence, with moderate insertion performing best. TagSpeech particularly improves diarization in overlapping speech by reducing missed speaker activity and countering baseline serialization bias.

  • Anchor interval analysis: Anchor intervals produce a U-shaped DER and cpWER pattern, with dense anchors disrupting semantic and speaker streams while sparse anchors provide insufficient grounding.The evaluated range spans every-frame insertion to every 32 frames, corresponding to 0.16–5.12 seconds at 6.25 Hz.
  • Anchor interval analysis: The best trade-off occurs around an anchor interval of 8, balancing temporal grounding with semantic coherence.
  • DER decomposition: Missed speech dominates DER variation across anchor settings, especially as anchors become sparse.Under MeetEval, missed speech includes silence predicted during active speech and undetected speakers in overlapped regions.
  • Overlapping speech: TagSpeech slightly trails the Pyannote + Whisper cascade in overall AMI DER by 1.8%, partly because end-to-end modeling introduces boundary errors relative to specialized VAD.
  • Overlapping speech: TagSpeech nearly improves overlap-region DER by 20% absolute on AliMeeting, outperforming the cascade when two to four speakers overlap.
  • Temporal cue design: Numeric anchors match textual-anchor performance at far lower token cost, using only 1–2 cached integer tokens without extra processing.Textual anchors perform slightly better but require substantially more tokens or compression engineering.
  • Qualitative failure analysis: Large end-to-end baselines show temporal blindness on very short inputs and linearity bias that serializes simultaneous speech.A 0.21-second sample elicited a hallucinated 28-second dialogue from Gemini-2.0-Flash, while TagSpeech recovered overlapping timing more accurately.

6 Conclusion

TagSpeech is presented as a simple, efficient end-to-end framework for jointly modeling who spoke what and when. It achieves strong DER results on AMI and AliMeeting while retaining low training cost, though several extensions remain future work.

  • TagSpeech directly addresses temporal hallucinations and diarization accuracy through unified end-to-end modeling of speaker, content, and timing.
  • TagSpeech achieves the best reported DER results on AMI and AliMeeting against strong baselines including Gemini and Qwen.
  • The framework requires no LLM vocabulary or backbone modifications, supporting low-cost training and deployment.
  • Future work targets speaker-encoder adaptation, LoRA-based content improvements, and context propagation for long-form and streaming scenarios.

Limitations

The main limitations concern evaluation scale, autoregressive inference speed, and the absence of rigorous paralinguistic annotations in the evaluated meeting corpora.

  • Data scale and generalizability: AMI and AliMeeting provide challenging speech conditions but contain relatively small training sets of 65 and 103 hours, limiting generalizability assessment.The authors recommend validation on larger synthetic datasets to study scaling behavior and occasional structural or hallucination failures.
  • Inference latency: Autoregressive generation makes inference slower than nonautoregressive and optimized cascaded systems, constraining strictly real-time or low-resource streaming deployment.Chunk-wise decoding supports 30–50-minute recordings, but overall speed still lags behind optimized cascades.
  • Multimodal understanding: The study cannot quantitatively evaluate gender, age, or emotion because AMI and AliMeeting lack rigorous annotations for these paralinguistic attributes.

A.1 Dataset and Metric Details

The appendix describes the evaluated datasets, filtering protocol, and error metrics used for reproducible comparison. DER decomposes diarization errors, while WER-family metrics quantify transcription errors and speaker assignment.

  • Evaluation protocol: Utterances shorter than 0.5 seconds or longer than 80 seconds are excluded, removing approximately 5% of speech duration from each dataset.The filtering addresses unstable behavior on extreme-duration inputs and supports fair comparison.
  • Datasets: Evaluation uses AMI-SDM and AliMeeting-Far, processed with Lhotse and summarized by training, validation, and test statistics.
  • Metrics: DER combines false alarm, missed speech, and speaker confusion durations under a 0 ms collar with no overlap forgiveness.
  • Metrics: WER counts substitutions, deletions, and insertions, while Mandarin evaluation uses Character Error Rate.
  • Metrics: Global WER concatenates utterances chronologically regardless of speaker labels, whereas cpWER evaluates speaker-assigned transcripts using optimal Hungarian matching.
  • Evaluation protocol: DER and cpWER are computed with the MeetEval Toolkit for reproducibility.

A.2 Reproducibility Details

The reproducibility setup specifies the acoustic and encoder configurations, numeric time-anchor interval, frozen LLM inference, training regimen, and single-GPU hardware used for experiments.

  • Acoustic Features: 80-dimensional log-mel spectrograms use a 25 ms window and 10 ms hop size.
  • Encoder Architecture: Zipformer encoders downsample outputs to 25 Hz with hidden dimension Denc = 768, followed by stride-4 projectors producing 6.25 Hz representations.Each encoder has approximately 156M parameters.
  • Time Anchor Mechanism: The numeric time-anchor interval is m = 8, corresponding to 1.28 physical seconds at the 6.25 Hz compressed frame rate.
  • Large Language Model: Qwen2.5-7B-Instruct is used in inference-only mode with greedy decoding and an added <|AUDIO|> placeholder token.
  • Training Setup: Training uses Adam with learning rate 0.001, mixed precision, SpecAugment, and a maximum input duration of 80 seconds.The model is trained with a bucketing sampler and scheduled learning rate.
  • Hardware: Experiments run on a single NVIDIA H200 or A100 GPU, requiring approximately 3 hours on H200 and 8 hours on A100 for convergence.

A.3 Baseline

The baseline section evaluates Gemini, Qwen-Omni, cascade systems, and representation alternatives, while documenting output-format constraints, failure behavior, and cross-dataset generalization.

  • Gemini: Gemini-2.0-flash is evaluated for end-to-end speaker diarization and multi-speaker ASR using structured prompts that encourage overlapping utterance segmentation.
  • Baseline Limitation: End-to-end baseline models generate strictly sequential segments without temporal overlap, even when overlapping speech is present, hindering precise DER evaluation.This limitation persists despite explicit prompting.
  • Qwen-Omni Models: Qwen2.5-Omni-7B and Qwen3-Omni-30B generate textual diarization and ASR outputs, with audio generation disabled and formats normalized for evaluation.Qwen models output speaker segments in strictly linear timelines, while Qwen3-Omni uses an SRT-like format.
  • Cascade Baselines: Pyannote and Whisper form a cascade baseline, but Pyannote often misses very short utterances and Whisper transcribes each predicted segment.
  • Cross-Lingual Generalization: Zero-shot cross-lingual testing finds competitive DER on successfully parsed samples, but cpWER exceeds 100% in both language-transfer directions.The model generates hypotheses in its training language under extreme language shift.
  • Condition-Wise Analysis: Performance degrades as speaker count increases, while extremely short and long utterances are especially challenging across models.The cascade is comparatively stable across conditions except for failures on extremely short segments.
  • DER Evaluation: DER trends remain consistent when evaluated with a standard 0.25 s collar rather than the strict 0.0 s collar.
Loading 2601.06896v2…