STAM-ASR: Speaker-Temporal Anchoring with Memory for Multi-Speaker ASR

1. INTRODUCTION

• Conversation's turn-taking, overlap, and speaker reappearances make ASR responsible for recovering both what was said and who spoke when.Source: §1 · p. 1
• Earlier multi-speaker ASR addressed attribution and overlapping speech through diarization, target-speaker conditioning, separation, permutation-invariant recognition, or serialized output training.Source: §1 · p. 1
• AudioLLMs, speaker caches, and long-context models have addressed speaker tracking and contextual history largely separately, leaving their joint preservation across turns open.Source: §1 · p. 1
• An internal diarization module estimates speaker activity and representations from pretrained audio-encoder features, supplying who and when cues to modulate the AudioLLM's semantic representation.This produces a speaker-conditioned view of mixed audio without external diarization or explicit waveform separation.Source: §1 · p. 1
• STAM-ASR combines speaker-temporal conditioning and fixed-size speaker and conversational memories in a pretrained AudioLLM, without explicit speech separation.The two memories preserve complementary speaker-specific and conversation-level context across turns.Source: §1 · p. 1
• Speaker-temporal conditioning and memory provide complementary benefits across close-talk, far-field, overlapping, and unseen cross-domain conditions.Speaker-temporal conditioning is more effective when current speaker evidence matters, while memory adds benefit in meetings where speaker and conversational history are informative.Source: §1 · p. 1

2.1. Audio Representation

• A pretrained AudioLLM supplies intermediate features for speaker modeling and semantic representations for ASR; long recordings are processed in segments with speaker and memory states propagated across them.Source: §2.1 · p. 2

2.2. Speaker Diarization and Temporal Anchoring

• A Transformer diarization module produces multi-label speaker activity and speaker-aware hidden states, from which activity-weighted pooling derives speaker representations.Multiple speakers may be active simultaneously; across segments, speaker representations are associated to maintain consistent identities, and turn timestamps select the corresponding representations.Source: §2.2 · p. 2

2.3. Speaker-Temporal Modulation

• For each target speaker, a residual FiLM transformation uses speaker representation and activity to modulate the semantic representation while preserving its sequence length and dimensionality.During overlap, the same mixed representation is conditioned separately for each active speaker without explicit waveform separation.Source: §2.3 · p. 2

2.4. Memory-Conditioned ASR

• A shared Q-Former maintains fixed-size speaker memory across turns by the same speaker and conversational memory across chronological turns, using memory-type embeddings to distinguish updates.ASR uses the memories available before the current turn's update; after transcription, both memories are updated for subsequent turns.Source: §2.4 · p. 2
• The AudioLLM generates the speaker-attributed transcript from the current turn representation together with the speaker and conversational memories.Source: §2.4 · p. 2

3.1. Data and Evaluation

• Training uses AMI, ICSI, and Mixer6, with held-out AMI and ICSI plus unseen LibriCSS and NOTSOFAR-1 for evaluation.AMI and ICSI provide ASR and diarization supervision; Mixer6 provides ASR supervision only, and ICSI training meetings with more than eight speakers are excluded.Source: §3.1 · p. 2

3.2. Training Data Preparation

• Chunks preserve complete speaker turns within at most 20 seconds, warm up memories on the first two turns, and retain overlap through multi-label speaker activity.The frozen Qwen2.5-Omni-7B audio encoder supplies intermediate speaker-modeling and final semantic representations; the intermediate layer is selected by a speaker probe.Source: §3.2 · pp. 2–3, Figure 2 · p. 3

3.3. Training Strategy

• Training begins with diarization-only warm-up, then jointly optimizes diarization, speaker-temporal FiLM, the memory Q-Former, and LoRA adapters while freezing the encoder and base LLM.Reference speaker activity is gradually replaced by predicted activity after a 100-step warm-up over the following 1,200 optimization steps.Source: §3.3 · p. 3

3.4. Baselines

• The comparison includes Qwen2.5-Omni-7B, a Whisper-plus-pyannote cascade, and TagSpeech-AMI, which uses separately trained Zipformer models for speaker-temporal information.STAM-ASR instead derives speaker activity and representations from AudioLLM internal features and maintains speaker and conversational memories.Source: §3.4 · p. 3

3.5. Evaluation Measures

• Speaker-attributed ASR is primarily assessed with cpWER and tcpWER@5, alongside WER and diarization error rate; full-session scoring also reports gDI-cpWER.DER uses a 0.25-second collar, while gDI-cpWER permits different hypothesis segmentation and uses greedy diarization-invariant assignment over the full recording.Source: §3.5 · p. 3

Key findings

• Reference speaker activity substantially reduces error across test sets, while LibriCSS analysis points to long-recording identity maintenance as a greater limitation than local speaker activity estimation.The findings identify robust speaker tracking as a key bottleneck.Source: §4 · pp. 3–4

Multi-speaker ASR

• Under VAD-based segmentation, STAM-ASR achieves the best cpWER on four of five test sets; TagSpeech is better on AMI-SDM (71.7% vs. 73.3%).The four sets are AMI-IHM, ICSI, LibriCSS, and NOTSOFAR-1.Source: §4 · p. 3, Table 3 · p. 4

When Does Each STAM-ASR Component Help?

• With predicted speaker activity, all components lower cpWER on AMI-IHM and ICSI, while speaker-temporal conditioning alone performs best on AMI-SDM and LibriCSS.On NOTSOFAR-1, speaker memory gives only a marginal cpWER gain, while speaker-temporal conditioning gives better tcpWER@5 and WER; configurations use the same checkpoint and require no retraining.Source: §4 · p. 3, Table 4 · p. 4

Session-level ASR

• On full-session gDI-cpWER, STAM-ASR outperforms TagSpeech on AMI-IHM, AMI-SDM, and LibriCSS, while the systems perform similarly on ICSI and NOTSOFAR-1.On AMI-SDM, TagSpeech has lower chunk-level cpWER (71.7 vs. 73.3), but STAM-ASR has lower full-session gDI-cpWER (49.7 vs. 54.8).Source: §4 · p. 3, Table 5 · p. 4

Auxiliary Diarization Analysis

• Under chunk-local scoring, Dϕ achieves 28.4% DER on AMI-IHM versus TagSpeech’s 39.9%; full-recording evaluation gives Dϕ 36.6% DER on AMI-IHM.Speaker confusion becomes the dominant error on many-speaker conditions such as ICSI.Source: §4 · p. 4, Table 6 · p. 4

• STAM-ASR augments a pretrained AudioLLM with speaker-temporal anchoring and fixed-size speaker and conversational memories for multi-speaker ASR.Source: §5 · p. 4
• Speaker-temporal conditioning and memory provide complementary benefits across different conversational conditions.Source: §5 · p. 4
• Predicted speaker activity remains a key source of error, underscoring the importance of jointly modeling current speaker evidence and persistent conversational context.Source: §5 · p. 4