Source-linked AI summary
Soft Posterior Speaker Injection for Multi-Talker Speech Recognition
Jian Zhu, Cheng Luo
TL;DR
Overlapping multi-talker speech challenges segmentation-based ASR, while SOT does not condition the encoder on speaker activity. SPSI injects continuous speaker posteriors into Whisper through FiLM and decoder prompts, improving controlled-overlap cpWER and enabling stronger transfer after freeze-posterior OV-heavy adaptation.
Problem
Hard segmentation is vulnerable to boundary errors, while SOT avoids explicit segmentation but typically leaves the acoustic encoder unconditioned on speaker activity.
Method
SPSI predicts frame-level speaker posteriors and injects the continuous speaker share into Whisper using multi-layer FiLM and decoder speaker-memory prompts.
Results
SPSI reduces controlled LibriSpeech overlap cpWER from 50.7% for SOT to 49.6%, while OV-heavy freeze-posterior adaptation reaches 32.4% versus 37.5% for SOT on held-out LibriCSS sessions 8–9.
Takeaways & Limitations
Encoder FiLM and decoder prompts are complementary, and the effective injection signal is a soft simplex-valued speaker share rather than a hard mask.
Takeaways & Limitations
Posterior calibration under domain shift remains unresolved because jointly updating the posterior head currently hurts transfer.
Abstract
from arXiv · showhide
Multi-talker automatic speech recognition (MT-ASR) remains challenging under overlapping speech. Hard diarization-based segmentation introduces irreversible errors, whereas serialized output training (SOT) avoids explicit segmentation but does not condition a pretrained encoder on speaker activity. We propose Soft Posterior Speaker Injection (SPSI): a lightweight head predicts frame-level speaker posteriors $\hat{\mathbf{P}}$ and injects them into Whisper through multi-layer feature-wise linear modulation (FiLM) and decoder speaker-memory prompts. On controlled two-speaker LibriSpeech overlap, SPSI reduces utterance-mean constrained permutation word error rate (cpWER) from 50.7\% (SOT) to 49.6\% (one-sided paired bootstrap $p{\approx}0.006$), with a larger reduction in the high-overlap bin (60.4\%$\to$58.8\%). Same-backbone speaker-auxiliary objectives and voice activity detection (VAD) pipelines do not outperform SOT; zero-shot (ZS) LibriCSS is comparable. Freeze-posterior adaptation with overlap-heavy (OV-heavy) continuation reduces held-out LibriCSS cpWER (sessions 8--9) to 32.4\% (versus 37.5\% for SOT). Ablations indicate complementary encoder FiLM and decoder prompts, and that the effective signal is a \emph{soft} simplex-valued speaker share.
1. INTRODUCTION
MT-ASR must handle overlapping speakers without the irreversible errors of hard segmentation, while SOT leaves Whisper's encoder unconditioned on speaker activity. SPSI addresses this gap by injecting continuous speaker shares through encoder FiLM and decoder prompts, improving cpWER on controlled overlap and under OV-heavy transfer.
- Hard VAD, diarization, or source-separation segmentation can truncate onsets and leak competing speech, while SOT avoids explicit segmentation but lacks speaker-activity conditioning.
- SPSI uses a mixture-normalized speaker share rather than a one-speaker-per-frame classifier, representing overlap as shared posterior mass.
- SPSI predicts frame-level speaker posteriors and injects them after encoder blocks 8, 16, and 24 with FiLM, while also using K=4 decoder speaker-memory tokens.
- SPSI avoids irreversible hard segmentation through a two-pass shared-backbone design that requires no external diarization or source separation at inference.
2. PROPOSED METHOD
SPSI uses a two-pass Whisper encoder: a posterior head estimates soft speaker shares, then shared-weight FiLM re-encodes the mixture and pooled posterior tokens condition the decoder. Training combines recognition and soft diarization objectives, with posterior freezing during LibriCSS adaptation to preserve calibration.
- Whisper-medium encodes an 80-bin log-Mel mixture into encoder memory, while SOT serializes two speakers as speaker-prefixed text streams.
- Soft posterior head: An unconditioned encoder pass feeds a convolutional MLP and linear projection, whose softmax output is a two-speaker posterior matrix.
- Soft posterior head: The default posterior is the estimated softmax share; alternatives include energy-ratio, exclusive hard, and multi-label VAD masks.
- Multi-layer FiLM: The second pass re-encodes with shared weights and applies FiLM after blocks 8, 16, and 24, using per-frame scale and shift parameters.
- Decoder prompts: Posterior statistics are mapped to four memory tokens, layer-normalized, prepended to encoder memory, and consumed through decoder cross-attention.
- Training and adaptation: Training minimizes LASR + λLdiar, while LibriCSS adaptation freezes the posterior head and continues with overlap-heavy data to retain posterior calibration.
3. EXPERIMENTS
Experiments show that SPSI improves synthetic-overlap recognition over SOT, especially at high overlap, while its components are complementary and its zero-shot LibriCSS performance is comparable to SOT. Freeze-posterior overlap-heavy adaptation provides the strongest held-out LibriCSS transfer result.
- Synthetic overlap: A 1.1-point cpWER reduction over SOT was observed overall, with a 1.6-point reduction in the high-overlap bin.The overall 95% CI was [0.2, 1.9] with p≈0.006; the high-overlap CI was [0.2, 3.0] with p≈0.01.
- Ablation study: Encoder FiLM and decoder prompts are complementary, while single-component, attention-based, convolutional, and hard-mask variants underperform the default SPSI design.The default SPSI score was 0.496; single-layer FiLM with prompts scored 0.498 overall but was weaker at high overlap, and hard-mask variants exceeded SOT.
- Synthetic overlap: SPSI outperformed SOT more often than the reverse on high-overlap utterances, with 172 wins versus 137 losses.The high-overlap subset contains 350 utterances, with 41 ties under the stated threshold.
- Domain transfer: Pooled zero-shot LibriCSS performance was comparable between SPSI and SOT, at 0.466 versus 0.467.The comparison covers 1,315 oracle-segmented windows across official overlap conditions.
- Domain transfer: Overlap-heavy continuation after freezing the posterior head reached 0.324 versus 0.375 for SOT on held-out LibriCSS sessions 8–9.This corresponds to a 5.1-point reduction with p<0.001; the best transfer protocol uses ω=1.
4. CONCLUSION AND FUTURE WORK
SPSI improves multi-talker recognition by injecting continuous speaker posteriors through complementary encoder FiLM and decoder prompts. Its strongest reported transfer result comes from freeze-posterior, overlap-heavy adaptation, while domain-shift calibration and broader deployment remain open challenges.
- SPSI injects continuous speaker posteriors through multi-layer FiLM and decoder prompts, avoiding hard masks.Ablations show encoder FiLM and decoder prompts are complementary, whereas hard-mask variants do not match the soft-share design.
- 5.1-point held-out cpWER reduction is achieved with freeze-posterior overlap-heavy adaptation versus SOT on LibriCSS sessions 8–9.The reported comparison is 32.4% versus 37.5%, with p<0.001.
- Table 4 evaluates held-out LibriCSS sessions 8–9 using top-level comparisons and injector ablations across joint, freeze, and OV-heavy settings.The table contains n=259 examples and includes matched SOT freeze/+OV-heavy schedules without a speaker head.
- Posterior calibration under domain shift remains unresolved because jointly updating the posterior head currently hurts transfer.The paper identifies adapting the head without collapsing the predicted speaker share as an open direction.