Source-linked AI summary
Over-Tightening-Aware Pseudo-Labeling for Tight-Boundary Speaker Diarization
Shota Horiguchi, Takanori Ashihara, Marc Delcroix, Naohiro Tawara, Alexis Plaquet
TL;DR
Loose diarization labels can produce loose predictions, while causal–anticausal pseudo-labeling can over-tighten and increase missed detections that affect downstream ASR. The paper addresses these issues with pause-focused tightening, burn-in, and non-causal-model-aware co-training. The proposed methods improve diarization and downstream multi-talker ASR performance.
Problem
Pseudo-labels estimated from causal and anticausal models can over-tighten, increasing missed detections that may propagate as errors to downstream tasks.
Source: §5 · p. 2, C. Non-causal-model-aware co-training · p. 4Method
The paper preserves loose boundaries while targeting pause filling, adds burn-in context for causal and anticausal predictions, and incorporates the non-causal model into co-training.
Source: §I · p. 1, C. Non-causal-model-aware co-training · p. 4Results
The proposed methods reduce excessive boundary shifts and missed detections, improve diarization, and bring multi-channel ASR performance closer to the tight-label system.
Source: A. Speaker diarization · p. 5, 2) Results · p. 6, §VI · p. 6Takeaways & Limitations
Because diarization missed detections can cause unrecoverable deletion errors in cascaded ASR, the proposed methods’ reduction of those errors brings ASR closer to tight-label performance.
Source: 2) Results · p. 6
Abstract
from arXiv · showhide
Training speaker diarization models on loose labels, such as speech segments with padded boundaries or filled pauses, often results in similarly loose model outputs. To obtain tighter boundaries, pseudo-labeling based on the averaged outputs of causal and anticausal models has been proposed. However, since the pseudo-labels are estimation-based, they can suffer from over-tightening, which increases missed detections that can propagate as unrecoverable errors to downstream tasks. This paper carefully analyzes the causes of over-tightening and proposes three approaches to address them: (i) removing pause filling rather than padding, (ii) introducing a burn-in phase to mitigate missed detections near the beginning of causal and anticausal predictions, and (iii) making pseudo-label-based co-training aware of the non-causal model used for final inference. Experimental results show that the proposed method reduces missed detections caused by over-tightening and improves both diarization accuracy and downstream multi-talker ASR performance.
I. INTRODUCTION
Diarization models trained on corpora with inconsistent boundary tightness can learn corpus-dependent outputs, making tight labels desirable when training data are mixed. The paper studies tight-boundary diarization from loose labels, where pseudo-labels can over-tighten and cause missed detections that affect downstream tasks.
I. INTRODUCTION
- Mixing corpora with different annotation tightness can produce unpredictable, corpus-dependent boundaries, particularly on out-of-domain data.Tight labels are considered advantageous because loose predictions can be recovered from tight ones using operations such as morphological closing.Source: §I · p. 1
- Causal–anticausal consistency provides a way to generate tight pseudo-labels from loose annotations without forced alignment, but the estimates can over-tighten and increase missed detections.Those missed detections can propagate to downstream tasks such as ASR.Source: §I · p. 1
- The proposed methods address over-tightening through pause-focused tightening, burn-in context for causal and anticausal models, and co-training that includes the final non-causal model.The paper reports reduced diarization error-rate gaps to ideal tight-label results and ASR performance close to that of diarization trained with tight labels.Source: §I · p. 1
A. Problem formulation
- End-to-end diarization models produce frame-wise speaker posteriors and are trained with speaker-aligned labels because output speaker order is arbitrary.The formulation uses powerset classes representing possible active-speaker sets, which are converted to speaker-wise posteriors.Source: A. Problem formulation · pp. 1–2
- Tight-boundary diarization seeks to train from loose annotations while predicting tight speech boundaries, despite models tending to imitate loose labels.The label noise considered includes speech-class regions added through padding or pause filling.Source: A. Problem formulation · p. 2
B. Pseudo-labeling via causal–anticausal consistency
The procedure generates tight pseudo-labels by combining causal and anticausal predictions, then uses those labels to train a non-causal model. It first pretrains the causal and anticausal models separately and then co-trains them.
B. Pseudo-labeling via causal–anticausal consistency
- Causal and anticausal outputs are combined to generate tight pseudo-labels for training a non-causal model.The approach exploits the models’ limitations on onset padding, offset padding, and pause filling.Source: B. Pseudo-labeling via… · p. 2
- The first stage trains causal and anticausal models independently using the original loose labels.The models are then co-trained from their pretrained weights.Source: B. Pseudo-labeling via… · p. 2
- During co-training, pseudo-labels generated at each iteration update both the causal and anticausal models.The resulting pseudo-labels are used in the later non-causal-model training stage.Source: B. Pseudo-labeling via… · p. 2
2 Align the speaker permutation of⃗P and⃖P to ˜Y
The speaker-confusion correction uses the loose label to guide frame-wise exchanges between missed-speaker and false-alarm-speaker posteriors.
2 Align the speaker permutation of⃗P and⃖P to ˜Y
- The correction targets speaker confusion by swapping posteriors of missed and false-alarm speakers.It refers to the loose label when making the swaps.Source: §2 · p. 2
- The loose label serves as the reference for deciding which speaker posteriors to exchange.The correction is applied to the causal and anticausal speaker-wise posteriors.Source: §2 · p. 2
- The method exchanges posteriors between speakers that were missed and speakers that were falsely detected.This correction is part of the pseudo-label generation process.Source: §2 · p. 2
5 Restore over-tightened segments in Ytight
The section examines over-tightening and proposes preserving loose boundaries while focusing tightening on pause filling. Its analysis finds pause filling larger than boundary padding and motivates the proposed approach.
5 Restore over-tightened segments in Ytight
- Tightened pseudo-labels are formed from thresholded average causal and anticausal predictions masked by the loose labels, with disappeared speech segments restored.The restoration assumes speech should not completely disappear within a loose segment; prior work used a more relaxed criterion, restoring only when more than 50% disappeared.Source: §5 · p. 2
C. Model architecture
- Figure 1 depicts pause filling or onset/offset shifts and gives corresponding AMI and AliMeeting histograms, with out-of-range values accumulated in the edge bins.Source: Figure 1 · p. 3
III. PROPOSED METHODS
- The paper analyzes multiple causes of over-tightening in speaker-counting-based tightening and proposes solutions aimed at diarization and downstream tasks.The proposed approach includes preserving loose boundaries, burn-in, and non-causal-model-aware co-training.Source: §III · p. 3
A. Pause-filling-focused tightening
- The AMI and AliMeeting validation-set analysis uses utterances with tight annotations available through forced alignment.After discarding alignment errors, it includes 8,643 AMI and 5,376 AliMeeting utterances.Source: 1) Analysis · p. 3
1) Analysis:
- Pause filling is much larger than boundary padding: mean onset and offset shifts are each below 0.1 s.The analysis reports 15,478 filled pauses averaging 0.336 s in AMI and 7,870 averaging 0.267 s in AliMeeting.Source: 1) Analysis · p. 3
A. Pause-filling-focused tightening
- The tightening method targets pause filling while leaving loose onset and offset boundaries unchanged to avoid over-tightening.This choice follows the finding that annotation looseness is primarily due to pause filling rather than boundary padding.Source: 1) Analysis · p. 3
1) Analysis:
- The method seeks to prevent early onset and offset over-tightening from being mistaken for convergence before pause filling has been sufficiently suppressed.The boundary-preservation steps retain loose onsets and offsets during tightening.Source: 1) Analysis · p. 3
2) Method:
The method retains a loose segment when its estimated speech-frame proportion falls below an ad hoc 0.5 threshold, while preserving the segment’s original boundaries. The threshold may not suit utterances of different durations.
2) Method:
- The 0.5 threshold is ad hoc and may be unsuitable across utterances of different durations.Short utterances are less likely to contain pauses, while long utterances are more likely to contain them.Source: 2) Method · p. 3
- When the speech-frame proportion falls below the threshold, the original loose segment remains the training target.Source: 2) Method · p. 3
- The procedure preserves each loose segment’s onset and offset boundaries, restoring the entire original segment if no tight segment is estimated inside it.It therefore does not attempt to reduce boundary padding.Source: 2) Method · p. 3
B. Causal and anticausal models with a burn-in phase
Causal and anticausal models have lower performance than non-causal models, with missed detections especially frequent near the start of their respective sequences. The method adds a burn-in phase to provide context before those models produce outputs used for diarization.
B. Causal and anticausal models with a burn-in phase
- The causal and anticausal models used to generate pseudo-labels generally perform worse than non-causal models.The analysis examines the conditions in which their errors occur most frequently.Source: 1) Analysis · p. 3
1) Analysis:
- Missed detections rise near the beginning of 10 s chunks: at 0–1 s for causal models and 9–10 s for anticausal models.The paper attributes this pattern to cold start, when models must extract features and detect speakers from limited samples.Source: 1) Analysis · p. 3
- A burn-in phase gives unidirectional models additional context before producing outputs used for diarization.For a causal model, the extra input consists of samples preceding the original input.Source: 2) Method · p. 4
2) Method:
For anticausal inference, the burn-in uses samples following the original input. Outputs associated with the extra input are separated from those associated with the original input and are not used for loss computation or inference.
2) Method:
- The anticausal model receives an extra input consisting of samples following the original input.The extra samples provide context for the anticausal model’s burn-in.Source: 2) Method · p. 4
- Outputs associated with the original input are distinguished from outputs associated with the extra input.Source: 2) Method · p. 4
- Outputs associated with the extra inputs are excluded from training loss and discarded during inference; the burn-in phase is set to 1 s.Source: 2) Method · p. 4
C. Non-causal-model-aware co-training
The best causal and anticausal models do not necessarily produce pseudo-labels that maximize the final non-causal model’s performance. The proposed co-training pipeline trains that model in parallel and uses its validation performance for early stopping.
C. Non-causal-model-aware co-training
- Pseudo-labels from the best causal and anticausal models may yield final outputs that are over-tightened or insufficiently tight.This is because the best-performing pseudo-label generators do not necessarily maximize the final non-causal model’s performance.Source: C. Non-causal-model-aware co-training · p. 4
- The proposed pipeline trains the non-causal model in parallel and bases early stopping on its validation performance.This aligns the stopping decision with the model used for final inference.Source: C. Non-causal-model-aware co-training · p. 4
- During co-training, causal and anticausal outputs generate pseudo-labels on the fly to update all three models, including the non-causal model.The non-causal model starts from loose-label training, avoiding a separate schedule to train it from scratch.Source: C. Non-causal-model-aware co-training · p. 4
IV. EXPERIMENTAL SETUP
The experiments use an EEND-vector clustering pipeline with local diarization trained on a five-corpus compound set. Tight labels are used for validation and evaluation on AMI and AliMeeting.
A. Diarization pipeline
- Local diarization uses pretrained ReDimNet encoders with an LSTM and linear backend, within an EEND-vector clustering pipeline.ResNet34 embeddings and VBx clustering aggregate local results.Source: A. Diarization pipeline · p. 4
B. Dataset
- The five-corpus training set combines loosely annotated AMI and AliMeeting ASR data with tightly annotated MSD and VoxConverse diarization data.AMI-MHM and AMI-SDM are both included.Source: B. Dataset · p. 4
- Tight labels are available for AMI and AliMeeting validation and test evaluation, although training does not require tight labels.Validation labels monitor convergence; test performance is measured using DER without collar tolerance.Source: B. Dataset · p. 5
A. Speaker diarization
Pause-filling-focused tightening improves average DER and mitigates corpus asymmetry, while burn-in reduces missed detections and non-causal-aware co-training trades fewer misses for more false alarms. Conventional pseudo-tightening systems outperform loose-label training on the reported multi-channel ASR metrics.
A. Speaker diarization
- Loose-label training performs substantially worse than the tight-label topline on originally loose corpora; over-tightening dominates AliMeeting, where a small threshold can yield overly tight outputs.AMI shows roughly twice the loose-label DER of AliMeeting, suggesting roughly twice as much pause filling.Source: A. Speaker diarization · p. 5
- Burn-in reduces missed detections near each unidirectional model’s context boundary, with approximately uniform DER across chunk positions.Without burn-in, misses rise by 2–3 percentage points near 0–1 s for causal models and 9–10 s for anticausal models.Source: A. Speaker diarization · p. 5, Figure 2 · p. 5
- Non-causal-model-aware co-training (P3) reduces missed detections at the cost of increased false alarms, with a modest improvement.The final non-causal model is trained in parallel with causal and anticausal co-training.Source: A. Speaker diarization · p. 5
- Pause-filling-focused tightening (P1) improves average DER and mitigates corpus asymmetry, while reducing false alarms on AMI and missed detections on AliMeeting.P1–P3 also mitigate excessive onset and offset shifts, especially on AliMeeting.Source: A. Speaker diarization · p. 5, Table III · p. 6
1) Evaluation protocol:
- On AMI, conventional pseudo-tightening systems C3 and C4 achieve better tcpWER and ORC-WER than loose-label system C1 in multi-channel ASR.The evaluated cascaded pipeline uses GSS followed by Whisper; both ASR metrics are reported.Source: 2) Results · p. 6, 1) Evaluation protocol · p. 6
2) Results:
The proposed systems substantially reduced deletion errors in multi-talker ASR, bringing performance closer to the tight-label system; in single-channel ASR, P2 and P3 outperformed conventional pseudo-label systems and surpassed the tight-label system. The conclusion reports improvements in both diarization and downstream multi-talker ASR.
2) Results:
- The proposed systems substantially reduced deletion errors, bringing multi-talker ASR performance closer to the tight-label C2 system.The paper notes that diarization missed detections may be unrecoverable in the cascaded system.Source: 2) Results · p. 6
- For single-channel ASR, P2 and P3 outperformed conventional pseudo-label systems C3 and C4, and surpassed tight-label system C2.The paper suggests SE-DiCoW's label-error augmentation may make it robust to diarization errors, while fully tight outputs may be out of domain for its loose-label training.Source: 2) Results · p. 6
VI. CONCLUSION
- The proposed methods improved both speaker diarization and downstream multi-talker ASR performance.The methods preserve loose boundaries during pseudo-label generation, add burn-in for causal and anticausal models, and use non-causal-model-aware co-training.Source: §VI · p. 6