Diarization Error Decomposition Under Pause Annotation Ambiguity

1. INTRODUCTION

• Manual annotation is costly, and accurate speech-boundary labeling remains difficult, motivating evaluation robust to annotation ambiguity.Diarization supports speech separation, ASR, conversation analysis, and data generation.Source: §1 · p. 1
• Pause-treatment differences can inflate DER or obscure genuine model errors, and collar tolerance cannot eliminate these effects.Pause-annotation discrepancies vary in importance by application.Source: §1 · p. 1
• The proposed method decomposes standard DER into pause-attributable errors and residual core errors without changing DER, including under overlapping speech.Its components vary monotonically with the pause threshold and saturate at finite durations.Source: §1 · p. 1
• Experiments demonstrate that the decomposition separates pause-attributable and core components, improving the interpretability and explainability of diarization performance.The introduction also references an example comparing the closing baseline and proposed method.Source: §1 · p. 1, Figure 1 · p. 1

2. RELATED WORK

• DER is duration-weighted, while alternatives such as JER, segment-based F-measure, conversational DER, and BER add hyperparameters or scoring complexity without isolating pause ambiguity.Word DER evaluates speaker attribution per word but requires ASR output and is not a standalone diarization metric.Source: §2 · p. 1
• Prior pause-ambiguity approaches use forced alignment or morphological closing, whereas this method exactly decomposes DER without modifying either input or the resulting score.Forced alignment requires an additional evaluation pipeline, and closing has limitations.Source: §2 · p. 1

3. DIARIZATION ERROR RATE

• At each time t, R(t) and H(t) denote the active reference and hypothesis speaker sets under the standard one-to-one speaker mapping.The sets provide the basis for defining the instantaneous error counts.Source: §3 · p. 1
• Missed speech, false alarm, and speaker confusion are the three instantaneous DER error counts.They are denoted M(t), F(t), and C(t), respectively.Source: §3 · p. 1
• DER is duration-weighted using the total duration of speech in the reference.The section also gives a breakdown for missed speech, false alarm, and speaker confusion.Source: §3 · p. 2

4.1. Information loss under complete pause invariance

• Complete pause invariance forces a score to depend only on closed inputs, discarding distinctions between pause-preserving and pause-filled hypotheses.The same loss prevents distinguishing annotation tightness from indiscriminate filling.Source: §4.1 · p. 2

4.2. Overlap-aware pause-fill decomposition

• The method splits missed-speech and false-alarm errors into pause-attributable and residual core parts, leaves speaker confusion unchanged, and preserves standard DER exactly.Attribution requires a compatible pause of duration at most τ, bridged by one activity interval in the other annotation; overlapping candidates are capped by the corresponding error count.Source: §4.2 · p. 2

4.3. Theoretical properties

• As τ increases, pause-attributable error is nondecreasing and core error is nonincreasing; both eventually saturate without changing standard DER.For fixed annotations with finitely many internal pauses, saturation occurs at a finite threshold; unlike closing, increasing τ does not make evaluation more permissive.Source: §4.3 · pp. 2–3

4.4. Remarks

• Core error is a diagnostic, not a replacement for DER, and values from references with different annotation tightness are not directly comparable.Reference tightness can change the denominator, speaker mapping, and compatible regions; collar settings also affect comparability, while the decomposition can extend to duration-based metrics such as JER.Source: §4.4 · p. 3

5.1. Controlled synthetic validation

• Only pause-fill errors enter the pause-attributable component; insertion, shift, and long-fill errors remain core, and the components sum exactly to standard DER.The synthetic transformations test these behaviors against original labels.Source: Table 1 · p. 3, §5.1 · p. 3

5.2. Loose–tight annotation differences

• At τ = ∞, the proposed core error saturates, while closing-based DER can be non-monotonic or nearly zero at large widths.95% saturation occurs at 2.02 s for AMI and 0.91 s for AliM; the paper therefore uses the saturated endpoint.Source: Figure 5 · p. 3, §5.2 · p. 3

5.3. Train–test label tightness mismatch

• Under train–test label-tightness mismatch, most of the DER increase is pause-attributable, while the decomposition preserves standard DER.The DiariZen-based system reproduces corpus-specific annotation tightness, confirming the effect reported in.Source: Table 2 · p. 4, §5.3 · p. 4

5.4. In-domain vs. out-of-domain evaluation

• DiPCo and LibriCSS-Mix cases where DiariZen has worse DER than pyannote can be largely accounted for by increased pause-attributable error.High absolute core error on some corpora, including DiPCo, ICSI, and AVA-AVD, suggests substantial domain mismatch and may help target annotation.Source: Table 3 · p. 4, §5.4 · p. 4

5.5. Tight-boundary speaker diarization

• The enhanced tight-boundary method reduces core error to a level comparable to or below the loose baseline, alongside a 21.47% word error rate for single-channel ASR on AMI-SD.The initial method increased core error and had a word error rate of 23.24%, versus 24.32% for the loose baseline.Source: Table 4 · p. 4, §5.5 · p. 4

• The proposed decomposition splits DER into pause-attributable and residual core components while accounting for overlapping speech.The analyses span synthetic transformations and four real-world settings.Source: §6 · p. 4
• The decomposition guarantees monotonicity and finite-threshold saturation.Source: §6 · p. 4
• Across the evaluated settings, the decomposition exposes error sources obscured by standard DER.Source: §6 · p. 4