PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond DER: Speaker Counting in Crowded End-to-End Diarization

Lahiru Samarakoon, Hongxu Zhu

arXiv:2609.27315v1eess.AS

TL;DR

Crowded EEND diarization under-counts, and duration-weighted DER can mask dropped low-activity speakers. The paper couples speaker existence and frame activity in a gated training loss; its speaker-weighted hybrid use improves counting on crowded recordings without cost to DER or harm to sparse recordings.

  • Problem

    Duration-weighted DER can mask under-counting because talkative speakers dominate the score, leaving dropped low-activity speakers insufficiently reflected.

    Source: §I · p. 1

  • Method

    A gated loss couples speaker existence with frame activity, and its speaker-weighted hybrid reduction supervises speakers equally regardless of airtime.

    Source: §I · p. 1

  • Results

    Speaker-weighted reduction significantly improves counting on crowded recordings without harming DER or sparse recordings.

    Source: §I · p. 1

  • Takeaways & Limitations

    Counting evaluation should complement DER with JER and explicit counting metrics to reveal omitted low-activity speakers.

    Source: §I · p. 1

    On CallHome, exact-count accuracy drops from 25 to 12, although ±1 accuracy is unchanged.

    Source: D. Missed-speaker and per-domain… · p. 6

Abstract

from arXiv · show

Speaker diarization must solve two problems: counting how many speakers are present in a given conversation, and assigning speech to each one. This becomes harder in crowded conversations with five or more speakers. We evaluate how end-to-end neural diarization models count speakers, and find systematic under-counting in crowded recordings. The standard diarization error rate (DER) hides this failure, because it is duration-weighted and barely penalizes the dropped, low-activity speakers. We therefore also report the Jaccard error rate (JER) and explicit counting metrics. We propose a gated loss that couples speaker existence with frame activity. This loss can be computed in two ways, duration-weighted or speaker-weighted, and the speaker-weighted variant, used as a regularizer, reduces the under-count. On real-world crowded recordings our method clearly improves the counting metrics, lowering JER by about 6% relative and DER by about 9% relative, while leaving sparse recordings unharmed.

I. INTRODUCTION

In crowded recordings with five or more speakers, EEND-EDA and EEND-TA systematically under-count, a failure duration-weighted DER can mask. The paper proposes a gated loss and speaker-weighted supervision to improve counting.

  • Both EEND-EDA and EEND-TA systematically under-count crowded recordings, while duration-weighted DER can mask dropped low-activity speakers.Source: §I · p. 1
  • The paper evaluates counting with JER, exact-count accuracy, ±1 accuracy, and count bias.Source: §I · p. 1
  • The gated loss couples speaker existence with frame activity and can be used alone or as a hybrid regularizer.Source: §I · p. 1
  • Speaker-weighted reduction significantly improves counting on crowded recordings without harming DER or sparse recordings, with gains consistent across domains.Source: §I · p. 1

II. SPEAKER COUNTING ANALYSIS

Both attractor-based EEND models degrade as speaker counts rise, with counting reliability declining sharply in crowded recordings. JER and explicit counting measures expose failures that DER can obscure.

  • JER is speaker-weighted and less forgiving of missing low-activity speakers than DER; exact-count, ±1 accuracy, and count bias complement it.Source: §II · p. 2
  • JER worsens more steeply than DER as speaker counts rise, while EEND-TA generally achieves lower error rates than EEND-EDA.Source: §II · p. 2
  • At eight speakers, exact-count accuracy falls below 20% and ±1 accuracy falls to approximately 35%.Source: Figure 2 · p. 2, §II · p. 2
  • Count bias reaches −1.9 at eight speakers, indicating roughly two-speaker under-prediction that DER scarcely penalizes.Source: Table I · p. 2, §II · p. 2
  • Both autoregressive EEND-EDA and non-autoregressive EEND-TA share the decline, suggesting the problem is not specific to attractor generation.Source: §II · p. 2
  • Because EEND-TA generally counts at least as well and achieves lower overall DER, the authors adopt it as the backbone for the rest of the paper.Source: §II · p. 2

III. PROPOSED METHOD

The method adds a training-only gated loss that couples speaker existence with frame activity. The loss can be reduced with duration- or speaker-weighting, and the count is read at inference using a first-stop rule.

  • The gated loss couples speaker existence and frame activity, addressing the standard objective’s separate supervision of these heads.Source: §III · p. 2
  • A training-only gated coupling loss is added to the Existence and Diar losses to form the Hybrid Loss.Its per-cell gated BCE is reduced either duration- or speaker-weighted.Source: §III · p. 2, Figure 3 · p. 3
  • At inference, the predicted count uses the first-stop rule on the existence head.Source: Figure 3 · p. 3

A. The EEND-TA backbone

EEND-TA encodes and temporally subsamples Log-Mel features before splitting them into a frame branch and an attractor branch. Frame-to-attractor dot products provide activity predictions, while speaker count is read from the existence head.

A. The EEND-TA backbone

  • Log-Mel features are encoded and temporally subsampled, then split into a frame branch and an attractor branch.Upsampling restores the frame branch to the original frame rate.Source: A. The EEND-TA backbone · p. 3
  • Diarization activity probabilities come from dot products between upsampled frame embeddings and ℓ2-normalized attractors.Source: A. The EEND-TA backbone · p. 3
  • The first-stop rule reads the speaker count from the existence logits, with attractors before the stop representing speakers.Training uses separate PIT activity BCE and existence BCE terms, with the per-recording Hungarian assignment reused in later terms.Source: A. The EEND-TA backbone · p. 3

B. Gated coupling loss

The gated coupling loss combines existence and activity into one posterior, penalizing disagreement between whether an attractor exists and whether it is active. It is used as a regularizer because using it alone over-constrains the model.

B. Gated coupling loss

  • The gated loss combines existence and activity into one posterior so an attractor counts as a speaker only when it both exists and is active.The separate heads can otherwise disagree, producing an attractor that exists but is silent or is active but marked absent.Source: B. Gated coupling loss · p. 3
  • Matched-attractor supervision aggregates per-cell gated BCE, while an empty-attractor penalty drives empty attractor existence toward zero.Source: B. Gated coupling loss · p. 3
  • On inactive frames, an attractor avoids the conjunctive penalty only by having low existence or low activity.Source: B. Gated coupling loss · p. 3
  • The gated coupling is kept as a regularizer because using it alone over-constrains the model.Source: B. Gated coupling loss · pp. 3–4

C. Hybrid loss

The hybrid loss retains the original decoupled supervision and adds gated coupling as a soft regularizer. The matched gated loss can be reduced by duration or by speaker, with speaker-weighting giving each attractor airtime-independent existence pressure.

C. Hybrid loss

  • The hybrid loss adds gated coupling as a regularizer while keeping the PIT and existence terms at full strength.The coupling acts as a soft constraint, while the decoupled terms directly supervise both heads.Source: C. Hybrid loss · p. 4

D. Duration-weighted vs. speaker-weighted reduction

  • Duration-weighted reduction gives every matched cell equal weight, so talkative speakers contribute more active-cell gradient than quiet speakers.This inherits the airtime bias of duration-weighted DER.Source: D. Duration-weighted vs.… · p. 4
  • Speaker-weighted reduction balances active and silent frames within each attractor before averaging, giving every speaker equal existence pressure.The two reductions compute the same gated loss, but speaker-weighting aligns with speaker-weighted evaluation such as JER and exact-count.Source: D. Duration-weighted vs.… · p. 4

A. Datasets

The evaluation combines simulated pretraining mixtures with several real-world diarization corpora, distinguishing the full set from the crowded 5–8-speaker subset. The experimental setup uses an EEND-TA model and selects checkpoints by DER.

A. Datasets

  • Pretraining uses 800,000 simulated mixtures spanning 1–8 speakers, totaling approximately 80,000 hours.There are 100,000 mixtures per speaker configuration.Source: A. Datasets · p. 4
  • The real-world evaluation pools seven diarization corpora, covering 811 recordings with up to eight speakers and a 149-recording crowded 5–8-speaker subset.The crowded subset is identified as the setting where counting is hardest.Source: A. Datasets · p. 4

B. Experimental Setup

  • EEND-TA uses a six-block Conformer encoder and a three-layer Transformer attractor decoder, with a maximum of eight emittable speakers.The input is 23-dimensional log-Mel features; the encoder subsamples time by a factor of 10.Source: B. Experimental Setup · p. 4
  • The EEND-TA backbone is fine-tuned for 150 epochs, and recordings are cropped to 600 seconds with chunk shuffling.All models share a backbone pretrained on simulated LibriSpeech mixtures.Source: B. Experimental Setup · p. 4
  • Inference averages the top 10 validation-DER checkpoints, and all error rates use a 0-second collar.Attractor-existence and diarization thresholds are both 0.5.Source: B. Experimental Setup · pp. 4–5

V. RESULTS

Among the tested loss objectives, the speaker-weighted hybrid regularizer improves counting metrics while keeping DER at baseline level. On the crowded subset, EEND-TA also outperforms EEND-EDA across the reported metrics.

A. Baselines

  • On the crowded 5–8-speaker subset, EEND-TA reaches 32.9% exact-count accuracy versus 20.8% for EEND-EDA, and has lower DER.EEND-TA outperforms EEND-EDA on every reported metric in both the full and crowded regimes.Source: A. Baselines · p. 5

B. Training Objectives

  • Replacing the decoupled losses with the gated loss alone raises DER from 13.24 to 13.92 overall and from 18.29 to 20.30 on the crowded subset.It yields no crowded exact-count gain, which remains 32.9%, and lowers ±1 accuracy to 59.7%.Source: B. Training Objectives · p. 5, Table III · p. 5
  • The duration-weighted hybrid keeps DER near baseline but crowded exact-count accuracy slips to 28.2%.The matched term pools over frames, leaving the quiet marginal speaker under-supervised.Source: B. Training Objectives · p. 5, Table III · p. 5
  • 2.5 points lower JER accompanies the speaker-weighted hybrid loss, while DER holds at 17.20 on the crowded subset and 13.14 overall.The reduction also shifts count bias from −1.03 to −0.62.Source: B. Training Objectives · p. 5, Table III · p. 5

C. Effect of the existence exponent α

The existence exponent controls how strongly speaker existence gates frame activity, and α=2.0 is the strongest tested operating point. Speaker weighting chiefly recovers quiet, silenced speakers; gains extend across most crowded domains, while CallHome is an exception and sparse recordings remain unchanged.

C. Effect of the existence exponent α

  • Crowded exact-count rises from 32.9% with EEND-TA to 41.6% at α=2.0, while DER shifts from 18.3% to 16.7%.The authors adopt the speaker-weighted hybrid model at α=2.0 as their final system.Source: C. Effect of the existence exponent α · p. 5, Table IV · p. 6
  • The α trend is not monotone: α=2.0 is strongest, while α=2.3 regresses on both DER and crowded exact-count; sparse recordings already have saturated speaker existence.On the crowded subset, exact-count rises from 33.6% at α=1.0 to 41.6% at α=2.0.Source: C. Effect of the existence exponent α · p. 5

D. Missed-speaker and per-domain analysis

  • Of 957 crowded speakers, EEND-TA misses 209 (21.8%) versus about 160 (16.6–16.7%) with speaker weighting; silence cases fall from 73 to 33–37, while overlap-only misses stay at 16.Absorptions decline more modestly, from 120 to 107–110.Source: D. Missed-speaker and per-domain… · p. 6, Table V · p. 6
  • Across four corpora with 5–8-speaker recordings, the speaker-weighted hybrid loss (α=2.0) broadly improves counting over EEND-TA: DIHARD-3 exact counts rise 25→42, while CallHome falls 25→12.Source: D. Missed-speaker and per-domain… · p. 6
  • CallHome’s missed speakers are mostly absorbed, which existence pressure cannot reverse; its exact-count dip rests on one of eight calls and merits caution.The added existence pressure raises false alarms there while missed speech falls, accounting for a small DER regression.Source: D. Missed-speaker and per-domain… · p. 6
  • Sparse recordings remain essentially unchanged: exact-count is 84–85%, ±1 accuracy is near 98%, and count bias is essentially zero.The fixed-roster AMI, AliMeeting, and MagicData domains already have 100% exact-count accuracy.Source: D. Missed-speaker and per-domain… · p. 6

VI. CONCLUSIONS

The paper finds that end-to-end diarization models under-count crowded recordings, a failure that duration-weighted DER conceals. A training-only speaker-weighted gated loss improves crowded speaker counting without changing the backbone or inference, and the authors report that Claude assisted with coding and figure design.

VI. CONCLUSIONS

  • Crowded exact-count rises from 32.9 to 41.6%, ±1 accuracy reaches 72.5%, and count bias roughly halves, with no cost to DER.The reported results are on pooled real-world data.Source: §VI · p. 6
  • The training-only EEND-TA remedy couples speaker existence with frame activity and gives every speaker an airtime-independent vote, leaving the backbone and inference unchanged.The gated loss is applied as a hybrid regularizer.Source: §VI · p. 6
  • Both EEND-EDA and EEND-TA systematically under-count crowded recordings, while duration-weighted DER can stay low as quiet speakers vanish.The conclusion says count bias reaches nearly two missing speakers at eight participants.Source: §VI · p. 6

VII. AI-GENERATED CONTENT DISCLOSURE

  • Claude assisted with coding, figure design, and narrative and language checks; the authors reviewed and verified all technical content and take responsibility for the final paper.Source: §VII · p. 7
Loading 2609.27315v1…