PaperlayerView exact revision on arXiv

Source-linked AI summary

Using the enrollment gallery as evidence: affine-invariant score calibration for target speaker tagging

Hee-Soo Heo, Minjae Lee, Youngki Kwon, Bong-Jin Lee

arXiv:2610.11272v1eess.AS

TL;DR

Target speaker tagging needs reliable verification decisions, but enrollment utterances cannot safely serve as a conventional normalization cohort when session clustering makes their statistics reflect gallery proximity rather than impostor behavior. The paper proposes gallery affinity verification, which calibrates with affine-invariant gallery evidence; it improves tagging accuracy with and without conventional normalization.

  • Problem

    Enrollment utterances share deployment conditions with test material, but their session clustering means gallery-based conventional normalization statistics reflect enrollment proximity rather than impostor behavior.

    Source: §1 · p. 1, §2.2 · p. 2

  • Method

    Gallery affinity verification calibrates the selected candidate’s verifier score using affine-invariant score-dispersion and enrollment-profile evidence across the gallery.

    Source: §1 · p. 1, §3 · p. 2, §3.1 · p. 2

  • Results

    The method improves tagging accuracy on both datasets with and without conventional normalization, with roughly 3–5pp gains over the unnormalized baseline and best EER in each externally normalized block.

    Source: §4.2 · p. 4

  • Takeaways & Limitations

    The approach can match external-cohort normalization without an external resource, which is attractive for on-premises deployments where distributing large cohort sets is impractical.

    Source: §5 · p. 4

    The evaluation places every enrolled speaker in every session’s candidate set, reflecting deployments without attendance information and making rejection harder.

    Source: §2.1 · p. 2

Abstract

from arXiv · show

Target speaker tagging (TST) assigns enrolled speaker identities to diarized segments of multi-speaker recordings. The enrollment utterances of the other registered speakers are an attractive source of evidence for calibrating each verification decision: they share the deployment domain of the test material and require no external cohort set. We show, however, that they cannot serve as the cohort of conventional score normalization. Enrollments tend to cluster by recording session, so the per-speaker cohort statistics reflect enrollment proximity to the rest of the gallery rather than impostor behavior, and normalization then rejects entire speakers. We propose gallery affinity verification, a cohort-free calibration that exploits the gallery through two affine-invariant terms, score dispersion and enrollment-profile deviance, and is therefore immune to such per-speaker shifts. On a synthetic benchmark and an in-house meeting corpus, it improves tagging accuracy with and without conventional score normalization and adds further gains when combined with it.

1. INTRODUCTION

TST offers enrollment utterances as deployment-matched evidence for verification, but session-clustered gallery enrollments make them unsuitable as a conventional normalization cohort. The paper proposes gallery affinity verification to use gallery evidence through affine-invariant terms and reports accuracy gains with and without conventional normalization.

1. INTRODUCTION

  • TST assigns enrolled identities to diarized single-speaker regions or rejects segments as non-target, supporting meeting transcription and multi-session analytics.The task combines segmentation and speaker recognition.Source: §1 · p. 1
  • Enrollment and test utterances often come from the same recordings and acoustic conditions, making the gallery a natural source of verification evidence.The enrollment utterances share the deployment pool with material to be tagged.Source: §1 · p. 1
  • The other enrolled speakers form a small, session-clustered cohort whose statistics reflect gallery proximity rather than impostor behavior, introducing speaker-dependent offsets that break cross-speaker score comparability.Identification relies on a shared decision threshold.Source: §1 · p. 1
  • Gallery affinity verification uses score dispersion and enrollment-profile deviance as affine-invariant terms, improving tagging accuracy with and without conventional score normalization on two datasets.The proposed terms are designed to withstand per-speaker offset and scale distortions.Source: §1 · p. 1

2. BACKGROUND AND PROBLEM ANALYSIS

The analysis finds that conventional normalization can fail when the enrollment gallery is its cohort because session-clustered enrollments distort speaker-wise statistics. It contrasts this with the TST pipeline and motivates using each segment’s score pattern across the gallery instead.

2.1. Target speaker tagging system

  • The system enrolls 12–60 seconds per speaker, diarizes test audio, selects a candidate with a separate identifier, and verifies it using cluster-level GMM-calibrated scores and a shared threshold.Every enrolled speaker is considered for every session, reflecting deployments without attendance information.Source: §2.1 · pp. 1–2

2.2. Limits of the gallery as a normalization cohort

  • The cohort analysis compares external VoxBlink2 utterances with other gallery enrollments, applying normalization before session-level GMM calibration while leaving the identifier unchanged.Z-norm and T-norm normalize the enrollment and test sides, respectively; S-norm averages them, and AS-norm uses top-K cohort members.Source: §2.2 · p. 2
  • On the in-house corpus, external-cohort Z-, T-, and S-norm improve DIR@FAR=1% by roughly 4 percentage points over baseline, while enrolled-cohort methods lose roughly half the baseline accuracy.Table 1 compares the cohort choices; AS-norm reports the best result across tried top-K sizes.Source: §2.2 · p. 2, Table 1 · p. 2
  • Session-shared channels cluster enrollments and inflate similarity statistics for some speakers, so normalization can reject their genuine trials while concentrating false alarms on others.The inflated enrollment-side mean reflects channel sharing with part of the gallery rather than impostor similarity.Source: §2.2 · p. 2
  • On TST-Bench, nearly uniform acoustic conditions yield no enrolled-cohort failure; neither cohort improves over baseline because miscalibration is limited and VoxBlink2 audio is acoustically distant.This control supports enrollment-condition clustering, rather than gallery size, as the source of the failure.Source: §2.2 · p. 2
  • The useful gallery evidence is the pattern of a segment’s scores across enrolled speakers, not per-speaker score statistics.This motivates calibrating from the segment’s gallery-wide score profile.Source: §2.2 · p. 2

3. PROPOSED GALLERY AFFINITY VERIFICATION

Gallery affinity verification leaves the identifier’s candidate decision unchanged and calibrates the selected candidate’s verifier score using competitor-score dispersion and enrollment-profile deviance. Its penalties use gallery score patterns while accounting for per-speaker affine distortions.

3. PROPOSED GALLERY AFFINITY VERIFICATION

  • The method calibrates only the selected candidate’s verifier score, subtracting penalties for competitor-score dispersion and deviation from the candidate’s enrollment profile.The identifier’s candidate decision remains unchanged.Source: §3 · p. 2

3.1. Competitor-profile dispersion

  • Dispersion penalizes the distance between the upper and lower ends of competitor scores, since a wide profile can make a non-target’s top score exceed the threshold.A square-root tail rule, d=⌊√m⌋, sets the quantile depth without a free hyperparameter and balances stable quantile estimates with robustness to a few extreme competitors.Source: §3.1 · pp. 2–3

3.2. Enrollment-profile deviance

  • Enrollment-profile deviance compares a segment’s competitor profile with the candidate’s enrollment profile through a non-negative affine fit, measuring shape mismatch after offset and scale are absorbed.The residual is judged against the candidate-specific median enrollment residual; the resulting penalty is zero within the enrollment range, then grows smoothly for excess deviation.Source: §3.2 · p. 3

3.3. Relation to score normalization

  • Unlike cohort normalization, which propagates cohort characteristics into corrected scores, gallery affinity uses affine fitting to remove per-speaker offset and scale and uses dispersion without enrollment statistics.The methods are complementary: normalization calibrates score scale against an impostor population, while gallery affinity tests whether score patterns fit gallery structure.Source: §3.3 · p. 3

4. EXPERIMENTS

The experiments evaluate gallery affinity verification on a synthetic benchmark and an in-house meeting corpus, then test its overall performance, component contributions, interaction with session-level calibration, and sensitivity to gallery size.

4.1. Setup

  • The evaluation uses TST-Bench and an in-house Korean meeting corpus, with the full gallery as the candidate set for every session.DIR is measured at fixed false alarm rates, with thresholds retuned for each condition; EER summarizes the full trade-off curve.Source: §4.1 · p. 3

4.2. Main results

  • Roughly 3–5pp gains across every operating point on both datasets without external-cohort normalization, with further improvements and the best EER in each block when combined with external-cohort normalization.The main results use external-cohort AS-norm with K=100; Table 2 reports DIR@FAR and EER in percent.Source: §4.2 · pp. 3–4, Table 2 · p. 4

4.3. Ablation: contribution of each term

  • At FAR=1% on TST-Bench, dispersion alone improves DIR by about 0.9pp, deviance by about 2.8pp, and their combination by 3.2pp.The combined configuration is strongest at every operating point except the strictest, where deviance alone is marginally better.Source: Table 3 · p. 4, §4.3 · p. 4

4.4. Ablation: interaction with session-level calibration

  • Without session-level GMM calibration, the proposed terms give gains of comparable magnitude at every operating point, and the strongest configuration uses both mechanisms.The authors conclude that gallery evidence is independent of session-level calibration rather than a byproduct of it.Source: Table 4 · p. 4, §4.4 · p. 4

4.5. Gallery-size sensitivity

  • On TST-Bench, average gains grow with gallery size: slightly negative at N=10, positive for every sampled gallery from N=30 onward, and about 3–4pp at N=75 and N=150.For N<150, results average four random speaker subsets; the configuration is unchanged across gallery sizes.Source: Table 5 · p. 4, §4.5 · p. 4

5. CONCLUSION

Gallery affinity verification uses affine-invariant dispersion and profile-deviance terms to calibrate TST decisions, improving tagging accuracy with or without conventional normalization. Its enrollment profiles assume a fixed gallery and must be regenerated when speakers are added or removed.

5. CONCLUSION

  • Gallery affinity verification uses affine-invariant dispersion and profile-deviance terms to exploit gallery evidence while avoiding per-speaker score shifts.The profiles assume a fixed gallery and must be regenerated when a speaker is enrolled or removed, though only at enrollment time.Source: §5 · p. 4
  • The method improves tagging accuracy both with and without conventional score normalization and matches external-cohort normalization without an external resource.The conclusion identifies on-premises deployments, where distributing large cohort sets is impractical, as an attractive use case.Source: §5 · p. 4
  • The paper traces gallery-cohort normalization failure to per-speaker statistics that reflect clustered enrollment conditions rather than impostor behavior.Source: §5 · p. 4
Loading 2610.11272v1…