Beyond EER: Multi-Dimensional Evaluation of Information Leakage in Speaker De-Identification

1. Introduction

• EER measures resistance to direct re-identification but does not reveal whether demographic attributes or population-level patterns can be recovered.The paper also notes that privacy metrics need to be considered alongside utility, since unintelligible speech undermines communication.Source: §1 · p. 1
• The framework spans verification resistance (EER), soft biometric leakage (SBLS), CMC re-identification, and embedding-space similarity via CCA and Procrustes alignment.The paper also extends SBLS to accent prediction and introduces subgroup protection analysis for demographic disparities.Source: §1 · pp. 1–2
• The cross-corpus evaluation covers four test sets from three Mixer corpora, with more than 22,000 segments per system and over 3.4 million verification trials.The tests cover multiple accents and sampling conditions.Source: §1 · p. 2
• The evaluation measures anonymization’s effect on intelligibility using word error rate and semantic similarity.The authors report a measurable trade-off between identity suppression and speech utility.Source: §1 · p. 2
• No system leads across all evaluated dimensions, so reliance on one metric can yield an incomplete or misleading privacy assessment.The metrics capture different aspects of privacy that can trade off against one another.Source: §1 · p. 2

2.1. Speaker De-Identification Systems and Evaluation

• VoicePrivacy benchmarks primarily use EER and log-likelihood ratio cost, leaving demographic leakage and privacy–utility trade-offs less examined.SDID approaches span signal-processing transforms and neural voice conversion.Source: §2.1 · p. 2

2.2. Identity Information Leakage in Anonymized Speech

• Anonymized speech may retain speaker-level identity signals and soft biometric attributes such as sex, age, and accent.Prior work on anonymized speech primarily tests speaker-level resistance, while findings on demographic attributes and structural similarity motivate complementary evaluation.Source: §2.2 · p. 2

3.1. Corpora and Test Conditions

• Four test sets span Mixer 3, 6, and 7, enabling sample-rate comparisons and analysis of Spanish and Hindi speakers producing English.The corpora provide sex, age-group, and native-language metadata unavailable in existing privacy-evaluation benchmarks; test-set demographic distributions are imbalanced.Source: §3.1 · p. 2, Table 1 · p. 3

3.2. Trials

• Five trial types assess three speaker-identification attacks, pseudo-identity consistency, and pseudo-identity distinctness.The SID trials compare original-to-anonymized, original-to-original, or anonymized-to-anonymized segments; the latter can show inflated EER when anonymized voices become acoustically similar.Source: §3.2 · pp. 2–3, Table 2 · p. 3

3.3. Speaker De-Identification Systems

• The study evaluates five IARPA ARTS systems: four performer submissions and one baseline, spanning diverse architectural approaches.The listed systems include VOXLET, RASP, SHADOW, PHORTRESS, and a baseline.Source: §3.3 · p. 3

3.4. Speaker Identification Backends

• Four independent speaker-identification backends assess anonymization effectiveness.They include NeMo TitaNet Large, NeMo ECAPA-TDNN, Hyperion, and OLIVE.Source: §3.4 · p. 4

3.5. Attribute Classifier and Automated Speech Recognition Systems

• VoxProfile serves as the soft-biometric attacker for sex, age-group, and accent prediction, while Whisper and Canary-1B transcribe speech for intelligibility assessment.The authors note that greater attacker diversity would offer a more complete assessment; Canary-1B was added because it exhibits fewer hallucinations during silent segments.Source: §3.5 · p. 4

4.1. Speaker Verification EER

• EER measures the point where false acceptance and false rejection rates are equal; higher SID-attack EER indicates stronger privacy, while lower consistency-trial EER indicates reliable pseudo-identities.The respective targets are 50% for SID attack trials and 0% for consistency trials.Source: §4.1 · p. 4

4.2. Soft Biometric Leakage Score (SBLS)

• SBLS combines attribute inference, residual dependence between true and predicted labels, and subgroup protection, with weights α = β = 0.4 and γ = 0.2.The authors describe the weighting choice as heuristic and slightly downweight subgroup robustness.Source: §4.2 · p. 4
• Attribute privacy uses one-vs-rest AUC, with Pattr = 1 representing chance-level inference and low leakage, and Pattr = 0 representing near-perfect recoverability and high leakage.When only hard predictions are available, macro balanced accuracy replaces mean AUC and is normalized by the chance-level baseline 1/Ka.Source: §4.2 · p. 4
• Subgroup protection uses subgroup leakage, and varying ω from 0.3 to 0.9 changes SBLS by at most 0.004 without changing rankings.When accent is included, demographic and accent subgroup protection are averaged equally.Source: §4.2 · pp. 4–5

4.3. CMC

• CMC records the rank of the first correct match when each anonymized embedding is compared by cosine distance with original embeddings; lower CMC indicates stronger anonymization.Both NeMo embeddings are used, and a permutation bootstrap establishes chance-level performance.Source: §4.3 · p. 5

4.4. CCA and Procrustes Alignment

• CCA reports the held-out mean of the top 10 canonical correlations, while Procrustes alignment measures mean cosine similarity after learning an orthogonal rotation.A random-permutation baseline gives CCA Top-10 ≈ 0.734 and Procrustes cosine ≈ 0.29–0.31; values near baseline indicate decorrelation, while values approaching 1 indicate linear predictability.Source: §4.4 · p. 5

4.5. Speech Intelligibility and Semantic Preservation

• Speech utility is evaluated using WER from two ASR systems and sentence-embedding cosine similarity, which complements WER because it penalizes errors equally regardless of semantic impact.WER counts substitutions, deletions, and insertions relative to the number of reference words; higher cosine similarity indicates stronger semantic preservation.Source: §4.5 · p. 5

5.1.1 Overall EER Landscape

• PHORTRESS is closest to the 50% target on oaoa, indicating the strongest resistance under the primary attack scenario.All systems exceed 50% on oaaa, a pattern attributed to anonymization-induced compression of the speaker identity space.Source: Table 4 · p. 5, §5.1.1 · p. 5
• Baseline, VOXLET, and SHADOW have low aaaa EER, while PHORTRESS has the highest, reflecting different levels of pseudo-speaker consistency.Across profiles, SHADOW's pseudo-identities are readily distinguishable, whereas VOXLET's are near chance.Source: §5.1.1 · p. 5

5.1.2 Privacy–Consistency Trade-off

• PHORTRESS offers stronger privacy but less reliable pseudo-identity consistency; SHADOW is more balanced, while VOXLET is consistent but weaker on privacy.Figure 1 defines the ideal region as high oaoa EER and low aaaa EER.Source: §5.1.2 · pp. 5–6, Figure 1 · p. 6

5.1.3 Accent Effects

• Across accent conditions, oaoa EER differences are generally modest relative to inter-team differences, but some oaoo and oaaa results shift more.Test 4 Hindi increases oaoo EER for every system.Source: §5.1.3 · p. 6

5.1.4 Sample Rate Effects

• On the same 76 English speakers, RASP's oaoa EER rises from 26.3% at 16 kHz to 45.3% at 8 kHz, while its aaaa EER worsens from 10.9% to 31.2%.Most other systems show stable oaoa performance across sample rates.Source: §5.1.4 · p. 6

5.2.1 Overall Results (Sex + Age)

• All systems improve over the unprocessed baseline, but protection remains incomplete and residual vulnerability is concentrated in subgroup protection.Passoc is consistently high, suggesting errors are not driven by simple, exploitable mappings.Source: §5.2.1 · p. 6
• Sex is harder to mask than age, with several systems yielding near-perfect sex recovery (AUC ≈0.95) even when age leakage is moderate.This asymmetry is consistent with deeply embedded sex-correlated acoustic cues such as fundamental frequency and formant structure.Source: §5.2.1 · p. 6
• SBLS rankings are stable across test sets: PHORTRESS ranks first and Baseline second in every condition.Other systems swap only in isolated cases.Source: §5.2.1 · p. 6

5.2.2 Accent as a Third Attribute

• Adding accent preserves top and bottom rankings but narrows mid-tier differences; PHORTRESS is near chance (AUC ≈0.51), while VOXLET shows substantial accent leakage (AUC ≈0.78).RASP and SHADOW improve through stronger accent subgroup protection.Source: §5.2.2 · p. 6

5.2.3 Weight Sensitivity

• Across seven SBLS weighting schemes, the top three and bottom one remain invariant; only RASP and SHADOW swap under equal and γ-heavy weighting.Removing or downweighting subgroup protection does not alter overall conclusions.Source: §5.2.3 · p. 6

5.2.4 Component Correlation

• With accent included, accent subgroup protection correlates weakly with Pattr (ρ = 0.20), Passoc (ρ = 0.31), and Psubgroup noaccent (ρ = 0.43), supporting accent as a distinct privacy dimension.For sex and age only, SBLS components are strongly positively correlated (Pearson r > 0.80), with perfect rank correlation between Pattr and Passoc (Spearman ρ = 1.0).Source: §5.2.4 · p. 6

5.3. CMC Analysis: Speaker Re-Identification

• PHORTRESS has the lowest CMC re-identification rates, near the random-permutation baseline, while VOXLET shows substantially elevated rates across ranks.Lower CMC rates indicate better privacy.Source: §5.3 · p. 6, Table 9 · p. 7
• TitaNet typically has slightly higher re-identification rates than ECAPA-TDNN, suggesting greater sensitivity to residual speaker cues.Source: §5.3 · p. 7

5.4. CCA and Procrustes: Embedding Subspace Analysis

• PHORTRESS is the only system with CCA below the random baseline, indicating stronger-than-random decorrelation between original and anonymized subspaces.Source: §5.4 · p. 8
• All other systems remain well above the CCA baseline, and Procrustes alignment follows the same overall trends.Source: §5.4 · p. 8

5.5. Speech Intelligibility: The Privacy–Utility Trade-off

• VOXLET achieves the best utility, while PHORTRESS has the largest utility degradation, illustrating a sharp privacy–utility trade-off.Source: §5.5 · p. 8
• The most privacy-protective system is also the least intelligible, whereas the most intelligible system offers weaker privacy on several dimensions.Source: §5.5 · p. 8

6. Discussion

• EER, CMC, embedding subspace similarity, and SBLS probe distinct identity-retention mechanisms: pairwise matching, gallery retrieval, structural preservation, and demographic predictability.Source: §6 · p. 8
• Weak correlations among SBLS components, especially accent subgroup protection, suggest accent attributes may occupy partially independent representational subspaces.Source: §6 · p. 8
• WER and semantic similarity do not measure perceived naturalness, speaker consistency, signal-level quality, or forensic detectability; complementary evaluations could address these omissions.Source: §6 · p. 8
• Subspace decorrelation below random-pair baselines suggests some anonymization strategies restructure representational geometry rather than merely perturbing it.The paper proposes formalizing this distinction with information-theoretic or geometric measures.Source: §6 · p. 8

• Five complementary metrics capture different aspects of identity leakage and can yield divergent privacy assessments.Source: §7 · p. 8
• These findings motivate multimetric evaluation as standard practice.Source: §7 · p. 8
• Anonymization methods should better balance privacy protection, speech utility, and broader measures of speech quality.Source: §7 · p. 8