Source-linked AI summary
VoxAnon: A balanced dataset for a fair evaluation of speech anonymization
Orane Dufour, Paul Magron, Mickael Rouvier
TL;DR
Attribute imbalances in attacker training and evaluation data can bias speech-anonymization privacy estimates. VoxAnon is a balanced dataset and evaluation framework that compares balanced and imbalanced conditions. The comparisons show that subgroup re-identification patterns depend on attribute composition, while disparities persist in balanced conditions, especially for accent.
Problem
Attribute imbalances in attacker training and evaluation data can bias privacy estimates for speaker groups, complicating fair evaluation of speech anonymization.
Source: §1 · p. 1Method
VoxAnon uses balanced VoxCeleb1 and 2 data and evaluates semi-informed attackers across anonymizers, with subgroup trials and distinct video sources intended to limit bias.
Source: §1 · p. 1, §3.1 · p. 3, §2.3.3 · p. 3Results
Attribute composition in training and test data changes which subgroups are hardest to re-identify, while balanced conditions retain disparities, most consistently for accent and dependent on the attacker-anonymizer pair.
Source: §1 · p. 1, §4.1 · p. 4, §4.2 · p. 4Takeaways & Limitations
VoxAnon supports fairer anonymization evaluation and provides a balanced benchmark for attribute-inference research.
Source: §5 · p. 4
Abstract
from arXiv · showhide
Speech anonymization is evaluated with automatic speaker verification attackers whose performance depends on the attribute distribution of both their training and evaluation data, which may bias privacy estimates for some speaker groups. To alleviate this issue, we introduce VoxAnon, a dataset built from VoxCeleb1 and 2 with training and evaluation splits balanced across gender, age, accent, and pitch. We extract and cross-check attributes using two complementary methods, balance trial lists per subgroup, and draw test/enrollment utterances from distinct video sources in order to limit linguistic and session bias. Comparing VoxAnon with unbalanced training and test data across four attackers and three anonymizers, we show that attribute imbalance changes which subgroups appear hardest to re-identify, in directions that depend on the attribute. In the balanced setting, residual disparities persist, mostly tied to accent and pitch, and depend on the anonymizer/attacker pair. Code and trial lists are publicly available.
1. INTRODUCTION
Speech anonymization privacy estimates can depend on attribute imbalances in both attacker training and evaluation data. VoxAnon addresses gaps in prior benchmarks with balanced data and shows that subgroup re-identification patterns vary with attribute composition.
1. INTRODUCTION
- Imbalances in attacker training and evaluation data can bias privacy estimates and make assessments unreliable or unfair for some speaker groups.Training imbalance may introduce learning biases, while evaluation imbalance may favor specific groups.Source: §1 · p. 1
- Prior work often lacks balanced target/non-target trials across subgroups and leaves attacker-training balance unaddressed.Existing benchmarks also vary in metadata coverage, and reported disparities differ across datasets.Source: §1 · p. 1
- VoxAnon uses VoxCeleb1 and 2 to balance gender, age, accent, and pitch across training and test splits while accounting for linguistic and video-source bias.The study compares balanced with imbalanced conditions and evaluates three recent VPC anonymizers against four attackers.Source: §1 · p. 1
- Subgroup re-identification performance varies with the attribute composition of both training and evaluation data, while residual disparities depend on the attribute and sometimes the anonymizer-attacker combination.These findings motivate controlling both data distributions when evaluating speech anonymization.Source: §1 · p. 1
2. VOXANON DATASET
VoxAnon is built from VoxCeleb1 and 2 with balanced speaker attributes and evaluation trials. Its construction combines cross-checked labels, subgroup balancing, and utterance selection intended to limit trial, linguistic, and video-source bias.
2. VOXANON DATASET
- VoxAnon is presented with dataset-generation code and repository.1 files containing speaker attributes and utterance lists; training and evaluation dataset statistics are reported in Table 1.Source: §2 · p. 1
2.1. Speaker Attributes
- SpeakerCard labels for gender, age, accent, and pitch are cross-checked against VoxCeleb metadata and VoxCeleb Enrichment, with stricter verification for evaluation speakers than training speakers.VoxCeleb1 evaluation speakers with unverifiable age or accent labels are excluded; VoxCeleb2 training speakers may be retained when age cannot be verified, but inconsistent accents are excluded.Source: §2.1 · p. 2
2.2. Attacker Training Set
- The VoxCeleb2 training set targets balanced attribute subgroups, but limited cross-checkable age labels mean 554 of 1,000 speakers retain extracted age ranges.The balanced set contains around 291 hours; a comparison set samples 1,000 speakers and around 308 hours without controlling attributes.Source: §2.2 · p. 2
2.3. Evaluation Set
- The VoxCeleb1 evaluation set selects 100 speakers for balanced attributes across test and enrollment, then selects utterances to limit linguistic and video-source bias.Table 1 reports training-set speakers, utterances, and duration alongside evaluation speakers and trial counts.Source: §2.3 · p. 2, Table 1 · p. 2
2.3.1. Balancing trial counts
- Test and enrollment sets contain 50 and 100 speakers, respectively, with subgroup labels evenly distributed; comparison sets downsample one subgroup per attribute.The evaluation design aims for equal target and non-target trial counts per speaker and subgroup, though subgroup sizes prevent exact ratios in every case; the downsampled groups are male, 60+, Indian, and high-pitched.Source: §2.3.1 · p. 2
2.3.2. Trial difficulty and residual bias
- Balancing trials on one attribute does not control attribute combinations; gender and pitch skew toward easy trials, though the within-attribute distribution remains fairly balanced overall.Figure 1 groups non-target trials by whether 0–1 or 2–3 additional attributes co-occur.Source: §2.3.2 · p. 2, Figure 1 · p. 3
2.3.3. Limiting linguistic and source bias
- Test and enrollment utterances are selected from distinct video sources to limit session-context and shared-vocabulary cues.The selection procedure chooses the least linguistically similar video pair meeting the utterance requirement; average pair similarity is 0.46 ± 0.09 for balanced and 0.45 ± 0.09 for unbalanced test sets.Source: §2.3.3 · p. 3
3. EVALUATION SETUP
The evaluation uses semi-informed attackers across anonymized and original speech, and measures subgroup disparities using EER. It also examines how the hardest and easiest subgroups change across balanced and imbalanced training and test conditions.
3.1. Attack scenario and systems
- The study evaluates 16 attack scenarios using four anonymizers, including original speech, and four attackers trained on data anonymized like the evaluation data.The attackers cannot know the random source-to-target speaker mapping used during voice conversion.Source: §3.1 · p. 3
- The attacker set comprises ResNet-101, RedimNet, WavLM-ECAPA, and ECAPA-ssl, spanning residual-block, reshaped spectrogram, and WavLM-based embedding approaches.ECAPA-ssl is the latest VPC baseline attacker.Source: §3.1 · p. 3
- The anonymizers are three standard baselines from the latest VPC: B3, B4, and B5.They use, respectively, perturbed phonetic features with resynthesis, semantic and acoustic tokens with a pseudospeaker, and quantized bottleneck features with a target-speaker vector.Source: §3.1 · p. 3
3.2. Bias and fairness evaluation
- Fairness is evaluated as the standard deviation of EER across subgroups for each attribute, while the study does not aim to establish or compare fairness metrics.For each attribute, it also tracks how often the undersampled subgroup in the imbalanced test set has the highest or lowest EER, across training conditions.Source: §3.2 · p. 3
4. RESULTS
Attribute imbalance changes which speaker groups appear hardest to re-identify, and balancing training data alone does not consistently remove bias. Even with balanced training and test sets, disparities remain, especially for accent, and vary by attacker and anonymizer.
4.2. Analysis under the balanced setting
- On balanced data, anonymizer ranking is stable across attackers (B5 > B3 > B4), but B4 scores roughly 8 points below RedimNet and ResNet with SSL-based attackers.Average EERs are 30.28% for B5, 28.88% for B3, and 21.48% for B4.Source: Table 2 · p. 4, §4.2 · p. 4
- Accent has the largest reported subgroup dispersion, peaking at 5.02 for RedimNet and B4, and US speakers are easiest to re-identify in 10 of 12 attacker/anonymizer combinations.Gender and age dispersion is smaller; gender and age disparities are more dependent on the attacker and anonymizer.Source: Table 2 · p. 4, §4.2 · p. 4
4.1. Balanced and unbalanced sets comparison
- Gender difficulty changes with test balance, while accent shows little bias on balanced tests but favors easier re-identification of Indian-accented speakers on unbalanced tests.With unbalanced training, male speakers were easiest on the balanced test in 75% of scenarios but hardest when undersampled in the test in 69%.Source: §4.1 · p. 3, Figure 2 · p. 4
- Balanced training worsens the 60+ age bias on the balanced test and barely reduces it on the unbalanced test; it also widens persistent bias favoring easier re-identification of high-pitched speakers.For speakers aged 60+, the reported change is 38% to 44% on the balanced test and 56% to 50% on the unbalanced test.Source: §4.1 · p. 4
- Subgroup bias can arise from training data, test data, or both, depending on the attribute; balanced training does not always mitigate it.The results support balancing both sets for fair privacy evaluation.Source: §4.1 · pp. 3–4
5. CONCLUSION
VoxAnon provides balanced training and evaluation data for speech anonymization. Comparisons show that attribute imbalance biases privacy evaluation, while disparities persist with balanced data and depend on the anonymizer and attacker.
5. CONCLUSION
- VoxAnon is a training and evaluation dataset for speech anonymization balanced across speaker attributes.The paper compares attacks on VoxAnon with attacks on an unbalanced dataset and finds that speaker-attribute imbalances bias privacy evaluation.Source: §5 · p. 4
- Across several attack scenarios, attribute-related disparities persist despite balanced data, with their extent depending on both the anonymizer and attacker.Source: §5 · p. 4
- VoxAnon also supports attribute-inference research, which the authors say lacks labeled, balanced benchmarks; future work will train attribute extractors on it to study their benefit to attackers.Source: §5 · p. 4