PaperlayerView exact revision on arXiv

Source-linked AI summary

VoxTubeS: Distributable Speaker-Anonymized Synthetic Speech Corpora and Their Analysis

Zhe Zhang, Yexin Lu, Junichi Yamagishi

arXiv:2609.12432v1eess.AS

TL;DR

Speaker-anonymized speech may reduce identity exposure, but whether it can remain useful, linguistically faithful, diverse, and fair as training data is an open corpus-level question. VoxTubeS constructs seven variants using three synthesis families and evaluates their privacy and utility trade-offs. The results show distinct operating points rather than a universally dominant method.

  • Problem

    The paper asks whether speaker-anonymized speech can reduce identity exposure while preserving downstream utility, linguistic fidelity, speaker-space structure, and fair subgroup performance.

    Source: §I · p. 1

  • Method

    VoxTubeS constructs seven synthetic variants using OHNN, SALT, and DAIEN-NCFG, then evaluates their privacy, utility, linguistic consistency, speaker diversity, and subgroup fairness.

    Source: §I · pp. 1–2

  • Results

    No method dominates: speaker-consistency training improves privacy with comparable utility and greater diversity, SALT offers strongest ASV utility, and DAIEN-NCFG can suppress identity at substantial utility and diversity cost.

    Source: §VI · p. 6, §V · p. 6

  • Takeaways & Limitations

    Corpus construction is a choice among operating points balancing privacy, utility, diversity, fairness, and responsible redistribution.

    Source: §VI · p. 6

    Findings may vary with alternative speaker encoders, attacker assumptions, or transcription systems because the evaluations depend on ECAPA embeddings and Whisper-generated transcripts.

    Source: §V · p. 6

Abstract

from arXiv · show

Large speech corpora support research, but recordings can expose speaker identity because voice remains a recognizable biometric. Meanwhile, speech data derived from media can be difficult to redistribute reliably. We present \emph{VoxTubeS}, a family of speaker-anonymized synthetic speech corpora designed for redistribution, comprising three method families and seven variants derived from the VoxTube corpus, which is distributed under CC BY-NC-SA 4.0, using 1.29M quality-filtered English utterances from 1,511 speakers. The synthesis methods span voice conversion, latent-space anonymization, and controllable text-to-speech. We evaluate VoxTubeS using utterance-level unlinkability, conversation-level linkability and singling-out, downstream speaker verification, linguistic consistency, speaker diversity, and fairness metrics for gender and accents. Our comprehensive analysis exposes a complex trade-off: stronger identity suppression often reduces linkability but sacrifices utility and population diversity, whereas speaker consistency training improves both utterance- and conversation-level privacy while retaining comparable utility and a broader speaker space. Fairness varies independently of aggregate performance. No method dominates; VoxTubeS therefore treats corpus construction as a choice among operating points that balances privacy, utility, diversity, fairness, and responsible redistribution under the source license.

I. INTRODUCTION

The introduction frames synthetic anonymized corpora as a response to biometric exposure and asks whether identity protection can coexist with useful, linguistically faithful, diverse, and fair training data. VoxTubeS supplies seven variants for comparing these operating points.

I. INTRODUCTION

  • The central question is whether speaker-anonymized speech can reduce identity exposure while preserving downstream utility, linguistic fidelity, speaker-space structure, and fair subgroup performance.The introduction notes that strong utterance-level anonymization does not necessarily yield useful training data.Source: §I · p. 1
  • VoxTubeS derives from VoxTube, distributed under CC BY-NC-SA 4.0, while treating biometric privacy and responsible redistribution as complementary requirements.The passage distinguishes licensing permission from protection against biometric exposure.Source: §I · p. 1
  • Corpus-level anonymization cannot be reduced to one privacy score: stronger identity suppression can destroy useful population structure, while high utility can coexist with residual cross-utterance linkage.The introduction presents privacy, utility, and population structure as competing considerations in choosing a corpus.Source: §I · p. 2
  • VoxTubeS comprises seven utterance-aligned synthetic variants from three method families and evaluates privacy, utility, linguistic consistency, speaker diversity, and subgroup fairness.The variants include OHNN, SALT, and DAIEN-NCFG; OHNN and SALT can process the multilingual inventory, while DAIEN-NCFG is English-limited.Source: §I · pp. 1–2

II. VOXTUBES CORPUS CONSTRUCTION

The corpus construction uses a quality-filtered English VoxTube subset to support comparison across the English-capable synthesis systems. The source corpus and VoxTubeS are distributed under the source license.

II. VOXTUBES CORPUS CONSTRUCTION

  • VoxTubeS inherits VoxTube’s CC BY-NC-SA 4.0 license, which permits non-commercial redistribution.The source corpus was collected from CC BY-licensed videos.Source: §II · p. 2

A. English Filtering Protocol

  • Whisper large-v3 transcripts and filtering address non-English speech, uninformative or repetitive transcripts, and a DAIEN-TTS preprocessing failure before release.The common English protocol supports comparison with systems whose TTS model and text-conditioning pipeline are English-limited.Source: A. English Filtering Protocol · p. 2
  • Filtering reduces the English subset from 1,334,157 to 1,292,167 utterances while retaining all 1,511 speakers, with a 9:1 train/development split.The protocol removes 41,990 utterances after language-confidence, transcript-content, repetition, and video-language checks.Source: A. English Filtering Protocol · p. 2

B. Speaker-Anonymized Corpora with Environmental Sound

The variants are constructed from the same filtered source while preserving environmental sound and aligned content. Three synthesis families implement different routes to speaker anonymization.

B. Speaker-Anonymized Corpora with Environmental Sound

  • Paired construction preserves linguistic content, speaker inventory, and split information across variants while retaining environmental sound.OHNN and SALT remix estimated environment audio after speech anonymization; DAIEN-NCFG synthesizes speech and environment jointly.Source: B. Speaker-Anonymized Corpora with… · p. 2
  • OHNN-HiFiGAN and OHNN-BigVGAN-SC share an OHNN transform, while the latter uses a BigVGAN vocoder trained with a speaker-consistency objective.The objective uses ECAPA-TDNN embeddings to encourage preservation of speaker characteristics represented by the input mel spectrogram and embedding.Source: B. Speaker-Anonymized Corpora with… · p. 2
  • SALT uses WavLM-based voice conversion with weighted reference-speaker features; SALT-k4 and SALT-k8 use four- and eight-speaker mixtures, respectively.Their kNN regression widths are k = 4 and k = 8.Source: B. Speaker-Anonymized Corpora with… · pp. 2–3
  • DAIEN-NCFG uses negative speaker-prompt guidance in DAIEN-TTS to suppress prompted identity, with variants at γ = −1.0, −0.75, and −0.5.A negative guidance weight moves generation away from the prompted speaker condition.Source: B. Speaker-Anonymized Corpora with… · p. 3

III. ANALYSIS OF THE VOXTUBES CORPORA

The analysis extends common voice-anonymization measures across six evaluation axes. It assesses utterance-level unlinkability with ASV and conversation-level privacy with linkability and singling-out measures.

III. ANALYSIS OF THE VOXTUBES CORPORA

  • The evaluation extends VoicePrivacy Challenge metrics across six axes of corpus analysis.The supplied passages describe privacy measurement at utterance and conversation levels.Source: §III · p. 3

A. Utterance-Level Unlinkability

  • Utterance-level unlinkability is measured by ASV equal-error rate using authentic recordings for enrollment and anonymized recordings for testing.Pretrained ECAPA-TDNN verifiers trained on VoxCeleb model an attacker attempting speaker recognition.Source: A. Utterance-Level Unlinkability · p. 3

B. Conversation-Level Privacy

  • Conversation-level privacy uses top-1 speaker matching for linkability and threshold-based isolation of exactly one test conversation for singling-out.The evaluation averages five runs and ten folds over conversation lengths 1, 3, and 30 and populations of 20, 100, and 1,000.Source: B. Conversation-Level Privacy · p. 3

C. Utility Metrics

The utility assessment combines downstream speaker-verification performance, linguistic consistency, and speaker diversity, using separate measures for each.

C. Utility Metrics

  • ASV utility is measured by training a separate ECAPA-TDNN model on each VoxTubeS variant and evaluating EER on VoxCeleb1-O trials.Lower EER indicates better utility.Source: C. Utility Metrics · p. 3
  • Linguistic consistency is assessed with pseudo-word error rate by comparing Whisper large-v3 transcriptions of anonymized and original audio on 29,109 development utterances.The measure is not equivalent to WER against ground-truth transcriptions.Source: C. Utility Metrics · p. 3
  • Speaker diversity is measured from speaker-centroid geometry using mean pairwise cosine distance and effective rank; larger values indicate greater separation and more widely distributed speaker spaces.Effective rank reflects the number of embedding directions with non-negligible variance.Source: C. Utility Metrics · p. 3

D. Subgroup Fairness

The fairness evaluation measures gender- and accent-based subgroup disparities in downstream ASV models; Table II summarizes the evaluation results.

D. Subgroup Fairness

  • Fairness is assessed for ASV models trained on VoxTubeS using female/male VoxCeleb2 trials and ten accent-specific VoxAccent trial sets.Source: D. Subgroup Fairness · p. 3
  • The fairness discrepancy rate (FDR) summarizes subgroup differences in false acceptance and false rejection rates at the pooled-EER threshold, with higher FDR indicating smaller gaps.Source: D. Subgroup Fairness · p. 3

IV. RESULTS

  • Table II summarizes the evaluation results, and the following subsections discuss the main findings.Source: §IV · p. 3

A. Privacy–Utility–Diversity Trade-offs

The results show trade-offs among privacy, utility, and speaker diversity: OHNN-BigVGAN-SC offers stronger privacy with comparable utility, while more extreme suppression comes at a cost. Conversation-level privacy is also tested as utterances are aggregated.

A. Privacy–Utility–Diversity Trade-offs

  • Auth has high utility and diversity but low privacy, whereas DAIEN-NCFG(−1.0) offers strongest privacy at substantial cost to ASV performance; SALT has highest utility among anonymized datasets and OHNN-BigVGAN-SC the widest speaker distribution.Source: A. Privacy–Utility–Diversity Trade-offs · p. 3
  • 38.02% unlinkability EER for OHNN-BigVGAN-SC exceeds OHNN-HiFiGAN and SALT-based methods; its conversation-level linkability is below all methods except DAIEN-NCFG(−1.0), with comparable downstream utility.Source: Figure 2 · p. 4, A. Privacy–Utility–Diversity Trade-offs · p. 4

B. Linkability under Utterance Aggregation

  • With 30 utterances per speaker and 1,000 speakers, original VoxTube is almost totally linkable; identity-related information becomes more detectable as utterances are aggregated, and OHNN-BigVGAN-SC has the best conversation-level privacy except for DAIEN-NCFG(−1.0).Source: Figure 3 · p. 4, B. Linkability under Utterance… · p. 4
  • DAIEN-NCFG(−1.0) is closest to chance for linkability and singling-out, but substantially degrades utility and speaker diversity.Source: B. Linkability under Utterance… · p. 4

C. Linguistic Consistency

SALT-k8 has the lowest pWER among the reported variants, while subgroup results show gender and accent disparities across anonymized systems. These results distinguish linguistic consistency from subgroup fairness.

C. Linguistic Consistency

  • 25.92% pWER for SALT-k8 is the lowest reported, followed by SALT-k4 at 27.17%, OHNN-BigVGAN-SC at 28.31%, and OHNN-HiFiGAN at 28.53%.The DAIEN-NCFG variants group around 31%; SALT-k8 preserves linguistic information despite reduced speaker diversity, while the OHNN pipelines have comparable linguistic consistency.Source: C. Linguistic Consistency · p. 4

D. Subgroup Fairness

  • Female EER is higher than male EER in every condition except collapsed DAIEN-NCFG(−1.0), and Spanish is the most challenging accent condition for most systems.OHNN-BigVGAN-SC and OHNN-HiFiGAN have the highest gender FDR among anonymized systems, but their subgroup EERs remain higher than Auth.Source: D. Subgroup Fairness · p. 4, Table III · p. 5
  • OHNN-BigVGAN-SC has the highest anonymized-system accent FDR at 0.895, below Auth at 0.930, while SALT variants range from 0.772 to 0.798.DAIEN-NCFG(−1.0) has gender EER values near 45% and accent EER values from 38.04% to 45.04%, illustrating that high gender FDR can reflect near-uniform failure rather than comparable effectiveness.Source: D. Subgroup Fairness · p. 4, Table III · p. 5

E. Speaker Diversity Analysis

Speaker diversity differs substantially across VoxTubeS variants, and that diversity broadly aligns with downstream ASV performance. Anonymization also reshapes gender-related structure in speaker space, sometimes preserving or concentrating clusters.

E. Speaker Diversity Analysis

  • OHNN-BigVGAN-SC has effective rank reff = 36.27, below Auth (97.25) but above the other synthetic variants.Its mean pairwise cosine distance and effective rank are the highest among anonymized variants, indicating broader but still substantially compressed speaker structure relative to Auth.Source: E. Speaker Diversity Analysis · p. 5
  • When training-corpus speaker diversity is substantially reduced, the resulting ASV model loses variation needed to learn robust speaker embeddings.OHNN-BigVGAN-SC retains more diversity than OHNN-HiFiGAN without necessarily increasing conversation-level linkability.Source: E. Speaker Diversity Analysis · p. 5

F. Gender Structure after Anonymization

  • Authentic centroids form male- and female-dominated clusters, and anonymization can preserve, blur, or collapse this gender-related speaker-space structure.OHNN-HiFiGAN and OHNN-BigVGAN-SC preserve authentic cluster assignment for 87.6% and 82.7% of speakers, respectively; SALT-k4 moves 1,321 speakers toward the female-dominated cluster, while DAIEN-NCFG(−1.0) places 1,504 in the male-dominated region.Source: F. Gender Structure after Anonymization · p. 5

V. DISCUSSION

The variants occupy distinct privacy–utility–diversity operating regions: SALT favors utility, DAIEN-NCFG offers stronger identity suppression at a cost, and speaker-consistency training improves the OHNN privacy profile. Correlations and methodological limitations further qualify these comparisons.

V. DISCUSSION

  • ρ = 0.82 links average inter-speaker separation with accent FDR; gender FDR is nearly uncorrelated with utility and diversity.Unlink. EER is negatively correlated with inversed ASV EER and inversed pWER; the two utility measures correlate with each other.Source: §V · p. 5, Figure 5 · p. 6
  • OHNN-BigVGAN-SC has stronger unlinkability and lower conversation-level linkability than OHNN-HiFiGAN, with nearly identical downstream ASV EER and pWER and more speaker diversity, though both trail Auth.Source: §V · pp. 5–6
  • SALT provides the strongest ASV utility, while DAIEN-NCFG(−1.0) offers the highest privacy at substantial utility and diversity cost; no variant dominates every axis.DAIEN-NCFG offers multiple operating points for different privacy levels.Source: §V · p. 6
  • Privacy scores depend on the attacker model, assumed knowledge, and evaluation protocol; findings may vary with alternative speaker encoders or transcription systems.The consistency objective and diversity measures use ECAPA embeddings, while linguistic consistency and DAIEN-NCFG conditioning use Whisper-generated transcripts.Source: §V · p. 6

VI. CONCLUSION

VoxTubeS exposes trade-offs among privacy, utility, diversity, and fairness rather than a single dominant system. Its results support choosing among operating points and jointly evaluating these axes alongside responsible redistribution.

VI. CONCLUSION

  • No method dominates every axis: SALT has the strongest ASV utility, DAIEN-NCFG can suppress identity at substantial utility and diversity cost, and fairness does not follow aggregate performance.Speaker consistency can improve privacy while retaining comparable utility and moderately greater speaker diversity.Source: §VI · p. 6
Loading 2609.12432v1…