PaperlayerView exact revision on arXiv

Source-linked AI summary

Is Broader Better? A Controlled Study of Multilingual Coverage and Pretraining Objective in Frozen SSL Encoders for Speech Deepfake Detection

Benjamin Hurt, Oscar O'Donnell

arXiv:2609.29138v1eess.AS

TL;DR

Prior encoder comparisons do not isolate capacity, pretraining objective, and multilingual coverage, leaving the drivers of out-of-domain generalization unresolved. The paper controls capacity and pipeline to test coverage and objective separately. Coverage is not monotonically beneficial, and masked prediction outperforms contrastive pretraining on matched data.

  • Problem

    Prior comparisons vary encoder capacity, pretraining objective, and multilingual coverage together, so they identify winning encoders without isolating why they generalize out of domain.

    Source: §1 · p. 1

  • Method

    The study fixes the pipeline and capacity, sweeps coverage across four wav2vec2-family encoders, and compares two matched-capacity encoders pretrained on identical data to isolate objective.

    Source: §1 · p. 1

  • Results

    Coverage helps through roughly 100 languages but not at 1406; masked prediction beats contrastive on matched data, and 315M XLS-R matches or surpasses 577M XEUS out of domain.

    Source: §1 · p. 1, §4.4 · p. 4

  • Takeaways & Limitations

    Within this fixed frozen-encoder recipe, encoder choice should favor sufficient rather than maximal multilingual coverage and a masked-prediction objective.

    Source: §6 · p. 4

    Across public checkpoints, multilingual coverage co-varies with pretraining data volume, so the study attributes effects to coverage and correlated data scale rather than language count alone.

    Source: §5 · p. 4

Abstract

from arXiv · show

Frozen self-supervised (SSL) speech encoders are strong, low-cost front ends for audio deepfake detection, and recent comparisons agree that large, multilingual, discriminative encoders generalize best out of domain. These comparisons fail to control for encoder capacity, pretraining objective, and multilingual coverage together, identifying which encoder wins without isolating why. We present a controlled decomposition with a fixed pipeline and trainable capacity. We vary multilingual coverage on four wav2vec2-family encoders, matched to ~315M parameters. We isolate the pretraining objective on two encoders matched on identical data. Coverage does not help monotonically, as out-of-domain error drops sharply at the ~100-language scale (XLS-R) but does not improve further at the 1406-language extreme (MMS). We find that a mid-coverage encoder is strongest on farther out-of-domain sets, matching or surpassing a 577M-parameter model at 315M. Its lead on these far sets, statistically significant under paired bootstrap, and on the official ASVspoof 5 cost metric holds under two backends. Separately, masked-prediction pretraining generalizes better than contrastive on identical data (In-the-Wild EER 26.5% vs. 46.8%). Within this fixed frozen-encoder recipe, we find that broader and larger models are not reliably better.

1. INTRODUCTION

Frozen SSL encoders are low-cost front ends for deepfake detection, but prior comparisons do not isolate which pretraining properties drive out-of-domain generalization. This study controls encoder capacity and pipeline to test multilingual coverage and pretraining objective separately.

1. INTRODUCTION

  • Prior comparisons vary encoder capacity, objective, and multilingual coverage together, identifying winners without isolating why they generalize out of domain.Reported models span roughly 4M–317M parameters, while some MMS evaluations use 1B-parameter checkpoints.Source: §1 · p. 1
  • At matched capacity and with a fixed pipeline, the study sweeps coverage across four wav2vec2-family encoders and isolates objective on two encoders trained on identical data.The coverage sweep ranges from monolingual wav2vec2-LV60 to 1406-language MMS-300M; the objective comparison uses roughly 315M-parameter encoders.Source: §1 · p. 1
  • Out-of-domain error drops sharply near 100-language coverage but does not improve at 1406 languages; the mid-coverage encoder is strongest on farther out-of-domain sets.XLS-R also matches or surpasses the 577M-parameter XEUS out of domain at 315M parameters.Source: §1 · p. 1
  • On identical pretraining data, masked-prediction pretraining generalizes better out of domain than contrastive pretraining.The study also evaluates across five benchmarks and two backends with multi-seed evaluation and paired-bootstrap significance testing.Source: §1 · p. 1

2. RELATED WORK

Prior work finds strong out-of-domain performance from large multilingual encoders, but often bundles scale, coverage, and objective. The paper situates its controlled comparisons against those bundled evaluations and other controlled evidence.

2. RELATED WORK

  • Spoof-SUPERB benchmarks 20 frozen SSL models and finds large multilingual discriminative encoders strongest, attributing the result to a bundle of scale, multilingual pretraining, and speaker-aware objectives.Other work also reports multilingual encoders outperforming monolingual ones, but compares 1B multilingual models with sub-350M monolingual models.Source: §2 · pp. 1–2
  • Prior comparisons often vary capacity, objective, and coverage together, while treating multilinguality as binary and leaving open whether additional coverage helps [3, 4, 5].RAPTOR controls scale at roughly 100M parameters and attributes robustness to multilingual pretraining rather than scale [6].Source: §2 · p. 2
  • A prior cross-corpus evaluation found MMS outperforming XLS-R on In-the-Wild, opposite to this paper’s matched-capacity result; checkpoint size, features, or downstream pipeline may explain the discrepancy.The cited comparison used 1B checkpoints, final-layer features, and a different downstream pipeline.Source: §2 · p. 2

3. METHOD

The method fixes the detection pipeline and trainable capacity while controlling feature scale, then varies multilingual coverage or pretraining objective. Evaluation uses two backends and five benchmarks of increasing distance from training.

3.1. Frozen-encoder pipeline

  • Frozen SSL features from the encoder’s last four transformer layers are fused with a CQCC spectral stream, then passed through a backend and linear bona-fide/spoof classifier.Only projections, fusion, backend, and classifier are trained; systems use the same ASVspoof 2019 LA training corpus and three seeds.Source: §3.1 · p. 2

3.2. Scale normalization as a control

  • A non-affine LayerNorm and shared fixed scalar normalize frozen features before projection, controlling cross-encoder scale while preserving the shared SSL-to-spectral balance.The normalization also resolves XLSR-53’s bimodal in-domain instability; the scale constant is not tuned, and comparison robustness to it follows by construction.Source: §3.2 · p. 2

3.3. Two controlled axes

  • Coverage is varied across four roughly 315M-parameter encoders sharing architecture and contrastive objective, while objective is compared using matched-capacity wav2vec2 and HuBERT trained on identical data.Across public checkpoints, coverage co-varies with pretraining data volume, so the coverage contrast cannot isolate language count alone.Source: §3.3 · p. 2

3.4. Capacity, reference, backends, metrics

  • The study includes 577M-parameter XEUS as an unmatched reference and evaluates MHFA and AASIST backends on five benchmarks, reporting pooled EER and ASVspoof 5 min-DCF.The benchmarks range from in-domain LA19 through LA21, DF21, and In-the-Wild to ASVspoof 5.Source: §3.4 · p. 2

4. RESULTS

The controlled results show that out-of-domain gains do not increase monotonically with multilingual coverage. Matched-data objective and backend comparisons further show that masked prediction performs better than contrastive learning, while XLS-R’s far-set advantage persists across backends.

4.1. Coverage helps up to a point

  • Every out-of-domain benchmark reaches its lowest EER at 128-language XLS-R, then rises at 1406-language MMS, so coverage gains are non-monotonic.Source: Figure 1 · p. 3, §4.1 · p. 3
  • 0.56 MHFA and 0.49 AASIST min-DCF are XLS-R’s best ASVspoof 5 costs, ahead of MMS at 0.65 and 0.60, respectively.Source: §4.1 · p. 3
  • 18.30 vs. 30.30 ITW EER: 315M XLS-R matches or surpasses 577M XEUS on out-of-domain benchmarks, though XEUS is strongest in-domain.Source: §4.1 · p. 3

4.2. Pretraining objective matters independently

  • 26.54 vs. 46.77 In-the-Wild EER: masked-prediction HuBERT outperforms contrastive wav2vec2-LV60 on identical Libri-Light-60k data.Source: §4.2 · p. 3

4.3. What is and is not backend-robust

  • XLS-R’s advantage over MMS on the two far sets is significant under both backends, but the XLS-R–MMS ordering reverses on near sets under AASIST.Source: §4.3 · pp. 3–4

4.4. Summary

  • Coverage peaks near 100 languages, masked prediction outperforms contrastive learning on matched data, and 315M XLS-R matches or surpasses 577M XEUS out of domain.Source: §4.4 · p. 4

5. DISCUSSION

The proposed explanation for the coverage peak is a breadth–representation-quality trade-off, but the study does not test that mechanism. Coverage also co-varies with pretraining data volume, and the findings have additional training-regime and benchmark-provenance boundaries.

5. DISCUSSION

  • The authors hypothesize that XLS-R balances phonetic and acoustic breadth against per-language representation quality, while MMS's 1406-language coverage may dilute representations.They explicitly state that their design does not test this mechanism.Source: §5 · p. 4
  • Public checkpoints make coverage effects inseparable from correlated data scale, so the authors attribute effects to both rather than language count alone.Pretraining data volume ranges from approximately 50K to 500K hours.Source: §5 · p. 4
  • The conclusions are scoped to ASVspoof 2019 LA training and a fixed detection recipe, while shared benchmark provenance may inflate absolute ASVspoof 5 performance.The authors cannot fully separate coverage-driven generalization from exposure to pretraining corpora.Source: §5 · p. 4
  • XLS-R surpassing the 577M-parameter XEUS out of domain is one data point, not evidence for a general scaling law.XEUS is a single larger reference from a different encoder lineage.Source: §5 · p. 4

6. CONCLUSION

Within the fixed frozen-encoder recipe, multilingual coverage helps up to roughly 100 languages rather than monotonically, and masked prediction generalizes better than contrastive pretraining. Whether these patterns hold across training corpora and detection recipes, and what produces the coverage peak, remain open questions.

6. CONCLUSION

  • Within the fixed recipe, the 315M-parameter XLS-R is the strongest out-of-domain choice on farther sets, while 1406-language coverage does not consistently help.The conclusion identifies XLS-R as a strong matched-capacity default.Source: §6 · p. 4
  • Coverage helps up to roughly 100 languages rather than monotonically, and masked-prediction pretraining generalizes better than contrastive pretraining on identical data.These findings come from isolating coverage at matched capacity and objective at matched data.Source: §6 · p. 4
  • Whether the observed effects generalize across training corpora and detection recipes, and what mechanism produces the coverage peak, remain open questions.Source: §6 · p. 4

7. COMPLIANCE WITH ETHICAL STANDARDS

The study evaluates audio deepfake detection using public benchmark datasets under their respective licenses, without collecting new speech or involving human subjects.

7. COMPLIANCE WITH ETHICAL STANDARDS

  • The work studies detection of synthetic speech, not its generation, using publicly available benchmark datasets under their respective licenses.Source: §7 · p. 5
  • No new speech was collected, and no human subjects were involved.Source: §7 · p. 5
  • The authors release no attack systems and specify their public checkpoints, backends, and training configurations in the paper.Source: §7 · p. 5
Loading 2609.29138v1…