PaperlayerView exact revision on arXiv

Source-linked AI summary

A Federated Deepfake Speech Detection Method Based on Layer-Wise Center-Guided Weighting Aggregation

Yingjian Yu, Haiyan Guo, Tianshun Wang, Zirui Ge, Chi Liu

arXiv:2610.01259v1eess.AS

TL;DR

Emerging ALM-based speech synthesis challenges deepfake speech detection, while centralized co-training brings resource and privacy concerns. FedDSD trains a global detector through local FedProx updates and layer-wise center-guided aggregation without sharing raw audio. Its reported EERs are comparable to co-training, outperform single-corpus models, and generalize across domains.

  • Problem

    Existing detectors struggle with ALM-based TTS speech, while centralized co-training requires substantial resources and raises privacy concerns around sharing raw audio.

    Source: §I · p. 1

  • Method

    FedDSD trains local client models with FedProx and aggregates their parameters using layer-wise center-guided weighting, without sharing raw audio.

    Source: §I · pp. 1–2

  • Results

    FedDSD significantly outperforms isolated single-corpus models, achieves EERs comparable to co-training, and generalizes across diverse cross-domain benchmarks.

    Source: §I · p. 2

  • Takeaways & Limitations

    Results across two local architectures support FedDSD's broad applicability and generalization across domains.

    Source: §V · p. 6

Abstract

from arXiv · show

The advancement of deep learning-based speech synthesis has significantly increased the diversity of deepfake speech, posing threats to voice authentication. While centralized training is effective for deepfake speech detection (DSD), it requires considerable computational resources and raises privacy concerns. To address these issues, we propose a Federated DSD (FedDSD) method that enables collaborative model training across decentralized speech datasets without sharing raw audio. Specifically, each client trains a local model using the FedProx algorithm to mitigate the effects of data heterogeneity and uploads model parameters to a central server. To improve global model aggregation, we further propose a layer-wise center-guided weighting aggregation (L-CGWA) strategy that adjusts each client's contribution per layer based on its distance to a reference center, capturing inter-client and inter-layer discrepancies and enhancing the robustness of model aggregation. Experimental results demonstrate that models trained under the proposed FedDSD method achieve equal error rates (EERs) comparable to those obtained via centralized co-training, while significantly out-performing models trained on individual corpora. Furthermore, the proposed FedDSD method demonstrates robust generalization capabilities across diverse cross-domain datasets.

I. INTRODUCTION

Emerging ALM-based speech synthesis challenges existing deepfake speech detectors, while centralized co-training raises resource and privacy concerns. FedDSD trains a shared detector without sharing raw audio, combining local FedProx training with layer-wise center-guided aggregation; reported results outperform single-corpus training and approach co-training.

I. INTRODUCTION

  • Existing detectors struggle with ALM-based TTS speech because most were trained on vocoder-generated fakes; Codecfake-trained models can detect ALM fakes but degrade on vocoder audio.The passage describes Codecfake as codec-specific datasets of speech re-synthesized with neural audio codecs.Source: §I · p. 1
  • Centralized co-training requires substantial server resources, and further training to address distribution shifts would require sharing privacy-sensitive raw speech.The passage notes that raw speech may contain sensitive personal information users are unwilling or legally prohibited from sharing.Source: §I · p. 1
  • FedDSD combines local training on private client data with parameter aggregation, without sharing raw audio.FedProx mitigates client drift, while L-CGWA weights each layer's client updates inversely to their distance from a reference center.Source: §I · pp. 1–2
  • FedDSD significantly outperforms isolated single-dataset training and achieves EERs comparable to co-training, with strong cross-domain generalization.The reported results support generalization across diverse cross-domain benchmarks.Source: §I · p. 2

II. RELATED WORK

Deepfake speech detection has moved from handcrafted features toward end-to-end and pretrained models, while federated aggregation research has explored weighting strategies at both model and layer levels. This paper uses two local architectures and proposes a center-guided layer-wise aggregation strategy.

A. Deepfake Speech Detection

  • DSD methods evolved from handcrafted features with classifiers to end-to-end raw-audio models, pretrained embeddings, and Mamba networks.The paper selects W2V2-AASIST and RawBMamba as local models, citing pretrained-model performance and Mamba's fast inference.Source: A. Deepfake Speech Detection · p. 2

B. Model Aggregation Schemes

  • Aggregation methods have weighted clients by dataset size, parameter divergence, or class disparities, while L-DAWA applies layer-wise cosine similarity.The paper's L-CGWA instead computes a reference center to guide layer-level local-model aggregation and account for layer-specific variations.Source: B. Model Aggregation Schemes · p. 2

III. PROPOSED METHOD

FedDSD follows a federated cycle in which clients train on local speech data and upload model parameters for server aggregation. Its objective aggregates client losses with weights based on each client's share of the total data.

A. Overall framework of FedDSD

  • Clients download a global model, update it locally on their labeled speech data, upload model parameters, and receive the aggregated global model for the next round.The client data include speech generated by different TTS or voice-conversion frameworks.Source: A. Overall framework of FedDSD · pp. 2–3
  • The server aggregates local parameters into a global DSD model while clients retain their raw speech data.Clients with limited memory or computational resources can participate by training locally and uploading only parameters.Source: A. Overall framework of FedDSD · pp. 2–3
  • The FedDSD objective weights each client's DSD loss according to its number of samples relative to the total across clients.The objective uses a general loss function and the global model parameterized by wG.Source: A. Overall framework of FedDSD · p. 3

B. Fedprox based Local DSD model training

FedProx regularizes local training to address heterogeneous client data, while L-CGWA aggregates model parameters using layer-specific, center-guided client weights.

B. Fedprox based Local DSD model training

  • FedProx is adopted for local training to alleviate optimization inconsistencies and convergence difficulties caused by heterogeneous data distributions.Its proximal term penalizes large deviations of local parameters from the global model, encouraging stable updates.Source: B. Fedprox based Local DSD model… · p. 3

C. L-CGWA based Global DSD model aggregation

  • L-CGWA addresses client representation divergence by adapting aggregation weights separately for each network layer.The method accounts for heterogeneity that varies across network depth rather than applying uniform weighting.Source: C. L-CGWA based Global DSD model… · p. 3
  • For each layer, L-CGWA computes a reference center from client parameters and gives greater weight to client parameters closer to that center.The reference is the arithmetic mean of corresponding parameters across participating clients.Source: C. L-CGWA based Global DSD model… · p. 3
  • The center-based weighting reduces the influence of anomalous or overly specialized client updates without explicit outlier detection.The design treats updates closer to the center as more likely to capture patterns shared across datasets.Source: C. L-CGWA based Global DSD model… · p. 3

D. Details of FedDSD algorithm

FedDSD repeats local client updates and server aggregation, retaining the global-model checkpoint with the lowest averaged client validation loss. The supplied tables identify EER evaluation scopes, but not their cell values.

D. Details of FedDSD algorithm

  • After each round, the server averages client validation losses and retains the global-model checkpoint with the lowest aggregated loss.That checkpoint is downloaded to clients after federated training terminates.Source: D. Details of FedDSD algorithm · p. 3
  • The algorithm iterates through communication rounds, with clients updating locally and uploading model parameters for server-side aggregation.The supplied algorithm passages show parallel client updates, parameter uploads, and transmission of the updated global model.Source: D. Details of FedDSD algorithm · pp. 3–4
  • Table I is labeled as reporting EER results for in-domain and cross-domain evaluations on 19LA and Codecfake.The supplied passage identifies the table's scope but gives no cell values.Source: Table I · p. 4
  • Table II is labeled as comparing aggregation schemes on cross-domain evaluations using W2V2-AASIST.The supplied passage identifies the comparison scope but gives no cell values.Source: Table II · p. 4

1) Datasets:

The experiments use 19LA and Codecfake, with additional cross-domain benchmarks and a two-client setup that pairs vocoder-driven and codec-based data. The implementation also identifies ablation and out-of-domain evaluation scopes.

1) Datasets:

  • Codecfake contains 740,747 training utterances from six codec models, with C7 reserved as an unseen method; its development and evaluation sets contain 92,596 and 224,873 samples.The 19LA evaluation set introduces seven additional vocoder types, yielding thirteen attack classes overall.Source: 1) Datasets · p. 4
  • The evaluation includes 21LA, 21DF, and In-the-Wild in addition to 19LA and Codecfake.The 19LA and Codecfake datasets are partitioned into standard training, validation, and test sets.Source: 1) Datasets · p. 4

2) Implementation Details:

  • The listed evaluation visuals cover W2V2-AASIST ablations and out-of-domain EER comparisons with state-of-the-art detectors.The supplied captions identify these scopes but do not provide table cell values.Source: Table III · p. 4, Table IV · p. 5
  • Figure 2 is labeled as plotting validation loss against communication rounds for FedDSD with W2V2-AASIST and RawBMamba local models.The supplied figure caption identifies the plotted measures and model variants.Source: Figure 2 · p. 5
  • The two-client setup assigns 19LA and Codecfake to separate clients, creating a highly heterogeneous, non-IID setting.Both clients use RawBMamba initially; W2V2-AASIST is also tested, with 20 communication rounds and 10 local epochs per round.Source: 2) Implementation Details · pp. 4–5

B. Experimental Results

FedDSD shows stable validation-loss decline and achieves strong EER results, including the lowest average EER among the compared aggregation methods. Its results remain slightly below centralized co-training, with heterogeneity offered as a possible explanation.

B. Experimental Results

  • Validation losses on clients A and B consistently decrease over communication rounds under FedDSD.The passage describes this trend as stable convergence and progressive global-model improvement.Source: B. Experimental Results · p. 5
  • 1.332% average EER for RawBMamba and 0.266% for W2V2-AASIST under FedDSD are both lower than their individual-corpus results.Both remain slightly inferior to centralized co-training.Source: B. Experimental Results · p. 5
  • Statistical heterogeneity of client data may cause inconsistent updates and hinder global-model convergence, potentially explaining the gap to centralized co-training.The paper frames this explanation as a possibility.Source: B. Experimental Results · p. 5

C. Comparison With Other Model Aggregation Schemes

  • 0.266% average EER makes FedDSD the lowest among the compared aggregation methods on 19LA and Codecfake.All methods were implemented on top of FedProx for a fair comparison.Source: C. Comparison With Other Model… · p. 5

D. Ablation Study

The ablation results support combining FedProx with L-CGWA, while out-of-domain comparisons show strengths and weaknesses across benchmarks. The conclusion describes FedDSD as matching centralized baselines and generalizing across domains.

D. Ablation Study

  • 0.324% average EER without FedProx and 0.675% without L-CGWA support retaining both components in FedDSD.The paper describes both FedProx and L-CGWA as indispensable to overall performance.Source: D. Ablation Study · p. 5

E. Comparison of Out-of-Domain EERs with Existing State-of-the-Art DSD Method

  • FedDSD outperforms several representative methods across 21LA, 21DF, and ITW, but trails the two XLSR-based approaches overall.It has lower EER than W2V2+AASIST2 and XLSR+AASIST on 21DF, but worse performance on 21LA; the two XLSR approaches achieve the lowest overall EERs.Source: E. Comparison of Out-of-Domain EERs… · pp. 5–6

V. CONCLUSION

  • FedDSD combines strong performance with broad applicability, matching centralized baselines and generalizing well across domains in experiments on two local architectures.Source: §V · p. 6
Loading 2610.01259v1…