PaperlayerView exact revision on arXiv

Source-linked AI summary

FedCFM: Federated Continual Domain Generalization for Fake Speech Detection via Conditional Flow Matching

Yingjian Yu, Haiyan Guo, Tianshun Wang, Xinzhou Xu, Zirui Ge, Chi Liu, Ziheng Liu

arXiv:2610.01242v1eess.AS

TL;DR

FSD models trained on fixed data can struggle with emerging spoofing types, and centralized or raw-data-sharing approaches do not fit privacy-constrained distributed clients. FedCFM exchanges client-specific CFM generators to synthesize spoof embeddings for classifier updates through generative replay and knowledge distillation. It reports lower EER than evaluated centralized and federated domain-generalization baselines across five in-domain and seven out-of-domain test sets.

  • Problem

    FSD models trained on fixed datasets face evolving spoofing attacks, while centralized training and raw-speech sharing are constrained by distributed data and privacy concerns.

    Source: §I · p. 1

  • Method

    FedCFM trains client-side CFM generators for spoof-type-specific embedding distributions, exchanges generators across clients, and updates classifiers with generative replay and knowledge distillation.

    Source: §I · pp. 1–2

  • Results

    FedCFM consistently improves EER over evaluated centralized and federated baselines on five in-domain and seven out-of-domain test sets.

    Source: §V · p. 6

  • Takeaways & Limitations

    The reported results demonstrate strong cross-domain generalization under the evaluated training configuration.

    Source: §V · p. 6

Abstract

from arXiv · show

The generalization ability of Fake Speech Detection (FSD) models is crucial for real-world deployment. Existing multi-dataset co-training methods rely on fixed training sets and cannot adapt to emerging spoofing types. Although con-tinual learning has been explored, many approaches overlook limited data storage at individual devices, thereby restricting practical applicability. To address this, we propose FedCFM, a Federated continual domain generalization framework via Conditional Flow Matching (CFM) for collaboration without sharing raw speech data across distributed clients facing diverse and evolving spoofing attacks. Each client trains a CFM-based generator to model spoof-type-specific embedding distributions, and cross-client generator exchange enables synthesis of unseen spoof-type embeddings for continual classifier updating through generative replay and knowledge distillation. With the same training datasets, FedCFM achieves lower EER than the eval-uated centralized and federated domain generalization baselines, demonstrating strong cross-domain generalization. Code will be released on https://github.com/jspycpp/FedCFM.

I. INTRODUCTION

FSD models trained on fixed or centralized datasets can struggle with unseen and evolving attacks, while privacy constraints complicate collaboration across distributed clients. FedCFM exchanges client-private spoof-specific generators and aggregates classifiers to support continual, generalizable FSD without sharing raw speech.

I. INTRODUCTION

  • Single-dataset training limits FSD generalization to unseen attacks, speakers, languages, and recording conditions, while centralized multi-dataset training assumes access to large, diverse datasets.These settings also involve data-collection costs, privacy concerns, and limited scalability as attacks evolve.Source: §I · p. 1
  • Continual FSD must adapt to emerging attacks and preserve prior knowledge, but existing approaches are primarily centralized despite distributed data and raw-speech privacy constraints.Enabling privacy-preserving collaboration for continual, generalizable FSD remains an open problem.Source: §I · p. 1
  • FedCFM combines client-side spoof-distribution generators with classifier aggregation, redistributing generators for embedding-level knowledge sharing without raw-speech exchange.Clients update classifiers using local and synthesized embeddings, and the server aggregates the classifier parameters.Source: §I · pp. 1–2
  • FedCFM models spoof-type-specific embedding distributions with a client-side CFM generator trained using joint supervised contrastive and flow-matching objectives.The objectives jointly enforce discriminative structure and conditional distribution modeling.Source: §I · p. 2

A. Federated Learning in Deepfake Detection

  • FedCFM frames FSD as federated continual domain generalization, exploiting client-level heterogeneity and CFM-based spoof-distribution generation to target unseen and evolving attacks without sharing raw speech.This differs from federated continual approaches described as addressing general class-incremental learning rather than client-specific spoofing distributions for cross-domain FSD.Source: A. Federated Learning in Deepfake… · p. 2

B. Federated Domain Generalization

Federated domain generalization seeks to generalize from decentralized, heterogeneous source domains under privacy constraints. FedCFM exchanges client-private spoofing generators to synthesize unseen spoofing embeddings and support continual adaptation.

B. Federated Domain Generalization

  • Federated domain generalization targets unseen domains from decentralized, heterogeneous sources when each client accesses only local data.Existing methods address heterogeneity at data or feature sharing, server aggregation, representation consistency, or optimization stages.Source: B. Federated Domain Generalization · p. 2
  • FedCFM exchanges client-private generators of class-conditional spoofing distributions rather than features or gradients, supporting synthesis of unseen spoofing embeddings and continual adaptation.The exchanged generators explicitly represent spoof-type-specific distributions across clients.Source: B. Federated Domain Generalization · p. 2

14 Server redistributes {ψt

The algorithm fragment indicates parallel client processing for classifier training and a generator update.

14 Server redistributes {ψt

  • Classifier training is performed across clients in parallel.The fragment refers to Eq. (6) for classifier training and Eq. (7) for a generator update.Source: §15 · p. 3

III. PROPOSED METHOD

Clients observe heterogeneous speech data over time and maintain local feature extractors, classifiers, and private CFM generators. Generators enable synthesis of locally absent attacks, while classifier parameters are aggregated into a global model each round.

III. PROPOSED METHOD

  • Each client maintains a feature extractor, classifier, and private CFM generator while speech data arrive over communication rounds.The client extracts embeddings from its observed speech and updates the generator to capture spoof-type-specific embedding distributions.Source: §III · p. 3
  • Clients use received generators to synthesize embeddings for attack types absent from their local datasets, then train classifiers on observed and synthesized embeddings.Source: §III · p. 3
  • Clients upload updated classifier parameters for standard federated averaging into a global model, and repeat the procedure over communication rounds.Source: §III · p. 3

A. Client-Side Generative Model Based on Conditional Flow Matching

Each client trains a CFM-based generator to model spoof-type-specific embedding distributions. The generator objective combines flow matching with supervised contrastive learning.

A. Client-Side Generative Model Based on Conditional Flow Matching

  • Each client trains a CFM-based generator to model the embedding distribution for locally observed spoof types.The attack type is encoded as a one-hot condition and concatenated with the embedding.Source: A. Client-Side Generative Model Based… · p. 3
  • CFM training interpolates between Gaussian noise and a target conditional embedding, then minimizes the error between predicted and target velocities.The generator uses a lightweight linear-attention variant to reduce model complexity under federated-learning constraints.Source: A. Client-Side Generative Model Based… · p. 3
  • A supervised contrastive term encourages embeddings from the same spoofing condition to cluster and those from different conditions to separate.The generator objective balances the generative and contrastive terms.Source: A. Client-Side Generative Model Based… · pp. 3–4

B. Client-Side Continual Training of Classifiers

Clients update classifiers using locally observed embeddings and generated embeddings for both unseen and previously observed attack types. A distillation loss is added to mitigate forgetting.

B. Client-Side Continual Training of Classifiers

  • Clients use exchanged generators to synthesize embeddings for attack types absent from their local observations.This adds distribution knowledge learned by other clients to local classifier training.Source: B. Client-Side Continual Training of… · p. 4
  • Because raw historical data are not retained, clients preserve bonafide speech and use their previous-round generator to replay embeddings for earlier attack types.A fixed number of embeddings is generated per attack type for continual training.Source: B. Client-Side Continual Training of… · p. 4
  • A knowledge-distillation loss is added to mitigate catastrophic forgetting, with α balancing adaptation to emerging attacks against preservation of prior knowledge.The classifier objective combines the task loss and distillation loss.Source: B. Client-Side Continual Training of… · p. 4

C. Server-Side Coordination and Model Aggregation

The server averages local classifier parameters and broadcasts the aggregate for the next round. It periodically redistributes generators to support cross-client knowledge sharing without direct raw-speech exchange.

C. Server-Side Coordination and Model Aggregation

D. Data locality and Privacy claim

FedCFM keeps raw speech and extracted XLSR embeddings on clients while transferring classifier parameters and generators for federated learning.

D. Data locality and Privacy claim

  • Raw speech and extracted XLSR embeddings remain local to each client.The server aggregates classifier parameters and redistributes generators instead of receiving speech or extracted representations.Source: D. Data locality and Privacy claim · p. 5
  • Clients train CFM-based generators on spoof-type-specific embedding distributions for model-level knowledge exchange.The passage states that this transfer mitigates privacy risks.Source: D. Data locality and Privacy claim · p. 5
  • Waveform reconstruction from intermediate representations typically requires dedicated inversion or synthesis modules.The passage cites prior studies for this qualification.Source: D. Data locality and Privacy claim · p. 5

E. Communication Cost

FedCFM is evaluated across four clients and seven out-of-domain test sets, with spoofing types introduced incrementally. Generator exchange every ten rounds raises communication overhead substantially above regular rounds.

E. Communication Cost

  • 19.6 MB per client per round is the average overhead over 70 rounds, versus 3.58 MB in regular rounds and 164.1 MB in generator-exchange rounds.Classifiers are exchanged every round and generators every M = 10 rounds; the reported overhead includes classifier communication.Source: E. Communication Cost · p. 5

A. Datasets

  • Evaluation uses four clients with distinct FSD datasets and seven out-of-domain test sets.The listed test sets include real-world, multilingual, codec-perturbed, and cross-attack benchmarks.Source: A. Datasets · p. 5
  • The continual-learning setup partitions 28 spoofing types into four disjoint groups of seven, introducing one previously unseen type per client at each of seven stages.Each stage lasts M = 10 communication rounds.Source: A. Datasets · p. 5

B. Implementation Details

FedCFM maintains competitive in-domain performance while achieving lower out-of-domain EER, and outperforms centralized static aggregation under identical dataset combinations. The implementation uses XLSR embeddings, AASIST classifiers, and CFM generators.

B. Implementation Details

  • Each of four clients uses XLSR-300M embeddings, an AASIST classifier, and a CFM generator adapted from Matcha-TTS.Training runs for 70 communication rounds in seven stages; classifiers are exchanged each round and generators every ten rounds. Models use Adam with a 10^-5 learning rate.Source: B. Implementation Details · p. 5

1) Comparison with Representative FSD Methods:

  • FedCFM maintains competitive in-domain performance and lower out-of-domain EER, while outperforming centralized static aggregation under identical dataset combinations.The paper attributes the latter comparison to distribution-level modeling rather than naive data pooling.Source: 1) Comparison with Representative FSD… · pp. 5–6

2) Comparison with Advanced FDG Methods:

FedCFM achieves the lowest EER among four FDG baselines and its CFM generator outperforms the evaluated VAE- and diffusion-based alternatives. Ablations show performance declines when key components are removed.

2) Comparison with Advanced FDG Methods:

3) Ablation Study:

  • Removing cross-client generator exchange, supervised contrastive loss, or knowledge distillation degrades performance.The reported effects are across datasets for generator exchange; the losses support feature discrimination and retention of previously learned knowledge, respectively.Source: 3) Ablation Study · p. 6

4) Effect of Different Generative Models:

  • The CFM-based generator yields the lowest EER across test datasets among the evaluated VAE-, diffusion-, and CFM-based models.The paper suggests CFM's conditional feature transformations and continuous trajectories may support more accurate structure-preserving synthesis.Source: 4) Effect of Different Generative Models · p. 6

V. CONCLUSION

  • Across five in-domain and seven out-of-domain test sets, FedCFM improves EER over the evaluated centralized and federated baselines.Source: §V · p. 6
Loading 2610.01242v1…