PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning as Deepfakes Evolve: RF-Prompt for Continual Audio Deepfake Detection

Yuankun Xie, Xiaoxuan Guo, Xiaopeng Wang, Siqing Qin, Shaole Li, Kong Aik Lee

arXiv:2609.37586v1cs.SDcs.AI

TL;DR

Continual audio deepfake detection must learn emerging attacks while retaining earlier knowledge, but dataset-based evaluations can change both real-speech domains and fake mechanisms. The paper proposes RAMI and RF-Prompt, which combines a shared real prompt with expanding fake experts. Across five task organizations, RF-Prompt has its lowest common-average and pooled EER under RAMI; on RAMI it achieves 10.110% average EER.

  • Problem

    Dataset-based continual evaluations can change both real-speech domains and fake mechanisms, making their effects difficult to distinguish while detectors must adapt to new attacks without forgetting earlier ones.

    Source: §1 · p. 2

  • Method

    The paper evaluates five task organizations over identical sample pools and proposes RAMI and RF-Prompt, which protects a shared real prompt while expanding inherited fake experts.

    Source: §1 · pp. 2–3

  • Results

    Across five organizations, RF-Prompt has its lowest common-average and pooled EER under RAMI; on RAMI, average EER is 10.110%.

    Source: §5.2 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    RAMI provides a controlled evaluation of task organization in a setting with recurring mixed-domain real speech and incrementally arriving fake mechanisms.

    Source: §1 · pp. 2–3

    Task boundaries are available during training, but task identity is unavailable at inference.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Continual audio deepfake detection requires learning newly emerging deepfake methods while retaining discrimination of previously encountered speech. Existing dataset-incremental evaluation changes both real-speech domains and deepfake mechanisms, making their effects difficult to distinguish. We construct five task organizations over identical training, development, and evaluation pools to study these factors under a controlled sample budget. Our proposed Real-Anchored Mechanism-Incremental (RAMI) protocol reflects the practical setting in which available real speech provides a recurring mixed-domain reference while new deepfake mechanisms arrive incrementally. We further propose RF-Prompt, an asymmetric continual prompt-learning method that preserves reusable real-speech knowledge through a shared real prompt and expands mechanism-specific knowledge through inherited fake experts with orthogonal residuals. Input-adaptive soft fusion combines the accumulated experts into a fixed number of injected tokens without requiring task identity at inference. On RAMI, RF-Prompt achieves 10.110% average EER and 10.370% pooled EER, outperforming all evaluated continual-learning baselines. Across the five controlled protocols, RAMI yields the lowest common-average and pooled EER. Component ablations, limited-data experiments, and cross-backbone evaluations further validate the proposed design.

1 INTRODUCTION

Continual audio deepfake detection must adapt to emerging generators without forgetting earlier attacks, but existing evaluations can confound changes in real-speech domains with changes in fake mechanisms. The paper proposes RAMI and RF-Prompt to study and address this challenge.

Motivation

  • Continual detection must incorporate emerging attacks while retaining earlier knowledge, since generalization to unfamiliar methods does not guarantee adaptation without forgetting.Deepfake generators evolve rapidly, making the detection target change over time.Source: §1 · pp. 1–2

Task organization

  • RAMI organizes fake samples by generation mechanism against recurring mixed-domain real speech, while four comparison protocols use the same sample pools.The protocols allow task-organization choices to be compared over identical training, development, and evaluation pools.Source: §1 · pp. 2–3

Method

  • RF-Prompt protects a shared real prompt and expands inherited fake experts with complementary residuals, using cosine anchoring, orthogonality, and input-adaptive fixed-length fusion.The approach is designed to preserve reusable real-speech knowledge while accommodating diverse deepfake mechanisms.Source: §1 · p. 3

Results

  • 10.110% average EER and 10.370% pooled EER are RF-Prompt’s results on RAMI, the lowest among the evaluated continual methods.The paper also compares task organizations, component ablations, limited-data settings, and speech backbones.Source: §1 · p. 3

2 RELATED WORK

Prior work broadens attack coverage, develops generalization and continual-learning methods, and explores prompt adaptation. This paper examines continual adaptation and retention under controlled task organizations.

ADD datasets and generalization

  • ADD datasets broaden coverage of deepfake methods and recording conditions, while generalization approaches aggregate real representations across domains and separate diverse fake representations.The cited work motivates asymmetric modeling and the study of source-domain arrival alongside mechanism organization.Source: §2 · p. 3

Continual ADD

  • Continual ADD methods develop knowledge distillation, real-embedding alignment, and adaptive update constraints to protect knowledge during successive adaptation.The paper compares these established approaches under controlled task organizations.Source: §2 · p. 3

Continual learning and prompt adaptation

  • Prompt-learning methods explore shared, task-specific, input-weighted, diverse, aligned, or sparse components; RF-Prompt assigns shared capacity to real knowledge and expanding capacity to fake knowledge.The review names L2P, DualPrompt, CODA-Prompt, RainbowPrompt, KA-Prompt, SinglePrompt, and SMoPE.Source: §2 · p. 3

Task-organization comparison

  • Figure 2 arranges five task organizations with real arrival varying by column and fake organization by row, identifying RAMI as Protocol 5.Source: Figure 2 · p. 4

3 RAMI: A REAL-ANCHORED MECHANISM-INCREMENTAL PROTOCOL

RAMI models arriving fake mechanisms against recurring mixed-domain real speech, using controlled task organizations over locked sample pools. Its learning setting, mechanism groups, and task layouts specify the comparison.

3 RAMI: A REAL-ANCHORED MECHANISM-INCREMENTAL PROTOCOL

  • RAMI combines mechanism-incremental fake tasks with a recurring mixed-domain real background to represent detector updates with new attacks and available real speech.Source: §3 · p. 4

3.1 LEARNING SETTING AND LOCKED SAMPLE POOLS

  • Training accesses the current task partition but no historical training audio; task boundaries are available during learning, while task identity is unavailable at inference.All five protocols use identical locked pools: 19,200 training, 19,200 development, and 40,000 evaluation samples.Source: §3.1 · p. 4

3.2 GENERATION MECHANISMS

  • Fake speech is grouped into four generation pathways: classical pipelines, neural acoustic pipelines, neural codec pipelines, and speech-language-model pipelines.M3 reconstructs existing speech, whereas M4 generates a new speech-token sequence; a downstream neural vocoder does not change either assignment to M2.Source: §3.2 · p. 4

3.3 FIVE TASK ORGANIZATIONS

  • The five organizations vary real arrival and fake grouping; RAMI uses a recurring four-domain real mixture with disjoint real utterances and one mechanism’s fake samples per task.Each RAMI training task contains 600 fresh real utterances from each domain and 2,400 fake utterances from one mechanism.Source: §3.3 · p. 5

4 RF-PROMPT: ASYMMETRIC CONTINUAL PROMPT LEARNING

RF-Prompt preserves shared real-speech knowledge while expanding fake experts through inherited knowledge and complementary residual learning. It fuses the experts into a fixed-length prompt sequence without requiring task identity at inference.

4 RF-PROMPT: ASYMMETRIC CONTINUAL PROMPT LEARNING

  • RF-Prompt protects a shared real prompt and expands fake experts through inherited knowledge and complementary residual learning, then fuses them into a fixed-length prompt sequence.The frozen speech backbone is prompted at each layer, and a trainable AASIST backend maps the final audio representation to binary logits.Source: §4 · p. 5, §4.1 · p. 5

4.1 RF-PROMPT FRAMEWORK

  • The shared real prompt and backend remain trainable, while historical fake experts are frozen; both prompt types participate in predictions for both classes.The real/fake names describe the allocation of shared and expanding capacity, not class-exclusive supervision.Source: §4.1 · p. 5

4.2 CONSISTENCY LEARNING FOR THE SHARED REAL PROMPT

  • The real-prompt consistency constraint anchors corresponding token directions across the first K layers to a detached reference from the previous selected prompt.It protects shared knowledge without historical audio or a teacher-feature forward pass; K = L by default.Source: §4.2 · p. 6

4.3 ORTHOGONAL LEARNING FOR FAKE PROMPTS

  • At task initialization, current-task fake-speech queries are averaged to select transferable knowledge, and the new expert inherits a base once before utterance-level fusion.The new expert starts with a small perturbation projected away from the historical complete-Prompt space.Source: §4.3 · p. 6
  • The current fake expert is optimized while its inherited base stays fixed, and normalized overlap between current and historical residual factors is penalized.Historical factors are detached; the orthogonality loss is zero for Task 1 and is a soft objective encouraging complementary new directions.Source: §4.3 · p. 6

4.4 INPUT-ADAPTIVE FUSION AND OPTIMIZATION

  • Input-adaptive fusion routes each utterance over all available complete fake experts, uses shared routing weights across layers, and requires no inference-time task identity.Five fused fake tokens and five shared real tokens keep the injected length fixed at ten per layer, while the expert bank grows with tasks.Source: §4.4 · pp. 6–7
  • Figure 4 arranges acquired-task EER by checkpoint after each task and acquired mechanism; lower EER is better, with final checkpoints boxed in purple.The figure shows Sequential and RF-Prompt on RAMI; complete trajectories for all continual methods are in Appendix C.Source: Figure 4 · p. 7

5 EXPERIMENTS

Experiments compare continual methods and task organizations, then test RF-Prompt’s components, limited-data behavior, and transfer across speech backbones. The evaluations use controlled protocols and matched settings to examine performance and retention.

5.1 EXPERIMENTAL SETUP

  • Experiments use frozen XLS-R 300M with a trainable AASIST backend; checkpoints are selected by development EER, and cross-protocol pooled EER uses the same 40,000 evaluation utterances.All reported results are single-seed observations; the paper makes no claim of statistical significance.Source: §5.1 · p. 7

5.2 MAIN COMPARISON AND CONTINUAL RETENTION

  • 10.110% average EER and 10.370% pooled EER: RF-Prompt beats RAWM on RAMI, although joint co-training remains better in aggregate by a 0.950% pooled gap.The Table 1 comparison uses XLS-R 300M and matched experiments; offline co-training is not counted when identifying the best continual method.Source: §5.2 · pp. 7–8, Table 1 · p. 8
  • RF-Prompt finishes with 10.40% EER on M1 and 13.12% on M3, versus Sequential’s 14.50% and 18.78%, and lowers average forgetting from 6.433% to 4.560%.Its M2 EER recovers to 8.40% after M4, close to 8.24% immediately after acquisition.Source: §5.2 · p. 8

5.3 EFFECT OF TASK ORGANIZATION

  • RAMI gives RF-Prompt its lowest aggregate errors across five organizations: 10.110% common-average EER and 10.370% pooled EER, versus 11.035% and 11.330% under Protocol 4.With real-arrival strategy held fixed, changing from dataset-wise to mechanism-wise fake tasks lowers common-average EER from 16.400% to 15.675% and pooled EER from 16.430% to 15.900%.Source: Table 2 · p. 8, §5.3 · p. 8

5.4 COMPONENT ABLATIONS

  • Removing any component increases average and pooled EER; removing residual orthogonal loss causes the largest pooled degradation at 2.435%, while uniform fusion raises it by 2.110%.These are controlled single-seed ablations, not estimates of independent additive component effects.Source: Table 3 · p. 9, §5.4 · p. 9

5.5 SENSITIVITY AND TRANSFER

  • 10.370% is the lowest pooled EER in the protection-depth sweeps, attained when the constraint is applied to all 24 layers.Both sweeps are non-monotonic, and the other constraint remains active across all 24 layers.Source: §5.5 · p. 9, Figure 5 · p. 9
  • With 100 unique fake training utterances per task, RF-Prompt reaches 12.375% pooled EER versus 17.645% for the strongest compared baseline, and outperforms four baselines at every tested budget.The budgets are 100, 500, or 1,000 fake utterances per task; real training data remains at 2,400 utterances.Source: §5.5 · p. 9, Figure 5 · p. 9
  • RF-Prompt improves average EER, pooled EER, and AF over Sequential on four alternative backbones, with pooled EERs of 12.920%, 10.040%, 9.570%, and 9.550%.The backbones are WavLM-Large, W2V-BERT 2.0, XLS-R 1B, and XLS-R 2B, respectively.Source: §5.5 · p. 9

6 CONCLUSION

The paper introduces RAMI and reports that RF-Prompt performs best among the five tested task organizations under RAMI, while identifying remaining open challenges.

6 CONCLUSION

  • Changing real domains, long task sequences, and unseen generators remain open challenges for the evaluated approach.The conclusion also reports that RF-Prompt’s components improve performance under RAMI and in matched low-resource and cross-backbone comparisons.Source: §6 · p. 9

AI USE STATEMENT

The authors describe their use of AI tools for manuscript preparation and explain how the benchmark’s fixed sample pool is organized into protocols.

AI USE STATEMENT

  • Generative AI assisted manuscript drafting and language polishing; authors reviewed the text and verified the experimental results, analyses, and scientific claims.The authors state they take full responsibility for the final content.Source: AI USE STATEMENT · p. 10

A BENCHMARK CONSTRUCTION AND DATA COMPOSITION

A.1 LOCKED SAMPLE POOLS

The five protocols hold the utterance pools and split membership fixed while varying how real speech and fake mechanisms are assigned to tasks. RAMI pairs a recurring mixed-domain real-speech reference with mechanism-wise fake tasks.

A.1 LOCKED SAMPLE POOLS

  • All five protocols use identical utterances, labels, audio paths, and split membership; training and development each contain 9,600 real and 9,600 fake utterances, and evaluation contains 20,000 of each.Each of the four real-source domains contributes 2,400/2,400/5,000 utterances across training, development, and evaluation.Source: §A.1 · p. 13

A.2 FIVE PROTOCOL ORGANIZATIONS

  • The protocols vary dataset-wise or mechanism-wise fake tasks and dataset-wise, source-matched, or recurring mixed-domain real arrivals; RAMI combines mixed-domain real speech with mechanism-wise fake tasks.Protocols 4 and 5 use an equal four-domain real mixture, while their fake tasks are dataset-wise and mechanism-wise, respectively.Source: §A.2 · p. 13

A.3 GENERATION MECHANISMS AND RAMI COMPOSITION

  • Each RAMI mechanism task contains 2,400 fake utterances for training and development and 5,000 for evaluation, but mechanism and source are not fully crossed in the locked pool.M1, M3, and M4 each draw training fakes from one source dataset, whereas M2 spans three; M1 and M4 also include cross-source examples in evaluation.Source: §A.3 · p. 14, Table 5 · p. 14

B.1 TRAINING CONFIGURATION

RF-Prompt freezes the speech backbone while updating a shared real prompt, the current fake expert, and the detector backend. Its task sequence inherits a selected fake expert and learns a residual while prior experts remain frozen.

B.1 TRAINING CONFIGURATION

  • RF-Prompt freezes the SSL backbone and trains the AASIST backend, shared real parameters, and current fake expert; it retains no historical audio or teacher-feature forward pass.Binary cross-entropy uses both classes, while the real consistency term compares parameters and is sample-independent.Source: §B.1 · p. 14, Table 7 · p. 15

B.2 MATCHED BASELINE SETTINGS

  • Baseline comparisons use matched frozen-backbone implementations; EWC, OWM, RegO, and LwF settings are taken from each run's active configuration.SMoPE is adapted to XLS-R attention projections and evaluated on the same four-task RAMI pool and training budget as other adapted prompt baselines.Source: §B.2 · p. 15

B.3 EXPERT CONSTRUCTION AND TENSOR DIMENSIONS

  • Each XLS-R 300M fake expert has shape 24 × 5 × 1024, and inference soft-routes all complete experts into five effective fake tokens per layer alongside five real tokens.At Task 4, routing combines a bank of four experts for each minibatch; selecting an inherited base at initialization is a separate operation from inference routing.Source: §B.3 · p. 15

B.4 ORTHOGONAL INITIALIZATION AND RESIDUAL REGULARIZATION

  • New residual directions are initialized by projecting away the historical complete-prompt space and scaling the perturbation relative to inherited tokens; training encourages residual orthogonality with a soft loss.Finite precision and QR rank deficiency can weaken the numerical interpretation, and the method gives no rank-adaptive guarantee for arbitrarily long task sequences.Source: §B.4 · pp. 15–16

B.5 TASK-LEVEL TRAINING PROCEDURE

  • Each task selects a historical base from current fake queries, initializes a residual, trains with input-dependent expert fusion, and carries forward the checkpoint selected by development EER.Prior fake experts remain frozen; the current expert is appended after evaluation, and ties in checkpoint selection are broken by development loss.Source: §B.5 · p. 16

C COMPLETE CONTINUAL-LEARNING RESULTS

On RAMI, RF-Prompt has the strongest final average and pooled EER among sequential continual-learning methods, while retaining acquired-task performance across updates. The section also examines design choices, limited fake-data budgets, and prompt-token counts.

C.1 RETENTION MATRICES ON RAMI (PROTOCOL 5)

  • RF-Prompt maintains the strongest final average and pooled EER among continual baselines on RAMI, with lower final M3 EER than every continual baseline and M2 EER close to its post-Task-2 value.Table 8 reports acquired-task trajectories after each update; unacquired tasks are omitted because they do not measure retention.Source: Table 8 · p. 17, §C.1 · p. 17

C.2 RF-PROMPT RETENTION MATRICES ON PROTOCOLS 1–4

  • Figure 6 shows RF-Prompt's acquired-task EER trajectories on Protocols 1–4, distinguishing dataset-wise fake tasks in Protocols 1 and 4 from mechanism-wise tasks in Protocols 2 and 3.The panels use a shared color scale and report each protocol's final native average and pooled EER.Source: Figure 6 · p. 17, §C.2 · p. 17

D DESIGN EXPLORATION FOR REAL AND FAKE PROMPTS

  • Parameter-level cosine anchoring reduces pooled EER from 11.180% to 10.370% versus SPD, while inherited residual construction reduces it from 12.355% to 10.370% versus Full-Prompt orthogonality.The residual construction transfers a selected historical base and separates only the newly learned residual, rather than pushing the inherited structure away.Source: Table 9 · p. 18, D DESIGN EXPLORATION FOR REAL AND FAKE… · p. 18, Figure 7 · p. 18
  • Complete experts have off-diagonal subspace overlap of 0.817–0.951, compared with 0.010–0.030 for residuals; these parameter-space measurements do not directly establish functional or mechanism-specific separation.The two-dimensional PCA plots are illustrative because their first two components explain less than 5% of variance.Source: D DESIGN EXPLORATION FOR REAL AND FAKE… · p. 18, Figure 8 · p. 19

E.1 UNIQUE-FAKE-UTTERANCE BUDGETS

  • The limited-data experiment restricts each task to 100, 500, or 1,000 unique fake utterances while retaining 2,400 real utterances and fixed development and evaluation sets.Resampling preserves class balance and optimization steps; the 2,400-example setting uses the complete fake task.Source: §E.1 · p. 19, Table 10 · p. 19

E.2 PROMPT-TOKEN COUNT

  • Five real and five fused fake prompt tokens per layer give the lowest evaluated average and pooled EER; the corresponding AF is 4.560%.This selected allocation injects ten prompt tokens in total per layer.Source: §E.2 · pp. 19–20, Figure 9 · p. 20

E.3 CROSS-BACKBONE CONFIGURATION AND RESULTS

Across four alternative backbones, RF-Prompt improves average EER, pooled EER, and AF relative to Sequential. The paper situates this comparison among work on deepfake datasets, detection, continual learning, and prompt adaptation.

E.3 CROSS-BACKBONE CONFIGURATION AND RESULTS

  • RF-Prompt improves average EER, pooled EER, and AF across all four alternative backbones compared with Sequential after four completed tasks.The comparison adapts prompt width and backend projection to each backbone.Source: Figure 10 · p. 20, §E.3 · p. 20

F EXTENDED RELATED WORK

  • The related-work overview reviews audio deepfake datasets and detection methods, then continual-learning approaches with emphasis on prompt-based adaptation.Source: F EXTENDED RELATED WORK · p. 20

F.1 AUDIO DEEPFAKE DATASETS

  • The reviewed datasets cover varied generation methods and recording conditions; unlike dataset-defined EVDA tasks, RAMI compares real-arrival and fake-organization choices on a locked sample pool.The cited resources include ASVspoof 2019, ASVspoof 5, ADD, CodecFake, and AT-ADD (Todisco et al., 2019; Wang et al., 2025; Yi et al., 2022; Xie et al., 2024b; 2026).Source: §F.1 · p. 21

F.2 AUDIO DEEPFAKE DETECTION METHODS

  • Detection methods aggregate real-speech representations across domains or model fake diversity, while scaling pretrained representations alone does not guarantee robustness under distribution shift.Continual ADD methods include DFWF, RAWM, RWM, and RegO; Oiso et al. (2024) study prompt adaptation for target-domain performance rather than retention across successive tasks.Source: §F.2 · p. 21

F.3 CONTINUAL LEARNING AND PROMPT-BASED ADAPTATION

  • EWC and OWM mitigate forgetting through parameter regularization or constrained updates, while prompt methods explore retrieval, shared and task-specific prompts, and input-conditioned composition.The cited prompt work also investigates prompt diversity, alignment, online selection, and sparse experts (Wang et al., 2022b;a; Smith et al., 2023; Hong et al., 2025; Xu et al., 2025; Park et al., 2026; Le et al., 2026).Source: §F.3 · p. 21

G ADDITIONAL METHOD DETAILS

  • Additional method details distinguish one-time parent selection from per-utterance inference routing and specify that offline joint co-training accesses all tasks simultaneously, making it a reference rather than a sequential learner or mathematical upper bound.The primary training setup uses frozen XLS-R 300M, a trainable AASIST backend, and development-EER checkpoint selection.Source: G ADDITIONAL METHOD DETAILS · p. 22

H PROTECTION DEPTH AND LIMITED-DATA ANALYSIS

RF-Prompt performs best against competing continual-learning methods across the tested fake-data budgets, and applying both constraints across all layers gives the lowest pooled EER. The limited-data experiment varies the number of unique fake training utterances while holding other training conditions constant.

H PROTECTION DEPTH AND LIMITED-DATA ANALYSIS

  • Applying both constraints across all 24 layers yields their lowest pooled EER, 10.370%, so RF-Prompt uses all-layer real prompt consistency and fake residual orthogonality.The comparison varies the first K layers receiving each constraint while keeping the other active across all 24 layers.Source: H PROTECTION DEPTH AND LIMITED-DATA… · p. 22
  • The limited-data experiment keeps 2,400 unique real training utterances per task and tests 100, 500, or 1,000 unique fake utterances, resampled to 2,400 instances per epoch.Development pools and evaluation sets remain unchanged; class balance, optimization steps, and learning-rate schedule are held constant.Source: H PROTECTION DEPTH AND LIMITED-DATA… · p. 22
  • RF-Prompt lowers pooled EER versus each budget’s strongest competitor by 5.270% (100), 4.100% (500), 1.660% (1,000), and 2.435% (2,400) unique fake utterances per task.It has the lowest pooled EER among all five continual-learning methods at every budget.Source: H PROTECTION DEPTH AND LIMITED-DATA… · p. 22
Loading 2609.37586v1…