PaperlayerView exact revision on arXiv

Source-linked AI summary

Preserving Knowledge across Space and Time for Continual Video Deepfake Detection

Taehoon Kim, Jongwook Choi, Heejae Jo, Byungmin Park, Jongwon Choi

arXiv:2609.03446v1cs.CV

TL;DR

Video deepfake detectors must adapt to evolving forgeries, but many existing continual approaches rely on image-level assumptions and miss video-specific cues. MSFD decomposes features into spatial, temporal, and spatiotemporal frequency modalities and adds cross-modality decorrelation. Experiments report strong adaptability while preserving prior knowledge across realistic continual deepfake detection scenarios.

  • Problem

    Detectors trained in a fixed distribution must instead adapt to sequentially emerging video forgeries, while existing continual approaches are described as image-centric.

    Source: §1 · pp. 1–2

  • Method

    MSFD separates spatial, temporal, and spatiotemporal frequency representations for distillation and uses decorrelation to keep joint representations orthogonal to single-modality cues.

    Source: §1 · p. 3, §3.4 · p. 7

  • Results

    Across realistic continual deepfake video scenarios, MSFD shows strong adaptability while preserving prior knowledge, with reported gains in generalization, robustness, and forgetting mitigation.

    Source: §1 · pp. 3–4

  • Takeaways & Limitations

    Different deepfake video types carry distinct spatial and temporal characteristics that need to be retained separately.

    Source: §1 · p. 3

    After learning KoDF, forgetting rises from near-zero to about 2 amid a race-specific distribution shift, and spatial and temporal branches still degrade at uneven rates.

    Source: §4.2 · p. 15

Abstract

from arXiv · show

The continuous emergence of high-quality video deepfakes requires detectors that continually adapt to new forgery patterns, yet existing approaches, which are designed for deepfake images, fail to capture video-specific cues. Unlike deepfake images that contain only spatial artifacts, deepfake videos leave distinct evidence along both spatial and temporal axes, necessitating the separate preservation of each modality during sequential model updates. To overcome this limitation, we introduce a continual deepfake video detection framework, Modality-Specific Frequency Distillation (MSFD), that explicitly decomposes video features into spatial, temporal, and spatiotemporal modalities in the frequency domain. This decomposition enables independent preservation of each modality, as different deepfake video types exhibit varying reliance on spatial and temporal cues across tasks. Furthermore, MSFD adopts a cross-modality decorrelation loss that encourages spatiotemporal representations to remain orthogonal to single-modality cues. Extensive experiments show that our framework achieves stronger adaptation and preserves performance more effectively than state-of-the-art methods across diverse continual deepfake video scenarios.

1 Introduction

Video deepfake detectors must adapt to evolving forgeries while retaining earlier knowledge, but image-centric continual methods do not capture video’s distinct spatial and temporal evidence. MSFD separates these modalities for preservation and is reported to improve adaptation and forgetting mitigation across continual video deepfake settings.

1 Introduction

  • Our method maintains the highest performance with the least degradation as training proceeds across six sequential datasets.The figure tracks initial-task FF++ performance and average performance across FF++, DFD, CDF, DFDCp, FFIW, and KoDF.Source: Figure 1 · p. 2, §1 · p. 4
  • Image-centric continual deepfake methods miss video-level spatiotemporal artifacts, while sequentially encountered forgeries require detectors to adapt over time.Video evidence includes spatial artifacts such as blending boundaries and temporal inconsistencies such as flickering or unnatural motion.Source: §1 · p. 2
  • Spatial-only and temporal-only training produce divergent updates and activations, while cue sensitivity varies across generation paradigms and even within a dataset.These observations motivate preserving spatial, temporal, and spatiotemporal representations separately.Source: §1 · pp. 2–3, Figure 2 · p. 3
  • MSFD decomposes features into spatial, temporal, and spatiotemporal frequency spectra and aligns each modality with the previous model’s spectra.The method uses frequency transforms to isolate distinct forgery signatures, including spatial boundary artifacts and fine-grained temporal irregularities.Source: §1 · pp. 3–4
  • Cross-Modality Decorrelation Loss encourages spatiotemporal representations to remain orthogonal to single-modality cues.The contributions report experiments across continual deepfake video settings showing improved generalization, robustness, and forgetting mitigation.Source: §1 · p. 4

2 Related Work

Prior continual deepfake detectors largely use image-level assumptions, while video continual-learning methods often preserve semantic rather than low-level forensic signals. The paper positions separate preservation of spatial, temporal, and spatiotemporal cues as a response to these gaps.

2 Related Work

  • Existing continual deepfake methods such as CoReD, DFIL, and SURLID preserve knowledge through distillation or feature alignment but rely on image-level assumptions.SARB-DF is identified as prior work on continual video deepfake detection, but it addresses only spatial cues and uses generic regularization.Source: §2 · p. 4
  • Video deepfakes provide complementary spatial and temporal evidence, making a single entangled representation insufficient for retaining both during continual updates.This motivates separately preserving spatial, temporal, and spatiotemporal forensic cues.Source: §2 · p. 4
  • Conventional video continual-learning methods preserve high-level semantics, whereas deepfake detection depends on fragile low-level forensic signals that semantic objectives can wash out.The paper motivates forensic-aware preservation targets for continual deepfake detection.Source: §2 · p. 5

3 Proposed Method

MSFD preserves prior knowledge across continual video tasks by decomposing intermediate features into spatial, temporal, and spatiotemporal spectra. It distills modality-specific signals and decorrelates joint cues from single-modality cues.

3 Proposed Method

  • At each task, MSFD learns the new deepfake domain from current data while using a frozen previous model and replay memory to retain earlier-task accuracy.Source: §3 · p. 5

3.2 Modality-wise Frequency Transform

  • FFT decomposes intermediate features into complementary temporal, spatial, and spatiotemporal spectra for modality-specific distillation.Source: §3.1 · p. 6, §3.2 · p. 6

3.3 Modality-Specific Adaptor

  • MSA recalibrates current-model spectra with learnable modality-wise residual weights and learns sparse binary masks to select data-discriminative frequency bands.Source: §3.3 · pp. 6–7

3.4 Training Objective

  • Frequency-domain distillation transfers knowledge in informative spectral regions by aligning the masked current and previous model spectra for each modality.Source: §3.4 · p. 7
  • CDL penalizes correlation between spatiotemporal spectra and spatial or temporal counterparts, keeping joint dynamics distinct; the total loss adds these terms to classification and logit distillation.Source: §3.4 · pp. 7–8

4 Experiments

Across three continual-learning protocols, MSFD achieves the strongest reported overall performance, including in the few-shot setting. Ablations support modality-specific frequency preservation and complementary components, while race-specific shifts and uneven modality retention remain failure modes.

4 Experiments

  • Evaluation covers ten video-deepfake datasets and three protocols: dataset-incremental, generation-paradigm incremental, and few-shot updates with 25 real and 25 fake videos per later task.Protocol 1 samples 200 real and 200 fake videos from each new dataset; reported metrics include video-level AUC, AUCall, and Favg, while Protocol 3 uses AA and AF.Source: §4 · pp. 8–9

4.1 Comparisons on Continual Deepfake Video Detection

  • MSFD achieves the highest AUCall under dataset- and fake-incremental protocols, and the highest AA with AF = −1.25 under few-shot incremental learning.Under Protocol 1 it maintains near-zero forgetting while AUC rises; in the few-shot setting, the negative AF indicates positive backward transfer.Source: §4.1 · pp. 10–11, Table 1 · p. 10, Table 2 · p. 11

4.2 Ablation Study and Analysis

  • Across memory sizes 50–400, MSFD outperforms baselines on AUCall and Favg; with 50 samples, it matches or surpasses DFIL and SURLID using 400.The authors attribute this memory efficiency to distillation focused on discriminative spectral components.Source: §4.2 · pp. 11–12, Figure 5 · p. 12
  • Multi-layer raw-space distillation yields only marginal improvement, whereas frequency-domain modality decomposition improves results and the full model performs best through complementary components.The full model combines MSFD, MSA, and CDL; the ablation describes decomposition as the key enabler, with MSA reducing noisy bands and CDL reducing cross-modality redundancy.Source: §4.2 · pp. 12–13, Table 4 · p. 13
  • All three frequency modalities contribute: temporal-only improves more than spatial-only, spatiotemporal-only is strongest, and combining modalities further improves performance.The authors relate temporal cues to more generalizable forensic signals and note that spatiotemporal cues capture joint dynamics unavailable to 1D or 2D transforms.Source: §4.2 · p. 13, Table 4 · p. 13
  • CDL improves accuracy and stability by reducing redundant cues in the spatiotemporal branch, with its largest reported improvement on DF40 face reenactment.The paper links reenactment artifacts to temporally inconsistent spatial deformations, making spatiotemporal cues especially relevant compared with face swap.Source: §4.2 · p. 13, Table 4 · p. 13
  • MSA’s reweighting and learnable masking work complementarily; fixed frequency filters perform worse, and the framework outperforms the baseline across tested backbones.MSR calibrates channel-wise spectral responses while MSDM selects discriminative frequencies; after MSA, real/fake spectral distributions separate more clearly even at Layer 1.Source: Table 4 · p. 13, §4.2 · p. 14, Figure 7 · p. 14
  • MSFD still shows race-specific forgetting after KoDF and uneven degradation between spatial and temporal branches.The authors associate the KoDF shift with differences in facial appearance changing previously learned spatial/frequency patterns; Fig. 6 defines retention as final AUC under perturbation relative to AUC when first learned.Source: Figure 6 · p. 12, §4.2 · p. 15

5 Conclusion and Future Work

MSFD preserves continual deepfake video knowledge through modality-specific frequency-domain transfer, and the paper reports stronger generalization, robustness, and stability than existing continual detectors. Future work will investigate video-specific memory updates and adaptive modality balancing.

5 Conclusion and Future Work

  • MSFD decomposes intermediate representations into spatial, temporal, and spatiotemporal spectra for modality-specific knowledge transfer.It also uses the Modality-Specific Adaptor for spectral calibration and masking, and Cross-Modality Decorrelation Loss to encourage orthogonal frequency representations.Source: §5 · p. 15
  • Across continual learning protocols, unseen forgery domains, and perturbation conditions, MSFD demonstrates stronger generalization, robustness, and stability than existing continual deepfake detection frameworks.Source: §5 · p. 15
  • Future work will explore memory updates that consider spatial and temporal information when selecting and maintaining exemplars.The paper also proposes adaptive mechanisms to balance modality importance per task, potentially improving robustness and reducing hyperparameter sensitivity.Source: §5 · p. 15

A.1 Additional Analysis

Grad-CAM visualizations compare attention on the initial FF++ task before and after continual training. MSFD retains attention around key forgery regions while existing methods show attention drift.

A.1 Additional Analysis

Generalization for Unseen Datasets

MSFD generalizes to unseen forgery datasets, remains stronger under spatial and temporal perturbations, and scales to a 12-task sequence. The results cover unseen-domain evaluation, four perturbations, and long-term forgery evolution.

Generalization for Unseen Datasets

  • MSFD achieves the highest average AUC across unseen forgery datasets, outperforming DFIL and SURLID.Models trained under Protocol 1 through DFDCp were evaluated on FFIW, KoDF, FSh, and DFo.Source: Generalization for Unseen Datasets · p. 16, Robustness Analysis · p. 17
  • MSFD maintains higher performance than existing continual-learning approaches across compression, resize, frame-shuffle, and grid-shuffle perturbations.The reported results include both temporal frame shuffling and spatial grid shuffling.Source: Robustness Analysis · p. 17
  • 87.44 AUCall is achieved by MSFD on the 12-task Fake-Incremental sequence, versus 67.43 AUCall for STSP.The paper notes that STSP’s lower nominal Favg comes at the cost of plasticity and adaptation to new forgeries.Source: Table C · p. 17, Scalability to long task sequences · p. 17

Additional Analysis for Effect of the Modality-Specific Adaptor (MSA)

The additional analyses examine modality calibration and decomposition, frequency choices, memory updates, robustness, deployment costs, mask generation, and evaluation settings. They report that modality-specific frequency processing separates real and fake spectra and supports complementary spatial and temporal cues.

Additional Analysis for Effect of the Modality-Specific Adaptor (MSA)

  • After MSA, real and fake frequency spectra become more separated across spatial, temporal, and spatiotemporal modalities at Layers 2–4.The visualization compares spectra before and after MSA for 100 real and 100 fake samples.Source: Additional Analysis for Effect of the… · pp. 17–18
  • For face reenactment, the temporal-only model outperforms the spatial-only model; the full model achieves the highest Average AUC across manipulation types.For face swap, all models achieve high accuracy, consistent with prominent artifacts in both spatial and temporal domains.Source: Analysis on Modality Decomposition… · p. 19, Table D · p. 19
  • Magnitude yields higher AUCall and lower Favg than phase; center-based memory updates show lower average forgetting than random replacement.Phase-based training frequently becomes unstable and its distillation loss can diverge, while AUCall remains largely consistent across memory strategies.Source: Analysis on Frequency Components · pp. 19–20, Analysis of Memory Update Strategy · p. 20
  • FFT-based modality decomposition outperforms both an FFT-free MSFD variant and per-modality 3D convolutions.The comparison indicates that frequency-domain decomposition contributes beyond axis separation alone.Source: Further Analysis on Modality… · pp. 20–21
  • MSFD retains higher AUC than iCaRL and DFIL under Gaussian blur, WebP compression, and a 3D-FFT low-pass filter; it is also competitive with static video detectors.The strongest static spatiotemporal-disentangled detectors cited, AltFreezing and PwTF, collapse under sequential fine-tuning, with Favg of 27.11 and 9.27, respectively.Source: Further Analysis on Modality… · p. 21, Robustness to Frequency-Suppressing… · pp. 21–22
  • MSFD uses more training memory, but its auxiliary modules are discarded after adaptation, leaving deployment FLOPs and memory identical to baselines.Replay storage is also reported as identical to other memory-based methods.Source: Computational Overhead · p. 22

B.1 Modality-Specific Distillation Mask Generation

  • Learnable radial-bin logits become binary pass/block decisions through Gumbel–Softmax, producing modality-specific masks for selective frequency preservation.The resulting masks form 1D bands for temporal, concentric radial bands for spatial, and spherical shells for spatiotemporal spectra.Source: §B.1 · p. 22, Figure D · p. 23

B.2 Dataset Configurations; B.3 Perturbation Settings

  • Tasks 2–6 sample 200 real and 200 fake training videos per dataset; modality retention compares final grid- or frame-shuffle AUC with AUC at first exposure.Retention closer to 1.0 indicates preservation of the model’s original discriminative capability in that modality.Source: §B.2 · p. 23, §B.3 · p. 24

B.4 Training Configuration

Training preserves clip-level temporal coherence through shared augmentation parameters, while evaluation uses only resizing and normalization. The configuration uses 32-frame 224 × 224 clips and an R3D-18 backbone.

B.4 Training Configuration

  • Training applies the same augmentation parameters across each clip, including a clip-level horizontal flip and a mask broadcast across frames.Evaluation applies only resizing and normalization.Source: §B.4 · p. 24
  • Each processed clip contains 32 frames at 224 × 224 resolution, and the adopted R3D-18 backbone uses four feature-extraction layers.Source: §B.4 · p. 24
  • The loss weights are λPD = 1.0, λFDL = 1.0, λCDL = 0.5, with modality weights λS = 0.5, λT = 1.0, and λST = 1.0.Training on Protocol 1 takes approximately 30 minutes per epoch on A100 GPUs.Source: §B.4 · p. 24

B.5 Configuration of Comparison Methods

This section describes reproduced or adapted comparison methods and their configured classifiers, replay strategies, and continual-learning objectives.

B.5 Configuration of Comparison Methods

  • NME computes fake probability by applying a sigmoid to the real–fake prototype-distance difference, using L2-normalized backbone features and exemplar-derived class means.Source: §B.5 · pp. 24–25
  • Figure E contrasts frame shuffling, which randomizes temporal order, with grid shuffling, which rearranges spatial patches while preserving temporal order.The overlaid numbers indicate original frame indices.Source: §B.5 · p. 25
  • ER replays randomly selected, class-balanced exemplars; EWC penalizes changes to important parameters, and iCaRL combines mixed-batch classification with memory-sample distillation.iCaRL reports both NME and fully connected classifier evaluations.Source: §B.5 · p. 25
  • IDER [29] trains an idempotent prediction stage, DCE fuses task-specific experts, vCLIMB regularizes temporal consistency, and Drift combines replay classification with distillation.Source: §B.5 · p. 26
  • HCE modulates feature-level distillation using hypercorrelation weights and adjusts cross-entropy using class-wise hypercorrelation variance ratios; subspace gradient projection constrains updates to orthogonal complements of previously learned input subspaces.The energy retention threshold is 0.99, and no replay buffer is used.Source: §B.5 · pp. 26–27
  • ESSENTIAL aligns semantic prompts with frame and sequence features; SARB-DF combines adaptive replay with EWC, while DFIL combines classification, contrastive, feature-distillation, and logit-distillation losses.Source: §B.5 · p. 27
  • HDP stores temporal universal adversarial perturbations as historical surrogates, whereas SURLID combines stable exemplar replay, distribution refilling, and classifier alignment.Source: §B.5 · pp. 27–28
Loading 2609.03446v1…