PaperlayerView exact revision on arXiv

Source-linked AI summary

AltFreezing for More General Video Face Forgery Detection

Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, Houqiang Li

arXiv:2307.08317v1cs.CV

TL;DR

Face forgery detectors have often focused on spatial or temporal artifacts, limiting coverage of both in one model. This paper combines AltFreezing with video-level fake-data augmentation to target both artifact types. It reports state-of-the-art performance across evaluations of unseen manipulations and datasets.

  • Problem

    Existing detectors often focus on spatial or temporal artifacts, leaving both artifact types insufficiently covered for general video face forgery detection.

    Source: §1 · p. 1

  • Method

    AltFreezing alternately freezes spatial- and temporal-related network weights, alongside video-level augmentations that generate fake clips with spatial-only or temporal-only artifacts.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    The method reports state-of-the-art performance across five benchmark datasets, including cross-manipulation and cross-dataset evaluations.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that their framework improves generalization to unseen manipulations and datasets while targeting both spatial and temporal artifacts.

    Source: §1 · p. 2, §5 · p. 8

    The method may miss artist-created Photoshop or realistic-editing forgeries and loses performance on heavily compressed videos.

    Source: D. Limitations · p. 11

Abstract

from arXiv · show

Existing face forgery detection models try to discriminate fake images by detecting only spatial artifacts (e.g., generative artifacts, blending) or mainly temporal artifacts (e.g., flickering, discontinuity). They may experience significant performance degradation when facing out-domain artifacts. In this paper, we propose to capture both spatial and temporal artifacts in one model for face forgery detection. A simple idea is to leverage a spatiotemporal model (3D ConvNet). However, we find that it may easily rely on one type of artifact and ignore the other. To address this issue, we present a novel training strategy called AltFreezing for more general face forgery detection. The AltFreezing aims to encourage the model to detect both spatial and temporal artifacts. It divides the weights of a spatiotemporal network into two groups: spatial-related and temporal-related. Then the two groups of weights are alternately frozen during the training process so that the model can learn spatial and temporal features to distinguish real or fake videos. Furthermore, we introduce various video-level data augmentation methods to improve the generalization capability of the forgery detection model. Extensive experiments show that our framework outperforms existing methods in terms of generalization to unseen manipulations and datasets. Code is available at https: //github.com/ZhendongWang6/AltFreezing.

1. Introduction

Face forgery detectors often focus on spatial or temporal artifacts, while naïve spatiotemporal training may neglect temporal evidence. AltFreezing and video-level fake-data augmentations target both artifact types, and the paper reports state-of-the-art performance across challenging evaluation settings.

1. Introduction

  • Prior detectors often target spatial or temporal artifacts alone, while naïve 3D ConvNets may favor obvious spatial cues and ignore temporal evidence.The paper attributes this tendency to spatial artifacts being more obvious than temporal incoherence.Source: §1 · pp. 1–2
  • AltFreezing alternately freezes spatial- and temporal-related weights to encourage a spatiotemporal network to capture both artifact types.The network’s spatial and temporal convolutional kernels are assigned to the corresponding weight groups.Source: Figure 1 · p. 1, §1 · p. 2
  • Video-level fake-data augmentations generate clips with temporal-only or spatial-only artifacts to encourage more general representations.The paper describes randomly dropping or repeating frames and blending a region from one real clip into another.Source: §1 · p. 2
  • The method reports state-of-the-art performance across five benchmark datasets, including cross-manipulation and cross-dataset evaluations.The introduction also reports performance in scenarios involving unseen forgeries and perturbations.Source: §1 · p. 2

2. Related Work

The related-work review covers image and video forgery detection, generalization to unseen manipulations, and data synthesis. It describes a shift from image-focused artifacts toward video-level approaches and spatiotemporal representations.

2.1. Image Face Forgery Detection

2.2. Video Face Forgery Detection

2.3. Generalization to Unseen Manipulations

  • Generalization approaches synthesize image forgeries through warping or blending [36] [33] [45], while video detectors differ in reliance on audio or temporal incoherence [22] [26] [55].Source: §2.3 · pp. 2–3

2.4. Data Synthesis for Face Forgery Detection

  • Earlier synthesis mainly targeted images; video-level synthesis remained limited, motivating augmentations designed for spatiotemporal networks.Source: §2.4 · p. 3

3. Method

Face forgery detectors may miss artifacts when they focus on only spatial or temporal cues, and naïvely trained spatiotemporal networks may rely on one easy artifact type. The method combines alternating weight freezing with video-level fake-clip augmentation to encourage learning both types of artifacts.

3.1. Motivation

  • Naïvely trained spatiotemporal networks may rely on one easy artifact type, while spatial- or temporal-focused detectors miss the other, limiting generalization to unseen datasets and manipulations.Some video-level manipulations can remain perceptually coherent, while still containing spatial artifacts.Source: §3.1 · p. 3

3.2. AltFreezing

  • AltFreezing divides network weights into spatial- and temporal-related groups and alternately freezes each group so the network learns to detect both artifact types.For 3D ResNet50, spatial kernels are 1×Kh×Kw and temporal kernels are Kt×1×1; other parameterized layers are treated as both-related.Source: §3.2 · p. 4
  • The iteration ratio Is:It controls emphasis on spatial versus temporal artifacts, with Is set larger than It because spatial artifacts are usually easier to learn.Each cycle freezes spatial weights for Is iterations, then temporal weights for It iterations.Source: §3.2 · p. 4

3.3. Fake Clip Generation

  • Video-level augmentation addresses the limited use of video augmentations by generating fake clips with temporal-only or spatial-only artifacts.Temporal dropout and repeat remove or duplicate random frames, shifting subsequent frames and filling or removing frames to preserve clip length.Source: §3.3 · p. 4
  • Clip-level blending combines corresponding frames from two real clips, retaining temporal coherence while introducing spatial artifacts at the blending boundary.Unlike image-level blending, which processes frames independently and yields temporal incoherence, clip-level blending preserves coherence.Source: §3.3 · p. 5

4. Experiment

Experiments train on FF++ and evaluate video-level AUC across unseen datasets, manipulations, perturbations, and ablations. AltFreezing generalizes across these settings, while ablations examine its training schedule, augmentations, and backbone.

4.2. Generalization to Unseen Datasets

  • A 2% average AUC gain, 94.7% → 96.7%, over the previous video-level state-of-the-art accompanies best results on CDF (89.5%), DFD (98.5%), and DFo (99.3%).The comparison evaluates models trained on FF++ against four unseen datasets; performance on FSh is competitive at 99.4%.Source: Table 1 · p. 5, §4.2 · p. 6

4.4. Robustness to Unseen Perturbations

  • About 3% and 4% AUC improvements at degradation levels 4 and 5, respectively, accompany AltFreezing's higher average performance across four perturbation types and all five levels.The evaluated perturbations are Gaussian blur, block-wise distortion, contrast change, and video compression.Source: Figure 3 · p. 6, §4.4 · p. 6

4.3. Generalization to Unseen Manipulations

  • 98.6% average AUC across the four FF++ manipulation types, including 99.8% on DF and 99.7% on FS, exceeds LipForensics on average, though results are slightly lower on F2F and NT.The evaluation uses a leave-one-out setting, training on three manipulation subsets and testing on the remaining subset.Source: §4.3 · p. 6, Table 3 · p. 6

4.5. Ablation Studies

  • AltFreezing improves in-domain and out-domain detection over vanilla 3D R50, whereas freezing only spatial or only temporal kernels provides no gain and can hurt performance.The alternating-freezing strategy adds no computation or parameters.Source: §4.5 · p. 7, Table 4 · p. 7
  • A 20:1 spatial-to-temporal freezing ratio gives the best generalization among tested ratios, and AltFreezing beats the baseline at every tested ratio.The tested ratios range from 1:1 to 30:1.Source: Table 6 · p. 7, §4.5 · p. 8
  • On synthetic sets containing only temporal incoherence or only spatial artifacts, AltFreezing performs strongly despite not seeing those artifact types during training and outperforms FTCN on temporal artifacts.The authors infer that spatial convolution also matters for temporal-artifact detection.Source: §4.5 · p. 7, Table 5 · p. 7
  • With fake video augmentations, AltFreezing achieves a +3.1% absolute CDF [37] AUC boost, from 86.4% to 89.5%; frame-level augmentation is not suitable to be directly applied to a spatiotemporal network.Source: §4.5 · p. 8
  • A 3D R101 backbone improves performance over 3D R50, while activation maps show AltFreezing focusing on manipulated face regions unlike vanilla 3D R50 and FTCN.Warmer activation-map colors indicate a higher probability of fake.Source: §4.5 · p. 8, Table 8 · p. 8, Figure 4 · p. 8

5. Conclusion and Discussion

AltFreezing separates spatial and temporal weights and alternately freezes them to encourage a model to capture both artifact types. Video-level fake-data augmentations and experiments support the framework’s effectiveness.

5. Conclusion and Discussion

  • AltFreezing divides spatial and temporal weights into two groups and alternately freezes them to encourage capture of both artifact types.Source: §5 · p. 8
  • The proposed video-level fake-data augmentations aim to help the model capture a more general representation of different manipulation types.Source: §5 · p. 8
  • Extensive experiments verify the effectiveness of the AltFreezing strategy and video-level data augmentations.Source: §5 · p. 8

A. More Implementation Details

The implementation detects and aligns faces, samples video clips, and uses a 3D ResNet50 backbone with consecutive spatial and temporal convolutions.

A. More Implementation Details

B. Additional Experiments

Additional ablations examine freezing ratios and fake-clip augmentations. Clip-level blending improves average AUC, while the best reported freezing ratio is 1:5 among ratios below 1.

B. Additional Experiments

C. Evaluation on Real-world Scenarios

The real-world evaluation uses carefully crafted YouTube deepfakes that are difficult for humans to distinguish. AltFreezing is reported to judge real versus fake more accurately than the compared models in these scenarios.

C. Evaluation on Real-world Scenarios

D. Limitations

The method has limited coverage across forgery types and perturbations, with performance dropping on heavily compressed videos.

D. Limitations

  • The method may fail to detect fake videos created with Adobe Photoshop or other realistic image-editing applications.The authors state that it cannot handle every type of face forgery.Source: D. Limitations · p. 11
  • The method is not fully robust to all perturbations.Source: D. Limitations · p. 11
  • On heavily compressed videos, its performance drops, as it does for other works.Source: D. Limitations · p. 11
Loading 2307.08317v1…