PaperlayerView exact revision on arXiv

Source-linked AI summary

ID-Reveal: Identity-aware DeepFake Video Detection

Davide Cozzolino, Andreas Rössler, Justus Thies, Matthias Nießner, Luisa Verdoliva

arXiv:2012.02512v3cs.CV

TL;DR

DeepFake detectors often generalize poorly from known to unseen facial manipulations, especially as new forgery types and low-quality videos emerge. ID-Reveal addresses this gap by learning identity-specific temporal facial behavior from pristine reference videos and real-only training. The approach improves generalization and robustness, with an average accuracy improvement of more than 15% reported for facial reenactment on highly compressed videos.

  • Problem

    Existing detectors are commonly trained for specific forgery methods and therefore struggle to generalize to unseen manipulations such as facial reenactment.

  • Method

    ID-Reveal compares biometric and temporal characteristics from a potentially manipulated test video with pristine reference videos, using metric learning and adversarial training on non-manipulated videos.

  • Results

    More than 15% average accuracy improvement was reported, with improved generalization across manipulation types and robustness to low-quality videos.

  • Takeaways & Limitations

    Identity-specific temporal behavior provides an example-based route to detecting varied facial manipulations without training on a specific forgery method.

  • Takeaways & Limitations

    The decision threshold is set experimentally using a one-time evaluation on 4 real and 4 fake videos from the original DFD.

Abstract

from arXiv · show

A major challenge in DeepFake forgery detection is that state-of-the-art algorithms are mostly trained to detect a specific fake method. As a result, these approaches show poor generalization across different types of facial manipulations, e.g., from face swapping to facial reenactment. To this end, we introduce ID-Reveal, a new approach that learns temporal facial features, specific of how a person moves while talking, by means of metric learning coupled with an adversarial training strategy. The advantage is that we do not need any training data of fakes, but only train on real videos. Moreover, we utilize high-level semantic features, which enables robustness to widespread and disruptive forms of post-processing. We perform a thorough experimental analysis on several publicly available benchmarks. Compared to state of the art, our method improves generalization and is more robust to low-quality videos, that are usually spread over social networks. In particular, we obtain an average improvement of more than 15% in terms of accuracy for facial reenactment on high compressed videos.

1. Introduction

DeepFake detectors often struggle with unseen manipulation methods and low-quality videos. ID-Reveal instead uses identity-specific temporal behavior from real reference videos to detect inconsistencies across manipulation types.

  • Supervised detectors trained on a specific forgery method can drastically lose performance on unseen manipulations, limiting practicality.
  • Facial reenactment changes expressions or lip movements while preserving identity, unlike face-swapping, which changes identity while preserving expression.
  • A DeepFake Detection Challenge winner showed performance drops on unseen manipulations and additional losses on low-quality compressed videos.
  • The method trains only on real videos and combines a facial feature extractor, temporal identity network, and adversarial network for person-specific motion.
  • ID-Reveal detects facial manipulations by comparing a test video’s biometric traits and temporal embedding with reference videos of the target person.
  • More than 15% average accuracy improvement was reported for generalization across manipulation types, including low-quality videos.

2. Related Work

Related work includes supervised, temporal, physiological, few-shot, and identity-based detectors. ID-Reveal differs by focusing on temporal behavior through 3D morphable-model features and adversarial learning.

  • Temporal approaches use LSTM, two-branch, or 3D CNN architectures to exploit artifacts arising across video time.
  • Most supervised methods perform well when training and test manipulations match but degrade dramatically on unseen tampering methods.
  • Few-shot approaches use labeled examples of a new forgery method to separate its embeddings from previously seen manipulations and pristine samples.
  • Physiological-feature methods detect artifacts involving eye blinking, head pose, face warping, heartbeat, and other biological signals.
  • Identity-based methods characterize individuals using biometric traits that generators may reproduce poorly, including facial and head movement patterns.
  • ID-Reveal uses 3D morphable-model facial features and adversarial learning to focus on temporal behavior, improving facial-reenactment detection while retaining face-swap detection.

3. Proposed Method

ID-Reveal detects facial manipulations by comparing identity-specific temporal biometrics, trained only on pristine videos, with test-video behavior. Its metric-learning Temporal ID Network is adversarially paired with a 3DMM Generative Network to emphasize temporal rather than visual cues.

  • ID-Reveal compares a potentially manipulated test sequence with pristine sequences of the same person using identity-specific biometric characteristics.A fixed threshold on minimum embedding-space distances determines whether behavioral properties match.
  • Training uses only non-tampered videos, allowing the approach to leverage a larger corpus than facial-manipulation datasets.The networks are trained on real videos containing many different subjects.
  • The method extracts compact per-frame facial representations with a 3D morphable model before temporal processing.The representation includes shape, expression, and rigid-pose parameters.
  • The Temporal ID Network applies temporal convolutions to 3DMM sequences and produces embedded vectors for metric-based identity comparison.Similarity is evaluated using squared Euclidean distance, followed by a log-loss over identity-based probabilities.
  • The metric-learning loss is less restrictive because one strong same-identity similarity can outweigh weaker similarities to other individuals.The probability considers all similarities relative to a pivot vector, but requires at least one sufficiently dominant coherent pair.
  • The 3DMM Generative Network adversarially generates identity-coherent features with another subject’s expressions to challenge the Temporal ID Network.Its objective combines adversarial and cycle-consistency losses, while the Temporal ID Network is trained to distinguish real identities from generated ones.

4. Results

ID-Reveal is evaluated through ablations and cross-dataset comparisons spanning manipulation types and video quality. It generally improves generalization across unseen manipulations and robustness to compression, especially for facial reenactment and low-quality videos.

  • Experimental design: The evaluation combines ablation studies of loss functions and adversarial training with comparisons against state-of-the-art detectors.Experiments cover different manipulation types and video qualities.
  • Ablation study: The ablation study evaluates accuracy and AUC for facial reenactment and face swapping under high-quality and low-quality compression.The setup includes leave-one-out pristine reference videos and video-level evaluation.
  • Ablation study: 5.5% average AUC improvement over multi-similarity loss and 2.8% over triplet loss are obtained with the proposed loss.Adversarial training adds around 3% AUC and 6% average accuracy for the most challenging facial-reenactment scenario.
  • Generalization analysis: Most supervised methods experience a major drop when tested on facial reenactment after training predominantly on face swapping, often approaching 50% accuracy.ID-Reveal and A&B maintain nearly identical performance across both forgery types.
  • Robustness analysis: ID-Reveal loses only a small amount of accuracy on low-quality videos and outperforms competing methods by a large margin.Across DFDCp and CelebDF, it improves over the best supervised approach by about 16% on low-quality videos, and over A&B by about 14% on high-quality and 13% on low-quality videos.
  • Generalization analysis: Supervised methods perform strongly when training and testing manipulations match but degrade sharply when those manipulations are excluded from training, especially for low-quality videos.Identity-based methods do not change performance based on which manipulation is included in training.

5. Conclusion

ID-Reveal uses low-dimensional 3DMM motion representations and adversarial training to improve DeepFake detection generalization, particularly on low-quality content. Its robustness to environmental variation and compression supports detection across different forgery methods.

  • 5. Conclusion: ID-Reveal leverages reference videos of a target person and adversarial training to detect manipulations using identity-specific motion.The method focuses on a low-dimensional 3DMM representation of facial motion.
  • 5. Conclusion: 3DMM representations contain less information than original 2D images but provide robustness that supports generalization across forgery methods.The representation is described as robust to different environments, lighting situations, and disruptive post-processing such as compression.
  • 5. Conclusion: ID-Reveal improves detection quality, especially on low-quality content, while using a training strategy focused solely on non-manipulated videos.The conclusion reports improved generalization capabilities alongside stronger detection quality.

Appendix

The appendix documents the architectures, comparison methods, additional generalization results, embedding-space separability, and a real web case.

  • Appendix: The appendix details the Temporal ID Network and 3DMM Generative Network architectures.It also describes the compared DeepFake methods.
  • Appendix: Additional appendix sections present results on generalization capability, subject separability in embedding space, and a real web case.The separability analysis uses scatter plots of videos from different subjects.

A. Architectures

The appendix describes temporal and generative network architectures built around dilated temporal convolutions and frame-wise 3DMM processing.

  • A. Architectures: The Temporal ID Network uses eleven layers, Group Normalization, LeakyReLU, and dilated convolutions along the temporal direction.Dilated convolutions increase receptive fields without increasing trainable parameters.
  • A. Architectures: The Temporal ID Network has a receptive field of 51 frames, corresponding to approximately 2 seconds.Its final layer reduces the channel count from 512 to 128.
  • A. Architectures: The 3DMM Generative Network concatenates two 3DMM feature vectors into a 124-channel input vector.The network expands this input to 512 channels before applying three residual blocks.
  • A. Architectures: The 3DMM Generative Network reduces 512 channels to 62 and sums its output with the input 3DMM feature vector.One-dimensional filters make the convolutions operate frame-by-frame.

B. Comparison methods

The comparison covers CNN, ensemble, temporal, and identity-based DeepFake detectors evaluated at video level on manipulated-video benchmarks.

  • B. Comparison methods: The comparison includes MesoNet, Xception, FFD, Efficient-B7, ISPL, and Seferbekov as learned-feature or frame-based methods.These methods use different architectures, attention or ensemble strategies, and augmentation procedures.
  • B. Comparison methods: The evaluations cover HQ and LQ videos with facial reenactment and face swapping, including FF++ comparisons trained on DFDC or VoxCeleb2 depending on method type.Table 5 compares video-level accuracy and AUC on FF++, while Table 4 evaluates frame-based aggregation on DFD.
  • B. Comparison methods: Seferbekov uses an ensemble of seven EfficientNet-B7 networks operating frame-by-frame.Its training includes augmentation such as cut-out operations that remove specific face regions.
  • B. Comparison methods: Temporal baselines include ResNet50 or EfficientNet-B1 frame features followed by an LSTM for video classification.The frame features are extracted from 20 uniformly sampled video frames.
  • B. Comparison methods: The Appearance and Behavior method combines face recognition with a head-movement network that encodes identity-related movement attributes.This provides an identity-based comparison to ID-Reveal.

C. Additional results

Additional experiments show that ID-Reveal generalizes to FOMM facial reenactment videos, despite FaceForensics++ lacking multiple videos per subject for identity-based evaluation.

  • FaceForensics++: The FaceForensics++ analysis evaluates ID-Reveal separately on facial reenactment and face swapping manipulations.Because FF++ does not provide multiple videos of the same subject, identity-based approaches face an evaluation constraint.
  • FOMM evaluation: 160 FOMM fake videos were created from pristine DFD videos using the official First-Order Motion Model code.The created examples are shown in Figure 8, with source videos, target sequences, and generated manipulations.
  • FOMM evaluation: 85.6% accuracy and 0.94 AUC on FOMM videos further demonstrate generalization to a new manipulation type.The evaluation used videos generated by First-Order Motion Model facial reenactment.

D. Robustness to different contexts

ID-Reveal remains robust when reference and test videos differ in environment, lighting, subject distance, and conversational context.

  • Context robustness: ID-Reveal shows robust performance when reference videos and test videos differ in contexts such as kitchens, podiums, outdoor scenes, and meetings.The analysis compares contexts including angry talking, talking against a wall, happy hugging, surprise, and serious meetings.

E. Visualization of the embedded vectors

The embedded temporal patterns are visualized through two-dimensional projections selected to separate actor identities and real from fake videos.

  • LDA visualization: Figure 10 uses 4-second video snippets from two DFD actors and applies LDA to select a 2D projection maximizing the relevant separations.The projection targets separation between the actors’ real videos and between real and fake videos.
  • Context comparison: Figure 9 reports average AUC across 28 actors under differing reference and test contexts for DFD facial reenactment and face swapping.The plotted contexts include kitchen, podium speech, outdoor laughing and talking, angry talking, and serious meetings.

F. A real case on the web

A YouTube case study compares embedding-space distances from real reference footage across real videos, DeepFakes, an imitator, and a DeepFake of the imitator.

  • YouTube case study: The analysis uses three real videos, four DeepFakes, one Nicolas Cage imitator, and one DeepFake applied to the imitator.Distances are computed as minimum pairwise squared Euclidean distances between 4-second snippets from reference and test videos.
  • Distance distributions: Real videos have the lowest distances, whereas DeepFake videos have higher distances and can be detected as fakes.The distributions are computed relative to a real Nicolas Cage reference video.
  • Imitator analysis: The imitator has a much lower distance, while a DeepFake driven by the imitator reduces the distance but remains detectable.The result indicates that imitation can bring test-video distances closer to those of the reference identity without eliminating detection.
Loading 2012.02512v3…