Source-linked AI summary
Exposing DeepFake Videos By Detecting Face Warping Artifacts
Yuezun Li, Siwei Lyu
TL;DR
DeepFake detection is difficult because synthesized faces are increasingly realistic, yet current pipelines affine-warp limited-resolution faces into source videos and leave distinctive artifacts. The paper simulates those artifacts with simple image processing, trains CNN detectors on them, and reports effective performance across multiple DeepFake datasets and comparisons. It identifies robustness to evolving video compression and more specialized network architectures as future work.
Problem
Realistic DeepFake videos require detectors that can distinguish synthesized faces from real videos without relying solely on DeepFake-generated negative training images.
Method
The method simulates affine face-warping artifacts by aligning, blurring, and inversely warping real faces, then trains CNNs to compare face regions with surrounding context.
Results
The method effectively distinguishes DeepFake videos from real videos across multiple datasets; ResNet50 outperforms Two-stream NN by approximately 16% on both UADFV and DeepfakeTIMIT.
Takeaways & Limitations
Simple image-processing operations can provide negative training data while avoiding the time and computing resources needed to generate DeepFake examples.
Takeaways & Limitations
The authors identify robustness to multiple video-compression settings and more efficient dedicated network architectures as ongoing work.
Abstract
from arXiv · showhide
In this work, we describe a new deep learning based method that can effectively distinguish AI-generated fake videos (referred to as {\em DeepFake} videos hereafter) from real videos. Our method is based on the observations that current DeepFake algorithm can only generate images of limited resolutions, which need to be further warped to match the original faces in the source video. Such transforms leave distinctive artifacts in the resulting DeepFake videos, and we show that they can be effectively captured by convolutional neural networks (CNNs). Compared to previous methods which use a large amount of real and DeepFake generated images to train CNN classifier, our method does not need DeepFake generated images as negative training examples since we target the artifacts in affine face warping as the distinctive feature to distinguish real and fake images. The advantages of our method are two-fold: (1) Such artifacts can be simulated directly using simple image processing operations on a image to make it as negative example. Since training a DeepFake model to generate negative examples is time-consuming and resource-demanding, our method saves a plenty of time and resources in training data collection; (2) Since such artifacts are general existed in DeepFake videos from different sources, our method is more robust compared to others. Our method is evaluated on two sets of DeepFake video datasets for its effectiveness in practice.
1. Introduction
DeepFake videos have become more realistic as machine learning reduces the need for manual editing. The paper detects them by targeting artifacts caused when fixed-size synthesized faces are affine-warped into source videos, training CNNs with simulated negative examples instead of generated DeepFakes.
- Motivation: DeepFake replaces a target individual’s faces with those of a source individual using neural networks trained to map facial expressions.Post-processing can make the resulting videos highly realistic.
- Core observation: Fixed-size synthesized faces must be affine-warped to match the source face, creating resolution inconsistencies with surrounding regions that reveal DeepFakes.The method treats these warping artifacts as the distinctive detection signal.
- Method: The method detects warping artifacts with a dedicated CNN comparing generated face areas with their surrounding regions.Faces are aligned using landmarks, blurred, and warped back to simulate the relevant inconsistency.
- Training strategy: Negative training data are produced with simple image-processing operations, avoiding the time and computing resources required to train DeepFake models.The authors argue that artifacts shared across DeepFake sources also improve robustness.
- Evaluation: The method is evaluated using four CNN architectures and tested on a DeepFake dataset as well as fake videos collected from YouTube.The models are VGG16, ResNet50, ResNet101, and ResNet152.
2. Related works
Prior work spans generative models, resampling forensics, and CNN-based forgery detection. The paper positions its approach as a simpler regional comparison that targets face-warping artifacts rather than estimating an exact resampling operation or relying on behavioral cues.
- AI-based video synthesis: GANs generate realistic images through adversarial training between a generator and discriminator, and related models underpin DeepFake face translation.Coupled GANs are identified as the basis of the DeepFake algorithm.
- AI-based video synthesis: DeepFake videos replace target faces with source faces using a GAN trained to translate between the two facial domains.Later methods add cycle-consistency, temporal information, or spatial cues.
- Resampling detection: Resampling-forensics methods detect transforms or estimate resampling algorithms, but compression and other post-processing steps are difficult to model.They commonly analyze whole images rather than focusing on face-versus-context differences.
- Resampling detection: Regional comparison can use surrounding image areas as relatively artifact-free references while synthesized face regions contain warping artifacts.This provides a simpler detection target than estimating the exact resampling operation.
- GAN-generated image/video detection: Other DeepFake detectors use missing eye blinking or head-pose inconsistency, but behavioral cues can be circumvented or may be weak in frontal faces.The cited eye-blinking approach can be bypassed by including closed-eye images during training.
3. Methods
The method simulates face-warping artifacts from real images, augments their appearance and shape, and trains CNNs on regions containing both faces and context. Its pipeline mirrors the transformations that create artifacts in DeepFake production.
- Artifact target: DeepFake detection targets artifacts created when fixed-size synthesized faces undergo affine transforms to match target poses.The artifacts arise from resolution inconsistency between facial and surrounding regions.
- DeepFake production pipeline: The production pipeline detects faces, extracts landmarks, computes a transform to normalize the face, synthesizes a face, warps it back, and post-processes the composite.Figure 1 shows the synthesized face being warped back with the same transform matrix.
- Negative data generation: Negative examples are generated by aligning faces at multiple scales, applying Gaussian blur, and affine-warping them back to the original size.This simulates varied resolution inconsistencies without running a DeepFake generator.
- Augmentation: Face-shape augmentation changes the retained warped region, including whole-face retention or a landmark-defined convex polygon.Brightness, contrast, distortion, and sharpness are also varied to enlarge training diversity.
- CNN training and inference: CNN inputs are regions of interest containing both the face and surrounding areas, selected using face landmarks and resized to 224 × 224.The networks include VGG16, ResNet50, ResNet101, and ResNet152; inference averages predictions across ten crops.
4. Experiments
The method is evaluated on UADFV and DeepFakeTIMIT using CNNs trained with dynamically simulated negative examples, achieving strong frame-level AUC and outperforming comparison methods.
- Evaluation setup: Training dynamically converts half of each batch’s positive examples into negative examples, diversifying data without pre-generating all negatives.The negative examples follow the simulated face-warping pipeline during training.
- UADFV: On UADFV, VGG16, ResNet50, ResNet101, and ResNet152 achieve frame-level AUCs of 83.3%, 97.4%, 95.4%, and 93.8%, respectively.ResNet models perform about 10% better than VGG16, while ResNet50 performs best among the ResNet variants.
- DeepFakeTIMIT: On DeepFakeTIMIT HQ, the same models achieve AUCs of 57.4%, 93.2%, 86.9%, and 91.2%, respectively.The HQ set uses 128 x 128 synthesis inputs and outputs.
- YouTube examples: On YouTube DeepFake videos, the ResNet50 model detects face-warping artifacts in example clips using frame output scores.A score of 0 corresponds to a frame free of the warping artifacts.
- Comparison with state of the art: ResNet50 outperforms Two-stream NN by approximately 16% and Meso-4 and MesoInception-4 by approximately 17% and 21% on both datasets.The comparison uses AUC on UADFV and DeepFakeTIMIT; the method also shows a notable advance on DeepFakeTIMIT’s HQ set.
5. Conclusion
The paper presents CNN detection of DeepFake-specific face-warping artifacts and reports effectiveness across available DeepFake video datasets. Future work targets robustness to multiple compression settings and more efficient dedicated network structures.
- Conclusion: The method detects resolution-inconsistency artifacts produced when limited-resolution synthesized faces are transformed to match source-video faces.A dedicated deep neural network captures these artifacts to distinguish DeepFake videos from real videos.
- Conclusion: The evaluation across several available DeepFake video datasets demonstrates the method’s effectiveness in practice.The conclusion states the overall empirical scope without specifying a single dataset or metric.
- Future work: Future work will evaluate robustness to multiple video-compression settings and explore dedicated network structures for more efficient detection.The current approach uses predesigned structures such as ResNet and VGG.