Source-linked AI summary
Adversarial Deepfakes: Evaluating Vulnerability of Deepfake Detectors to Adversarial Examples
Shehzeen Hussain, Paarth Neekhara, Malhar Jere, Farinaz Koushanfar, Julian McAuley
TL;DR
DNN-based Deepfake detectors are designed to distinguish generated videos from real ones, yet their vulnerability to adversarial examples has been insufficiently addressed. The paper modifies fake videos frame by frame and evaluates white-box and black-box attacks, including variants robust to compression. These attacks can cause fake videos to be classified as real, including robust white-box success rates of 98.07% for XceptionNet and 99.83% for MesoNet in compressed format.
Problem
DNN-based Deepfake detectors face an insufficiently addressed vulnerability to adversarial examples that can make fake videos classify as real.
Method
The paper generates adversarial examples for each frame of fake videos from multiple synthesis methods and combines them into adversarially modified videos for white-box and black-box attacks.
Results
Robust white-box attacks achieved average compressed-video success rates of 98.07% for XceptionNet and 99.83% for MesoNet.
Takeaways & Limitations
Deepfake detectors should be evaluated against adaptive adversaries with different attack capabilities and developed toward provable robustness.
Takeaways & Limitations
Supervised detectors trained on existing Deepfake-generation methods cannot be reliably secure against novel methods unseen during training.
Abstract
from arXiv · showhide
Recent advances in video manipulation techniques have made the generation of fake videos more accessible than ever before. Manipulated videos can fuel disinformation and reduce trust in media. Therefore detection of fake videos has garnered immense interest in academia and industry. Recently developed Deepfake detection methods rely on deep neural networks (DNNs) to distinguish AI-generated fake videos from real videos. In this work, we demonstrate that it is possible to bypass such detectors by adversarially modifying fake videos synthesized using existing Deepfake generation methods. We further demonstrate that our adversarial perturbations are robust to image and video compression codecs, making them a real-world threat. We present pipelines in both white-box and black-box attack scenarios that can fool DNN based Deepfake detectors into classifying fake videos as real.
1. Introduction
Deepfake generation has become accessible enough to support both benign creative uses and malicious disinformation, while DNN-based detectors remain vulnerable to adversarially modified fakes. The paper evaluates attacks that make fake videos appear real, including under compression and black-box access.
- Deepfakes modify a subject’s face into a target face to simulate realistic footage of events that never occurred.
- Fake videos can support movies and storytelling but can also spread disinformation, harass individuals, or defame public figures.
- DNN-based detectors typically track and crop faces before classifying video content as real or fake with CNNs.
- Adversarial examples can fool neural detectors into classifying fake videos as real, exposing a vulnerability largely ignored in existing Deepfake-detection work.
- The proposed pipelines perturb each frame of a fake video, combine the modified frames, and produce videos that bypass detectors in white-box and black-box settings.
- The attacks remain effective after image and video compression, which matters because videos shared on social media are usually compressed.
2. Background
The paper situates Deepfake detection within CNN-based multimedia forensics and evaluates adversarial attacks against detectors trained to identify facial-manipulation artifacts. It uses multiple synthesis methods and datasets to test this vulnerability.
- 2.1. Generating Manipulated Videos: The experiments generate adversarial examples for FaceSwap, Face2Face, DeepFakes, and NeuralTextures videos.
- 2.1. Generating Manipulated Videos: FaceForensics++ provides curated manipulated videos for these synthesis methods, while DFDC is used to evaluate attacks against sequence-based detectors.
- 2. Background: Traditional multimedia forensics relied on hand-engineered features or prior knowledge of natural-image statistics and physical properties.
- 2. Background: Recent CNN-based detectors extract visual features from video frames to identify facial tampering and artifacts from Deepfake-generation methods.
- 2. Background: Several leading detectors formulate Deepfake detection as per-frame binary classification, including XceptionNet, which was reported to perform well on uncompressed and compressed videos.
- 2. Background: Adversarial examples intentionally alter machine-learning inputs so models make mistakes, often using gradients to change outputs while moving samples toward low-density regions.
3. Methodology
The attack targets frame- and sequence-based Deepfake detectors by crafting adversarial face crops that make manipulated videos classify as Real, including under compression-aware transformations.
- Victim Models: Deepfake Detectors: The evaluated victim models include frame-level CNN classifiers and a sequence-based 3-D EfficientNet detector operating on tracked face crops.Frame-level models classify each cropped face independently; the sequence model exploits temporal dependencies across face-crop sequences.
- Threat Model: The perturbation is constrained with the L∞ metric and optimized using iterative gradient-sign updates to keep modifications quasi-imperceptible and computationally efficient.Iterations stop after success or a maximum iteration count.
- Attack Pipeline: For each fake-video frame, the pipeline perturbs the cropped face, reinserts it into the original face bounding box, and repeats this process across the video.The resulting adversarial frames are combined into an adversarially modified fake video.
- White-box Attack: The white-box attack assumes complete access to the detector, including face extraction, classifier architecture, and parameters.It optimizes a loss whose minimization increases the classifier’s pre-softmax score for the target label Real.
- White-box Attack: Uncompressed-frame attacks achieve an average attack success rate of 99.05%, but their perturbations are not robust to MJPEG compression.This limitation motivates the compression-robust approach described next.
- Robust White-box Attack: Robust white-box attacks optimize expected target-class probability over transformations including blur, noise, translation, and resizing.The loss is estimated from sampled transformations, enabling gradients with respect to the input.
- Black-box Attack: In the black-box setting, the attacker queries Real/Fake probabilities and estimates gradients because classifier architecture and parameters are unavailable.The attacker still assumes knowledge of the detection pipeline structure and face-tracking model.
4. Experiments
The experiments evaluate white-box and black-box attacks against XceptionNet and MesoNet on FaceForensics++ videos, including robustness to compression. Robust transformation-based attacks preserve high success rates on compressed videos, whereas basic attacks degrade substantially after compression.
- Setup: The evaluation uses pre-trained XceptionNet and MesoNet victim models on FaceForensics++ HQ test videos generated by four manipulation techniques.The dataset contains 70 videos and 29,764 frames per manipulation technique.
- Setup: Adversarial videos are evaluated in Raw, MJPEG, and H.264 formats, with success rate measuring frames classified as Real.Primary evaluation uses Raw and MJPEG; H.264 compression levels are studied for the robust white-box attack.
- White-box Setting: 99.85% and 98.15% average success rates are achieved against XceptionNet and MesoNet respectively in Raw format by the white-box attack.With MJPEG compression, success rates fall to 58.46% and 92.72% for XceptionNet and MesoNet respectively.
- White-box Setting: 98.07% and 99.83% compressed-video success rates are achieved against XceptionNet and MesoNet respectively by the robust white-box attack.At equal L∞ distortion of 0.008, robust white-box success rate is 84.96% versus 74.69% for simple white-box attacks across all datasets.
- Black-box Setting: 97.04% and 86.70% Raw-format success rates are achieved against XceptionNet and MesoNet respectively by the non-robust black-box attack.The average query count is 985 per frame, and success drops significantly after compression.
- Black-box Setting: Robust black-box attacks improve compressed-video success for XceptionNet, but the authors observe no significant improvement for MesoNet despite a higher overall success rate.The robust black-box attack requires an average of 2153 network queries per frame.
5. Evaluation on Sequence Based Detector
The attacks are also evaluated against a 3D CNN that classifies seven-frame face sequences. Robust white-box attacks completely fool the detector after compression, while robust black-box attacks improve compressed-video success over naive black-box attacks.
- Detector and Dataset: The sequence-based detector applies 3D convolution to seven consecutive face-crops and is evaluated on 200 Fake videos from the DFDC public validation set.The evaluated checkpoint was trained on the DFDC training set and released by the NTech-Lab team.
- White-box Attacks: Both white-box and robust white-box attacks achieve 100% success on uncompressed videos.The attacks back-propagate loss through the entire sequence model to obtain gradients with respect to the input frames.
- White-box Attacks: The robust white-box attack completely fools the sequence detector on compressed videos, outperforming the standard white-box attack.Compared with frame-by-frame detectors, fooling this sequence model requires larger perturbations in both white-box settings.
- Black-box Attacks: 51.02% versus 24.43% SR-C is obtained by robust versus naive black-box attacks on compressed videos.The robust black-box attack improves compression robustness, although black-box success falls after compression.
6. Discussion and Conclusion
Deepfake detectors must be evaluated against adaptive adversaries because current state-of-the-art methods can be bypassed with complete or partial detector knowledge. Adversarial-example robustness alone is insufficient, motivating provably robust detectors and adaptive training.
- Current state-of-the-art Deepfake detectors can be easily bypassed when attackers have complete or partial knowledge of the detector.
- Adaptive-adversary evaluation should account for attackers who know the defense and intentionally attempt to fool it.
- Robustness to adversarial examples is necessary but not sufficient for securing DNN-based Deepfake detectors.
- Attackers may also modify a Deepfake generator’s training objective to include a loss term based on the detector score.
- Supervised detectors trained on existing Deepfake methods cannot be reliably secure against novel generation methods absent from training.
- The paper recommends adversarial training in which detectors continually improve against newly generated Deepfakes that bypass them.