Source-linked AI summary
Evading Deepfake-Image Detectors with White- and Black-Box Attacks
Nicholas Carlini, Hany Farid
TL;DR
Highly realistic synthetic images can support fraudulent social-media identities and dis-information, motivating neural forensic classifiers. This paper tests those classifiers with white-box and black-box attacks and finds severe vulnerability, including when classifier details are hidden. The authors argue that studying these restrictive attacks is useful for understanding and improving forensic reliability.
Problem
Synthetic images can support fraudulent social-media profiles and dis-information, creating a need to assess the robustness of neural classifiers used to detect them.
Method
The paper evaluates white-box and black-box adversarial attacks that imperceptibly modify synthetic or real images to subvert image-forensic classifiers.
Results
White-box attacks reduced AUC from 0.95 to below 0.1, while black-box attacks reduced ROC performance to below 0.22.
Takeaways & Limitations
These results reveal significant vulnerabilities in certain image-forensic classifiers and show that black-box attacks remain effective without direct access to classifier parameters.
Takeaways & Limitations
The studied attacks focus on restrictive, low-distortion perturbations, although the authors note that real adversaries may have a larger action space.
Abstract
from arXiv · showhide
It is now possible to synthesize highly realistic images of people who don't exist. Such content has, for example, been implicated in the creation of fraudulent social-media profiles responsible for dis-information campaigns. Significant efforts are, therefore, being deployed to detect synthetically-generated content. One popular forensic approach trains a neural network to distinguish real from synthetic content. We show that such forensic classifiers are vulnerable to a range of attacks that reduce the classifier to near-0% accuracy. We develop five attack case studies on a state-of-the-art classifier that achieves an area under the ROC curve (AUC) of 0.95 on almost all existing image generators, when only trained on one generator. With full access to the classifier, we can flip the lowest bit of each pixel in an image to reduce the classifier's AUC to 0.0005; perturb 1% of the image area to reduce the classifier's AUC to 0.08; or add a single noise pattern in the synthesizer's latent space to reduce the classifier's AUC to 0.17. We also develop a black-box attack that, with no access to the target classifier, reduces the AUC to 0.22. These attacks reveal significant vulnerabilities of certain image-forensic classifiers.
1. Introduction
Synthetic images and fake social-media identities create risks for dis-information, increasing the need for reliable forensic detection. This paper shows that neural forensic classifiers can be attacked so that imperceptibly modified images are classified incorrectly.
- Motivation: A fictional congressional candidate used a StyleGAN2-generated profile image to create a convincing social-media identity.The account included a website, candidate tagline, and Twitter verification.
- Motivation: Fake accounts with synthesized profile photographs have been associated with dis-information campaigns by hackers, trolls, and fraudsters.The paper connects these campaigns to threats against democratic institutions, civil society, and economic security.
- Detection: Forensic techniques detect synthetic media using either semantically meaningful features or pixel-level artifacts from the synthesis process.The paper distinguishes high-level cues such as eye blinks and head pose from low-level pixel artifacts.
- Detection: Prior work reported classifiers that generalize across unseen image generators and remain robust to JPEG compression, blurring, and resizing.One classifier trained on ProGAN detected images from nearly a dozen previously unseen architectures, datasets, and training methods.
- Contribution: The paper studies white-box and black-box attacks that modify images imperceptibly to change forensic classifiers’ real-or-fake decisions.The contribution frames forensic classifiers as lacking adversarial robustness.
- Results: 0.95 to below 0.1 AUC was achieved by white-box attacks, while black-box attacks reduced ROC performance to below 0.22.A randomly guessing classifier has an AUC of 0.5 in the stated comparison.
2. Background & Related Work
The paper reviews how synthetic images are generated and detected, then places its attacks within adversarial machine learning. It examines multiple forensic classifiers and uses a simpler classifier to generate adversarial examples for attacking others.
- Synthetic Image Generation: GANs generate images by training a generator and discriminator against the distribution of a dataset.The generator synthesizes images consistent with the training data, while the discriminator evaluates whether they belong to that distribution.
- Detection Model: An image generator maps latent vectors from Z to color images in X, while a forensic classifier maps images to real-valued fake-likelihood scores.Larger classifier outputs indicate a higher likelihood that an image is synthetic.
- Forensic Classifiers: The study evaluates Wang et al.’s and Frank et al.’s learning-based forensic classifiers to test whether the findings extend beyond one detector.Most experiments focus on Wang et al.’s classifier.
- Forensic Classifiers: Wang et al.’s classifier uses ImageNet-pretrained ResNet-50 and training data split evenly between real images and ProGAN-synthesized images.The training set contains 720,000 images, with 4,000 validation images, and includes blur and JPEG augmentation.
- Forensic Classifiers: Frank et al.’s classifier is motivated by a spatial-frequency artifact associated with image upsampling in GAN synthesis.The cited passage describes this artifact as common across GAN-synthesized images.
- Forensic Classifiers: The authors’ auxiliary classifier is trained on 1,000,000 ProGAN images and is used to create adversarial examples for attacking other classifiers.Its simpler pipeline has an error rate roughly three times higher than Wang et al.’s classifier.
- Adversarial Machine Learning: Adversarial attacks add perturbations that cause a classifier to misclassify an input, commonly using gradient-descent optimization.For binary real-or-fake classification, the attack can minimize the classifier score when targeting the real class.
- Adversarial Machine Learning: The paper also motivates strengthening forensic classifiers because attacks and counter-measures help assess their reliability.Concurrent work addressed strengthening forensic classifiers.
3. White-Box Attack
The paper evaluates white-box attacks that directly perturb images or generator latents, showing that forensic classifiers can be reduced from strong detection to near-chance or reversed decisions. These attacks include image-specific, universal-patch, and latent-space manipulations, with substantial effects under very small or shared perturbations.
- Attack setting: White-box attacks assume complete classifier access, enabling gradients to construct perturbations that make fake images appear real or real images appear fake.The study applies three established image attacks and introduces a latent-space attack on the generator.
- Distortion-minimizing attack: An ℓ2 distortion of 0.02 reduces true positive rate to chance at a fixed 5% false positive rate, while 0.1 reduces it to 0.1%.For comparison, an ImageNet classifier of the same architecture and image size requires over 16 times more distortion to reach chance performance.
- Distortion-minimizing attack: Changing 2% of pixels causes 71.3% of fake images to be misclassified as real, rising to 89.7% at 4% and nearly all below 11%.The reverse attack requires just under 7% of pixels for 50% of real images to be classified as fake, versus 1% for fake images.
- Loss-maximizing attack: Flipping the lowest-order bit of 40% of pixels reduces AUC from 0.966 to 0.27 for uncompressed images.The figure compares ROC trade-offs across modified-pixel fractions and JPEG versus PNG storage.
- Universal attacks: A universal 1% patch reduces AUC from 0.966 to 0.085, while a universal latent-space attribute reduces AUC from 0.99 to 0.17.The patch is shared across fake images; latent-space adversarial images remain nearly indistinguishable and photorealistic.
4. Black-Box Attack
The paper tests whether attacks can evade forensic classifiers when the target model’s parameters are hidden. By exploiting adversarial-example transferability, a separately trained source classifier reduces the target’s AUC below chance.
- Black-box attacks assume the adversary knows the classifier type and defense strategy but not the target model’s exact details.
- The attack transfers adversarial examples from a source classifier to a target classifier trained for the same real-versus-fake task.Transferability is attributed to sufficiently similar decision boundaries across different classifiers.
- The authors train a ResNet-18 source classifier on ProGAN and real images, differing from the target’s ResNet-50 architecture, pretraining, and training setup.The source classifier uses one million ProGAN-generated images and one million corresponding real images.
- 0.22 target AUC: transferring source-generated adversarial examples reduces Wang et al.’s classifier from 0.96 AUC to 0.22.The source classifier’s AUC falls from 0.85 to 0.003 under the same attack.
- Although weakest among the presented attacks, the black-box attack drives the forensic classifier below chance and is described as the most realistic threat model.
5. Discussion
The discussion frames forensic detection as an adversarial, time-sensitive problem in which low-distortion attacks expose detector sensitivity and potential countermeasures remain important.
- Over ten percentage points of true-positive-rate loss can result from resizing, rescaling, cropping, or recompression alone.
- The study’s low-distortion attacks impose stricter constraints than realistic adversaries may face, making their success a stringent vulnerability test.
- On internet timescales, a tweet’s minute-scale half-life gives the forger an inherent advantage during the first few hours.
- The forensic classifier requires perturbations roughly 10 times smaller than ImageNet classifiers, with regular structure aligned to the 8×8 JPEG lattice.
- Adversarial-training and randomized-smoothing defenses are identified as among the most effective defenses studied for standard image classification.
6. Conclusions
The conclusions argue that neural-network forensic classifiers are vulnerable to adversarial manipulation, including attacks that do not require access to classifier parameters.
- Forensic classifiers need adversarial models that extend beyond recompression, resizing, blurring, and white-noise laundering attacks.
- The classifiers’ power and flexibility can also be manipulated to create adversarial images that subvert detection.
- Black-box attacks remain effective when classifier details are unknown, making withheld model information an insufficient protection.
- The attacks were demonstrated on two previously published forensic classifiers and one classifier created by the authors.
- If deployed classifiers can be defeated, the false sense of security they provide may be worse than having no classifier at all.
- Even vulnerable classifiers may increase the difficulty and time required to create convincing fakes.
Acknowldgements
The authors acknowledge contributors who assisted with detector runs, discussions, and manuscript feedback, and identify DARPA funding for the research.
- The authors thank Joel Frank, Andrew Owens, Alexei Efros, and Sheng Yu Wang for discussions and assistance running detectors.
- David Berthelot, Andreas Terzis, and Carey Radebaugh provided feedback on earlier drafts.
- The research was funded by the Defense Advanced Research Projects Agency under grant FA8750-16-C-0166.