Source-linked AI summary
Two-Stream Neural Networks for Tampered Face Detection
Peng Zhou, Xintong Han, Vlad I. Morariu, Larry S. Davis
TL;DR
Face tampering is difficult to detect because realistic splicing can evade inspection and existing methods often focus on a single evidence source. The paper combines visual-artifact and local-noise streams, evaluates them on a new 2010-image tampering dataset, and reports stronger performance than other methods while identifying small-face and end-to-end-design limitations.
Problem
Existing face-tampering detectors are limited by focusing on particular evidence sources, while available face-tampering datasets may be small or visually distinguishable.
Method
The method combines a GoogLeNet face-classification stream for tampering artifacts with a steganalysis-feature patch triplet stream for local noise and camera characteristics.
Results
The authors report that their two-stream approach outperforms other methods on a challenging face-tampering dataset containing 2010 tampered images.
Takeaways & Limitations
Fusing high-level tampering artifacts with hidden noise-residual features provides the paper’s supported approach to face-tamper detection.
Takeaways & Limitations
The method is less robust for very small tampered faces and requires separate steganalysis-feature extraction rather than an end-to-end design.
Abstract
from arXiv · showhide
We propose a two-stream network for face tampering detection. We train GoogLeNet to detect tampering artifacts in a face classification stream, and train a patch based triplet network to leverage features capturing local noise residuals and camera characteristics as a second stream. In addition, we use two different online face swapping applications to create a new dataset that consists of 2010 tampered images, each of which contains a tampered face. We evaluate the proposed two-stream network on our newly collected dataset. Experimental results demonstrate the effectiveness of our method.
1. Introduction
Face tampering is increasingly accessible and difficult to detect because realistic splicing can evade visual inspection and existing face verification. The paper addresses limitations of single-evidence detectors with a two-stream architecture and a challenging new dataset.
- Motivation: Face swapping through mobile applications or open-source software makes realistic face tampering inexpensive and widely accessible.Splicing may require adjustments to shape, boundaries, illumination, and scale, complicating detection.
- Motivation: Existing approaches can fail when they rely on one tampering cue, such as local noise or CFA patterns.The paper specifically notes failures after careful post-processing and resizing.
- Approach: The proposed two-stream network combines a GoogLeNet face-classification stream with a patch triplet stream based on steganalysis features.The streams target high-level tampering artifacts and low-level noise residual or camera-characteristic evidence.
- Approach: The two streams are fused to recognize tampered faces using complementary visual and local-noise evidence.The face stream classifies tampered versus authentic faces, while the patch stream classifies learned patch features.
- Dataset: The new dataset contains 2010 tampered images generated with two face-swapping applications, plus 1400 authentic images for each subset.The authors retain only tampered images of good quality and use parallel sets with the same source and target faces across algorithms.
- Contributions: The work contributes both a detector that learns tampering artifacts and local noise residual features and a challenging face-tampering dataset.These contributions are presented as the paper’s two main contributions.
2. Related Work
Prior tampering detectors target distinct cues, including noise, JPEG compression, CFA patterns, illumination, and steganalysis features. CNN-based methods broaden learned representations but can degrade under multiple post-processing operations.
- Existing approaches: Tampering-detection methods have targeted local noise, double JPEG compression, CFA patterns, illumination, and steganalysis features.These approaches differ according to the image evidence they model.
- Existing approaches: Local-noise methods identify inconsistencies between global and local noise characteristics but can fail when post-processing alters those characteristics.The cited example uses noise statistics and kurtosis-related properties to expose hidden tampered regions.
- Existing approaches: Double-JPEG methods distinguish regions based on whether compression is applied twice, but depend strongly on the double-JPEG assumption.The methods include aligned and non-aligned double-JPEG settings.
- Existing approaches: CFA-based methods distinguish tampered and authentic regions by estimating differences in camera filter patterns or related prediction errors.Their premise is that regions may reflect different imaging devices or low-level tampering artifacts.
- Existing approaches: Steganalysis features encode local noise-residual information through co-occurrence statistics from filtered pixel neighborhoods.Prior work used these features for Gaussian modeling, anomaly detection, and CFA-related estimation.
- CNN methods: CNN tampering detectors use learned filtering or median-filter layers, but performance can degrade when multiple post-processing techniques are applied.The paper positions its approach as combining learned artifact evidence with refined steganalysis features.
3. Approach
The approach combines a CNN face-classification stream that learns tampering artifacts with a patch triplet stream that models in-camera processing and local noise characteristics. Their scores are combined to produce the final face tampering score.
- Face Classification Stream: The face classification stream fine-tunes a deep CNN to classify each face as tampered or authentic.Its output is the tampering score F(q_i) for face q_i.
- Patch Triplet Stream: The patch triplet stream uses steganalysis features to learn embeddings that distinguish patches from the same image from patches from different images.The learned function f is modeled by a two-layer fully connected neural network, with squared-distance comparisons between embeddings.
- Patch Triplet Stream: 15000 randomly sampled triplets of 128 × 128 patches are used instead of online hard-negative mining because weak supervision can make same-camera patches pseudo negatives.The sampling procedure uses authentic images and includes anchor, positive, and negative patches.
- Patch Triplet Stream: The triplet embedding treats authentic and tampered regions within an image as different classes, enabling an image-specific SVM to predict tampered regions.Patch features are extracted with sliding windows, while non-face patches provide negative samples from the same image.
- Two-stream Score Fusion: The final score for face q combines the outputs of the two streams, with λ balancing their score scales.The patch contribution depends on N_q, the number of patches inside the face.
4. Experiments
Experiments evaluate the two-stream network on a newly collected SwapMe and FaceSwap dataset, compare it with alternative methods, and analyze learned artifacts, fusion behavior, and protocol robustness. The method improves AUC over a steganalysis-feature SVM baseline and outperforms other methods, while remaining less robust for very small tampered faces.
- Dataset: The experiments use SwapMe and FaceSwap images generated by two face-swapping applications, with 1005 tampered images per technique and splits for training and testing.The dataset includes realistic tampering and authentic images, with 705 tampered images used for training and 300 for testing per application.
- Experiment Setup: The face classification stream fine-tunes GoogLeNet on resized face inputs, while the patch triplet stream learns representations from 128×128 patches and steganalysis features.The patch stream uses triplets of patches and produces a learned representation for SVM-based face-level scoring.
- Comparison with Other Methods: 0.875 AUC was achieved by the patch triplet stream, improving from 0.794 for steganalysis features + SVM on the SwapMe test set.CFA-pattern and double-JPEG methods perform poorly under the dataset’s resizing and compression conditions.
- Comparison with Other Methods: The face classification stream learns application-created artifacts including boundary stitching, strong lip edges, and blurring near eyes when glasses are involved.Class Activation Maps visualize these learned tampering artifacts and indicate that the network extracts useful detection features.
- Discussion: Fusing the two streams improves performance over either individual stream because one stream can detect tampering when the other fails.The streams respond to complementary evidence, including unnatural visual edges and different noise-residual distributions.
- Discussion: The method fails on very small tampered faces, such as a 50×50 face, because upsampling loses visual information and 128×128 patches include substantial authentic content.This limits robustness when the tampered face occupies only a small region of the input.
5. Conclusion
The two-stream technique combines low-level patch inconsistencies with explicit tampered-face detection. It outperforms other methods on a challenging dataset, while requiring steganalysis feature extraction.
- The technique combines low-level inconsistencies between image patches with explicit tampered-face detection.
- The evaluation uses a challenging dataset in which multiple post-processing operations are applied to the spliced region.
- The approach outperforms other methods by learning both tampering artifacts and hidden noise residual features.