Source-linked AI summary
Multi-task Learning For Detecting and Segmenting Manipulated Facial Images and Videos
Huy H. Nguyen, Fuming Fang, Junichi Yamagishi, Isao Echizen
TL;DR
Manipulated facial images and videos require both authenticity detection and localization of altered regions. The paper addresses both tasks with a semi-supervised Y-shaped autoencoder that shares information across classification, segmentation, and reconstruction. Experiments report effectiveness on seen and unseen attacks, including improved adaptation after fine-tuning with limited data.
Problem
Detecting whether facial media is manipulated and locating its altered regions are important digital-media-forensics tasks involving attacks such as removal, copy-move, and splicing.
Method
A semi-supervised multi-task Y-shaped autoencoder uses encoded features for classification, one decoder branch for segmentation, and another for reconstructing inputs.
Results
The network was effective for classification and segmentation, handled mismatch conditions for seen attacks, and achieved around 25% higher classification and 8% higher segmentation accuracy after fine-tuning on a small amount of data.
Takeaways & Limitations
Segmentation information can provide an important clue alongside classification for judging the authenticity of images and videos involving unseen attacks.
Takeaways & Limitations
The method assumes equal weighting for classification, segmentation, and reconstruction because the authors consider classification and segmentation equally important and reconstruction important for segmentation.
Abstract
from arXiv · showhide
Detecting manipulated images and videos is an important topic in digital media forensics. Most detection methods use binary classification to determine the probability of a query being manipulated. Another important topic is locating manipulated regions (i.e., performing segmentation), which are mostly created by three commonly used attacks: removal, copy-move, and splicing. We have designed a convolutional neural network that uses the multi-task learning approach to simultaneously detect manipulated images and videos and locate the manipulated regions for each query. Information gained by performing one task is shared with the other task and thereby enhance the performance of both tasks. A semi-supervised learning approach is used to improve the network's generability. The network includes an encoder and a Y-shaped decoder. Activation of the encoded features is used for the binary classification. The output of one branch of the decoder is used for segmenting the manipulated regions while that of the other branch is used for reconstructing the input, which helps improve overall performance. Experiments using the FaceForensics and FaceForensics++ databases demonstrated the network's effectiveness against facial reenactment attacks and face swapping attacks as well as its ability to deal with the mismatch condition for previously seen attacks. Moreover, fine-tuning using just a small amount of data enables the network to deal with unseen attacks.
1. Introduction
Deepfakes create a societal threat, while manipulated-region shapes can provide clues about the attack type. The paper proposes a semi-supervised multi-task network that detects facial manipulations and segments their regions while reconstructing inputs.
- Deepfake videos pose a major societal threat because attackers can create computer-generated spoofing material.
- Manipulated facial-region masks may reveal the type of attack, including removal, copy-move, and splicing.
- Most existing forensic segmentation methods process full-scale images or repeatedly classify sliding windows to locate manipulated regions.
- The proposed semi-supervised autoencoder uses an encoder and Y-shaped decoder for simultaneous classification, segmentation, and input reconstruction.
- Information shared among classification, segmentation, and reconstruction tasks is intended to improve the network’s overall performance.
2. Related Work
Prior work developed methods for generating convincing manipulated videos, detecting them from frames, and locating manipulated regions. These approaches include handcrafted and deep learning detectors, transfer learning, autoencoder adaptation, semantic segmentation, and sliding-window classification.
- 2.1. Generating Manipulated Videos: Facial reenactment and image-based methods can generate spoof videos, with speech synthesis or voice conversion enabling synchronized audiovisual spoofs.
- 2.2. Detecting Manipulated Images and Videos: A typical detection strategy treats videos as sequences of image frames and processes the frames as image inputs.
- 2.2. Detecting Manipulated Images and Videos: Prior detectors use handcrafted noise features, CNN feature extraction, fine-tuning, transfer learning, constrained layers, statistical pooling, two-stream networks, and lightweight architectures.
- 2.2. Detecting Manipulated Images and Videos: Transferability benchmarks and autoencoder-like architectures have been used to improve adaptation to unseen attacks.
- 2.3. Locating Manipulated Regions in Images: Manipulated regions can be located through full-image segmentation, semantic segmentation, boundary boxes, or repeated sliding-window classification.
3. Proposed Method
The proposed method is a semi-supervised multi-task autoencoder that detects manipulated facial inputs while segmenting their manipulated regions. Its encoder and Y-shaped decoder share information across classification, segmentation, and reconstruction.
- The network outputs both spoofing probabilities and segmentation maps for each frame, treating videos as sets of frames and resizing cropped faces to 256 × 256 pixels.
- The autoencoder uses encoded-feature activations for classification, while separate decoder branches perform segmentation and input reconstruction.
- Partitioned latent features and the Y-shaped decoder share information among classification, segmentation, and reconstruction tasks through three losses.The losses are activation, segmentation, and reconstruction losses.
- The selection block passes only the class-specific half of the latent features to both decoders, zeroing the off-class half before decoding.This forces the segmentation and reconstruction branches to learn decoding schemes from the selected latent representation.
- Segmentation uses cross-entropy against ground-truth masks, while reconstruction uses L2 distance between reconstructed and original images.
- The total objective is a weighted sum of activation, segmentation, and reconstruction losses, with all three weights set to 1.The equal weighting reflects the stated importance of classification and segmentation and the role of reconstruction in segmentation.
4. Experiments
Experiments evaluated the multi-task autoencoder on matched and mismatched seen attacks, unseen attacks, and small-data fine-tuning conditions using FaceForensics and FaceForensics++. Results showed strong segmentation, robustness to mismatch, and improved transfer after fine-tuning.
- Datasets and settings: Each dataset was split into training, validation, and testing videos, with segmentation masks and three H.264 compression levels available.The experiments used only light compression, with quantization = 23, and extracted fixed numbers of frames from each video.
- Datasets and settings: The experiments used FaceForensics and FaceForensics++, with Face2Face, Deepfake, and FaceSwap attacks represented across training and testing conditions.FaceForensics included matched and mismatched Face2Face tests; FaceForensics++ added Deepfake and FaceSwap tests.
- Seen attacks: Deeper networks substantially outperformed shallower networks for classification, while the No Recon and Proposed New methods achieved higher segmentation accuracy than Proposed Old.Among deeper networks, classification differences were not substantial; the newer weighting settings improved segmentation.
- Seen attacks: All methods degraded slightly under mismatch conditions, but FT Res and Proposed New showed the best adaptation ability; reconstruction also produced the highest segmentation score.The results associate better adaptation with residual images and, for Proposed New, the reconstruction branch.
- Unseen attacks: Unseen attacks substantially reduced classification accuracy and increased EERs, while segmentation accuracy remained high, especially for FaceSwap.Deeper methods produced nearly random classification results in unseen tests, but segmentation could provide an additional authenticity clue.
- Fine-tuning: Fine-tuning with only ten frames per video increased classification accuracy by around 25% and segmentation accuracy by around 8%, except for Proposed Old segmentation.The fine-tuning experiment used 100 videos per class for training and 40 per class for evaluation.
5. Conclusion
The proposed Y-shaped autoencoder effectively performs classification and segmentation without sliding windows, while task sharing particularly improves mismatch-condition performance. It also adapts to unseen attacks using few fine-tuning samples.
- The network performs classification and segmentation without using a sliding window.
- Information sharing among classification, segmentation, and reconstruction improves overall performance, especially for seen attacks under mismatch conditions.
- Fine-tuning with only a few samples enables adaptation to unseen attacks.