Source-linked AI summary
Capsule-Forensics: Using Capsule Networks to Detect Forged Images and Videos
Huy H. Nguyen, Junichi Yamagishi, Isao Echizen
TL;DR
Forged media is increasingly easy to create, while existing detectors often target limited forgery domains. This paper uses capsule networks for broad forged-image and video detection, achieving strong results across replay, face-swapping, reenactment, and computer-generated-image tests.
Problem
Forged images and videos can bypass facial authentication and create fake news, while existing detection methods are generally targeted at particular domains and may become obsolete as new attacks emerge.
Method
The method applies a capsule network to detect replay attacks and fully or partially computer-generated images and videos, processing videos by classifying frames and averaging their posterior probabilities.
Results
Across four major datasets, the method achieved zero HTER for replay attacks, highest accuracy for face swapping, comparable performance for facial reenactment, and best performance for computer-generated images.
Takeaways & Limitations
The experiments support the feasibility of a general detector for a wide range of forged image and video attacks and demonstrate capsule networks beyond computer vision.
Abstract
from arXiv · showhide
Recent advances in media generation techniques have made it easier for attackers to create forged images and videos. State-of-the-art methods enable the real-time creation of a forged version of a single video obtained from a social network. Although numerous methods have been developed for detecting forged images and videos, they are generally targeted at certain domains and quickly become obsolete as new kinds of attacks appear. The method introduced in this paper uses a capsule network to detect various kinds of spoofs, from replay attacks using printed images or recorded videos to computer-generated videos using deep convolutional neural networks. It extends the application of capsule networks beyond their original intention to the solving of inverse graphics problems.
1. INTRODUCTION
Forged media threaten facial authentication and information integrity, while existing detectors often target particular attack types. The paper introduces a capsule-network method intended to cover replay and computer-generated forgeries.
- Forged images and videos can bypass facial authentication and contribute to the rapid spread of manipulated fake news.
- Advanced generation methods have simplified facial forgery creation using short videos, ID photos, or synthesized audio-video mappings.
- Existing countermeasures are often attack-specific, with replay-focused LBP methods and deepfake-focused eye analysis showing different coverage.
- The proposed capsule-network detector targets replay attacks and fully or partially computer-generated images and videos.
2. RELATED WORK
Related work is organized around replay attacks and computer-generated media, while capsule networks provide a proposed alternative for forensics. Dynamic routing made capsule architectures practical and may support detection of complex forgeries.
- Forgery detectors are grouped into replay-attack methods and computer-generated image or video methods according to features and target attacks.
- Replay detection progressed from LBP and local-speed-pattern methods to deep-learning systems using CNN features and classifiers.
- Computer-generated forgery detection includes hand-crafted noise features, CNN implementations, fine-tuned CNNs, and methods for deepfake and reenactment attacks.
- Capsule Networks: Capsule networks were proposed to address CNN limitations in inverse graphics, but became practical after dynamic and expectation-maximization routing algorithms.
- Capsule Networks: Dynamic-routing agreement between capsules could boost detection of complex and nearly flawless forged images and videos.
3. CAPSULE-FORENSICS
Capsule-Forensics processes images or video frames through VGG-19 features, primary capsules, and dynamically routed real/fake output capsules. Its design adds statistical pooling, training noise, an extra squash operation, and cross-entropy loss.
- Overview: Video inputs are split into frames, classified individually, and combined by averaging posterior probabilities; image inputs follow the same remaining pipeline.
- Overview: Faces are detected and scaled to 128 × 128 before part of VGG-19 extracts latent features for the capsule network.
- Capsule Design: The network uses three primary capsules and two output capsules representing real and fake images, with statistical pooling applied to latent features.
- Capsule Design: Dynamic routing connects primary-capsule outputs to real/fake output capsules, and the network contains approximately 2.8 million parameters.
- Capsule Design: Gaussian noise is added to the weight tensor during training, while an additional squash operation is used before iterative routing.
- Capsule Design: Cross-entropy loss replaces the loss used by Sabour et al., with y denoting the ground-truth label and ˆy the predicted label.
- Capsule Design: Equation 3 promotes separation between the two output capsules on each dimension instead of relying only on capsule-output length.
4. EVALUATION
The proposed capsule-network method was evaluated across replay attacks, face swapping, facial reenactment, and computer-generated images. It achieved strong results across these settings, including zero HTER for replay attacks and highest or best performance in several comparisons.
- 4.1. Replay Attack Detection: HTER of zero was achieved by Capsule-Forensics-Noise and the authors’ previous method on the REPLAY-ATTACK dataset.The comparison included eight state-of-the-art detection methods.
- 4.2. Face Swapping Detection: Capsule-Forensics-Noise had the highest accuracy for face-swapping detection at both frame and video levels on the deepfake dataset.The evaluations corresponded to Tables 2 and 3.
- 4.3. Facial Reenactment Detection: On FaceForensics facial-reenactment detection, the proposed method had performance comparable on average to the best-performing state-of-the-art methods.The evaluation used the authors’ processing guidelines and included the results reported in Table 4.
- 4.3. Facial Reenactment Detection: Using only the first ten video frames, the proposed method outperformed Afchar et al.’s MesoNet network at the video level.This comparison was conducted on the FaceForensics dataset.
- 4.4. Fully Computer-Generated Image Detection: The proposed method had the best performance for discriminating computer-generated images from photographic images and achieved perfect accuracy on full-size test images.The comparison used the dataset proposed by Rahmouni et al.
5. CONCLUSION
The conclusion reports that the experiments support a general detection method for a wide range of forged image and video attacks. It also identifies capsule networks and training-phase random noise as useful elements, while leaving adversarial and mixed-attack robustness for future work.
- 5. CONCLUSION: The experiments demonstrated the feasibility of a general detection method effective for a wide range of forged image and video attacks.The evaluation included facial reenactment results reported across three compression levels in Table 4.
- 5. CONCLUSION: The experiments demonstrated that capsule networks can be used in domains beyond computer vision.The paper frames digital forensics as an application beyond the networks’ original computer-vision purpose.
- 5. CONCLUSION: Random noise during training was beneficial in most cases.The conclusion identifies this training design as an empirically useful component of the proposed method.
- 5. CONCLUSION: Future work will evaluate resistance to adversarial machine attacks, improve robustness against mixed attacks, and assess random noise at test time.The stated future directions include raising the issue of mixed-attack robustness in the research community.