Source-linked AI summary
Use of a Capsule Network to Detect Fake Images and Videos
Huy H. Nguyen, Junichi Yamagishi, Isao Echizen
TL;DR
Malicious users can create and distribute manipulated images and videos, while many existing detectors are narrow and ineffective against other domains or new attacks. This paper presents a lightweight capsule-network detector with detailed forensic analysis, reporting equivalent or better performance than state-of-the-art methods on tested tasks while using fewer parameters.
Problem
Manipulated images and videos can be used to bypass facial authentication and violate privacy and social trust, while many existing detectors target specific domains and attacks.
Method
The paper develops a capsule-network forensics pipeline for images and videos, using input-dependent preprocessing, capsule analysis, visualization, and score aggregation across patches or frames.
Results
The method achieved performance equivalent to or better than state-of-the-art methods on the tested digital-forensics tasks while using fewer parameters.
Takeaways & Limitations
Capsule-Forensics provides a lower-parameter approach for detecting computer-manipulated or generated images and videos, including presentation attacks, within the tested tasks.
Abstract
from arXiv · showhide
The revolution in computer hardware, especially in graphics processing units and tensor processing units, has enabled significant advances in computer graphics and artificial intelligence algorithms. In addition to their many beneficial applications in daily life and business, computer-generated/manipulated images and videos can be used for malicious purposes that violate security systems, privacy, and social trust. The deepfake phenomenon and its variations enable a normal user to use his or her personal computer to easily create fake videos of anybody from a short real online video. Several countermeasures have been introduced to deal with attacks using such videos. However, most of them are targeted at certain domains and are ineffective when applied to other domains or new attacks. In this paper, we introduce a capsule network that can detect various kinds of attacks, from presentation attacks using printed images and replayed videos to attacks using fake videos created using deep learning. It uses many fewer parameters than traditional convolutional neural networks with similar performance. Moreover, we explain, for the first time ever in the literature, the theory behind the application of capsule networks to the forensics problem through detailed analysis and visualization.
1. INTRODUCTION
Advances in computer graphics and deep learning have made realistic manipulated media easier to create and distribute, increasing risks to privacy, security, and social trust. The paper proposes Capsule-Forensics as a lightweight, general-purpose detector and analyzes its behavior across attack types.
- Motivation: Computer graphics, social networks, and advanced hardware enable malicious users to create and widely publish fake images and videos.Such media can also be used to bypass facial authentication.
- Motivation: Deepfakes can be created from short videos or even a single identity photo, enabling users to impersonate other people.Speech synthesis and speech-to-lip-motion mapping further support fully synthesized audio-video impersonation.
- Prior countermeasures: Existing countermeasures span presentation attacks and computer-generated or manipulated media, but CNN-based detection has become central as attack realism improves.Earlier presentation-attack detectors relied on handcrafted features, whereas CNNs provide automatic feature extraction.
- Contribution: The paper aims to build a lightweight, general-purpose detector that handles different attacks with reasonable performance relative to task-specific detectors.Its architecture uses a newer CNN type with fewer parameters than many traditional CNN detectors.
- Contribution: Capsule-Forensics is analyzed through visualizations of capsule activations and is enhanced with architectural modifications and two regularizations.The analysis examines the novelty and behavior of the proposed capsule network across several attack types.
2. RELATED WORK
Related work covers increasingly realistic face manipulation, presentation-attack detection, manipulated-media forensics, and the emergence of capsule networks. These strands motivate a detector designed for varied attacks and an analysis of capsule behavior in digital media forensics.
- 2.1. Face Manipulation: Face manipulation methods can transfer expressions, alter lip movements, synthesize speech, animate portraits, and generate realistic videos from few images.Several approaches substantially reduce the input needed to create convincing audiovisual or facial manipulations.
- 2.1. Face Manipulation: Deep-learning face-swapping tools have become widely available, allowing ordinary users to create impersonation and pornographic videos.The related work also notes applications that generate realistic nude images from clothed photographs.
- 2.2. Presentation Attack Detection: Presentation attack detection originally used handcrafted features such as local binary patterns, while later methods adopted pretrained or customized CNNs.These methods target attacks intended to interfere with biometric capture-system operation.
- 2.3. Computer-Generated Image/Video Detection: Detector generalization to unseen attacks is difficult because attack techniques and their variations are increasing rapidly.Related directions include adversarial training, domain adaptation, semi-supervised learning, and multi-task learning.
- 2.4. Capsule Networks: Capsule networks became more practical after dynamic and expectation-maximization routing algorithms addressed earlier limitations.Their capsule agreements encode hierarchical relationships and pose information while preserving more information than CNNs with a fraction of the data.
- 2.4. Capsule Networks: Prior capsule-network applications across multiple domains motivated Capsule-Forensics for detecting modified or generated media and analyzing intermediate activations.The paper positions this work as a continuation of pioneering capsule-network research in digital media forensics.
3. CAPSULE-FORENSICS
Capsule-Forensics combines input-dependent preprocessing, a VGG-19 feature extractor, capsule processing, and score averaging to detect manipulated images and videos. Its capsule architecture uses statistical pooling and dynamic routing to aggregate evidence from primary capsules into real or fake outputs.
- 3.1. Overview: Input-dependent preprocessing separates video frames, divides generated images into patches, or crops faces before classification.The method uses 300 × 300 images in its described setup, balancing information and computational cost.
- 3.1. Overview: A truncated pre-trained VGG-19 network extracts features before the capsule network, helping guide training and reduce overfitting.The extractor runs through the third max-pooling layer rather than using the full network.
- 3.1. Overview: The model averages patch scores for generated images and frame scores for videos to produce the final output.This post-processing step mirrors the corresponding preprocessing choice.
- 3.2. Detailed Architecture: Each primary capsule combines 2D convolution, statistical pooling of filter means and variances, and 1D convolution, making the architecture adaptable to different input sizes.The statistical layer produces an output suitable for 1D convolution and avoids redesigning the architecture for different image sizes.
- 3.3. Dynamic Routing Algorithm: Dynamic routing calculates agreement among primary-capsule features and routes the results to real and fake output capsules for binary classification.The output probabilities are based on output-capsule activations, while routing uses transformed, squashed, and dropout-regularized capsule outputs.
- 3.4. How Capsule-Forensics Works: Primary capsules often attend to facial regions such as the eyes, nose, mouth, and contours, while routing aggregates their partial evidence.The capsules can miss regions or fail to detect manipulation individually, but their agreement produces a more consolidated focus.
4. EVALUATION
The evaluation tests Capsule-Forensics across manipulated media, fully computer-generated images, and presentation attacks, while analyzing capsule activations and comparing variants with a large CNN baseline. Improvements such as larger inputs, dropout, more primary capsules, and frame aggregation generally strengthened performance, with results that were competitive across tasks and used fewer parameters.
- Capsule Analysis: Activation visualizations compare three capsules with the whole network across deepfake, Face2Face, and FaceSwap inputs, including manipulated regions and activation differences.Additional figures examine real inputs, capsule agreement, and a case where two capsules compensated for one capsule's failure.
- Detecting Computer-Manipulated Images/Videos: The evaluation covers FaceForensics++ manipulations, including deepfake, Face2Face, and FaceSwap, across three compression levels.The database was divided into training, validation, and test sets.
- Detecting Computer-Manipulated Images/Videos: Larger inputs, dropout, additional primary capsules, and frame aggregation improved Capsule-Forensics performance, especially for multi-class and video classification.Combining random noise and dropout produced performance nearly matching XceptionNet with five times fewer parameters.
- Detecting Computer-Manipulated Images/Videos: Face2Face was the most difficult manipulation for both networks, while XceptionNet performed better on deepfake detection with 94.81% accuracy.Capsule-Forensics showed more balanced performance across labels overall.
- Detecting Fully Computer-Generated Images: Capsule-Forensics and its earlier version achieved 100% classification accuracy on the large-scale CGI-versus-photographic-image dataset.Both versions outperformed three other state-of-the-art classifiers.
- Detecting Presentation Attacks: Capsule-Forensics achieved perfect frame-level results on the Idiap Replay-Attack database without mistakes on any frames.The videos were center-cropped from 320 × 240 to 240 × 240 before processing.
5. CONCLUSION
The paper concludes that Capsule-Forensics supports digital image and video forensics, including computer manipulation, computer generation, and presentation attacks. Its performance is equivalent to or better than state-of-the-art methods on tested tasks while using fewer parameters, and visualization provides an analysis of capsule behavior.
- Scope: Capsule-Forensics applies to detecting computer-manipulated or computer-generated images and videos as well as presentation attacks.The conclusion describes the method as applicable across these digital forensics tasks.
- Performance and Efficiency: On the tested tasks, the method achieved performance equivalent to or better than state-of-the-art methods while using fewer parameters.The authors connect the reduced parameter count with lower computation cost.
- Analysis: Activation visualization and primary-capsule agreement analysis explain how Capsule-Forensics behaves across several digital forensics tasks.The paper presents this analysis as contributing to understanding the mechanism behind its performance.