PaperlayerView exact revision on arXiv

Source-linked AI summary

ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection

Davide Cozzolino, Justus Thies, Andreas Rössler, Christian Riess, Matthias Nießner, Luisa Verdoliva

arXiv:1812.02510v2cs.CV

TL;DR

CNN-based forgery detectors can perform well on known manipulations but often fail to transfer to unseen ones, creating a need for reliable detection with little new labeled data. ForensicTransfer learns an autoencoder-based forensic embedding for cross-domain detection and reports improved transferability, including strong performance with few target examples.

  • Problem

    CNN detectors trained for one manipulation often overfit its artifacts and require substantial new labeled data when manipulation methods change.

  • Method

    ForensicTransfer uses a novel autoencoder-based architecture to learn a forensic embedding that transfers between manipulation domains.

  • Results

    ForensicTransfer generalizes better to unseen manipulations and improves with few target examples, with accuracies above 90% within five shots in both dataset pairings.

  • Takeaways & Limitations

    The method supports forgery detection when no or only a handful of training samples for a new manipulation are available.

  • Takeaways & Limitations

    The method assumes source-domain training data and corresponding target-domain manipulation definitions, including original or same-typology real images when originals are unavailable.

Abstract

from arXiv · show

Distinguishing manipulated from real images is becoming increasingly difficult as new sophisticated image forgery approaches come out by the day. Naive classification approaches based on Convolutional Neural Networks (CNNs) show excellent performance in detecting image manipulations when they are trained on a specific forgery method. However, on examples from unseen manipulation approaches, their performance drops significantly. To address this limitation in transferability, we introduce Forensic-Transfer (FT). We devise a learning-based forensic detector which adapts well to new domains, i.e., novel manipulation methods and can handle scenarios where only a handful of fake examples are available during training. To this end, we learn a forensic embedding based on a novel autoencoder-based architecture that can be used to distinguish between real and fake imagery. The learned embedding acts as a form of anomaly detector; namely, an image manipulated from an unseen method will be detected as fake provided it maps sufficiently far away from the cluster of real images. Comparing to prior works, FT shows significant improvements in transferability, which we demonstrate in a series of experiments on cutting-edge benchmarks. For instance, on unseen examples, we achieve up to 85% in terms of accuracy, and with only a handful of seen examples, our performance already reaches around 95%.

1. Introduction

Recent advances have made image manipulation more widespread and sophisticated, while CNN detectors often fail to transfer from known to unseen manipulation methods. ForensicTransfer addresses this gap with a transferable forensic embedding and reports strong detection with few or no target-domain examples.

  • Motivation: Recent machine learning advances have made sophisticated image manipulation broadly accessible beyond high-budget production settings.The trend includes synthetic faces, facial-expression editing, style transfer, image composition, and semantic image completion.
  • Motivation: CNN detectors perform well on trained manipulation types but deteriorate sharply on unseen methods because they overfit manipulation-specific artifacts.Fine-tuning can help, but requires large amounts of new labeled data.
  • Proposed solution: ForensicTransfer targets generalization across related manipulations, including settings with no or only a few labeled examples from the new method.The authors frame this as knowledge transfer between manipulation domains.
  • Proposed solution: The method learns a forensic embedding with a novel autoencoder-based neural architecture designed to transfer between manipulation domains.The architecture is presented as a learned model for a derived forensic embedding.
  • Evaluation: The work includes an ablation study and reports state-of-the-art detection accuracy when no or only a handful of new-manipulation training samples are available.The supplied introduction identifies both methodological analysis and few-shot robustness as contributions.

2. Related work

Prior forgery-detection research spans model-based forensic clues, supervised deep networks, and transfer-learning approaches. These methods commonly depend on assumptions or manipulation-aligned training data, motivating ForensicTransfer’s focus on adapting with few samples to new manipulation types.

  • Traditional Media Forensics: Traditional media-forensics methods use pixel-level, physical, or geometric inconsistencies, trading strong effectiveness for sensitivity to assumptions or reduced realistic performance.Pixel-level clues include compression, demosaicking, lens aberration, and camera noise; physics-based methods use illumination or perspective inconsistencies.
  • Learned Media Forensics: Learned forensic systems exploit high- and low-level image features through high-pass filters, residual features, two-stream networks, and deep architectures.The literature includes fixed or learned filters and CNNs recasting hand-crafted residual features.
  • Learned Media Forensics: Learning-based detectors require fine-tuning on datasets whose manipulations align with the test set, while compression and dataset polarization remain important concerns.The passage identifies compressed social-media data and limited camera diversity as practical challenges.
  • Recent Manipulation Detection: Recent deepfake and GAN-image detectors often target specific manipulation categories, whereas ForensicTransfer is designed to adapt to new manipulations from only a few samples.The related-work distinction is between manipulation-specific detectors and the proposed broader transfer setting.
  • Transfer and Few-Shot Learning: Transfer-learning research addresses domain shift through domain adaptation and autoencoder-based latent embeddings, while few-shot learning studies generalization from limited labeled examples.ForensicTransfer applies these ideas to the binary forensic distinction between pristine and forged images.

3. Proposed Method

ForensicTransfer uses an autoencoder-based CNN to separate real and fake information in a transferable latent embedding, then adapts the detector to new manipulation domains with few target samples. Reconstruction preserves image information while activation-based supervision supports class separation and transferability.

  • Forensic embedding: ForensicTransfer trains an autoencoder-based detector that disentangles real and fake information in a latent space for transfer across manipulation domains.The source-domain detector is fine-tuned on a target domain using available examples rather than being retrained from scratch.
  • Network architecture: Input images are high-pass filtered into residuals before encoding, emphasizing forensic information used for forgery detection.The preprocessing applies a third-order derivative in image space.
  • Forensic embedding: The encoder maps each image to a latent vector split into real and fake parts, whose activation strengths determine the predicted class.The class activation is measured using the ℓ1-norm of the corresponding latent-space part.
  • Losses: The training objective combines reconstruction and activation losses, with γ weighting the reconstruction term and set to 0.1 in all experiments.The reconstruction loss uses the ℓ1 difference between the input and decoder reconstruction, while activation loss supervises class-specific latent responses.
  • Transferability: Reconstruction forces the latent representation to retain input information instead of encoding only source-manipulation artifacts, supporting adaptation to novel manipulations.The method also reduces intra-class variance through its activation-based loss, which the authors associate with improved transferability.
  • Network architecture: The network uses mirrored encoder-decoder structures with five 3 × 3 convolutional layers and a latent space containing 128 feature maps.The latent space assigns 64 feature maps to real and 64 to fake, while the decoder reconstructs the input after downsampling and upsampling.

4. Datasets

The experiments evaluate transfer between related manipulation methods across synthetic, inpainting, and computer-graphics face datasets. Source and target manipulations are paired by relatedness, with target examples used for fine-tuning.

  • Dataset design: The study focuses on computer-graphics and deep-learning manipulations, including partial edits of real images and fully synthetic image generation.The source datasets are large, and target fine-tuning examples are randomly drawn from training and validation sets.
  • Synthetic images: Five synthetic datasets contain 30,000 images each generated by Progressive GAN, CycleGAN, StyleGAN, Glow, and StarGAN.The datasets cover high-resolution face generation, image-to-image translation, and facial-attribute manipulation.
  • Inpainted images: Two inpainting datasets contain 20,000 images each, with manipulation applied to the central 128 × 128-pixel region.One dataset uses the method of Iizuka et al., and the other uses Yu et al.'s method.
  • CG-based manipulated faces: The FaceForensics experiment pairs Face2Face as the source manipulation with FaceSwap as the target manipulation using 1,004 real and 1,004 fake videos.Videos are divided into 704 training, 150 validation, and 150 testing videos, with H.264 compression applied.

5. Results

The experiments evaluate ForensicTransfer through ablations, cross-domain generalization, few-shot adaptation, and multi-source transfer. FT maintains stronger transferability than reference methods, achieving high accuracy with very few target-domain samples.

  • The study examines design choices, state-of-the-art comparisons, few-shot adaptation, and transfer from multiple source domains.
  • 5.1. Ablation study: The reconstruction constraint provides a 15% initial gain in one dataset pairing, while the proposed method exceeds 90% accuracy within five shots for both pairings.
  • 5.1. Ablation study: Activation scatter plots show that single-shot adaptation separates target classes reasonably, becoming nearly perfect with 100 samples, whereas cross-entropy performs much worse.
  • 5.2. Comparison with state-of-the-art: Source-domain accuracy is close to 100% for all methods, but most reference methods fall near 50% on unseen target manipulations; FT reaches 90% from Cycle to Style-GAN.
  • 5.1. Ablation study: On Pro/Cycle-GAN, FT exceeds 90% at four shots and reaches 100% before 100 target images, outperforming competitors by a large margin.
  • 5.3. Multi-source experiments: Multi-source training improves zero-shot detection of unknown target forgeries by at least 7% and beats baselines by more than 25% with fewer than five target samples.

6. Conclusion

ForensicTransfer is introduced to improve transferability between image-manipulation domains. The authors report higher detection rates when no or only a few target-domain samples are available.

  • ForensicTransfer addresses CNNs’ poor detection of different manipulation methods, even when those edits are related.
  • The method achieves significantly higher detection rates than traditional learning techniques when no or only a few target-domain training samples are available.
  • The paper positions ForensicTransfer as a step toward forgery detectors requiring only a few target-domain training samples.

Supplemental Material

The supplemental document visualizes transferability, reports additional ablations, compares few-shot learning methods, and provides reproducibility details.

  • The supplement uses class activation maps to examine transferability relative to a classic classifier.
  • It also reports additional ablation studies and comparisons with few-shot learning methods.
  • The document includes hyperparameter details for reproducibility.

A. Ablation studies

The ablation studies vary the forensic embedding dimension and show that a 128-dimensional space provides the strongest transfer performance, including in few-shot adaptation.

  • 128 dimensions is the sweet spot for zero-shot transfer on the inpainting dataset, outperforming lower and higher-dimensional embeddings.The embedding uses 64 features for pristine images and 64 for manipulated images.
  • Transferability decreases with embedding dimensions below or above 128, but remains better than or comparable to baseline methods.
  • The 128-dimensional feature space consistently outperforms other configurations in few-shot adaptation experiments.The configurations converge as the number of target-domain samples increases.

B. Comparison to few-shot learning methods

The paper compares ForensicTransfer with several few-shot learning methods that use frozen feature extractors, adversarial alignment, or distance-based prediction. On the inpainting pairing, these alternatives perform poorly and are slower than ForensicTransfer.

  • Few-shot learning is especially relevant here because ForensicTransfer addresses a two-class pristine-versus-forged forensic setting.This differs from traditional computer-vision few-shot learning, which often targets many classes.
  • Chen19 freezes a source-trained feature extractor and fine-tunes only a cosine-similarity classifier on target examples.
  • FADA aligns source and target feature distributions adversarially before classification.
  • MatchingNet, ProtoNet, and RelationNet extract features and classify by comparing inputs with a small labeled support set using different distance metrics.MatchingNet uses cosine similarity, ProtoNet Euclidean distance to class means, and RelationNet a trained CNN.
  • Accuracy for these few-shot methods is close to 50% on the inpainting pairing and grows only slowly with more examples.They are generally slower than ForensicTransfer, particularly FADA because of adversarial learning.

C. Baseline methods

The evaluation uses CNN-based forensic detectors and few-shot comparison figures to assess transferability, while standardizing training through shared optimization and validation procedures.

  • Few-shot comparison plots report accuracy against the number of fine-tuning images, with values averaged over 10 runs.The figures cover embedding dimensions and comparisons with recent few-shot methods for inpainting.
  • Bayar16 suppresses high-level image content through a constrained convolutional layer in an eight-layer network.
  • Cozzolino17 reconstructs handcrafted forensic features within a CNN framework.
  • Rahmouni17 integrates statistical feature extraction into a CNN and uses the best-performing Stats-2L network.
  • MesoInception-4 detects DeepFakes with Inception-inspired modules and uses mean squared error rather than cross-entropy loss.
  • XceptionNet uses depth-wise separable convolutions with residual connections and ImageNet pretraining.
  • Training stops after validation accuracy fails to improve for 10 consecutive epochs, retaining weights from the best validation accuracy.

D. Activation maps

Activation maps illustrate how ForensicTransfer adapts its forensic cues across Face2Face and FaceSwap manipulations, including after fine-tuning on only four target images.

  • XceptionNet achieves 98.13% on Face2Face and 98.30% on FaceSwap when trained on the corresponding manipulation type.
  • ForensicTransfer is designed to detect novel unseen manipulation methods without requiring a large amount of training data.
  • The activation-map comparison includes Face2Face training and testing, FaceSwap testing, and Face2Face training followed by four-image FaceSwap fine-tuning.
  • The method classifies a target image by comparing its forensic embedding with source-domain real and fake images when manipulation artifacts are shared.
  • ForensicTransfer adapts its activation regions from the nose for Face2Face to the eyebrows and mouth for FaceSwap.Fine-tuning on four FaceSwap images further supports reliance on these different artifacts.
Loading 1812.02510v2…