PaperlayerView exact revision on arXiv

Source-linked AI summary

FakeCatcher: Detection of Synthetic Portrait Videos using Biological Signals

Umur Aybars Ciftci, Ilke Demir

arXiv:1901.02212v3cs.CV

TL;DR

Fake portrait videos threaten information integrity, while existing detectors are limited by particular generators, datasets, people, or handcrafted features. FakeCatcher instead uses biological signals in portrait videos, finding spatial and temporal inconsistencies in synthetic content and analyzing their classification, but generalization remains sensitive to dataset structure.

  • Problem

    Existing fake-content detectors are limited by specific generative models, datasets, people, or handcrafted features, motivating a source-independent approach to portrait-video authenticity.

  • Method

    FakeCatcher exploits biological signals in portrait videos, transforming them into features and aggregating segment-level authenticity predictions into video labels.

  • Results

    99.39% accuracy was achieved for separating pairs of original and synthetic videos, while experiments identified spatial coherence, temporal consistency, and transformed biological-signal artifacts as relevant findings.

  • Takeaways & Limitations

    Synthetic portrait videos lack coherent biological signals across facial parts and stable PPG across frames, providing explainable authenticity cues for detection.

  • Takeaways & Limitations

    Generalizability is constrained when datasets are biased toward known generative models; random fake-real pairing can conceal target features and reduce accuracy in the wild.

Abstract

from arXiv · show

The recent proliferation of fake portrait videos poses direct threats on society, law, and privacy. Believing the fake video of a politician, distributing fake pornographic content of celebrities, fabricating impersonated fake videos as evidence in courts are just a few real world consequences of deep fakes. We present a novel approach to detect synthetic content in portrait videos, as a preventive solution for the emerging threat of deep fakes. In other words, we introduce a deep fake detector. We observe that detectors blindly utilizing deep learning are not effective in catching fake content, as generative models produce formidably realistic results. Our key assertion follows that biological signals hidden in portrait videos can be used as an implicit descriptor of authenticity, because they are neither spatially nor temporally preserved in fake content. To prove and exploit this assertion, we first engage several signal transformations for the pairwise separation problem, achieving 99.39% accuracy. Second, we utilize those findings to formulate a generalized classifier for fake content, by analyzing proposed signal transformations and corresponding feature sets. Third, we generate novel signal maps and employ a CNN to improve our traditional classifier for detecting synthetic content. Lastly, we release an "in the wild" dataset of fake portrait videos that we collected as a part of our evaluation process. We evaluate FakeCatcher on several datasets, resulting with 96%, 94.65%, 91.50%, and 91.07% accuracies, on Face Forensics, Face Forensics++, CelebDF, and on our new Deep Fakes Dataset respectively. We also analyze signals from various facial regions, under image distortions, with varying segment durations, from different generators, against unseen datasets, and under several dimensionality reduction techniques.

1 INTRODUCTION

FakeCatcher addresses the difficulty of detecting realistic synthetic portrait videos by using biological signals as authenticity descriptors. It transforms these signals, learns interpretable and neural classifiers, and aggregates segment predictions into video-level decisions.

  • Motivation: Deep fakes threaten information integrity because generated visuals can exceed human ability to judge authenticity.The paper motivates detection as a preventive response to societal, legal, and privacy risks.
  • Motivation: Biological signals provide natural authenticity cues because synthetic videos do not reliably preserve their spatial coherence or temporal consistency.The approach searches for priors in authentic content rather than assumptions about particular fake generators.
  • Approach: The system extracts biological signals from facial regions, transforms them across domains, computes feature sets and PPG maps, and trains probabilistic SVM and CNN classifiers.Segment-level authenticity probabilities are aggregated into a binary video-level decision.
  • Contributions: The paper contributes signal transformations for pairwise and generalized authenticity classification, a biological signal map construction, and an in-the-wild portrait-video dataset.These contributions target interpretable detection that operates independently of the source of creation.
  • Evaluation: 91.07% accuracy was achieved on the authors’ Deep Fakes Dataset, while the CNN achieved 96% on Face Forensics and 94.65% on Face Forensics++.The evaluation used more than 140 online videos totaling 30GB, with comparisons against baseline architectures.

2 RELATED WORK

Prior fake-detection methods largely analyze visual artifacts, model-specific signatures, or semantic realism, while FakeCatcher focuses on biological signals in portrait videos. The related work also motivates diverse datasets and remote PPG extraction as foundations for evaluating this approach.

  • Image Forensics: Earlier image-forensics methods detect distortions, compression artifacts, frequency signatures, or image-quality differences, but deep-fake artifacts are harder for humans and handcrafted analyses to detect.The related work contrasts earlier easily recognizable forgeries with more realistic synthetic content.
  • Detection Strategies: Existing detection strategies include pure deep-learning detectors tied to particular generative models and semantic methods that assess facial realism.The paper identifies generalization beyond specific models, people, or datasets as a challenge for these approaches.
  • Biological Signals: Remote biological-signal extraction uses subtle color and motion changes in RGB videos, including remote PPG and head-motion-based ballistocardiograms.The paper focuses on PPG because it is more robust to dynamically changing scenes and actors than BCD.
  • Biological Signals: PPG signals from multiple facial locations are more interconsistent in real videos than synthetic ones, while multiple signals help regularize illumination, occlusion, and motion effects.The paper uses combinations including green-channel PPG to exploit these properties.
  • Datasets: The evaluation combines UADFV, Face Forensics, Face Forensics++, Celeb-DF, and the authors’ Deep Fakes Dataset to span dataset generations and generative-model diversity.The paper frames generated-video datasets as necessary for evaluating detection methods.

3 BIOLOGICAL SIGNAL ANALYSIS ON FAKE & AUTHENTIC VIDEO PAIRS

The analysis extracts six biological signals from three facial regions and applies time-, frequency-, and correlation-based transformations to distinguish authentic from synthetic video pairs. Cross-power spectral density features provide the strongest pairwise separation, reaching 99.39% accuracy.

  • Signal extraction: Six signals combine green-channel and chrominance PPG measurements from the left cheek, right cheek, and mid-face regions.The signals are denoted GL, GR, GM, CL, CR, and CM.
  • Signal transformations: The analysis compares original and synthetic signals using raw values, derivatives, logarithmic scaling, filtering, power spectral density, and discrete cosine transforms.These transformations examine signal behavior across time and frequency domains.
  • Power spectra: 79.33% accuracy results from thresholding power spectral density in linear and logarithmic scales.The result uses the power spectrum density transformation.
  • Power spectra: 77.41% accuracy is obtained using the DC and first three AC components of the discrete cosine transform, increasing to 91.33% with only the zero-frequency DC value.The DC-only formulation performs better than the stated multi-component DCT configuration.
  • Spatio-temporal coherence: 94.57% and 97.28% accuracy result from comparing maximum and mean cross-correlation values of power spectral densities, respectively.Log-scale cross-correlation reaches 98.79%, while cross-power spectral density reaches 99.39%.
  • Spatio-temporal coherence: The final cross-power spectral density formulation reaches 95.06% accuracy on Face Forensics and 83.55% on the Deep Fakes Dataset.The pairwise formulation identifies authentic videos as having more spatio-temporally coherent biological signals in 99.39% of pairs.

4 GENERALIZED CONTENT CLASSIFIER

The generalized classifier examines transformed biological signals and feature sets, then uses CNN-based PPG maps to capture spatial and temporal inconsistencies in synthetic content. Segment probabilities are aggregated into video-level authenticity decisions.

  • Generalized classifier: The generalized classifier uses interpretable signal transformations and feature vectors with an RBF-kernel SVM rather than a DNN.The experiments cover time- and frequency-domain transformations and feature sets designed without assumptions about fake content.
  • Generalized classifier: 126 combined features produced 75% accuracy on Face Forensics and 76.78% on the Deep Fakes Dataset.The features combine F1, F3, and F4 sets with log transforms, pairwise cross spectral densities, and spectral autocorrelations.
  • Probabilistic video classification: 82.55% video accuracy on Face Forensics and 80.35% on the Deep Fakes Dataset followed probabilistic aggregation of segment predictions.The method uses SVR-learned segment probabilities and weighted voting rather than only discrete segment labels.
  • CNN-based classification: 90.66% video accuracy on Face Forensics and 82.69% on the Deep Fakes Dataset were obtained with a three-layer CNN trained on PPG maps.The CNN uses pooling layers, dense connections, ReLU activations, dropout, and a sigmoid output; spectral PPG maps further improved video accuracy to 96% and 91.07%.

5 RESULTS

The evaluation introduces a diverse in-the-wild dataset and tests FakeCatcher across datasets, architectures, biological-signal representations, and cross-dataset settings. Results support the use of spatially coherent and temporally consistent biological signals for synthetic portrait-video detection.

  • 5.1 Deep Fakes Dataset: 142 videos totaling 32 minutes and 30 GB constitute the diverse Deep Fakes Dataset collected from online sources.The dataset varies in generative model, resolution, compression, illumination, motion, pose, cosmetics, occlusion, content, and context.
  • 5.2 Comparisons: FakeCatcher outperformed the best baseline architecture by 8.85% on the comparison reported in Table 6.The comparison uses the same 60% training and 40% test split of the Deep Fakes Dataset; the table compares image, face-image, and video networks.
  • 5.2 Comparisons: 87.62% accuracy with PPG maps exceeded 46.89% for input frames and 54.56% for cropped faces.ConvLSTM reached 47.65% on entire frames and 44.82% on face frames, supporting the reported importance of biological-signal spatial and temporal properties.
  • 5.2 Results: Biological-signal artifacts include spatial inconsistency across synthetic facial parts and temporal incoherence in PPG signals.The paper reports that combining these artifacts improves explainable feature-based detection, while CNN performs better than SVM when feature dimensionality cannot be clearly reduced.
  • 5.3 Cross Dataset/Model Experiments: 18.73% higher accuracy resulted when training on the diverse Deep Fakes Dataset and testing on Face Forensics than in the reverse direction.The authors also report that increasing diversity from FF to FF++ increased accuracy on DF by 16.9%.

6 ANALYSIS

The analysis examines segment duration, facial regions, and face-detection quality to identify robust settings for FakeCatcher. It finds that shorter segments and the default ROI offer practical accuracy, while imperfect detection remains workable.

  • Segment Duration: ω = 300 (10 sec) is long enough to detect strong correlations without including too many artifacts for video labels.
  • Segment Duration: ω = 180 (6 sec) increases video classification accuracy while remaining long enough to extract biological signals.Longer segments can accumulate noise, whereas mislabeled segments may be compensated when enough segments are aggregated.
  • Face Regions: The default ROI is a generalizable choice with good accuracy across pairwise, segment, and video tasks.The comparison covers very small, small, default, big, and whole-face regions on FF and DF datasets.
  • Face Detection Dependency: FakeCatcher processes multiple faces separately, classifying a two-face fake video with an average confidence of 91.7%.
  • Face Detection Dependency: 99.97% fake confidence was obtained on a fake video with 3% face-detection accuracy, while a real video with 17% accuracy was classified as real with 65%.

6.3 Image Quality Analysis

The image-quality analysis tests whether FakeCatcher depends on high-quality facial imagery or precise face contours. The method relies on spatio-temporal biological-signal consistency, which can persist under moderate image degradation.

  • Signal Robustness: Low SNR is unnecessary for fake detection because spatio-temporal inconsistency in PPGs is sufficient.Even low-quality authentic videos can preserve this consistency.
  • ROI Robustness: ROI contours compare whole-face, big, default, small, and few-pixel regions across left, middle, and right facial areas.
  • Face Detection: Face-detection edge cases include multiple faces and very low detection confidence.
  • Image Distortions: Accuracy stays relatively unaffected through a 7x7 blur kernel before declining with larger filtering kernels.The experiments use Gaussian blur and median filtering on Celeb-DF with ω = 128.

6.4 Blind Source Separation

This analysis examines feature separability and classification errors across dimensionality-reduction settings and dataset classes. It also highlights the trade-off between false positives and false negatives in defensive deployment.

  • Feature Space: No significantly high Fisher-criterion ratio was observed for LDA, motivating kernel-based SVMs and further feature-space exploration.PCA and CSP were also applied to reduce feature-space dimensionality.
  • Class Accuracies: Dataset accuracy combines correct fake detection with correct real-video acceptance, while class errors have different operational costs.False positives may be more tolerable than false negatives in systems with secondary manual review, but excessive false positives can overwhelm review capacity.
  • Class Accuracies: FakeCatcher shows a slight general tendency toward false positives across the evaluated datasets and sources.

6.6 Signal Processing Enhancements

The signal-processing analysis evaluates normalization, frequency bands, quantization, and DCT feature cutoffs. It indicates that preserving broad signal information is important, while some alternative feature choices are surpassed by the main configuration.

  • Experimental Setup: The analysis documents representative preprocessing and postprocessing choices while holding the signal and feature set constant in small experiments.
  • Normalization Effects: All frequencies and their ranges are needed because normalization may remove signal characteristics that differentiate original and fake videos.The pairwise analysis uses ω = 300 and 387 segments on the toy FF dataset.
  • Frequency Bands: The study compares below, acceptable, high-heart-rate, and additional high-frequency bands for pairwise separation on the entire FF dataset.The bands are defined by thresholds at 1.05 Hz, 3.04 Hz, and 4.68 Hz.
  • Quantization: 256 bins were verified as the optimum quantization choice for the best configuration on the entire FF dataset.
  • DCT Features: DCT feature accuracies with cutoffs N = {1, 2, 3, 4} were surpassed by the configuration reported in Section 5.2.3.

7 IMPLEMENTATION DETAILS

FakeCatcher extracts and processes biological signals from portrait-video segments using filtering, spectral analysis, and PPG-specific procedures rather than standard heart-rate denoising.

  • Each segment is Butterworth-filtered over [0.7, 14] and analyzed with Welch’s method before collecting all frequencies across heart-beat ranges.The pipeline uses PPG extraction methods for six facial regions: GL, GM, GR, CL, CM, and CR.
  • Detection-oriented PPG signals retain frequencies and temporal changes that heart-rate estimation would normally treat as noise.Heart-rate estimation applies stronger denoising and componentization to fit expected ranges and periods, whereas detection uses information beyond those cleaned signals.

8 FUTURE WORK

Future work extends FakeCatcher beyond current portrait-video detection, improves its CNN and dataset design, and explores biologically plausible generation models.

  • FakeCatcher currently focuses on portrait videos; general videos without humans would require other faithfully extractable spatiotemporal signals, such as illumination or wind.The proposed extension targets signals that can be extracted from original videos.
  • A more complex CNN architecture is identified as one route for improving FakeCatcher.
  • A BioGAN could incorporate biological-signal preservation into a generative model through an additional discriminator and differentiable extraction process.The paper notes that formulating a differentiable loss matching the signal-processing steps is not straightforward.
  • Future dataset work should reduce bias toward known generators and investigate learning from random fake–real pairs for better generalizability.The authors report that random pairing can conceal the features of interest and reduce accuracy in the in-the-wild setting, whereas paired data focuses learning on generative differences despite other artifacts.

9 CONCLUSION

FakeCatcher detects fake portrait videos by exploiting biological-signal coherence and temporal consistency, with strong results across pairwise and video-level evaluations. The paper also contributes an in-the-wild analysis and releases the Deep Fakes Dataset for further research.

  • 99.39% pairwise separation accuracy, 96% constrained video classification accuracy, and 91.07% in-the-wild video classification accuracy were achieved.The reported results cover pairwise separation and segment- and video-level authenticity classification on Face Forensics and the Deep Fakes Dataset.
  • FakeCatcher uses physiological changes and novel PPG maps to support a CNN-based classifier that is agnostic to the generative model.The conclusion states that spatial coherence and temporal consistency of biological signals are not well preserved in GAN-generated content.
  • The paper provides an in-depth analysis of deep fakes in the wild and presents the first experimental study described here of human signals in synthetic portrait videos.The authors also make the Deep Fakes Dataset available to the research community.
Loading 1901.02212v3…