PaperlayerView exact revision on arXiv

Source-linked AI summary

DeepRhythm: Exposing DeepFakes with Attentional Visual Heartbeat Rhythms

Hua Qi, Qing Guo, Felix Juefei-Xu, Xiaofei Xie, Lei Ma, Wei Feng, Yang Liu, Jianjun Zhao

arXiv:2006.07634v2cs.CV

TL;DR

Realistic, accessible DeepFakes create an urgent detection problem because raw pixel-based methods may weaken as synthesis improves. DeepRhythm monitors heartbeat rhythms with motion representations and dual spatial-temporal attention, and experiments report effectiveness, robustness, and generalization across datasets and manipulations.

  • Problem

    DeepFake realism and accessibility threaten multimedia integrity, while detectors relying only on raw pixels may become less effective as generation methods mature.

  • Method

    DeepRhythm detects DeepFakes by modeling disrupted heartbeat rhythms with motion-magnified spatial-temporal representations and dual spatial-temporal attention.

  • Results

    DeepRhythm outperforms four state-of-the-art detectors on FaceForensics++ and is robust to JPEG compression, noise, and blur, with results also demonstrated on DFDC-preview.

  • Takeaways & Limitations

    Heartbeat-rhythm patterns provide an effective DeepFake detection signal that generalizes across datasets and different DeepFake generation techniques.

  • Takeaways & Limitations

    The authors identify combining DeepRhythm with other detectors and studying applications beyond DeepFake detection as future work.

Abstract

from arXiv · show

As the GAN-based face image and video generation techniques, widely known as DeepFakes, have become more and more matured and realistic, there comes a pressing and urgent demand for effective DeepFakes detectors. Motivated by the fact that remote visual photoplethysmography (PPG) is made possible by monitoring the minuscule periodic changes of skin color due to blood pumping through the face, we conjecture that normal heartbeat rhythms found in the real face videos will be disrupted or even entirely broken in a DeepFake video, making it a potentially powerful indicator for DeepFake detection. In this work, we propose DeepRhythm, a DeepFake detection technique that exposes DeepFakes by monitoring the heartbeat rhythms. DeepRhythm utilizes dual-spatial-temporal attention to adapt to dynamically changing face and fake types. Extensive experiments on FaceForensics++ and DFDC-preview datasets have confirmed our conjecture and demonstrated not only the effectiveness, but also the generalization capability of \emph{DeepRhythm} over different datasets by various DeepFakes generation techniques and multifarious challenging degradations.

1 INTRODUCTION

DeepFakes have become accessible and realistic enough to threaten multimedia integrity, while pixel-based detectors may weaken as generation improves. DeepRhythm addresses this gap by detecting disruptions in heartbeat rhythms using motion amplification and dual spatial-temporal attention.

  • Motivation: GAN-based tools have lowered the expertise required to create realistic face swaps and other synthetic multimedia.Pre-trained DeepFake generators make such manipulation widely accessible.
  • Motivation: DeepFakes can threaten the integrity of multimedia information by enabling targeted misinformation and manipulated elections.
  • Detection gap: Raw pixel-domain detectors may become less effective as DeepFake generation methods produce increasingly realistic images and videos.The paper therefore calls for a fundamentally different detection method.
  • Proposed idea: DeepRhythm conjectures that blood-flow-related heartbeat rhythms in real face videos are disrupted or broken by DeepFake manipulation.The approach is motivated by remote visual photoplethysmography, which tracks tiny periodic skin-color changes caused by blood pumping.
  • Method: DeepRhythm combines a heart-rhythm motion amplification module with learnable spatial-temporal attention to handle changing faces and degradations.
  • Results: DeepRhythm outperforms four state-of-the-art detectors on FaceForensics++ and remains robust to JPEG compression, noise, and blur.The reported detectors are Bayer’s method, Inception ResNet V1, Xception, and MesoNet.

2 RELATED WORK

Prior DeepFake research spans face-wrapping, GAN synthesis, handcrafted and neural forensic features, and temporal modeling. DeepRhythm differs from anti-spoofing approaches by comparing real and fake heartbeat patterns rather than merely checking whether a pulse exists.

  • DeepFake generation: Early face-wrapping systems used tracked mouth motion, face 3D models, face warping, or monocular capture for video replacement.
  • DeepFake generation: GANs such as PGGAN, StyleGAN, and StyleGAN2 enable near-realistic face synthesis and attribute manipulation, while remaining easy to use.
  • Pulse-based cues: Current GAN-based DeepFake methods do not explicitly preserve pulse signals, motivating pulse-based discrimination between real and manipulated videos.
  • Detection methods: DeepFake detection remains challenging because near-realistic GAN faces can evade state-of-the-art digital forensics.
  • Detection methods: Detection research has progressed from handcrafted forensic features to discriminative cues, classifiers, and increasingly advanced deep neural networks.
  • Temporal modeling: CNN-RNN and attention-based methods combine spatial, temporal, or informative-region features to classify real and fake videos.
  • Anti-spoofing: rPPG-based anti-spoofing methods use pulse information for liveness detection, including pulse differences, face-background comparisons, and auxiliary rPPG signals.
  • Novelty: Unlike liveness detection, DeepRhythm distinguishes real and fake heartbeat patterns because manipulated videos may retain diminished but differing rhythms.

3 METHOD

DeepRhythm detects DeepFakes by highlighting heartbeat-related sequential signals in face videos and classifying their real/fake patterns. Its MMSTR representation and dual-spatial-temporal attention address weak real–fake signal differences, interference, changing faces, and varied fake types.

  • 3.1 DeepRhythm for DeepFake Detection: DeepRhythm predicts whether a face video is real or fake from heartbeat rhythm signals represented by an MMST map.MMSTR highlights motion-magnified temporal variations across face regions and color channels.
  • 3.3 Dual-Spatial-Temporal Attentional Network: Dual-spatial-temporal attention assigns weights across MMST-map positions by decomposing attention into spatial and temporal components with N + T parameters instead of N · T.The decomposition indicates where and when the input map should be used and is intended to adapt to changing faces and fake types.
  • 3.2 Motion-Magnified Spatial-Temporal Representation: MMSTR magnifies spatial-temporal face variations after landmark-based preprocessing, motion magnification, ROI blocking, and average pooling over RGB channels.Each MMST-map row represents one ROI block’s motion-magnified temporal variation.
  • 3.2 Motion-Magnified Spatial-Temporal Representation: MMST maps expose real–fake differences that are difficult to judge from raw frames, providing discriminative information for DeepFake detection.The paper reports that these differences can be easily found on MMST maps.
  • 3.3.1 Dual-Spatial Attention: Dual-spatial attention combines fixed prior attention over six robust ROI blocks with face-adaptive attention that responds to environmental variation.The specified regions are four blocks under the eyes and two blocks between the eyes; adaptive attention handles changing effective ROIs.
  • 3.3.2 Dual-Temporal Attention: Dual-temporal attention combines block-level and frame-level signals so more significant or faker-looking frames contribute more to final classification.An LSTM produces block-level attention from MMST-map rows, while a temporal-attention network independently scores each motion-magnified frame.

4 EXPERIMENTS

Experiments evaluate DeepRhythm on FaceForensics++ and DFDC-preview, comparing baselines, testing component contributions, and measuring robustness to common degradations. DeepRhythm achieves strong accuracy and generalization, while MMSTR and attention modules provide complementary benefits.

  • Dataset and Experiment Setting: FaceForensics++ supplies training and testing data across four manipulation subsets, while DFDC-preview serves as an additional cross-dataset test set.The evaluation uses DFD, DF, F2F, FS, and ALL subsets, with an 8:1:1 partition and augmented real videos excluded from testing.
  • Baseline Comparison on Accuracy: DeepRhythm achieves the highest accuracy across FaceForensics++ datasets and training configurations, outperforming the baseline methods.When trained on ALL, it reaches 0.96 accuracy on FS versus 0.719 for MesoNet and exceeds Xception on F2F and FS.
  • Baseline Comparison on Accuracy: On DFDC-preview, DeepRhythm reaches 0.641 accuracy, exceeding Xception’s 0.612 but trailing MesoNet’s 0.745.The models are trained on FaceForensics++ and evaluated on a balanced DFDC-preview test set of 500 real and 500 fake videos.
  • Ablation Study on Accuracy: MMSTR improves DR-st accuracy by 0.328 on ALL and raises accuracy by an average of 0.217 across the five FaceForensics++ datasets.The DR-st baseline remains near random performance, whereas DR-mmst benefits substantially from the proposed representation.
  • Ablation Study on Accuracy: Adaptive and block-level temporal attention improve DR-mmst by average gains of 0.061 and 0.0632, while dual-spatial and dual-temporal attention add average gains of 0.033 and 0.178.End-to-end training of the attention modules achieves the highest accuracy across all testing datasets.
  • Ablation Study on Robustness: MMSTR remains nearly stable across JPEG, blur, and noise, whereas temporal sampling is challenging; combining MMSTR with MesoNet provides complementary robustness.The final combined model shows comprehensive robustness across the tested degradations, although temporal sampling remains a weakness relative to Xception and MesoNet.

5 CONCLUSIONS

DeepRhythm’s experiments confirm that heartbeat rhythms in real face videos are disrupted in DeepFakes, while robustness is evaluated against multiple degradations and model variants. Future work includes combining DeepRhythm with other detectors and exploring related applications.

  • Experiments on FaceForensics++ and DFDC-preview confirm that DeepFake videos disrupt normal heartbeat rhythms in real face videos.
  • Figure 6 compares DeepRhythm with Xception and MesoNet under JPEG compression, Gaussian blur, temporal sampling, and Gaussian noise.Average accuracy and standard deviation across degradation degrees are reported for each sub-figure.
  • Figure 7 evaluates MesoNet and seven DeepRhythm variants, grouped according to whether MesoNet supplies frame-level temporal attention.The figure reports average accuracy and standard deviation across degradation degrees.
  • Future work will study combining DeepRhythm with other DeepFake detectors and applying it to non-traditional adversarial attacks and tracking-based feature mining.
Loading 2006.07634v2…