PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Ambiguous Visual Cues: Studying Physiological Disruptions and Cross-Modal Inconsistencies in Deepfake Videos

Chenxi Yang, Yassine Ouzar, Larbi Boubchir

arXiv:2609.12668v1cs.CV

TL;DR

Deepfake detectors face ambiguous visual cues, benchmarks without physiological ground truth, and limited modeling of the relationship between rPPG and facial behavior. The paper constructs manipulations on real rPPG datasets and applies bidirectional co-attention; it reports strong results across face swapping and motion transfer, while identifying broader generalization and measurement as future needs.

  • Problem

    Many deepfake benchmarks lack physiological ground truth, and existing methods often use rPPG independently or with shallow fusion rather than modeling its relationship with facial behavior.

    Source: §1 · p. 1

  • Method

    The authors construct face-swap and motion-transfer videos on real rPPG datasets and use bidirectional co-attention to jointly model rPPG and facial-behavior tokens.

    Source: §1 · pp. 1–2

  • Results

    92.80% AUC on face swapping, while co-attention improves performance to 95.13% in the nearly saturated motion-transfer task.

    Source: §4.3 · p. 6

  • Takeaways & Limitations

    The results show manipulation-dependent physiological disruption and support using token-level interaction between pulse and facial-behavior representations in these constructed datasets.

    Source: §5 · p. 6, §4.3 · p. 6

    The authors call for broader physiological measures and stricter generalization studies across datasets, generators, and video conditions; reference-free objectives may help where contact-sensor ground truth is unavailable.

    Source: §5 · pp. 6–7

Abstract

from arXiv · show

Recent deepfake detection studies increasingly suggest remote photoplethysmography (rPPG) signals as an authenticity cue. However, existing benchmarks lack physiological ground truth, and current detectors underexplore the cross-level relationship between facial features and physiological dynamics, often relying on late fusion or rPPG features alone. In this paper, we construct high-fidelity deepfake manipulations on established real rPPG datasets (COHFACE and UBFC-rPPG) to investigate how forgeries disrupt natural physiological signals and facial behavior at the same time. Building on this analysis, we propose a bidirectional co-attention fusion detector that jointly models rPPG and facial behavior tokens. This mechanism explicitly captures the cross-level dependencies between pulse dynamics and facial motion to learn a robust, joint authenticity representation. Extensive experiments using a subject-disjoint 5-fold evaluation demonstrate the superiority of our approach. Achieving a 92.80\% AUC on constructed datasets using face swapping and 96.78\% AUC on motion transfer, our model outperforms both the rPPG-only single modality baseline and the best feature-level fusion methods. Furthermore, transfer-learning result of the fusion detector on Celeb-DF-v2 while keeping both feature extractors fixed achieves 91.20\% accuracy and 86.08\% AUC, which suggests applicability under target-domain adaptation.

1 Introduction

Visual cues can become ambiguous as deepfake generation improves, while physiology-aware detection lacks benchmarks with physiological ground truth and often underexplores links between pulse dynamics and facial behavior. The paper constructs manipulations on real rPPG datasets and proposes bidirectional co-attention; experiments report gains over single-modality and shallow-fusion baselines.

1 Introduction

  • The study addresses increasingly ambiguous visual cues and the lack of physiological ground truth in widely used deepfake benchmarks.Existing physiology-aware methods often rely on handcrafted rules, independent classifiers, or late fusion, leaving links between physiology and facial behavior underexplored.Source: §1 · p. 1
  • The authors create face-swap and motion-transfer manipulations on real rPPG datasets with synchronized physiological references.This setup supports analysis of how manipulations affect physiological signals and facial dynamics.Source: §1 · p. 1
  • The detector jointly models rPPG and facial-behavior tokens, using bidirectional co-attention to exchange complementary temporal evidence before classification.Behavior tokens include action units, pose, and gaze; the resulting fusion is compared with single-modality and shallow-fusion baselines.Source: §1 · pp. 1–2
  • 92.80% AUC on face swapping; bidirectional co-attention outperforms rPPG alone and shallow fusion, including concatenation.The result is reported for face swapping, where behavior alone is close to chance.Source: §1 · p. 2

2 Related Work

Related work spans identity and facial-motion manipulations, artifact- and behavior-based detection, and emerging use of rPPG. The paper highlights gaps in physiological ground-truth data and cross-modal modeling, then constructs manipulations on real rPPG datasets and models bidirectional dependencies between physiology and behavior.

2.1 Deepfake Generation

  • Face swapping transfers source identity onto a target video while largely preserving target motion and scene context, whereas reenactment or motion transfer preserves target identity while driving facial dynamics from another source.Examples include GAN-based FaceShifter, SimSwap, and FSGAN [17] [25], diffusion-based DiffFace [16], and video-driven First Order Motion Model, Thin-Plate Spline Motion Model, and Face-vid2vid [34] [35] [38].Source: §2.1 · p. 2

2.2 Deepfake Detection

  • Detection methods have used synthesis artifacts and blending boundaries [1] [18] [20] [33], then expanded to temporal and behavioral cues such as blinking, pose, and lip motion [11] [19] [23] [36].The cited work notes that low-level cues can generalize poorly as generation and post-processing evolve.Source: §2.2 · p. 2

2.3 Remote Photoplethysmography (rPPG) for Deepfake Detection

  • Physiology-aware methods use rPPG as complementary evidence [5] [12] [31], but many benchmarks lack physiological ground truth and existing detectors often use independent streams or shallow fusion.A prior UBFC1-F study examined swap quality and compression but was limited to one dataset and one identity-swapping technique; this paper addresses the gaps with real-rPPG manipulations and bidirectional token dependencies.Source: §2.3 · pp. 2–3

3 Methods

The method aligns pulse-sensitive rPPG tokens with frame-level facial-behavior tokens, then uses bidirectional co-attention to exchange information between modalities. Evaluation uses subject-disjoint folds and includes physiological recovery measures and waveform discrepancy descriptors.

3.1 Aligned Physiology and Behavior Tokens

  • PhysFormer yields pulse-sensitive rPPG tokens per frame, while OpenFace provides aligned behavior tokens containing pose, gaze, and action-unit measurements.The rPPG tokens retain temporal features rather than reducing each clip to a single estimated heart rate; failed OpenFace frames are masked.Source: §3.1 · p. 3

3.2 Bidirectional Physiology–Behavior Co-Attention

  • Separate projections and layer normalization map the two streams into a shared space, where bidirectional attention lets each modality retrieve evidence from the other before temporal aggregation.The contextualized sequences are subsequently pooled and used for classification.Source: §3.2 · p. 3

3.3 Evaluation Protocol and Measures

  • Evaluation uses subject-disjoint 5-fold cross-validation, with identities disjoint across folds and motion-transfer driving sequences partitioned between training and testing.The protocol also computes physiological recovery measures and waveform discrepancy descriptors after processing recovered signals.Source: §3.3 · p. 3, Table 1 · p. 3
  • The method’s evaluation materials include an overview of the deepfake manipulations applied to rPPG datasets.The supplied table caption identifies its scope but does not specify its entries.Source: Table 2 · p. 4

4 Experiments

Experiments use controlled face-swap and motion-transfer manipulations of rPPG recordings, with synchronized pulse references for physiological analysis. Bidirectional co-attention performs best among the compared fusion methods, while transfer evaluation on Celeb-DF-v2 reports 91.20% accuracy and 86.08% AUC.

4.1 Deepfake generation

  • The study retains recordings from UBFC-rPPG and COHFACE as real samples and constructs identity-swapped and motion-transfer subsets; identity swapping uses two frameworks and motion transfer uses external driving videos.Identity-swapping source-target pairs are divided approximately equally between FaceFusion and SimSwap; each motion-transfer source is paired with three driving videos after quality control.Source: §4.1 · p. 4

4.2 Ground-truth Biological Signal Analysis on Real and Manipulated Videos

  • Real videos recover physiology best; face swapping raises HR MAE from 3.13 bpm to 5.70 bpm and reduces MACC from 0.753 to 0.672, while PSD correlation remains 0.702.The comparison uses dataset-provided ground-truth pulse signals as the reference.Source: Table 3 · p. 4, §4.2 · p. 4
  • Motion transfer produces the strongest disruption: MACC drops from 0.753 to 0.474 and PSD correlation from 0.820 to 0.341, with negative SNR.The negative SNR indicates that the recovered signal no longer concentrates energy around the ground-truth heart-rate frequency and its harmonics.Source: §4.2 · p. 4
  • 62.00% accuracy and 0.655 AUC separate real from face-swap clips using Raw MAE alone; real versus motion transfer reaches 86.16% accuracy and 0.934 AUC.FaceSwap versus MotionTransfer reaches 77.98% pairwise accuracy and 0.860 AUC; raw waveform discrepancies are especially discriminative for comparisons involving motion transfer.Source: Table 4 · p. 5, §4.2 · p. 5

4.3 Physiology-Behavior Fusion Analysis

  • 85.32% accuracy and 92.80% AUC on face swapping; on motion transfer, bidirectional co-attention reaches 95.13% accuracy, with the other fusion variants near 93%.For face swapping, its AUC gains are 3.53 points over rPPG alone and 2.65 points over concatenation; motion-transfer behavior alone already reaches 90.30% accuracy and 92.99% AUC.Source: Table 5 · p. 6, §4.3 · p. 6
  • On Celeb-DF-v2, the fine-tuned detector reaches 91.20% video-level accuracy and 86.08% AUC while keeping both feature extractors fixed.Celeb-DF-v2 lacks synchronized physiological ground truth and is excluded from physiological recovery analysis.Source: §4.3 · p. 6

5 Conclusion and Future Work

Face swapping preserves more physiological structure than motion transfer, which causes the strongest signal degradation; the bidirectional co-attention detector achieves strong results on both manipulations. The paper calls for broader physiological characterization and stricter generalization studies, noting that Celeb-DF-v2 transfer does not establish cross-dataset robustness.

5 Conclusion and Future Work

  • 92.80% AUC on face swapping and 96.78% AUC on motion transfer were achieved by the bidirectional co-attention detector under subject-disjoint 5-fold evaluation.Motion transfer caused the strongest physiological degradation, while face swapping preserved more of the target’s physiological structure.Source: §5 · p. 6
  • The Celeb-DF-v2 result of 86.08% AUC after retraining shows promise beyond controlled rPPG datasets but does not establish strict cross-dataset robustness.The authors call for train-on-one/test-on-another studies, evaluation on unseen generators and manipulation families, and systematic analysis of recording and image conditions.Source: §5 · p. 6
  • Future work should broaden physiological degradation analysis beyond heart-rate error, waveform agreement, SNR, PSD correlation, and discrepancy measures.Proposed measures include pulse-rate variability, phase coherence, spatial consistency, harmonic structure, signal-quality indices, and uncertainty-aware estimates.Source: §5 · p. 6
  • Reference-free physiological consistency objectives and modality-reliability gating may improve deployment when contact-sensor ground truth is unavailable and a modality is unreliable.Source: §5 · p. 7

Ethical considerations and data availability.

The generated videos were made solely for deepfake-detection research using public recordings under the datasets’ applicable access conditions.

Ethical considerations and data availability.

Loading 2609.12668v1…