Source-linked AI summary
Improving the Efficiency and Robustness of Deepfakes Detection through Precise Geometric Features
Zekun Sun, Yujie Han, Zeyu Hua, Na Ruan, Weijia Jia
TL;DR
Deepfakes detection must uncover manipulation that remains difficult to distinguish visually while handling temporal cues, compression, noise, and model complexity. The paper proposes LRNet, which calibrates facial landmarks and models their temporal dynamics with a two-stream RNN. The authors report efficient, robust detection and identify remaining generalizability and interpretability limitations.
Problem
Deepfakes can become highly realistic, while existing detectors may rely on bypassable appearance defects, struggle with temporal modeling, lose performance under compression, or require costly models.
Method
LRNet calibrates facial landmarks to improve geometric-feature precision and uses a two-stream RNN to model temporal characteristics for Deepfakes detection.
Results
LRNet combines dimensionality reduction and corruption invariance from landmarks with increased feature expressiveness from deep RNNs, balancing cost and performance.
Takeaways & Limitations
Facial geometric information and its dynamic characteristics are useful clues for efficient and robust in-the-wild Deepfakes detection.
Takeaways & Limitations
The framework still has room for improvement in generalizability, and its captured temporal features and real-versus-fake movement differences are difficult to interpret.
Abstract
from arXiv · showhide
Deepfakes is a branch of malicious techniques that transplant a target face to the original one in videos, resulting in serious problems such as infringement of copyright, confusion of information, or even public panic. Previous efforts for Deepfakes videos detection mainly focused on appearance features, which have a risk of being bypassed by sophisticated manipulation, also resulting in high model complexity and sensitiveness to noise. Besides, how to mine the temporal features of manipulated videos and exploit them is still an open question. We propose an efficient and robust framework named LRNet for detecting Deepfakes videos through temporal modeling on precise geometric features. A novel calibration module is devised to enhance the precision of geometric features, making it more discriminative, and a two-stream Recurrent Neural Network (RNN) is constructed for sufficient exploitation of temporal features. Compared to previous methods, our proposed method is lighter-weighted and easier to train. Moreover, our method has shown robustness in detecting highly compressed or noise corrupted videos. Our model achieved 0.999 AUC on FaceForensics++ dataset. Meanwhile, it has a graceful decline in performance (-0.042 AUC) when faced with highly compressed videos.
1. Introduction
Deepfakes can look highly realistic frame by frame while retaining subtle temporal and facial-movement artifacts. LRNet addresses this gap with calibrated geometric features and temporal modeling designed for efficient, robust detection.
- Motivation: Existing detectors face challenges in exposing intrinsic manipulation characteristics, resisting compression-related performance drops, and reducing costly model complexity.Many methods rely on powerful DCNNs or data augmentation, while several models perform poorly on compressed videos.
- Motivation: Deepfakes can appear realistic in individual frames yet exhibit subtle unnatural expressions or facial-organ movements.These artifacts arise because forged videos are generated frame by frame without strong constraints on behavior patterns and time continuity.
- Contributions: LRNet uses landmarks for dimensionality reduction and corruption invariance while RNNs expand feature expressiveness, balancing computational cost and performance.The authors describe this combination as reducing model redundancy and improving the cost-performance balance.
- Approach: LRNet models temporal characteristics on precise geometric features rather than relying only on appearance information.Facial landmarks describe the shape and position of facial organs and are suitable for modeling facial dynamic behavior.
- Approach: The framework combines a landmark-calibration module with a two-stream RNN to reduce jitter and extract deep temporal features from landmark sequences.Calibration enhances geometric-feature discriminability and supports reliable combination of geometric and temporal features.
2. Related Work
Deepfakes detection has progressed from frame-level visual artifacts toward temporal and geometric cues. LRNet focuses on precise landmarks because frame-level landmark detectors may be accurate yet too variable for reliable temporal modeling.
- Frame-level detection: Frame-level detectors commonly target manually selected visual defects or deep features extracted by DCNNs.Examples include iris colors, facial shadows, missing eye or teeth details, and mesoscopic image properties.
- Video-level detection: Video-level methods exploit temporal information, but manually selected geometric features can be less discriminative and limit performance.Prior examples include eye-blinking frequency and inconsistencies between head and face directions.
- Geometric features: Facial landmarks outline iconic facial parts and provide representative geometric information for detecting synthetic faces and videos.Earlier landmark detectors include AAM, CLM, and CSR-based methods.
- Landmarks detection: Accuracy means low bias, whereas precision means low variance; precision is more important here because jittering severely disturbs temporal modeling.Current landmark detectors mostly operate frame by frame and therefore may not achieve high precision.
3. Methodology
LRNet preprocesses video frames into calibrated facial landmarks, embeds them into feature vectors, and uses RNNs to judge authenticity from temporal movement patterns. Its calibration combines optical-flow tracking, consistency checks, and Kalman filtering to reduce landmark jitter and noise.
- Framework overview: LRNet comprises face preprocessing, landmark calibration, feature embedding, and RNN classification components.The framework detects abnormal facial movement patterns and time discontinuities, while only the RNN component requires training.
- Preprocessing: Preprocessing detects faces, extracts 68 facial landmarks, and aligns them to a preset position using affine transformation.The resulting landmarks outline iconic facial profiles and provide geometric information for later processing.
- Landmark calibration: The calibration module predicts landmark positions across successive frames with Lucas-Kanade optical flow and merges valid predictions with detections using a customized Kalman filter.This procedure denoises frame-by-frame detections and produces landmarks with higher precision.
- Tracking: Pyramidal Lucas-Kanade tracking matches corresponding image patches across frames to estimate each landmark’s displacement.The method seeks a displacement vector that minimizes the difference between same-size patches in successive frames and iteratively updates it to convergence.
- Tracking: A forward-backward check discards predictions whose backward-tracked positions differ substantially from their original locations.This removes imprecise optical-flow predictions before the detection and prediction results are integrated.
- Filtering: Because optical flow can introduce noise, the customized Kalman filter integrates detections and predictions through an estimated optimal landmark position.The filter uses a weighted combination rather than relying only on Lucas-Kanade outputs.
14 end
The calibration module combines landmark detection with Lucas–Kanade tracking and Kalman filtering to reduce landmark instability across frames. Calibrated landmark sequences are transformed into shape and inter-frame-difference features for a two-stream RNN that produces video-level predictions.
- Landmarks calibration: The calibration module uses Lucas–Kanade operation to predict landmark positions between successive frames.The operation tracks feature-point movement between frames and provides a prediction alongside the landmark detection result.
- Landmarks calibration: Kalman filtering combines the LK prediction with landmark detection to estimate optimized landmark positions.The Kalman gain weights the prediction and detection according to their respective variances.
- Landmarks calibration: Because P and D are difficult to calculate directly, the method replaces D with an empirically defined approximate relative variance Dr and sets Q = 0.3.The difficulty arises because neither LK operation nor the landmark detector has a simply representable mathematical model.
- Landmarks calibration: Calibration proceeds sequentially, using optimized landmarks from one frame to calibrate the next frame.The calibrated landmarks of frame2 are used to calibrate frame3 and subsequent frames.
- Feature embedding and classification: The method embeds calibrated landmarks as shape vectors and inter-frame position-difference vectors before two-stream temporal modeling.One RNN models facial-shape movement, while the other models landmark differences or speed patterns associated with temporal discontinuity.
- Feature embedding and classification: Clip-level predictions are aggregated to produce the video-level detection result.Each video is segmented into fixed-length clips before aggregation.
4. Experiments
The experiments evaluate LRNet’s efficiency on multiple benchmarks and analyze factors influencing its performance.
- The experimental section evaluates LRNet on several benchmarks and analyzes factors influencing the framework.
4.1. Experiment Setting
The evaluation uses four representative Deepfakes datasets with varied sizes, origins, and compression settings. The implementation uses Dlib preprocessing, bidirectional GRU-based streams, fixed-length clips, and an 8:2 train–test split, while Table 1 reports AUC results and model characteristics.
- Datasets: The evaluation uses UADFV, FaceForensics++, Celeb-DF, and DeeperForensics-1.0 as four representative datasets.The datasets include early manipulated-video data, compression variants, and newer high-visual-quality benchmarks.
- Datasets: FaceForensics++ contains 1000 original videos with corresponding manipulated versions in raw, c23, and c40 compression conditions.
- Implementation: Dlib performs face and landmark detection, while each bidirectional GRU stream uses 64 output units.
- Implementation: The model uses fully connected layers with 64 and 2 units, dropout rates of 0.25 and 0.5, an 8:2 train–test split, and 60-frame clips.
- Evaluation: Table 1 reports general performance using AUC scores on different testing datasets and indicates that LRNet is relatively lightweight, requires no data augmentation, and performs best on FF++.
4.2. Performance Evaluation
LRNet is evaluated across cross-dataset generalization, compression and noise robustness, and training efficiency. The experiments indicate strong performance on FF++ and relatively stable behavior under corruption with lower training cost.
- 99.9 AUC is achieved on the FF++ training dataset in the Celeb-DF benchmark evaluation.
- LRNet is evaluated for robustness to video compression against Xception and X-Ray, with models trained on original or specified compressed data settings.
- LRNet’s performance is relatively more invariant to video compression than the compared methods.
- 99.2% AUC and 97.74% accuracy are achieved on DF1.0 under the same training and testing dataset, while LRNet has the least performance decline under noise.
- LRNet consumes acceptable training time, about 2 hours beyond a 6-hour basic requirement, while being faster and less memory intensive during training.
4.3. Framework Analysis
Framework analyses examine calibration, the two-stream RNN, and input length as factors affecting LRNet’s robustness and effectiveness. Calibration reduces detector dependence, the two streams provide complementary signals, and suitable sequence length helps under distribution shifts.
- 4.3.1 Effect of calibration module: Calibration enhances overall detection performance while preserving robustness in LRNet.
- 4.3.1 Effect of calibration module: Calibration prevents performance drops when landmarks from Dlib and OpenFace are exchanged without retraining.It also shortens the gap between landmark detections and ground-truth positions.
- 4.3.2 Effect of network architecture: The two-stream RNN outperforms either single stream by combining abnormal facial movement and time-discontinuity information.The g2 stream contributes time-discontinuity clues, while g1 contributes robustness through abnormal-movement signals.
- 4.3.3 Influence of input length: 60 successive frames are adopted as the input length, improving effectiveness and robustness across different data distributions such as compressed samples.Input length has little effect when training and testing use the same dataset.
5. Discussion
The discussion identifies limited generalizability and difficult interpretability of LRNet’s temporal features. It also frames geometric features as efficient and robust but harder to generalize than appearance features.
- LRNet still has room for improvement in generalizability based on the general evaluation results.
- The temporal features captured by LRNet are difficult to interpret and the movement differences between real and fake faces are difficult to visualize.
- Appearance features are high-dimensional and expressive but less robust and more costly, whereas geometric features are more robust and lower-cost but harder to generalize.
- Combining appearance and geometric features is identified as a future direction for improving efficiency without incurring high cost.
6. Conclusion
The work identifies facial geometric information and its dynamic characteristics as essential clues for efficient and robust in-the-wild Deepfakes detection.
- Facial landmarks and temporal features provide a fast and robust test for Deepfakes.