PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpretable Deepfake Detection in Videos via Explicit Forensic Features and Temporal Modeling

Chahira Benhama, Mohand Saïd Allili, Assia Hamadene

arXiv:2610.03380v1cs.CV

TL;DR

Deepfake videos can appear consistent frame by frame even when their temporal behavior is irregular. This paper combines identity-consistent facial trajectories, explicit forensic descriptors, and LSTM sequence modeling; it reports F1-scores of 98.0%, 91.0%, 97.6%, and 96.2% across four benchmarks.

  • Problem

    Deepfake videos may appear realistic at the frame level while exhibiting temporal inconsistencies, making video detection challenging.

    Source: §I · p. 1

  • Method

    The framework encodes identity-preserving face sequences with 68 explicit forensic features and uses an LSTM to model temporal dependencies.

    Source: §I · p. 2

  • Results

    F1-scores were 98.0% on FaceForensics++, 91.0% on Celeb-DF v2, 97.6% on DFDC, and 96.2% on DeeperForensics.

    Source: B. Dataset-wise Main Results · p. 6

  • Takeaways & Limitations

    The framework is presented as a transparent, computationally lightweight alternative to black-box representations, with balanced detection behavior on ForgeryNet and WildDeepfake.

    Source: §VI · p. 8, D. Cross-Dataset Generalization · p. 8

    Sequential models cannot directly exploit facial-topology relations, and recurrent architectures may not capture long-range temporal anomalies.

    Source: §VI · p. 9

Abstract

from arXiv · show

Deepfake detection in videos remains challenging, as manipulated content may appear visually consistent at the frame level while exhibiting subtle temporal inconsistencies. This paper introduces an interpretable deepfake detection framework that models spatially and temporally coherent facial features in video sequences. Unlike end-to-end deep models relying on implicit representations, the proposed approach explicitly encodes physically grounded forensic cues, enabling transparent analysis and improved multi-dataset generalization. The pipeline transforms videos into identity-consistent facial trajectories, segments them into fixed-length temporal windows, and represents each frame using 68 structured descriptors spanning four complementary domains: photometric, textural, geometric, and compression-based features. These descriptors provide a compact multi-domain representation of manipulation artifacts and are processed by a Long Short-Term Memory (LSTM) network to capture temporal dependencies and subtle irregularities. Evaluation on four benchmark datasets, FaceForensics++, Celeb-DF v2, a curated subset of the DeepFake Detection Challenge (DFDC), and DeeperForensics, yields strong and consistent F1-scores of 98.0%, 91.0%, 97.6%, and 96.2%, respectively. The approach also demonstrated a good cross-dataset generalization, providing a robust and interpretable solution for video deepfake detection.

I. INTRODUCTION

Realistic deepfakes may look convincing frame by frame while exhibiting temporal inconsistencies, and the paper proposes explicit forensic features with temporal modeling to address this challenge. It positions this approach at the intersection of temporal modeling and explainability.

I. INTRODUCTION

  • Frame-level realism can obscure temporal irregularities in motion, appearance, identity, illumination, geometry, and compression, motivating coherent analysis over time.The introduction also identifies limited interpretability, dataset-specific overfitting, and independently sampled frames as weaknesses of existing approaches.Source: §I · p. 1
  • The framework tracks identity-consistent faces, segments them into fixed-length windows, and encodes each frame with 68 features across four forensic domains.Photometric, textural, geometric, and compression descriptors represent complementary manipulation traces; an LSTM models temporal dependencies.Source: §I · pp. 1–2
  • The approach combines explicit forensic descriptors with sequence learning over identity-consistent trajectories, linking detection decisions to observable inconsistencies.The paper states that this combination can capture coherent temporal patterns overlooked by frame-based methods while reducing overfitting to dataset-specific visual cues.Source: §I · p. 2
  • The contribution summary describes evaluation on four datasets and additional cross-dataset testing on ForgeryNet and WildDeepfake.The supplied introduction states the evaluation scope but does not give results for these tests.Source: §I · p. 2

II. RECENT LITERATURE AND POSITIONING

  • The paper frames multibenchmark evaluation, cross-dataset generalization, temporal modeling, and interpretability as central directions in recent deepfake-detection research.It presents its sequence-based framework as a complementary, more interpretable paradigm rather than a replacement for end-to-end architectures.Source: §II · p. 2

III. OVERALL FRAMEWORK

The framework harmonizes video timing, detects and tracks faces, and forms identity-preserving facial sequences for downstream analysis. Its tracking formulation predicts face-box state and updates it using detections.

III. OVERALL FRAMEWORK

  • Frames are temporally resampled to a target rate of 25 fps, and detected face crops are resized to 224 × 224 with aspect ratio preserved by padding.The detector produces candidate faces for each frame.Source: §III · p. 2

A. Face Tracking Formulation

  • Detected faces are linked across frames into identity-preserving tracks represented by bounding-box center, aspect ratio, height, and their temporal velocities.The measurement contains observed location and size, while the state prediction uses a linear motion model.Source: A. Face Tracking Formulation · p. 2
  • After a detection is associated with a track, a Kalman update corrects the predicted state using the new observation, with the gain controlling the correction.The paper describes this update as supporting robust and smooth temporal tracking of facial regions.Source: A. Face Tracking Formulation · p. 3

B. Identity Association Cost

The identity-association stage combines motion and appearance consistency to match detections to active tracks. The resulting tracks are constrained into ordered, overlapping sequences for temporal analysis.

B. Identity Association Cost

  • The identity-assignment cost combines motion consistency, measured by Mahalanobis distance, with appearance consistency, measured by cosine distance between embeddings.The motion term uses prediction uncertainty and predicted observations; the relative weights sum to one.Source: B. Identity Association Cost · p. 3
  • The Hungarian algorithm obtains the optimal assignment, after which valid sequences contain 20 consecutive frames from the same tracked face.Sequences preserve temporal order, normalize frames to 224 × 224, and use sliding windows of length 20 with stride 10.Source: B. Identity Association Cost · p. 3

IV. EXPLICIT PER-FRAME FEATURE FAMILIES

Each face frame is encoded with 68 descriptors from four complementary forensic families. The photometric family makes color, illumination, and facial symmetry measurable through explicit statistics.

IV. EXPLICIT PER-FRAME FEATURE FAMILIES

  • Each face is represented by 68 descriptors: 18 photometric, 24 textural, 14 geometric, and 12 compression-related features.The decomposition is intended to capture diverse forensic cues rather than rely on a single source.Source: §IV · p. 3

A. Photometric Family (18 Variables)

  • Photometric features represent color in RGB, HSV, and YCrCb spaces and compute channel means; RGB-channel standard deviations are also included.The paper says these color-space views make manipulation-induced inconsistencies more directly accessible.Source: A. Photometric Family (18 Variables) · p. 3
  • The analyzed regions include the segmented face for color and texture, detected landmarks for geometry, and hair-skin contact and jaw boundaries.The facial midline partitions the face into left and right halves for symmetry assessment.Source: Figure 1 · p. 3
  • Color ratios capture relative chromatic relationships, while left-right facial partitions support luminance and saturation asymmetry measures.A small constant prevents division by zero in the symmetry analysis.Source: A. Photometric Family (18 Variables) · p. 3
  • Highlight proportion is included among the photometric descriptors, with a luminance threshold defining highlight pixels within the facial region.The supplied text does not specify the threshold value.Source: A. Photometric Family (18 Variables) · p. 3

B. Textural Family (24 Variables)

The textural family combines co-occurrence statistics with regional texture and frequency descriptors to capture facial texture and left-right differences. Fake samples show broader texture distributions and greater variability.

B. Textural Family (24 Variables)

C. Geometric Family (14 Variables)

The geometric family uses landmark distances, facial proportions, and symmetry measures to describe structural consistency. These features complement appearance cues by capturing subtle inconsistencies and instability, including across frames.

C. Geometric Family (14 Variables)

D. Compression-Related Family (12 Variables)

The compression-related family measures recompression residuals, block artifacts, and boundary discontinuities over meaningful facial regions. The feature families differ between real and fake samples, with compression cues providing particularly strong discrimination even in high-quality deepfakes.

D. Compression-Related Family (12 Variables)

E. LSTM-Based Deepfake Prediction

The model applies an LSTM to explicit forensic-feature sequences to detect temporal irregularities rather than only anomalous frame values. Its sequence representation preserves feature-family structure and temporal coherence.

E. LSTM-Based Deepfake Prediction

  • The LSTM captures short-term variations and longer-range temporal patterns, including instability, drift, and incoherent co-evolution across feature families.Source: E. LSTM-Based Deepfake Prediction · p. 5
  • A temporal sample stacks 20 consecutive frames from the same tracked face into a 20 × 68 matrix, with rows encoding time and columns representing forensic variables.Source: E. LSTM-Based Deepfake Prediction · p. 5
  • A two-layer unidirectional LSTM feeds its final hidden state to fully connected layers; dropout regularization is used, and training uses early stopping based on validation loss.Source: E. LSTM-Based Deepfake Prediction · pp. 5–6

V. EXPERIMENTAL RESULTS

The experiments standardize videos into identity-consistent temporal samples and evaluate the pipeline separately across four datasets. Performance is strong across datasets, though it is lower on the more visually realistic Celeb-DF v2 benchmark.

A. Dataset-Wise Corpus Construction and Standardization

  • Each dataset is processed independently through the same pipeline, so results reflect dataset-specific behavior rather than a merged-corpus aggregate.The datasets span different manipulation pipelines, acquisition conditions, compression levels, and visual realism.Source: A. Dataset-Wise Corpus Construction… · p. 6, B. Dataset-wise Main Results · p. 6
  • Preprocessing standardizes inputs as 20-frame sequences of one tracked face, encoded in 20 × 68 matrices, with video-level splits to prevent temporal leakage.Frames are resampled to 25 fps, valid face crops are resized to 224 × 224, and the sequence stride is 10 frames.Source: A. Dataset-Wise Corpus Construction… · p. 6

B. Dataset-wise Main Results

  • 98.0% F1-score on FaceForensics++, 97.6% on the DFDC subset, 96.2% on DeeperForensics, and 91.0% on Celeb-DF v2.The associated accuracy values are 98.0%, 97.6%, 96.2%, and 91.0%, respectively.Source: Table II · p. 6, B. Dataset-wise Main Results · p. 6
  • The last row of Table II is an unweighted dataset average, not the global test-set result; Table IV is a reference comparison, not a strict leaderboard.Table IV limits comparisons to metrics explicitly reported by prior papers because protocols and evaluation settings differ.Source: Table II · p. 6, B. Dataset-wise Main Results · p. 6, Table IV · p. 7
  • On a balanced illustrative subset, 982 real and 976 fake sequences were correctly classified, while 18 real and 24 fake sequences were misclassified.The matrix may differ slightly from overall dataset-level performance due to sampling effects.Source: B. Dataset-wise Main Results · pp. 6–7, Table III · p. 7

C. Comparison with Recent Literature

The proposed method reports the highest average AUC in its reference comparison, while prior methods show varying performance across datasets. Zero-shot evaluation on ForgeryNet and WildDeepfake tests transfer without retraining, with lower scores on the latter.

C. Comparison with Recent Literature

  • Prior temporal-coherence and spatiotemporal-adaptation methods perform strongly on controlled datasets but degrade on DFDC, while frequency- and temporal-cue approaches vary across datasets.Context-aware methods are reported to improve robustness in multi-face scenarios.Source: C. Comparison with Recent Literature · p. 7
  • The proposed method achieves the highest average AUC (98.0%) and a strong pooled AUC (99.2%), with consistently high precision, recall, and F1-score.Source: C. Comparison with Recent Literature · p. 7

D. Cross-Dataset Generalization

  • ForgeryNet is the validation set and WildDeepfake the held-out test set, evaluated using weights learned from the original four training datasets.The evaluation is described as strictly zero-shot cross-dataset testing.Source: D. Cross-Dataset Generalization · p. 7, Table V · p. 8
  • Without retraining or fine-tuning, the model achieves 93.8% F1-score on ForgeryNet and 89.5% on WildDeepfake.The corresponding accuracy values are 93.8% on ForgeryNet and 89.4% on WildDeepfake.Source: D. Cross-Dataset Generalization · p. 7, Table V · p. 8
  • WildDeepfake’s lower performance is associated with uncontrolled acquisition, repeated post-processing, and compression variability that can weaken or obscure low-level forensic cues.The passage also states that performance is sustained under substantial distribution shifts.Source: D. Cross-Dataset Generalization · pp. 7–8

E. Ablation Study

The ablation favors combining all four forensic feature families, while the conclusion emphasizes interpretable temporal modeling and identifies further directions for extending the framework.

E. Ablation Study

  • Compression and photometric features have the largest impact, while textural and geometric features add complementary cues, especially when post-processing weakens dominant artifacts.The importance ranking is compression > photometric > textural > geometric on both datasets.Source: E. Ablation Study · p. 8
  • All four feature families together perform best on ForgeryNet and WildDeepfake, and removing any one degrades every reported metric.Removing compression features lowers F1-score by 3.2 points on ForgeryNet and 3.9 points on WildDeepfake.Source: Table VII · p. 8, Table VIII · p. 8, E. Ablation Study · p. 8

VI. CONCLUSION AND DISCUSSION

  • The conclusion presents explicit, physically meaningful features and identity-consistent tracking with LSTM aggregation as an interpretable alternative to black-box representations.The features are described as computationally lightweight and explainable without post-hoc methods.Source: §VI · p. 8
  • Extending analysis beyond predominantly frontal faces and incorporating expression dynamics are proposed directions for improving robustness and detecting unnatural affective transitions.The proposed extensions include arbitrary orientations, scales, and poses.Source: §VI · p. 8
  • The discussion proposes graph-based facial-topology reasoning, multimodal signals, Transformer sequence modeling, and hybrid features as further extensions.The proposed hybrid would combine handcrafted features with learned deep representations while retaining transparency and physical grounding.Source: §VI · p. 9
Loading 2610.03380v1…