PaperlayerView exact revision on arXiv

Source-linked AI summary

Exposing Deep Fakes Using Inconsistent Head Poses

Xin Yang, Yuezun Li, Siwei Lyu

arXiv:1811.00661v2cs.CV

TL;DR

Deep Fakes create a detection challenge because synthesized face regions can introduce landmark inconsistencies when spliced into original imagery. The paper detects this cue by comparing 3D head poses estimated from whole-face and central-face landmarks, then classifies the resulting differences with an SVM. Experiments report effective discrimination on real and Deep Fake image and video datasets.

  • Problem

    Deep Fakes create a serious misinformation problem, while existing detectors have focused on synthesis artifacts or direct neural-network classification.

  • Method

    The method compares 3D head poses estimated from whole-face and central-region landmarks, using their differences as SVM features.

  • Results

    The SVM achieves AUROC 0.89 on UADFV and 0.843 on the DARPA GAN Challenge dataset for frame-level classification.

  • Takeaways & Limitations

    Landmark errors introduced by splicing synthesized faces can be revealed through inconsistent estimated 3D head poses and used for Deep Fake detection.

Abstract

from arXiv · show

In this paper, we propose a new method to expose AI-generated fake face images or videos (commonly known as the Deep Fakes). Our method is based on the observations that Deep Fakes are created by splicing synthesized face region into the original image, and in doing so, introducing errors that can be revealed when 3D head poses are estimated from the face images. We perform experiments to demonstrate this phenomenon and further develop a classification method based on this cue. Using features based on this cue, an SVM classifier is evaluated using a set of real face images and Deep Fakes.

1. INTRODUCTION

Deep Fakes pose a serious social problem, creating an urgent need for effective detection. The paper proposes exposing them through landmark inconsistencies revealed by comparing head poses from different face regions.

  • Deep Fakes insert faces synthesized by deep neural networks into original images or videos, creating digital impersonations with negative social impact.
  • Existing detection methods have targeted synthesis artifacts such as unrealistic eye blinking, mismatched color profiles, or direct neural-network classification.
  • The proposed cue is a mismatch between facial landmarks of the original and synthesized faces, which may arise because synthesis models preserve expression but not landmark consistency.
  • Comparing head poses estimated from all landmarks and central-region landmarks can reveal errors that are not directly visible to human observers.
  • The authors use differences between the two estimated head poses as feature vectors for an SVM classifier distinguishing original images from Deep Fakes.

2. DEEP FAKE PRODUCTION PIPELINE

Deep Fakes are produced by aligning, synthesizing, and reinserting a cropped face, while the paper estimates 3D head pose from facial landmarks to support detection. The pose-estimation procedure recovers camera transformation parameters by fitting 3D landmark coordinates to their 2D image coordinates.

  • The production pipeline detects a face and landmarks, warps the face using affine transformation M, synthesizes a 64 × 64 face, reverses the warp with M −1, and applies post-processing.
  • 3D head pose represents the rotation and translation from world coordinates to camera coordinates.
  • The overview contrasts real and fake images by comparing poses estimated from whole-face landmarks with poses estimated from central-face landmarks.
  • The camera-coordinate transformation uses a 3 × 3 rotation matrix R and a 3 × 1 translation vector t.
  • Head-pose estimation solves for scale s, rotation R, and translation t from corresponding 2D image and 3D world coordinates of facial landmarks.
  • The optimization is solved with the Levenberg-Marquardt algorithm, and the head pose is obtained by reversing the estimated camera rotation as R^T.

4. INCONSISTENT HEAD POSES IN DEEP FAKES

Deep Fake face swapping can leave central facial landmarks inconsistent with the unchanged outer face contour, producing discrepant head poses. Comparing whole-face and central-region orientation vectors reveals this discrepancy, with cosine-distance distributions separating real and fake images.

  • INCONSISTENT HEAD POSES IN DEEP FAKES: The method compares head poses estimated from all facial landmarks with those estimated from central-region landmarks.The orientation vectors are derived from the corresponding estimated rotations.
  • INCONSISTENT HEAD POSES IN DEEP FAKES: Face swapping can shift central landmarks while leaving outer-contour landmarks unchanged, creating a mismatch in fake faces.Across 795 image pairs, the mean landmark shift was 1.540 pixels with a standard deviation of 0.921 pixels.
  • INCONSISTENT HEAD POSES IN DEEP FAKES: Cosine distance between the two orientation vectors is computed as 1 − va·vc/(∥va∥∥vc∥), where zero means agreement.The distance lies in [0, 2], and smaller values indicate closer vectors.
  • INCONSISTENT HEAD POSES IN DEEP FAKES: Real images have cosine distances concentrated up to 0.02, whereas most Deep Fake values fall between 0.02 and 0.08.The figure presents these distributions for fake and real face images.

5. CLASSIFICATION BASED ON HEAD POSES

The paper trains RBF-kernel SVM classifiers on differences between whole-face and central-region 3D head poses using two real-versus-Deep-Fake datasets. AUROC reaches 0.89 on UADFV and 0.843 on the DARPA GAN Challenge dataset, while richer pose features improve ablation performance.

  • CLASSIFICATION BASED ON HEAD POSES: An SVM uses differences between head poses estimated from 68 whole-face and central-region landmarks as classification features.The pose estimates use standard 3D landmark models, with RBF kernels and five-fold cross-validation for training.
  • CLASSIFICATION BASED ON HEAD POSES: The evaluation uses UADFV real and Deep Fake videos plus real and Deep Fake images from the DARPA MediFor GAN Challenge.UADFV training uses 21,694 frames, while held-out UADFV frames and all DARPA GAN images are used for testing.
  • CLASSIFICATION BASED ON HEAD POSES: 0.89 AUROC is achieved on UADFV, while 0.843 AUROC is achieved on the DARPA GAN Challenge dataset using individual frames.The authors report AUROC as the performance metric.
  • CLASSIFICATION BASED ON HEAD POSES: Feature ablation on the Deep Fake dataset improves AUROC from 0.738 with orientation differences to 0.840 with rotation-matrix differences.Adding translation differences yields 0.866 and 0.890 AUROC for the corresponding feature combinations.

6. CONCLUSION

The paper proposes exposing Deep Fakes through errors introduced when synthesized face regions are spliced into original images and detected via 3D head-pose estimation. Experiments demonstrate this cue and support an SVM classification method.

  • The method detects Deep Fakes by revealing splicing-induced errors through 3D head-pose estimation.The cue arises from synthesized face regions being inserted into original images.
  • Experiments demonstrate the head-pose inconsistency phenomenon and support a classification method based on this cue.
  • The proposed approach is evaluated on real face images and Deep Fakes using SVM classification.
Loading 1811.00661v2…