Source-linked AI summary
In Ictu Oculi: Exposing AI Generated Fake Face Videos by Detecting Eye Blinking
Yuezun Li, Ming-Ching Chang, Siwei Lyu
TL;DR
Realistic AI-generated face videos challenge conventional forensic cues, motivating detection methods based on physiological signals that synthesis may omit. This paper detects missing eye blinking with an LRCN-based video pipeline, and reports promising performance on DeepFake videos, including an ROC result of 0.99 on its testing data.
Problem
AI-generated fake face videos challenge traditional signal-, physical-, and semantic-level forensic methods, creating a need for dedicated detection approaches.
Method
The method detects eye blinking by aligning faces, extracting eye-region sequences, and applying an LRCN model that captures temporal blinking dynamics.
Results
0.99 ROC performance was reported for LRCN on the authors’ testing data, compared with 0.98 for CNN and 0.79 for EAR.
Takeaways & Limitations
Eye blinking can provide a physiological cue for exposing neural-network-generated fake face videos, with promising performance reported on DeepFake content.
Takeaways & Limitations
Sophisticated forgers may create realistic blinking through post-processing, advanced models, and more training data.
Abstract
from arXiv · showhide
The new developments in deep generative networks have significantly improve the quality and efficiency in generating realistically-looking fake face videos. In this work, we describe a new method to expose fake face videos generated with neural networks. Our method is based on detection of eye blinking in the videos, which is a physiological signal that is not well presented in the synthesized fake videos. Our method is tested over benchmarks of eye-blinking detection datasets and also show promising performance on detecting videos generated with DeepFake.
I. INTRODUCTION
Advances in generative deep neural networks have made realistic fake face videos easier to synthesize, creating an urgent need for forensic detection methods. The paper targets these videos by detecting missing physiological signals, especially eye blinking.
- Motivation: Digital video tampering has become easier to create, edit, and propagate, enabling falsified information.The spread of cellphones, social networks, and video-sharing portals contributes to this broader risk.
- Motivation: Generative deep neural networks can synthesize realistic face videos from large training datasets with minimal manual editing.DeepFake uses GANs to replace one subject’s face with another, potentially falsifying identity and violating privacy.
- Research gap: Detecting AI-generated fake videos is presented as a pressing need for digital media forensics.The motivation is linked to realistic synthetic videos that can falsify a subject’s identity.
- Research gap: Traditional signal-, physical-, and semantic-level forensic cues face challenges when applied to AI-generated fake face videos.The paper therefore focuses on physiological signals that synthesis processes may overlook.
- Approach: The proposed method detects missing eye blinking using a deep learning model combining CNN-based appearance features with RNN-based temporal modeling.Eye blinking is treated as a physiological signal that is not well captured in synthesized videos.
II. BACKGROUNDS AND RELATED WORKS
Eye blinking is a physiological eyelid movement with variable timing, while generative models can omit closed-eye examples and produce videos lacking this cue. The paper situates blinking detection within broader GAN-based video synthesis and forensic research.
- Eye blinking: Blinking is the rapid closing and opening movement of the eyelid, with spontaneous, reflex, and voluntary forms.The paper describes spontaneous blinking as occurring without conscious effort and serving corneal and conjunctival maintenance.
- Illustration: An original video shows a detectable blink within 6 seconds, whereas the DeepFake example does not.The caption characterizes the latter absence as physiologically abnormal.
- Eye blinking: AI-generated faces may lack eye blinking because training datasets often contain few faces with closed eyes.The paper identifies absent blinking as a telltale sign of synthetic face videos.
- Generative models: GAN research has developed increasingly capable image-generation methods through adversarial training and architectural extensions.The cited work includes Laplacian pyramid GANs, DCGANs, Wasserstein objectives, and conditional adversarial networks.
B. Eye Blinking Detection
The paper extends CNN-based eye-state classification with LRCN, modeling blinking as a temporal process rather than relying only on individual frames. Its pipeline detects and aligns faces, extracts eye sequences, and predicts eye openness over time.
- Prior work: Eye blinking detection has been studied for fatigue detection and face spoof detection using several computer-vision methods.Prior approaches include conditional random fields, active shape models, frame differences, optical flow, and parametric eye-shape models.
- Proposed model: LRCN incorporates temporal relationships between consecutive frames to model blinking from opening to closing and reduce noise from single images.This extends CNN-based classification by using long-term dynamics.
- Pipeline: The pipeline detects faces, aligns them to discount head movement and orientation changes, extracts eye regions, and applies LRCN to quantify eye openness.The resulting eye-region sequence provides the input for blinking detection.
A. Pre-processing
Pre-processing converts video frames into aligned eye-region sequences so that head motion and orientation changes do not dominate blinking analysis. The procedure detects facial landmarks, normalizes faces, and crops enlarged eye regions.
- Face detection and landmarks: Facial landmarks are extracted from detected face areas to locate structural features including the eyes, nose, mouth, and cheek contours.These landmarks provide the basis for subsequent face alignment.
- Face alignment: Landmark-based alignment maps faces into a unified coordinate space to reduce distractions from head movement and changing orientation.The transformed face is centered, rotated so the eyes are horizontal, and scaled similarly.
- Scope: The method focuses on one subject per video frame, although the authors state that multiple subjects can be handled.This is an explicit scope assumption of the described analysis.
- Eye-region extraction: Eye crops are formed from landmark bounding boxes enlarged by 1.25 horizontally and 1.75 vertically.The enlargement is intended to ensure that the eye region remains inside each cropped input.
B. Long-term Recurrent Convolutional Networks (LRCN)
LRCN combines CNN-based eye-feature extraction, recurrent sequence learning, and state prediction to model the temporal dependencies of blinking.
- LRCN captures temporal dependencies in human eye blinking, which progresses from opening to closing across frames.
- The model contains feature extraction, sequence learning, and state prediction components.Its feature extractor uses a VGG16-based CNN without fc7 and fc8 layers.
- The LSTM updates memory through forget, input, and output gates, with the cell state and hidden state carried through time.The implementation uses 256 hidden units in the LSTM cell.
- Aligned eye-region sequences are processed by LRCN for dynamic eye-state prediction.
- A fully connected layer converts each LSTM output into probabilities for eye-open and eye-closed states, represented by 0 and 1.
C. Model Training
LRCN training proceeds in two stages: first learning image-based eye states, then jointly training recurrent and prediction components.
- The VGG-based CNN is first trained on labeled eye regions showing open and closed eyes.Training uses back-propagation with stochastic gradient descent and dropout probability 0.5 in fully connected layers.
- The LSTM-RNN and fully connected components are trained jointly in the second stage using back-propagation.
- Cross entropy loss is used for the binary open-versus-closed eye classification objective.
IV. EXPERIMENTS
The experiments train LRCN on eye-state image datasets and test blinking detection on authentic videos and DeepFake-generated videos, including a collected video set.
- The LRCN model is trained on image datasets containing open-eye states and evaluated for blinking detection in authentic and DeepFake videos.
- Because blinking is temporal, the study notes the lack of an accessible video dataset specifically designed for eye-blinking detection.The CEW dataset provides 1,193 closed-eye images and 1,232 open-eye images, but is an image dataset.
B. Generating Fake Videos
The authors generate DeepFake videos with alignment, landmark-guided masking, and boundary smoothing to reduce visible compositing artifacts.
- The generation pipeline detects faces, extracts landmarks, aligns faces, and uses aligned faces as inputs to DeepFake.
- The cited ZJU Eyeblink Video Database exists but is inaccessible for evaluation.
- Directly affine-warping a generated rectangular face back into the source image leaves visible boundary color differences.
- A convex polygon mask retains generated content inside facial landmarks, and Gaussian blur smooths the mask boundary.
- This post-processing procedure produces 49 fake videos.
1) Data preparation:
The method prepares eye-image and eye-sequence data for CNN and LRCN training, using augmentation and a pretrained VGG16 feature extractor.
- Eye sequences are generated by cropping eye areas from each video frame.
- Training augmentation includes horizontal flipping and changes to contrast, brightness, and color distortion.Sequence augmentation is kept consistent to preserve temporal relationships.
- VGG16 is trained as a binary eye-state classifier using cropped eye images and the CEW dataset.The input size is 224x224, with batch size 16 and training capped at 100 epochs.
- ROC curves are illustrated for CNN, LRCN, and EAR.
- The trained VGG16 replaces its fc7 and fc8 layers to provide LRCN features.
- LRCN training uses variable-length sequences of 10 to 20 eye images containing at least one blink, while CNN parameters remain fixed.The remaining LSTM and fully connected layers are trained for up to 100 epochs.
C. Evaluations
The evaluation compares LRCN, CNN, and EAR for eye-state detection. LRCN achieves the strongest reported performance by incorporating temporal dynamics, while EAR depends on potentially unreliable landmarks.
- EAR analyzes eye state from landmark-based ratios but depends fully on eye landmarks, which are unreliable in many cases.
- 0.99 is the reported performance for LRCN, compared with 0.98 for CNN and 0.79 for EAR.These values are shown for the three methods' ROC-curve evaluation on the authors' testing data.
- CNN distinguishes eye state from individual images but does not consider temporal knowledge from previous states.
- LRCN uses long-term dynamics to produce smoother and more accurate eye-state predictions when individual eye images are ambiguous.Its temporal information can help infer subsequent states from prior blinking or closing trends.
V. CONCLUSION
The paper detects AI-generated fake face videos through eye blinking, a physiological signal that synthesized videos may not capture well. It reports benchmark testing and promising DeepFake-detection performance, while identifying remaining limitations and future directions.
- Eye blinking is used as a physiological cue for exposing neural-network-generated fake face videos.The method targets a signal that is not well presented in synthesized videos.
- The method is tested on eye-blinking detection benchmarks and shows promising performance on DeepFake-generated videos.
- The current method uses only the lack of blinking as a detection cue.
- Future work includes modeling physiologically unlikely blinking dynamics and exploring other physiological signals.The paper also proposes investigating other deep neural-network architectures for detecting closed eyes.
- Sophisticated forgers may create realistic blinking effects through post-processing, advanced models, and more training data.
- LRCN produces smoother and more accurate results than CNN in the illustrated Trump-video comparison.