Source-linked AI summary
Lips Don't Lie: A Generalisable and Robust Approach to Face Forgery Detection
Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, Maja Pantic
TL;DR
Face forgery detectors can fail on unseen manipulations, while some generalisation cues are vulnerable to common perturbations. LipForensics uses lipreading-pretrained mouth representations and temporal modelling to target high-level motion irregularities, reporting stronger generalisation and corruption robustness than prior methods. Its scope is limited when mouths are occluded or unaltered, motion is limited, inputs are isolated images, or large-scale labelled pretraining data is unavailable.
Problem
Face forgery detectors perform well on matched distributions but can substantially underperform on unseen forgeries, while some generalisation cues are vulnerable to post-processing.
Method
LipForensics pretrains a spatio-temporal network for lipreading, freezes its feature extractor, and finetunes a temporal network on mouth embeddings for forgery detection.
Results
LipForensics significantly outperforms prior methods in generalisation to unseen forgeries and robustness to common corruptions, including a 2.2% AUC gain over Face X-ray, 94.9% →97.1%.
Takeaways & Limitations
High-level mouth-motion representations learned through lipreading support face forgery detection that is both generalisable across manipulations and robust to common corruptions.
Takeaways & Limitations
LipForensics cannot be applied to isolated images and may miss videos with occluded or unaltered mouths; it also requires large-scale labelled pretraining data.
Abstract
from arXiv · showhide
Although current deep learning-based face forgery detectors achieve impressive performance in constrained scenarios, they are vulnerable to samples created by unseen manipulation methods. Some recent works show improvements in generalisation but rely on cues that are easily corrupted by common post-processing operations such as compression. In this paper, we propose LipForensics, a detection approach capable of both generalising to novel manipulations and withstanding various distortions. LipForensics targets high-level semantic irregularities in mouth movements, which are common in many generated videos. It consists in first pretraining a spatio-temporal network to perform visual speech recognition (lipreading), thus learning rich internal representations related to natural mouth motion. A temporal network is subsequently finetuned on fixed mouth embeddings of real and forged data in order to detect fake videos based on mouth movements without overfitting to low-level, manipulation-specific artefacts. Extensive experiments show that this simple approach significantly surpasses the state-of-the-art in terms of generalisation to unseen manipulations and robustness to perturbations, as well as shed light on the factors responsible for its performance. Code is available on GitHub.
1. Introduction
Face forgery detectors often fail on unseen manipulations and post-processing, motivating LipForensics, which detects semantic irregularities in mouth motion using lipreading-based representations. Experiments report stronger generalisation and corruption robustness than prior methods.
- Motivation: Deep detectors achieve excellent in-distribution accuracy but can substantially underperform on novel forgery types.This creates a deployment concern because real-world test distributions may differ from training data.
- Motivation: Face X-ray generalises across manipulations but relies on imperceptible patterns susceptible to low-level post-processing.Its blending-boundary cue can therefore be corrupted by routine distortions.
- Motivation: Many face forgeries alter mouth movements, producing irregular phoneme closure, movement speed, mouth shape, or interior appearance.These anomalies arise across identity, speech, and expression manipulations.
- Approach: LipForensics pretrains on visual speech recognition, freezes the feature extractor, and finetunes only a temporal network on mouth embeddings.This design targets rich spatio-temporal mouth representations while discouraging reliance on unstable low-level artefacts.
- Results: LipForensics significantly outperforms previous methods on unseen forgeries and remains robust to common corruptions, including heavy compression.The paper also uses ablations and comparisons with other pretraining tasks to examine the design choices.
2. Related Work
Prior work uses architectural constraints, multimodal correspondence, anomaly modelling, frequency cues, and self-supervised forgery synthesis to improve detection. LipForensics instead uses lipreading-pretrained mouth representations and targets high-level temporal irregularities less tied to manipulation-specific artefacts.
- Face forgery detection: Earlier detectors use mesoscopic features, manipulated-region prediction, frequency anomalies, or deep unconstrained CNNs.These approaches represent several distinct strategies for face forgery detection.
- Face forgery detection: Some methods exploit visual-audio correspondence, whereas LipForensics is visual-only and requires neither audio nor a reference video at inference.The cited biometric alternative requires a reference video, unlike the proposed approach.
- Generalisation to novel manipulations: Generalisation methods include autoencoder-like reconstruction, forgery segmentation, multiband-frequency modelling, and blur or compression augmentation.These methods address performance degradation when training and test manipulations differ.
- Generalisation to novel manipulations: Self-supervised methods synthesize targeted irregularities such as affine warping or blending boundaries to improve cross-dataset generalisation.The approach can encode assumptions about specific steps in forgery pipelines.
- LipForensics: LipForensics passes clips through a lipreading-trained feature extractor that produces embeddings relatively invariant to low-level artefacts.A temporal network then uses these embeddings for forgery detection, as illustrated in Figure 2.
- Generalisation to novel manipulations: Blending-boundary artefacts can be susceptible to common perturbations, limiting the suitability of that strategy for real-life scenarios.This limitation motivates seeking cues that are more semantically high-level.
3. LipForensics
LipForensics detects face forgeries through anomalous mouth movements rather than manipulation-specific low-level artefacts. It pretrains a spatiotemporal lipreading model, then freezes its feature extractor while finetuning temporal forgery classification on mouth embeddings.
- LipForensics targets natural-versus-anomalous mouth movements as forgery evidence that can generalise across generation methods and resist common perturbations.The approach hypothesises that fake videos contain irregular mouth motion and that high-level semantic cues are less easily corrupted.
- The model first jointly trains a spatiotemporal feature extractor, temporal network, and linear classifier for lipreading on real word-utterance videos.Lipreading pretraining uses labelled clips in which each label identifies the spoken word.
- The forgery detector transfers the feature extractor and temporal network, replaces the classifier with a binary classifier, and finetunes the temporal network while keeping the feature extractor fixed.The detector is trained with binary cross entropy on real and fake clips; the fixed extractor is intended to reduce reliance on unwanted artefacts.
- Architecture: The feature extractor is a temporally padded ResNet-18 that produces a 512-D vector per frame, while an MS-TCN combines short- and long-term temporal information before classification.Temporal global average pooling precedes the task-specific linear classifier.
4. Experiments
Experiments evaluate LipForensics across datasets, unseen manipulations, corruptions, compression settings, and ablations. Results show strong generalisation and robustness, while isolating lipreading pretraining, frozen features, and temporal modelling as important design factors.
- Experimental setup: Experiments use FaceForensics++ and additional datasets, reporting accuracy and video-level AUC for fair comparison.FaceForensics++ provides diverse training forgeries; evaluation includes DeeperForensics, FaceShifter, Celeb-DF-v2, and DFDC.
- Cross-manipulation generalisation: LipForensics improves cross-manipulation generalisation, outperforming Face X-ray by 2.2% AUC, from 94.9% to 97.1%.FaceSwap remains the exception for Face X-ray, but LipForensics reaches 90.1% AUC without assuming a blending boundary.
- Cross-dataset generalisation: LipForensics surpasses all compared methods on every cross-dataset benchmark, with especially strong results on FaceShifter and DeeperForensics.Lipreading pretraining adds 8.4% on average and freezing the feature extractor adds 4.9%; DFDC remains difficult under domain shift.
- Robustness to unseen perturbations: LipForensics remains robust to blur, pixelation, and compression except at the most severe levels, while other methods deteriorate substantially.It is more robust than previous methods to all tested corruptions except block-wise distortions, which affect high-level semantic content.
- Learning on compressed data: On compressed data, LipForensics outperforms other methods because it targets high-level spatio-temporal cues rather than fragile blending artefacts.Face X-ray suffers the most under compression, whereas LipForensics is substantially less affected.
- Ablation study: Increasing forgery diversity reaches 95.1% AUC on FaceSwap, while freezing the feature extractor outperforms finetuning the whole network.Without lipreading pretraining, the model does not achieve adequate generalisation.
- Ablation study: High-level temporal information is crucial for generalisation, with short-term dynamics helping FaceShifter but not DeeperForensics.Large improvements occur when the MS-TCN is added and when the ResNet feature extractor remains frozen.
- Ablation study: Lipreading pretraining outperforms pretraining on Kinetics-400, IG-65M, and face-recognition datasets for cross-manipulation generalisation.The comparison finetunes models on FF++ and tests them on FaceShifter and DeeperForensics.
5. Conclusion
LipForensics detects forged face videos by targeting semantically high-level inconsistencies in mouth movements, using representations learned through lipreading. It achieves strong generalisation to unseen forgery types and robustness to common corruptions.
- LipForensics targets inconsistencies in semantically high-level mouth movements using representations learned via lipreading.The approach is designed to detect forged face videos through mouth-motion evidence.
- LipForensics achieves state-of-the-art generalisation to unseen forgery types.
- LipForensics is significantly more robust than other methods to various common corruptions.
A.1. Architecture details
The architecture uses a multi-scale temporal convolutional network block composed of one-dimensional convolutions, batch normalisation, PReLU activations, and dropout.
- The MS-TCN block uses Conv1D layers with specified output channels and kernel sizes, same padding, and stride 1.
- BatchNorm1D applies one-dimensional batch normalisation with momentum 0.1.
- PReLU uses a separate learnable parameter for each input channel.
- Dropout(x) applies dropout with probability x.
A.2. Datasets
The experiments use multiple face-forgery datasets, standardised face and mouth preprocessing, and implementations of several detector baselines and pretraining alternatives.
- FF++ uses the provided training, validation, and test splits, while FaceShifter samples are downloaded separately at c23 compression.
- Face crops use RetinaFace detection, the largest face, and an enlarged crop; mouth crops use FAN landmarks, smoothing, affine warping, and resizing.
- Face X-ray training combines blended images with FF++ fake frames and applies random flipping, JPEG compression, and pixelation augmentations.
- The experiments use official code or pretrained models for several baselines, with implementation details varying by method.
- Robustness experiments include seven perturbation types evaluated at five severity levels.
- Baseline implementations include CNN-GRU, Multi-task, DSP-FWA, CNN-aug, Patch-based, Xception, R(2+1)D-18, ip-CSN-152, and SE-ResNet50.
- Table 7 compares tight full-face crops with mouth crops using video-level accuracy on FaceShifter and DeeperForensics after training on FF++.
A.5. Robustness experiments
The robustness experiments apply corruptions using the DeeperForensics code and evaluate all considered corruptions across all severity levels.
- The robustness experiments use the DeeperForensics code to apply the considered corruptions.
- All considered corruptions are evaluated at all severity levels and depicted in Figure 8.
B. Full Face Versus Mouth Crops
The appendix compares larger face inputs with mouth crops and shows that lipreading pretraining changes which input choice generalises best. With pretraining, mouth crops outperform full-face inputs, whereas training from scratch favors full faces on the reported datasets.
- Input design: When training from scratch, full faces generalise better than mouth crops to FaceShifter and DeeperForensics.
- Input design: With lipreading pretraining, mouth crops generalise better than full-face inputs.
- Pretraining: Lipreading pretraining significantly improves accuracy for both full-face and mouth-crop inputs.
C. Qualitative Analysis
The qualitative analysis shows that LipForensics focuses on semantically meaningful mouth irregularities and consistently uses the mouth region, unlike comparison models. Failure cases are associated with difficult motion, pose, or movement conditions.
- Failure cases: Failure cases commonly involve rapid head movements, uncommon poses relative to FF++, or very limited mouth movements.
- Occlusion sensitivity: Occlusion sensitivity visualises model reliance by covering frame regions and recording the resulting correct-class probabilities as a heatmap.The procedure uses a 40×40×t block, shifts it pixel by pixel, averages probabilities, and overlays the normalised heatmap on the first frame.
- Occlusion sensitivity: Unlike Xception, LipForensics consistently relies on the mouth region, while a spatiotemporal network trained without lipreading pretraining often relies on other regions such as the nose.This non-mouth reliance is especially observed for Deepfakes and FaceSwap.
- Qualitative Analysis: Mouth anomalies include insufficient closure, limited yet anomalous movement, inconsistent teeth or lip shape, and temporal irregularities.Subtle anomalies are easier to observe in video than in individual frames.