PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Real Talking Faces via Self-Supervision for Robust Forgery Detection

Alexandros Haliassos, Rodrigo Mira, Stavros Petridis, Maja Pantic

arXiv:2201.07131v3cs.CV

TL;DR

Face forgery detectors struggle with unseen manipulation methods and common corruptions such as compression. RealForensics learns temporally dense facial representations from audiovisual correspondence in unlabelled real talking-face videos, then uses them as auxiliary targets during forgery classification. The paper reports strong cross-manipulation generalisation and corruption robustness, while noting higher training cost, video-only input, and calibration concerns.

  • Problem

    Face forgery detectors often generalise poorly to unseen manipulation methods and may rely on low-level cues vulnerable to common corruptions.

  • Method

    RealForensics uses cross-modal self-supervision on real talking-face videos to learn temporally dense video representations, then predicts them alongside binary forgery classification.

  • Results

    RealForensics achieves strong cross-manipulation generalisation and robustness to common corruptions, with state-of-the-art performance reported in the evaluated experiments.

  • Takeaways & Limitations

    Large amounts of unlabelled real talking-face video are a promising resource for developing more robust face forgery detectors.

  • Takeaways & Limitations

    The method requires higher training-time computation, accepts videos rather than single images, and can produce confidently wrong predictions.

Abstract

from arXiv · show

One of the most pressing challenges for the detection of face-manipulated videos is generalising to forgery methods not seen during training while remaining effective under common corruptions such as compression. In this paper, we examine whether we can tackle this issue by harnessing videos of real talking faces, which contain rich information on natural facial appearance and behaviour and are readily available in large quantities online. Our method, termed RealForensics, consists of two stages. First, we exploit the natural correspondence between the visual and auditory modalities in real videos to learn, in a self-supervised cross-modal manner, temporally dense video representations that capture factors such as facial movements, expression, and identity. Second, we use these learned representations as targets to be predicted by our forgery detector along with the usual binary forgery classification task; this encourages it to base its real/fake decision on said factors. We show that our method achieves state-of-the-art performance on cross-manipulation generalisation and robustness experiments, and examine the factors that contribute to its performance. Our results suggest that leveraging natural and unlabelled videos is a promising direction for the development of more robust face forgery detectors.

1. Introduction

Face forgery detectors often fail to generalise to unseen manipulation methods and can rely on low-level cues vulnerable to compression. RealForensics uses unlabelled real talking-face videos to learn facial behaviour and appearance representations, then uses them to guide forgery detection.

  • Detectors can achieve high in-distribution accuracy but often perform poorly on manipulation methods absent from training.
  • Frame-based approaches still underperform on novel forgeries or depend on low-level cues corrupted by compression.
  • High-level facial movements, expressions, and form changes may remain more resilient to compression or blurring than low-level content.
  • RealForensics first learns temporally dense video representations through self-supervised visual-auditory correspondence in natural videos.
  • The detector jointly performs forgery classification and predicts learned video representations, encouraging attention to facial behaviour and appearance cues.
  • The paper reports state-of-the-art cross-manipulation generalisation and robustness to common corruptions.

2. Related Works

Prior work addresses unseen-forgery generalisation through architectural constraints, augmentation, auxiliary tasks, frequency cues, self-consistency, and temporal modelling. Related audiovisual and self-supervised methods motivate RealForensics’ use of temporally dense cross-modal supervision with real videos.

  • Cross-manipulation generalisation: Cross-manipulation methods include augmentation, reconstruction, frequency mining, classifier truncation, self-consistency, and spatio-temporal networks.
  • Cross-manipulation generalisation: LipForensics improves generalisation and corruption robustness but requires labelled pre-training and focuses only on the mouth region.
  • Cross-manipulation generalisation: RealForensics uses audio only for cross-modal supervision and remains visual-only at test time.
  • Audiovisual self-supervision: Audiovisual action-recognition methods commonly learn one representation per clip, which is less suited to fine-grained speaking-face movements.
  • Generalisation via self-supervision: Self-supervised auxiliary tasks have previously been used to improve generalisation on a main task.

3. Method

RealForensics learns dense facial representations from paired real video and audio, then transfers the video representation to a multi-task detector. The detector classifies real versus fake videos while matching frozen teacher targets on real samples.

  • Stage 1: representation learning: Stage 1 learns temporally dense embeddings from real talking faces using cross-modal audiovisual student-teacher self-supervision.
  • Stage 1: representation learning: Teacher networks generate modality-specific targets, while students predict targets from the other modality using corresponding dense embeddings.
  • Stage 1: representation learning: The student-teacher objective uses stop-gradient targets and exponential-moving-average teacher updates without contrasting negatives.
  • Stage 2: multi-task forgery detection: Only real videos contribute to the auxiliary target-prediction loss, with the stage-1 video teacher frozen.
  • Stage 1: representation learning: Random masking improves representations by forcing students to infer missing information from context rather than relying on specific input features.
  • Stage 2: multi-task forgery detection: Stage 2 initializes the detector backbone from the video student and jointly performs binary forgery classification and video-target prediction.

4. Experiments

The experiments evaluate RealForensics on unseen manipulations, cross-dataset transfer, parameter efficiency, and robustness to common video corruptions. Across these settings, it achieves strong generalisation and corruption robustness without auxiliary labelled supervision.

  • Experimental setup: The evaluation uses LRW as an unlabelled auxiliary dataset and reports video-level accuracy and AUC metrics across several forgery datasets.LRW contains 500,000 talking-face videos, while the forgery benchmarks include FF++, FaceShifter, DeeperForensics, CelebDF-v2, and DFDC.
  • Cross-manipulation generalisation: Table 1 measures cross-manipulation generalisation by training on some FF++ manipulation types and testing on the remaining type.The evaluated manipulation types are Deepfakes, FaceSwap, Face2Face, and NeuralTextures.
  • Cross-manipulation generalisation: RealForensics performs on par with state-of-the-art methods without auxiliary labelled supervision, extensive network constraints, or audio at test-time.It also outperforms a CSN baseline trained directly on forgery data with the same augmentations.
  • Cross-dataset generalisation: RealForensics achieves state-of-the-art AUC on CelebDF-v2, DFDC, FaceShifter, and DeeperForensics after training on FF++ alone.It also beats the CSN baseline by a large margin and obtains higher generalisation accuracy on FaceShifter and DeeperForensics than related methods, with fewer test-time parameters.
  • Robustness to common corruptions: RealForensics suffers significantly less from saturation, contrast, occlusion, noise, blur, pixelation, and compression than compared methods.The robustness evaluation averages AUC across five intensity levels for each corruption type.

5. Ablations

The ablations examine the framework, representation-learning choices, auxiliary-data scale, and auxiliary-dataset identity. They show that the two-stage design, dense representations, auxiliary loss, and abundant real talking-face data contribute to generalisation.

  • Framework ablation: Removing the two-stage framework lowers accuracy by about 14%, while omitting the stage-2 auxiliary loss lowers it by about 2%.The auxiliary loss requires the detector to predict learned video representations alongside forgery classification, providing a positive regularisation effect.
  • Representation learning ablation: Dense representations outperform global representations, while dense learning with a predictor does not benefit from adding negatives.Without negatives and without a predictor, representation collapse is observed; global learning without negatives also struggles to achieve competitive performance.
  • Effect of number of real samples: RealForensics benefits substantially from increasing the number of unlabelled LRW real samples, more than an imbalanced-classification baseline.The baseline also improves with more real samples, but its increase in generalisation is significantly smaller.
  • Using a different auxiliary dataset: Using VoxCeleb2 provides about 1 million talking-face videos with varied identities as an alternative source of auxiliary real samples.The method is trained with the same hyperparameters as for LRW, and results are reported on CelebDF-v2, DFDC, and other datasets.

6. Limitations / Societal Impact

RealForensics has higher training-time computational demands and only operates on videos, while its predictions can be confidently wrong. The paper also cautions against relying on a single detection method in deployment.

  • Higher training-time computational demands are the cost of RealForensics’ strong generalisation, although test-time demands are unaffected.
  • The detector accepts videos rather than single images, defining a scope boundary for its use.
  • Confidently wrong predictions mean the model’s output probabilities should be interpreted with care.
  • Deployed systems should use an ensemble of independent approaches rather than relying exclusively on one detection method.

7. Conclusion

The paper concludes that RealForensics uses large amounts of unlabelled real data to detect fake videos. It reports strong cross-manipulation generalisation and robustness to common corruptions, and encourages further research using real faces.

  • RealForensics uses large amounts of unlabelled real data to detect fake videos.
  • The method simultaneously achieves strong cross-manipulation generalisation and robustness to common corruptions.
  • The authors hope the study encourages future research on leveraging real faces for robust forgery detection.

A. More Experiments

Additional experiments show that RealForensics is competitive in-distribution and generalises strongly to the ForgeryNet dataset. It performs especially well under higher compression compared with other methods.

  • A.1. In-distribution performance: RealForensics is on par with state-of-the-art methods under no or low compression in in-distribution evaluation.Training and testing use corresponding splits at raw, c23, and c40 compression levels.
  • A.1. In-distribution performance: RealForensics outperforms other methods on more compressed in-distribution data.
  • A.2. Generalisation to ForgeryNet: RealForensics significantly outperforms publicly available LipForensics and FTCN models on generalisation to ForgeryNet.All compared models were trained on FaceForensics++.

A.3. Detailed analysis of robustness

RealForensics degrades less abruptly as corruption severity increases, with particularly strong results for video compression and block-wise distortions. Full-face processing helps when the mouth is occluded.

  • A.3. Detailed analysis of robustness: RealForensics’ AUC deteriorates less abruptly than other methods as corruption severity increases.The comparison plots AUC as a function of corruption severity.
  • A.3. Detailed analysis of robustness: Video compression produces especially noteworthy robustness results for RealForensics.Video compression is described as ubiquitous on social media.
  • A.3. Detailed analysis of robustness: RealForensics achieves significantly higher results than LipForensics on block-wise distortions such as occlusions.This may be influenced by using the whole face rather than only the mouth, since other facial regions can remain visible when the mouth is occluded.
  • A.3. Detailed analysis of robustness: Figure 6 illustrates a clean FaceForensics++ frame alongside versions corrupted by various perturbations.

A.4. More ablations

The ablations examine how input region, clip size, backbone, predictor design, contrastive learning, and audiovisual inputs affect RealForensics generalisation. Results favor full-face inputs, temporally aware prediction, and cross-modal representation learning.

  • Full face versus mouth: Full-face training consistently outperforms mouth-crop training for RealForensics, unlike the reported result for LipForensics.The comparison covers both representation-learning stages and uses Table 9.
  • Effect of clip size: Generalisation improves as video clip size increases, up to a point.Table 10 evaluates accuracy on FaceShifter and DeeperForensics after training on FaceForensics++.
  • Different backbone: RealForensics significantly outperforms LipForensics with the same backbone and auxiliary dataset, without auxiliary labels.The comparison uses a ResNet+MS-TCN backbone and evaluates FaceShifter and DeeperForensics.
  • Projector and predictor: A transformer predictor improves results over the MLP variant, while the 1-block transformer performs slightly better than the 2-block version.The comparison concerns the stage-1 dense representation-learning predictor.
  • Different contrastive baselines: The proposed non-contrastive method is better suited for forgery detection than the evaluated Perfect Match contrastive baselines.The baselines use audiovisual contrastive learning, including the symmetrised Perfect Match++ variant.
  • Visual-only representation learning: Cross-modal representation learning strongly benefits generalisation compared with visual-only BYOL-style learning.The authors attribute this likely advantage to audiovisual correspondence providing a richer signal for natural facial movements and expressions.

B. Further Implementation Details

The implementation uses standardised face alignment and several established forgery datasets, with FaceForensics++ providing the main split and multiple datasets supplying manipulation variants.

  • Preprocessing: Faces are detected with RetinaFace, landmarks are extracted with a 2-D FAN, and frames are aligned to LRW’s mean face using an affine warp.Landmark means are computed over a 12-frame window and eight stable points are used for the warp.
  • Datasets: All listed dataset licenses permit research use.The paper provides additional dataset details for reproducibility.
  • Datasets: FaceForensics++ uses official train, validation, and test splits containing 720, 140, and 140 videos, respectively.These are the dataset splits used in the experiments.
  • Datasets: FaceShifter and DeeperForensics use real videos originating from FaceForensics++ at compression c23.FaceShifter is treated separately from FaceForensics++, consistently with the original paper.
  • Datasets: CelebFD-v2 is used from its official webpage, while DFDC experiments use a single-subject subset with successful face and landmark detection.The DFDC subset excludes videos where detectors failed, often because of extreme perturbations.

B.3. Architecture and training details

The architecture preserves temporal information in modified video and audio backbones and combines cosine-based supervised learning with dense audiovisual representation-learning components. Visualisation uses occlusion sensitivity to inspect which facial regions influence predictions.

  • Supervised loss: The supervised head uses a cosine classifier with logit adjustment, and the supervised loss is standard binary cross entropy on the adjusted scores.The cosine score scales similarity by s = 64 and uses π as the fake-sample prior.
  • Training augmentation: Random masking independently augments video frames, video-frame sequences, audio frames, and mel filters.The procedure applies random erasing with probability 0.5 and samples bounded ranges for erased elements.
  • Backbones: The modified CSN video backbone uses temporal stride 1 throughout to preserve the temporal dimension.The architecture is selected for video-recognition accuracy and relatively low parameter count.
  • Backbones: The modified ResNet18 audio backbone subsamples time by four in its stem and performs no further temporal subsampling.Its temporal output is matched to that of the video backbone.
  • Projector and predictor: Ablation MLPs use a 3-layer projector and 2-layer predictor with specified hidden and output dimensions and no final ReLU activations.Batch normalisation is applied after projector layers, while the predictor output has neither batch normalisation nor ReLU.
  • Contrastive experiments: Dense contrastive experiments produce 25 frame-level embeddings and randomly enqueue one embedding as a negative sample.Shuffling batch normalisation is used to prevent cheating on the pretext task.
  • Visualisation: Occlusion sensitivity heatmaps measure how sliding-window erasures affect predictions; expression-editing methods usually highlight the mouth, whereas face swaps vary across facial regions.The examples are reported for FaceForensics++ manipulations.
Loading 2201.07131v3…