PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-Supervised Video Forensics by Audio-Visual Anomaly Detection

Chao Feng, Ziyang Chen, Andrew Owens

arXiv:2301.01767v2cs.CV

TL;DR

Manipulated speech videos can contain subtle audio-visual inconsistencies, while labeled datasets cannot cover every real-world manipulation. The paper learns autoregressive distributions over synchronization features using only real, unlabeled video and flags low-probability sequences. Despite training entirely on real videos, it achieves strong performance on manipulated speech-video detection.

  • Problem

    Labeled datasets cannot fully capture the possible manipulations encountered in the wild, motivating detection methods trained from real data.

  • Method

    The method learns autoregressive distributions over audio-visual synchronization features from real, unlabeled videos and flags low-probability sequences.

  • Results

    The model obtains strong performance on FakeAVCeleb and KoDF despite training entirely on real videos, with AP over 90% in most evaluated categories.

  • Takeaways & Limitations

    Audio-visual synchronization features support self-supervised, zero-shot fake-video detection that generalizes across spoken languages and remains robust to compression and blurring.

  • Takeaways & Limitations

    The synchronization-based design is not well suited to manipulations that preserve motion-sound synchronization, such as changing a speaker’s appearance without substantially changing mouth motion.

Abstract

from arXiv · show

Manipulated videos often contain subtle inconsistencies between their visual and audio signals. We propose a video forensics method, based on anomaly detection, that can identify these inconsistencies, and that can be trained solely using real, unlabeled data. We train an autoregressive model to generate sequences of audio-visual features, using feature sets that capture the temporal synchronization between video frames and sound. At test time, we then flag videos that the model assigns low probability. Despite being trained entirely on real videos, our model obtains strong performance on the task of detecting manipulated speech videos. Project site: https://cfeng16.github.io/audio-visual-forensics

1. Introduction

The paper frames manipulated-video detection as audio-visual anomaly detection, using synchronization features learned from real, unlabeled videos to identify unlikely audio-visual sequences.

  • 1. Introduction: The method detects manipulated videos by flagging low-probability sequences of audio-visual synchronization features learned from real, unlabeled video.It targets inconsistencies between video and audio streams without requiring manipulated examples during training.
  • 1. Introduction: Naturally occurring encoding or recording shifts make simple out-of-sync detection unreliable, motivating anomaly detection over richer synchronization features.Real videos may contain a consistent temporal shift of a few frames, so misalignment alone is not necessarily evidence of manipulation.
  • 1. Introduction: The proposed feature sets capture temporal alignment through time delays, delay distributions, and audio-visual network information.Figure 1 illustrates using the time delay between each video frame and the audio stream as one feature set.
  • 1. Introduction: The model requires neither manipulated training examples nor previously observed test speakers, unlike prior supervised or speaker-verification approaches.This enables zero-shot detection for speakers not present in the training data.
  • 1. Introduction: The method achieves strong performance on FakeAVCeleb and KoDF, generalizes across spoken languages without retraining, and remains robust to compression and blurring.The reported evaluations concern lip-synced, audio-driven face reenactment, and some faceswap videos.

2. Related Work

Prior work spans supervised audio-visual forensics, self-supervised representation learning, visual face forensics, and anomaly detection; this paper combines audio-visual synchronization features with anomaly modeling trained only on real data.

  • Audio-visual forensics: Earlier audio-visual forensics methods use supervised manipulation labels, manipulated-video finetuning, or speaker-identity verification, whereas this approach uses only real data.The paper positions its method as not requiring labels or fake-video examples.
  • Audio-visual representation learning: Self-supervised audio-visual representation methods learn multimodal or synchronization features for tasks such as sound localization, active-speaker detection, source separation, and lip reading.The proposed method uses an off-the-shelf audio-visual synchronization model for anomaly detection.
  • Visual face forensics: Visual face-forensics research has developed manipulation datasets and detectors targeting synthetic content and inconsistent visual or JPEG artifacts.These methods provide a visual-forensics context distinct from the paper’s audio-visual focus.
  • Anomaly detection: Anomaly-detection methods learn a distribution and flag unusual or low-probability examples, a formulation adapted here to manipulated-video detection.The paper uses features intended to capture properties that may be difficult for a forger to reproduce accurately.

3. Method

The method extracts synchronization features from an audio-visual model, learns their temporal distribution with an autoregressive Transformer, and flags sequences assigned low probability.

  • 3. Method: The pipeline models audio-visual examples through features designed to convey subtle synchronization properties that manipulated videos may not accurately capture.Rather than fitting the raw video-audio data directly, it learns a distribution over synchronization-relevant features.
  • 3. Method: The synchronization network produces time-delay scores and related representations from paired video and audio clips, using encoders and a fusion function.The synchronization score is computed across audio-visual pairs in a temporal window and interpreted as synchronization probability.
  • 3. Method: At test time, videos with low probability under the learned feature distribution are flagged as potential fakes.The same anomaly-detection principle is illustrated by the model overview in Figure 2.
  • 3. Method: For each frame, an autoregressive model estimates the next feature from preceding features, with a Transformer implementing the prediction function.Training maximizes sequence log probability through a per-frame loss.
  • 3. Method: The method evaluates multiple feature representations, including discrete most-likely delays, full distributions over delays, and audio-visual network activations.Discrete delays use the highest-probability alignment, while delay-distribution features preserve ambiguity across possible offsets.

4. Results

The model is trained on real, unlabeled speech videos and evaluated for manipulation detection, cross-dataset and language generalization, robustness, and feature-design choices. It performs strongly across these evaluations, with multimodal synchronization features and distribution-based modeling providing the most useful signals.

  • Evaluation setup: The system uses a decoder-only autoregressive Transformer to model synchronization-feature sequences and flag low-probability examples.The evaluated implementation uses 50-frame sequences from 2.0-second videos and is trained on real, unlabeled speech video.
  • Cross-manipulation generalization: AP exceeds 90% in most FakeAVCeleb categories, while the method substantially outperforms self-supervised baselines and sometimes matches or surpasses supervised detectors.These results hold despite training without labeled supervision or fake examples; some supervised methods collapse to near-chance on particular categories.
  • Cross-dataset generalization: The method generalizes to KoDF audio-driven synthesis videos and spoken Korean despite being trained on real videos from other datasets.The evaluation compares models trained on FakeAVCeleb with the proposed model trained on LRS2.
  • Robustness to unseen perturbations: The self-supervised detector is more robust on average and under worst-case unseen visual perturbations than supervised baselines, except RealForensics.Perturbations include compression, blur, noise, distortion, and color changes across six intensity levels.
  • Feature set analysis: Distribution-based time-delay features outperform discrete delays, while soft cross-entropy and multimodal activation features provide the strongest feature-set variants.The multimodal activation model significantly outperforms the visual-only activation model, indicating that both modalities contribute useful information.
  • Ablation study: Increasing the maximum time-delay offset initially improves performance, after which results plateau or slightly decrease as the distribution spreads across unlikely delays.The tested offsets are τ ∈ {5, 10, 15, 20, 25}, corresponding to distribution lengths from 11 to 51.

5. Conclusion

The paper detects video manipulation through self-supervised anomaly detection over audio-visual synchronization features, using only real video for training. It performs strongly on two datasets but is poorly suited to manipulations that preserve motion-sound synchronization.

  • Self-supervised anomaly detection over audio-visual synchronization features detects manipulated videos using unlikely feature sequences.The method learns a distribution from real video and flags low-probability examples.
  • Strong performance on FakeAVCeleb and KoDF is achieved despite training only on real video.
  • The approach is robust to visual postprocessing and videos containing other spoken languages.
  • Manipulations that preserve synchronization between motion and sound, such as appearance changes with little mouth-motion alteration, remain a limitation.

A.1. Video Results

Qualitative results visualize synchronization-model outputs, autoregressive predictions, and fake probabilities for FakeAVCeleb samples.

  • FakeAVCeleb samples are shown with synchronization-model outputs, autoregressive predictions over time, and fake-probability scores.The time-delay distribution is displayed as a heat map.

A.2. Cross-dataset Generalization

The method generalizes to LipGAN-manipulated videos using continuous time-delay distributions and training only on real LRS2 videos. It outperforms many supervised methods on the reported test set.

  • The method outperforms many supervised methods on 100 real and 100 LipGAN-generated FakeAVCeleb test videos.The reported evaluation uses AP and AUC scores.
  • The experiment uses the continuous time-delay distribution feature set, indicating generalization to another audio-driven video-editing method.
  • Training uses real LRS2 videos, while the test videos are manipulated with LipGAN.

A.3. Ablation Study

The ablation study evaluates PCA dimensionality for audio-visual feature activations and finds that increasing the number of principal components reduces anomaly-detection accuracy.

  • Accuracy decreases as the number of PCA components D increases across the tested values {11, 31, 32, 64, 128, 256, 512}.The authors suggest that higher-dimensional prediction is more challenging.
  • The ablated model projects audio-visual feature activations into a lower-dimensional space while keeping other hyperparameters fixed.
  • The PCA ablation reports average AP and AUC scores on the FakeAVCeleb dataset.

A.4. Feature Set Variations

The paper compares autoregressive models for audio-visual synchronization features, including binary cross-entropy and discrete probability-grid formulations.

  • Feature-set modeling: The feature-set implementations use autoregressive probability modeling to represent temporal synchronization between frames and audio.The section introduces additional details for building the autoregressive model on these feature sets.
  • Binary cross entropy model: The BCE model treats the 2τ +1 possible time delays at each step as independent and optimizes their probabilities with binary cross entropy.Predictions are constrained to [0, 1] with a sigmoid function.
  • Discrete 2D probability model: The discrete model generates a quantized 2D frame/time-delay probability matrix autoregressively in raster-scan order.K-means with K = 8 converts each probability entry into a discrete bin, and the model uses a PixelCNN-like loss.

A.5. Implementation Details

The implementation trains synchronization and anomaly-detection components with staged sampling, learned projections, positional encodings, and specified optimization settings.

  • Audio-visual synchronization model: Synchronization training uses curriculum learning, first sampling negatives from different videos and then sampling them randomly within the same videos.The second stage samples four 5-frame short clips per video.
  • Anomaly detection model: The anomaly detector projects input features to 256 dimensions before autoregressive modeling and projects embeddings back into feature space afterward.Learnable positional encodings are used for both synchronization and autoregressive models.
  • Hyperparameters: The models use Adam optimization with staged synchronization batch sizes of 16 and 40, while the anomaly detector uses learning-rate decay, batch size 16, and dropout 0.1.The anomaly detector uses learning rate 1 × 10^-3 and weight decay 1 × 10^-6.
  • Perturbation setup: The implementation details include robustness testing with image corruptions such as saturation, blur, white Gaussian noise, JPEG compression, and video compression changes.The corruptions are evaluated at intensity level 5 in the visualization referenced by the section.

A.6. Visualization of perturbed images

The experiments visualize temporal localization and robustness to perturbations, showing higher fake scores on manipulated frames and sustained performance under Gaussian audio noise.

  • Perturbed images: The perturbation visualization includes unseen corrupted images used for robustness testing.The figure illustrates examples of the corruptions considered at intensity level 5.
  • A.7. Temporal localization: Temporal localization evaluates random 5-frame FakeAVCeleb subsequences whose frames are manipulated with Wav2Lip audio.The experiment uses these manipulated subsequences because the benchmark datasets lacked relevant temporal-localization videos.
  • A.7. Temporal localization: 92.0% top-5 accuracy accompanies significantly higher fake scores on manipulated frames, indicating temporal localization of manipulations.The cited comparison states that supervised and unsupervised baselines cannot temporally localize manipulated frames.
  • A.8. Robustness to background noise: The detector basically maintains performance as Gaussian-noise SNR decreases, despite never seeing audio Gaussian noise during training.Figure 10 reports AUC as a function of SNR for all categories and fake-video categories.
Loading 2301.01767v2…