PaperlayerView exact revision on arXiv

Source-linked AI summary

Audio-Visual Person-of-Interest DeepFake Detection

Davide Cozzolino, Alessandro Pianese, Matthias Nießner, Luisa Verdoliva

arXiv:2204.03083v3cs.CVcs.SDeess.AS

TL;DR

Deepfake detectors face diverse manipulation methods and poor cross-dataset performance, especially in low-quality settings. POI-Forensics learns identity-specific audio-visual representations from real video and compares them with POI references. The authors report 7%-14% AUC and accuracy improvements over SOTA approaches, especially for compressed and adversarially attacked videos.

  • Problem

    Existing detectors struggle to generalize across diverse manipulation methods, datasets, low-quality videos, and adversarial attacks.

    Source: §1 · p. 2

  • Method

    POI-Forensics uses contrastive learning on audio and video segments to learn identity-specific representations and compares test segments with POI reference videos.

    Source: §1 · p. 2, §3.1 · p. 4

  • Results

    AUC and accuracy improve by 7%-14% over SOTA approaches, especially on compressed and adversarially attacked videos.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that the method handles a wide variety of manipulations without relying on patterns introduced by specific deepfake generators.

    Source: §5 · p. 8

    Testing assumes a reference set of at least 10 pristine POI videos, each about 30 seconds long.

    Source: §3.2 · p. 5

Abstract

from arXiv · show

Face manipulation technology is advancing very rapidly, and new methods are being proposed day by day. The aim of this work is to propose a deepfake detector that can cope with the wide variety of manipulation methods and scenarios encountered in the real world. Our key insight is that each person has specific characteristics that a synthetic generator likely cannot reproduce. Accordingly, we extract audio-visual features which characterize the identity of a person, and use them to create a person-of-interest (POI) deepfake detector. We leverage a contrastive learning paradigm to learn the moving-face and audio segment embeddings that are most discriminative for each identity. As a result, when the video and/or audio of a person is manipulated, its representation in the embedding space becomes inconsistent with the real identity, allowing reliable detection. Training is carried out exclusively on real talking-face video; thus, the detector does not depend on any specific manipulation method and yields the highest generalization ability. In addition, our method can detect both single-modality (audio-only, video-only) and multi-modality (audio-video) attacks, and is robust to low-quality or corrupted videos. Experiments on a wide variety of datasets confirm that our method ensures a SOTA performance, especially on low quality videos. Code is publicly available on-line at https://github.com/grip-unina/poi-forensics.

1. Introduction

POI-Forensics uses person-specific audio-visual representations to detect a broad range of manipulations, addressing the poor cross-dataset and low-quality performance of existing detectors. The authors report improved AUC and accuracy, particularly for compressed and adversarially attacked videos.

1. Introduction

  • Existing detectors struggle with diverse and unseen manipulations, cross-dataset performance, low-quality video, and adversarial attacks.Deepfake manipulation can include synthesized speech, altered expression, identity swaps, or changes to what a person says.Source: §1 · pp. 1–2
  • POI-Forensics uses contrastive learning to bring same-identity audio and video segment representations together and compares test features with POI reference videos.Joint audio-video similarity indices are included to improve discrimination.Source: §1 · p. 2
  • Training only on real videos avoids dependence on manipulation-specific artifacts and is intended to generalize to known and unseen methods.The stated scope includes face swapping, facial reenactment, and other manipulation methods.Source: §1 · p. 2
  • The multimodal detector covers video-only, speech-only, and joint audio-video manipulations.Joint-modality analysis improves performance when video and audio are manipulated together.Source: §1 · p. 2
  • The authors report robustness to compressed and maliciously attacked videos, conditions they identify as common in real-world scenarios.Social-network sharing can further compress video and remove relevant audio-visual information.Source: §1 · p. 2
  • AUC and accuracy improve by 7%-14% over SOTA approaches, especially on compressed and adversarially attacked videos.The reported comparison highlights the most challenging scenarios.Source: §1 · p. 2

2. Related Work

Related work spans supervised detectors built around manipulation artifacts, audio-visual inconsistency methods, and person-of-interest approaches that model identity. The reviewed methods face limits involving unseen manipulations, degraded video quality, audio coverage, or the amount of POI data required.

Single-modality methods

  • Supervised detectors commonly learn video artifacts from real and fake examples, but can perform poorly on manipulation types absent from training.Repeatedly expanding datasets or acquiring examples of new manipulations remains a challenge.Source: §2 · pp. 2–3
  • Artifact-based approaches can offer clues about manipulation, but quality impairment and adversarial attacks can obscure the low-level features they use.The reviewed approaches include frequency analysis, blending artifacts, and attention to spatial or temporal inconsistencies.Source: §2 · p. 3

Multimodal analysis

  • Audio-visual methods exploit synchronization or cross-modal inconsistencies, yet realistic synchronized speech and lips make synchronization analysis challenging.The reviewed multimodal methods require real and fake videos for training, which the authors say could impair generalization.Source: §2 · p. 3

POI-based detection

  • POI-based detection models pristine identity and treats deviations as anomalies, framing detection as whether a video depicts the claimed person.The review describes this identity-verification framing as applicable to both cheapfakes and deepfakes.Source: §2 · p. 3
  • Earlier POI methods use facial or head motion, mouth-speech consistency, or inner-versus-outer face identity cues, with some requiring several hours of POI video.The reviewed facial-motion method is described as more effective for face swapping than facial reenactment.Source: §2 · p. 3

3. Method

POI-Forensics learns identity-characterizing audio and video embeddings from real videos, then evaluates whether test segments match pristine references for the person of interest. Contrastive learning and normalized similarity scores support single-modality and joint audio-video testing.

3. Method

  • POI-Forensics treats manipulation as an identity inconsistency, learning audio-visual identity features from real videos rather than detecting generation artifacts.The training dataset contains more than 5,000 identities.Source: §3 · pp. 3–4
  • Each 3-second test segment yields 256-component audio and video embeddings that are compared with embeddings from pristine POI videos.The audio network uses an amplitude spectrogram, while the video network uses cropped faces from 25 frames.Source: §3 · p. 4, Figure 3 · p. 4

3.1. Contrastive Learning

  • Contrastive losses pull same-identity segments together and separate different identities across video, audio, and joint modalities; the joint loss helps offset modality shortcomings.The training scheme computes three N × N similarity matrices, one for each modality view.Source: §3.1 · p. 4, Figure 4 · p. 5

3.2. Testing

  • Testing assumes at least 10 pristine POI videos of about 30 seconds each, yielding 100 disjoint reference segments for nearest-match comparisons.Source: §3.2 · p. 5
  • The method normalizes each modality's POI similarity using pristine-reference statistics, takes the minimum of the three normalized indices as fusion, and averages target-video segment scores.Under the real-video hypothesis, normalized scores are modeled as standard Gaussian variables to set decision thresholds for a desired false-alarm rate.Source: §3.2 · p. 5

4. Experimental Results

The experiments assess POI-Forensics across datasets, metrics, ablations, and comparisons with state-of-the-art methods. The reported results include stronger performance on low-quality video and robustness to an adversarial-attack setting.

4.1. Datasets and Metrics

  • Evaluation uses pDFDC, DF-TIMIT, FakeAVCelebV2, and KoDF, reporting video-level AUC, accuracy at a fixed 0.5 threshold, and Pd@10%.The datasets include face-swapping, facial reenactment, and—in FakeAVCelebV2—cloned fake audio.Source: §4.1 · p. 6

4.2. Ablation Study

  • Adding audio similarity improves AUC by 15% to 20% on average over video-only features, while fusion of all three indices performs best on average; the joint contrastive loss gives limited but consistent gains.Audio similarity can reduce performance when the audio is uninformative, and the loss’s improvement is especially notable for Pd@10% with fusion.Source: §4.2 · p. 6, Table 1 · p. 6
  • For face-swapping, a single 3-second snippet reaches perfect detection; facial reenactment starts at 70% AUC and plateaus at 80% with about 20 seconds of video.The figure varies test-video length and reports results separately for face-swapping and facial reenactment.Source: §4.2 · pp. 6–7, Figure 5 · p. 6
  • A single POI reference video yields at most 84% AUC, whereas multiple videos raise AUC above 90%, reaching 92% with at least 40 segments; variety matters more than sheer data size.Facial reenactment results are worse than face-swapping results in this analysis.Source: Figure 5 · p. 6, §4.2 · p. 7
  • In t-SNE projections, embedded vectors from different identities form relatively compact, separated clusters, while most manipulated counterparts lie distant from the corresponding real videos.The selected vectors concatenate audio and video features; dots represent video-segment features.Source: §4.2 · p. 7, Figure 6 · p. 7

4.3. Comparative Performance Analysis

  • On low-quality videos, POI-Forensics gains 11% (AUC) and 12% (Accuracy) over the second best; on high-quality videos, it leads by 2.3% (AUC) and 1.5% (Accuracy) over the respective best references.Table 2 covers pDFDC, DeepfakeTIMIT, FakeAVCelebV2, and KoDF, with high- and low-quality results.Source: §4.3 · p. 7, Table 2 · p. 8
  • Under adversarial attack, all methods’ metrics fall dramatically, while POI-Forensics and, to a lesser extent, ICT-ref and ID-Reveal retain reasonable performance.For fairness, POI-Forensics does not use audio in this experiment because audio is not attacked.Source: Table 3 · p. 8, §4.3 · p. 8

5. Conclusions

POI-Forensics learns an identity representation from real videos to assess whether a test video depicts its claimed person. The authors report that audio-visual information handles varied manipulations without relying on generator-specific patterns, and suggest adding textual data.

5. Conclusions

  • POI-Forensics is trained on real videos to learn a person-of-interest representation that tests whether the video depicts its claimed identity.The conclusion frames this as an audio-visual deepfake detection method.Source: §5 · p. 8
  • Experiments show that audio-visual information can handle a wide variety of manipulations without relying on patterns introduced by specific deepfake generators.The authors characterize the learned model as a real-world data model.Source: §5 · p. 8
  • The authors suggest that the multimodal analysis could be enriched with information from other domains, such as textual data.Source: §5 · p. 8

Supplemental Material

The supplemental material adds ablations, robustness results, method descriptions, dataset details, and a summary of the method’s limitations.

Supplemental Material

A. Additional ablation studies

The additional experiments test POI-Forensics on person identification and deepfake detection against audio-visual speaker-verification methods. They report strong performance as the number of identities or attack scenarios increases.

Reference methods

  • The identification comparison includes SyncNet, which separates slowly changing identity from quickly changing content, and two audio-visual embedding methods using attention or an MLP for fusion.The methods described are SyncNet [44], [53], and.Source: A. Additional ablation studies · p. 9

Person identification

Deepfake detection

  • On deepfake detection, POI-Forensics averages 6 percentage points higher AUC and 12 points higher Pd@10% than the second-best method.The comparison covers four scenarios involving video manipulation, audio manipulation, and audio inconsistency, with references trained on the same dataset.Source: A. Additional ablation studies · p. 9, Table 5 · p. 9

B. Additional robustness analysis

The robustness analysis tests Gaussian noise on audio, video, or both, then compares video-only detection with state-of-the-art methods. Performance drops under intense noise, while the method retains reported advantages.

Gaussian noise

  • With audio and video both attacked by noise, POI-Forensics retains an AUC of around 74%, despite a sharp performance reduction at the tested noise levels.Audio SNR ranges from 5 to 15 dB, and video PSNR from 13 to 27 dB.Source: B. Additional robustness analysis · p. 10
  • The experiments add Gaussian noise to a KoDF subset and evaluate attacks on video alone, audio alone, or both while the detector uses both modalities.The analysis extends earlier tests of compression and adversarial attacks.Source: B. Additional robustness analysis · p. 10

Comparison with state of the art

  • Against video-only state-of-the-art detectors on noisy KoDF, POI-Forensics gains over 8% AUC and 16% Pd@10% over ICT-Ref.The comparison uses only the video signal for POI-Forensics, matching the reference methods.Source: B. Additional robustness analysis · p. 10

C. State-of-the-art methods

The paper compares POI-Forensics with methods based on identity cues, temporal artifacts, audio-visual consistency, and learned representations. The listed approaches vary in their modalities and detection mechanisms.

State-of-the-art methods

  • Seferbekov uses an ensemble of seven EfficientNet-B7 models, while FTCN [65] combines short-term flickering cues with long-term temporal incoherence.Seferbekov analyzes frames individually; FTCN uses a Fully Temporal Convolution Network and Temporal Transformer.Source: C. State-of-the-art methods · p. 10
  • LipForensics [24] detects semantic irregularities in mouth movement using a spatio-temporal network pretrained for visual speech recognition.Source: C. State-of-the-art methods · p. 10
  • Real Forensics [23] uses a teacher-student network for real/fake classification and feature extraction, with the teacher supplying target features.Source: C. State-of-the-art methods · p. 10
  • MDS-based FD [9] measures audio-visual similarity to capture inconsistencies, while Joint AV synchronizes separately labeled audio and video representations.Source: C. State-of-the-art methods · p. 10
  • ICT [20] detects identity inconsistencies between inner and outer face regions, and ICT-Ref assumes access to reference real videos; ID-Reveal [12] uses 3D Morphable Models and an adversarial game.Source: C. State-of-the-art methods · p. 10

D. Deepfake video datasets

The experiments use four deepfake video datasets with differing manipulation types and sample counts. The section also identifies pDFDC videos used in examples of failures.

pDFDC

FakeAVCelebV2

  • FakeAVCelebV2 includes 500 real videos and about 20,000 fakes spanning face swaps, reenactment, and audio manipulation, separately or in combination.The fakes include Faceswap, FSGAN, Wav2Lip, and SV2TTS-generated audio.Source: D. Deepfake video datasets · p. 10

KoDF

DF-TIMIT

  • DF-TIMIT contributes 290 real and 580 fake videos of at least four seconds, with two manipulated versions per real video.The fakes use a GAN-based face-swapping method at two input dimensions.Source: D. Deepfake video datasets · p. 11

E. Limitations

POI-Forensics avoids fake-video training but requires audio-visual data from multiple subjects and pristine references for each test identity. Performance also drops on profile faces, which the authors tentatively connect to frontal-pose-heavy training.

E. Limitations

  • Training requires a large audio-visual dataset covering several subjects, plus around 10 pristine reference videos of each target subject at test time.Source: E. Limitations · p. 11
  • Performance drops on profile faces, which the authors tentatively attribute to the training set’s predominance of frontal-pose videos.They suggest balancing training samples or using suitable augmentation as possible remedies.Source: E. Limitations · p. 11
Loading 2204.03083v3…