PaperlayerView exact revision on arXiv

Source-linked AI summary

ExposeAnyone: Personalized Audio-to-Expression Diffusion Models Are Robust Zero-Shot Face Forgery Detectors

Kaede Shiohara, Toshihiko Yamasaki, Vladislav Golyanik

arXiv:2601.02359v1cs.CV

TL;DR

Detecting unseen face manipulations is difficult because fake-sample training can bias detectors toward known patterns, while existing self-supervised methods struggle to learn discriminative representations. ExposeAnyone personalizes an audio-to-expression diffusion model to reference subjects, and reports results on traditional deepfakes, Sora2 videos, and common corruptions.

  • Problem

    Detectors trained on fake or pseudo-fake samples can become biased toward seen manipulations, while existing self-supervised approaches struggle to learn discriminative representations.

    Source: §1 · pp. 1–2

  • Method

    ExposeAnyone pre-trains an audio-to-expression diffusion model on unlabeled videos, then personalizes it to a subject using reference videos for identity-based authentication.

    Source: §1 · p. 2

  • Results

    ExposeAnyone generalizes to traditional deepfakes and Sora2 videos and is robust to common corruptions; it achieves 95.22% average AUC on traditional deepfake benchmarks.

    Source: §1 · p. 2, §4.4 · p. 7, §6 · p. 8

  • Takeaways & Limitations

    Personalizing audio-to-expression diffusion models to specific subjects enables person-of-interest face forgery detection without prior knowledge of face forgeries.

    Source: §6 · p. 8

    Inference depends on off-the-shelf models and incurs computational overhead from iterative 3DMM extraction and diffusion reconstruction.

    Source: §5 · p. 8

Abstract

from arXiv · show

Detecting unknown deepfake manipulations remains one of the most challenging problems in face forgery detection. Current state-of-the-art approaches fail to generalize to unseen manipulations, as they primarily rely on supervised training with existing deepfakes or pseudo-fakes, which leads to overfitting to specific forgery patterns. In contrast, self-supervised methods offer greater potential for generalization, but existing work struggles to learn discriminative representations only from self-supervision. In this paper, we propose ExposeAnyone, a fully self-supervised approach based on a diffusion model that generates expression sequences from audio. The key idea is, once the model is personalized to specific subjects using reference sets, it can compute the identity distances between suspected videos and personalized subjects via diffusion reconstruction errors, enabling person-of-interest face forgery detection. Extensive experiments demonstrate that 1) our method outperforms the previous state-of-the-art method by 4.22 percentage points in the average AUC on DF-TIMIT, DFDCP, KoDF, and IDForge datasets, 2) our model is also capable of detecting Sora2-generated videos, where the previous approaches perform poorly, and 3) our method is highly robust to corruptions such as blur and compression, highlighting the applicability in real-world face forgery detection.

1. Introduction

Detectors trained on known fake patterns often struggle with unseen manipulations, while existing self-supervised methods have difficulty learning discriminative representations. ExposeAnyone personalizes an audio-to-expression diffusion model for person-of-interest detection and reports strong results on traditional and Sora2-generated videos.

1. Introduction

  • Methods trained on fake or pseudo-fake samples often become biased toward seen manipulations, while existing self-supervised approaches struggle to learn discriminative representations.Source: §1 · pp. 1–2
  • ExposeAnyone pre-trains an audio-to-expression diffusion model on unlabeled video, personalizes it with a subject-specific adapter, and uses content-agnostic authentication focused on identity discrepancy.The paper also introduces an expression feature extraction strategy intended to disentangle face expression from face shape.Source: §1 · p. 2
  • 95.22% average AUC on traditional deepfake benchmarks, outperforming previous state-of-the-art approaches.Source: §1 · p. 2
  • ExposeAnyone detects Sora2-generated videos, where previous methods perform poorly, and is highly robust to corruptions including severe compression.Under severe compression, AltFreezing drops 36.71 percentage points in AUC, while ExposeAnyone drops 2.0 percentage points.Source: §1 · p. 2
  • The authors describe ExposeAnyone as a self-supervised method achieving competitive performance with prior state-of-the-art approaches.Source: §1 · p. 2

2. Related Work

Prior detectors trained on predefined forgery patterns can struggle with future manipulations, and self-supervised approaches face challenges in learning discriminative signals. ExposeAnyone combines self-supervision, a general prior from unlabeled videos, and personalization to model subject-specific talking identity.

2. Related Work

  • Forgery types in synthesized training data remain limited to predefined patterns, making detection performance difficult to maintain on future manipulations.Source: §2 · p. 2
  • Prior self-supervised methods face a vast, unconstrained search space, while ExposeAnyone instead focuses on talking-identity discrepancy, described as manipulation-invariant and robust on high-fidelity deepfakes.Source: §2 · p. 2
  • ExposeAnyone compares reconstruction distances with and without a subject-specific adapter after personalization, and trains in a self-supervised fashion during pre-training and personalization.Source: §2 · p. 3
  • Reference-based methods vary in whether they use fake samples, a general prior, and subject personalization; training from scratch can require long reference videos, while identity-feature methods are limited by available labeled subjects.PWL requires one hour of reference video per subject for accurate disentanglement; VoxCeleb2 provides about 6K identities.Source: §2 · p. 3
  • ExposeAnyone uses no fake samples, learns a general prior from unlabeled large-scale videos, and personalizes to capture finer identity-related facial dynamics.Source: §2 · p. 3

3. Proposed Framework: ExposeAnyone

ExposeAnyone pre-trains an audio-to-expression diffusion model, personalizes it to subjects, then authenticates suspected videos by comparing reconstruction distances with and without identity information. Its framework combines expression-data curation, unlabeled pre-training, lightweight personalization, and content-agnostic scoring.

3. Proposed Framework: ExposeAnyone

  • The framework authenticates suspected videos by comparing reconstruction distances with and without subject identity information after pre-training and personalization.The pipeline first trains EXAM on curated audio-expression data, then personalizes it to specific subjects.Source: §3 · p. 3

3.1. ExposeAnyone Model (EXAM)

  • EXAM generates facial expression sequences from speech, predicting 53 FLAME coefficients with a diffusion model conditioned on audio.The coefficients cover 50 principal expression parameters and three jaw-pose parameters; TiLM conditions sequences with O(L) time complexity.Source: §3.1 · pp. 3–4

3.2. Audio-4D Data Curation with Refinement

  • The curated training set contains 200,359 eight-second videos totaling 445 hours, with a shared mean face shape used to disentangle shape from expression.The source videos come from VoxCeleb2, AVSpeech, and Acappella; frames are initialized with SPECTRE and then refined using a single shape per clip.Source: §3.2 · p. 4

3.3. Pre-training

  • Pre-training learns the denoising model on curated, unlabeled videos, without identity-annotated training data.The model predicts the noise added to clean expression coefficients from noisy coefficients, diffusion timestep, and waveform.Source: §3.3 · p. 4

3.4. Personalization to Specific Subjects

  • Personalization inserts subject-specific learnable tokens into attention and trains only the tokens and their key/value projections.This minimal update retains the pre-trained model’s knowledge and stores 528,384 parameters per subject rather than a separately fine-tuned base model.Source: §3.4 · pp. 4–5

3.5. Video Authentication with Content-Agnosticity

  • Content-agnostic authentication scales identity-conditioned reconstruction error by error without identity conditioning to reduce variation due to video content.Sampling diffusion timesteps from 201 to 800 improves discriminability by excluding low-noise and high-noise extremes.Source: §3.5 · p. 5

3.6. Implementation Details

  • With eight reference videos, personalization takes under 15 minutes per subject, and eight-second video inference takes 25 seconds excluding feature extraction.The experiments use one NVIDIA A100 GPU; authentication samples 60 timesteps and 64 noise sequences per timestep.Source: §3.6 · p. 5

4. Experiments

ExposeAnyone is evaluated in a person-of-interest setting with pristine subject references across traditional deepfake datasets and Sora2-generated videos. It achieves the highest reported average AUC across the traditional benchmarks, generalizes to Sora2, and remains robust to corruptions; framework analyses examine its authentication and personalization choices.

4.1. Problem Setting / 4.2. Setup

  • The person-of-interest setting gives the detector pristine reference videos for subjects appearing in suspected videos, and evaluation uses video-level AUC across four audio-visual datasets.The four datasets are DF-TIMIT, DFDCP, KoDF, and IDForge; the average AUC measures generalization across them.Source: §4.1 · p. 5, §4.2 · p. 6

4.3. Comparison with Previous Methods

  • 95.22% average AUC across DF-TIMIT, DFDCP, KoDF, and IDForge gives ExposeAnyone the highest reported generalization ability over these datasets.The result exceeds the previous best methods by approximately 5%.Source: Table 2 · p. 6, §4.3 · pp. 6–7

4.4. Comparison on Sora2-Generated Videos

  • On S2CFP, ExposeAnyone achieves 94.44% average AUC on Sora2-generated videos, while previous detector methods fail to generalize.S2CFP provides 40 reference videos and 12 real plus 12 fake test videos per subject.Source: §4.2 · p. 6, §4.4 · p. 7

4.6. Framework Analysis

  • The quotient of pre-training and personalization objectives distinguishes real from fake samples, whereas either objective used directly does not.The authentication comparison is shown in Figure 4.Source: Figure 4 · p. 7, §4.6 · pp. 7–8

4.5. Robustness to Perturbations

  • Across seven corruption types and five severity levels, ExposeAnyone maintains consistent AUC under corruption, especially blur and compression.LipForensics and AltFreezing perform slightly better without corruption but drop when severity exceeds zero.Source: §4.5 · p. 7, Figure 3 · p. 7

4.6. Framework Analysis

  • Expression sequences refined to singularize face shape yield 93.45% AUC on DFDCP, versus 46.82% without refinement.The authors attribute the difference to SPECTRE predicting different face-shape parameters across frames, entangling shape and expression.Source: §4.6 · p. 7
  • Longer reference videos improve ExposeAnyone more than prior reference-assisted methods; 1 versus 64 noise sequences yields 93.25% versus 93.45% performance.The study tests reference durations of 15, 30, 60, and 120 seconds and noise-sequence counts of 1, 4, 16, and 64.Source: §4.6 · p. 8
  • Eight adapter tokens per subject perform best: AUC rises from 91.79% with one token to 93.45% with eight, then falls to 93.37% with 16.The authors attribute the slight decline at 16 tokens to overfitting to reference data.Source: §4.6 · p. 8, Figure 5 · p. 8

5. Limitations

ExposeAnyone’s limitations are its dependence on off-the-shelf model capabilities and its computational overhead during inference.

5. Limitations

  • The method depends on the capabilities of FLAME for face representation.The authors also identify SPECTRE and Wav2Vec 2.0 as dependencies.Source: §5 · p. 8
  • Inference has computational overhead, mainly from iterative 3DMM extraction and diffusion reconstruction.The reconstruction runs over multiple timesteps and noise sequences.Source: §5 · p. 8
  • Face representation, feed-forward initialization, and audio encoding rely on off-the-shelf models.The specified models are FLAME, SPECTRE, and Wav2Vec 2.0, respectively.Source: §5 · p. 8

6. Conclusion

ExposeAnyone personalizes an audio-to-expression diffusion model for person-of-interest face forgery detection. The authors report generalization to unseen manipulations, Sora2 videos, and common corruptions.

6. Conclusion

  • Personalizing the audio-to-expression diffusion model to specific subjects enables person-of-interest forgery detection without prior knowledge of forgeries.Source: §6 · p. 8
  • The approach significantly outperforms previous state-of-the-art methods on unseen manipulations, including traditional deepfakes and Sora2.Source: §6 · p. 8
  • The authors report robustness to common corruptions and identify real-world face forgery detection as a promising direction.Source: §6 · p. 8

A. Evaluation Dataset

The evaluation uses reference-assisted datasets selected for audio, identity labels, and multiple clips per subject, while excluding some datasets for comparability. It also introduces a three-subject Sora2 Cameo dataset and describes how its real and generated videos were assembled.

Evaluation dataset selection

  • Evaluation datasets must include audio, identity labels, and multiple clips per subject; datasets built on VoxCeleb2 are also excluded because some evaluated methods train on it.Videos without good face tracking are excluded from experiments.Source: A. Evaluation Dataset · p. 9

DF-TIMIT and DFDCP

  • DF-TIMIT uses six Session 1 videos per subject for reference and four later-session videos for testing, while DFDCP uses eight reference videos and keeps scene indexes separate across reference and test sets.DF-TIMIT uses the higher-quality, more challenging fake videos; DFDCP’s test set contains 128 real and fake videos.Source: A. Evaluation Dataset · p. 9

KoDF and IDForge

  • KoDF evaluation uses face-reenacted videos rather than face-swapped videos because each subject has only one scene, which could let reference-assisted methods rely on background or clothing.IDForge follows its official reference/test split; the passage reports 67 evaluation subjects and sampling up to 64 reference videos per subject.Source: A. Evaluation Dataset · p. 9

S2CFP

S2CFP construction

  • For each S2CFP subject, eight real videos are split into reference and test material, with speaking segments clipped into eight-second units.Reference videos yield ten clips each and test videos yield three clips each.Source: A. Evaluation Dataset · p. 10
  • Generated S2CFP videos use captions and transcriptions from corresponding real videos as prompts, with some sensitive transcription words replaced by abstract terms.GPT-5 generates frame captions, Whisper transcribes speech, and Sora2 generates the Cameo videos.Source: A. Evaluation Dataset · p. 10

B. Curated Dataset for Pre-training

The pre-training dataset combines curated talking and singing videos from VoxCeleb2, AVSpeech, and Acappella. The authors describe identity and face tracking filters used to prepare these sources.

VoxCeleb2

  • VoxCeleb2 videos are filtered for identity consistency using ArcFace similarity, yielding 140,194 videos.Videos with any frame-pair identity similarity below 0.4 are excluded.Source: B. Curated Dataset for Pre-training · p. 10

AVSpeech

Acappella

C.1. ExposeAnyone Model

ExposeAnyone uses TiLM layers for sequential conditioning in a DiT- and EDGE-like architecture. Its inference guidance uses learnable unconditional audio and identity vectors, while personalization and authentication share a reference-derived face shape across videos of a subject.

C.1. ExposeAnyone Model

  • TiLM layers provide sequential conditioning in an architecture similar to DiT and EDGE.The model also uses learnable unconditional vectors for audio and identity conditions during training.Source: §C.1 · p. 11
  • Audio and identity guidance strengths are set to 0.5 and 0.25, respectively.The paper reports that large guidance strengths harm detection performance.Source: §C.1 · p. 11
  • For personalization and authentication, the first reference video's extracted face shape is fixed during optimization of other videos for the same subject.Pre-training instead assigns one shared shape per clip, assuming different videos have different face shapes, though identities may overlap.Source: §C.1 · p. 11

C.2. Baselines

The baseline descriptions specify training or implementation choices for several comparison methods. The supplied captions also identify a timestep-sampling study, a model-complexity analysis, and KoDF accuracy results.

C.2. Baselines

  • EfficientNet-b4 is trained on FF++ real and manipulated videos using the preprocessing, augmentations, and inference strategy of SBI.Face X-ray is re-implemented with its backbone frozen for 50K iterations, then all layers updated for 150K iterations on real and blended images.Source: §C.2 · p. 11
  • Table 7 reports timestep-sampling variants and states that excluding early and late timesteps improves results.The caption identifies the highlighted gray setting as the default.Source: Table 7 · p. 11
  • The supplied captions identify Table 9 as a model-complexity analysis and Table 8 as KoDF accuracy results.The captions do not provide values or comparisons.Source: Table 9 · p. 11, Table 8 · p. 11
  • UCF uses DeepfakeBench’s implementation, while the other baseline methods use their official implementations.Face X-ray blended images are generated with the authors’ unofficial implementation.Source: §C.2 · p. 11

D. Additional Experiments

Additional experiments examine timestep selection, threshold-based classification, audio conditioning, model complexity, S2CFP comparisons, and temporal authentication behavior. They report improved AUC with a middle timestep range, reduced performance without audio, and some score instability during silent frames.

D. Additional Experiments

  • 95.22% average AUC with timesteps [201, 800] exceeds 94.57% with [1, 1000], while [301, 700] yields 95.20%.The paper recommends [201, 800] for stable performance across datasets; excessive exclusion slightly reduces performance.Source: D. Additional Experiments · p. 11
  • For KoDF thresholding, the method uses a subject-specific Gaussian score distribution estimated from eight real validation videos and reports consistent accuracy across thresholds.Thresholds are set at µ + σ, µ + 2σ, and µ + 3σ; the authors note that DF-TIMIT and DFDCP have too few videos per subject for this validation setup.Source: D. Additional Experiments · p. 11
  • Removing audio conditioning lowers AUC on every test set, although the audio-free model retains state-of-the-art generalization ability.The authors link audio conditioning to better expression-coefficient prediction and detection performance.Source: D. Additional Experiments · p. 12, Table 10 · p. 12
  • On S2CFP, ExposeAnyone achieves the best result by a large margin, including against methods specialized for diffusion-generated images.Table 11 is the comprehensive S2CFP comparison.Source: D. Additional Experiments · p. 12, Table 11 · p. 12
  • Inference is slower than previous methods, taking 22.2 seconds for 3DMM extraction and 23.6 seconds for diffusion authentication per eight-second video on one A100 GPU.The authors suggest feed-forward 3DMM extraction and fewer noise sequences as possible ways to reduce overhead.Source: D. Additional Experiments · p. 12
  • Authentication scores are slightly unstable at silent frames, but the model distinguishes real videos from visually similar deepfakes across subjects and scenes.The authors interpret this as evidence that it focuses on semantic talking identities.Source: D. Additional Experiments · p. 12, Figure 8 · p. 13
Loading 2601.02359v1…