PaperlayerView exact revision on arXiv

Source-linked AI summary

MOTIF: Person-of-Interest Deepfake Detection Beyond 3DMM Coefficients

Giovanni Affatato, Sara Mandelli, Paolo Bestagini, Stefano Tubaro

arXiv:2610.09830v1cs.CV

TL;DR

POI detectors often use full 3DMM coefficient vectors without measuring which parts carry detection signal. MOTIF tests coefficient and reconstructed-mouth representations under a controlled setup, then improves on both state-of-the-art POI detectors in every benchmark cell at both quality levels.

  • Problem

    POI deepfakes are especially harmful, while reference-based detectors use whole 3DMM coefficient descriptions without measuring which parts carry the detection signal.

    Source: §I · p. 1, §V · p. 6

  • Method

    MOTIF is a visual-only POI detector that tests 3DMM representations in a controlled study, using a temporal identity-contrastive encoder trained on real videos without manipulated footage or target-POI data.

    Source: §I · pp. 1–2

  • Results

    MOTIF improves on both state-of-the-art POI detectors in every benchmark cell at both quality levels.

    Source: §V · p. 6

  • Takeaways & Limitations

    The shape block alone recovers nearly all full-vector accuracy, while reconstructed mouth geometry complements coefficients on weak lip-sync and reenactment cases.

    Source: A. Dissecting the Coefficient Vector · p. 4, B. Exploiting the Reconstructed Surface · p. 5

    Clip-consistent 3DMM fitting and facial regions beyond the mouth remain open directions.

    Source: §V · p. 6

Abstract

from arXiv · show

Video deepfakes targeting a specific individual, the Person-of-Interest (POI), are the most harmful ones, and, since a public figure is abundantly recorded, a detector can be built from genuine footage of that individual. Such detectors commonly describe a subject through a 3D Morphable Model (3DMM) and adopt its coefficients as a whole, so which part of that description carries the signal has never been measured. We dissect it, holding the encoder, the training corpus and the enrollment protocol fixed and varying only what the encoder observes. The groups of coefficients prove largely redundant, since the shape block alone recovers almost all the accuracy of the full vector, and their temporal evolution contributes a real but bounded amount. We further show that the dense surface the same fit returns, which these detectors discard, carries identity information that the coefficients do not, and that it helps precisely where they are weakest. We assemble the best configuration into MOTIF, a visual-only detector trained on real videos only, with no manipulated video and no POI-specific data. It improves on both state-of-the-art POI detectors in every dataset and manipulation of our benchmark and at two quality levels. Our experimental code will be released at https://github.com/polimi-ispl/MOTIF.

I. INTRODUCTION

Deepfakes targeting a specific Person-of-Interest (POI) can enable disinformation and fraud, yet prior detectors using 3DMMs adopted the full coefficient representation without measuring which parts matter. The paper isolates the representation’s contributions and tests whether reconstructed mouth geometry adds information.

I. INTRODUCTION

  • Prior 3DMM-based detectors used shape, expression, and pose coefficients together, leaving their individual and temporal contributions unmeasured.The paper frames the open questions as which coefficient groups provide effective, complementary or redundant signals, whether temporal evolution adds information, and whether the reconstructed surface carries information absent from the coefficients.Source: §I · p. 1
  • MOTIF tests these representation choices while holding the encoder, training corpus, and POI enrollment protocol fixed.The encoder is trained on real videos of many subjects, without manipulated videos or footage of the target POI.Source: §I · p. 1
  • The study introduces reconstructed mouth geometry as a signal that coefficient-based detectors leave unused.The paper extracts the mouth vertices from the dense surface returned by the same 3DMM fit.Source: §I · pp. 1–2
  • MOTIF is compared with six state-of-the-art detectors across four datasets, multiple manipulation families, and two video-quality levels.The comparison includes face swap, reenactment, and lip-sync manipulations.Source: §I · p. 2

II. PROPOSED METHODOLOGY

The method frames POI detection as comparing a test video with pristine references for the target individual. Its global descriptor represents each video as windows of 3DMM coefficients describing the face.

A. POI Deepfake Detection

  • A test video is classified by thresholding how well its descriptors align with those from the POI’s pristine reference videos.The comparison uses a subject-similarity score between the test video and reference set.Source: A. POI Deepfake Detection · p. 2
  • The detector is open-set and POI-agnostic: training does not assume known forgery techniques or use data specific to the target identity.The encoder is therefore not tailored to a particular identity.Source: A. POI Deepfake Detection · p. 2

1) Global descriptor extraction:

  • The global descriptor concatenates per-frame 3DMM shape, expression, head-pose, and translation coefficients into windows of L consecutive frames.Shape encodes neutral facial geometry; expression encodes facial gestures; pose and translation describe head motion relative to the camera. Windows overlap by 50%.Source: 1) Global descriptor extraction · p. 2

2) Mouth descriptor extraction:

The mouth branch describes lip motion from the dense surface reconstructed by the 3DMM, rather than relying only on the whole-face coefficient vector.

2) Mouth descriptor extraction:

  • The mouth descriptor uses only 3D lip vertices reconstructed frame by frame from the 3DMM.It subtracts the window’s temporal mean, removing neutral lip shape while retaining how the lips move.Source: 2) Mouth descriptor extraction · p. 2
  • Mouth descriptors are organized as window-level inputs alongside the global branch’s whole-face descriptors.The paper denotes each branch’s window descriptors as W_b(V) and its input dimension as L × D_b.Source: 2) Mouth descriptor extraction · p. 2

C. Encoder and Training Objective

MOTIF uses separate temporal-transformer encoders for the global-face and mouth branches, trained with an identity-contrastive objective. The objective uses multiple encoder depths and training choices that discourage identity matching through recording-specific cues.

1) Temporal transformer:

  • Each branch uses a separate temporal transformer whose attention relates frames across a window, producing an L2-normalized CLS embedding.The branches share architecture but not weights, and differ in tokenizer input dimension.Source: 1) Temporal transformer · p. 2

2) Identity-contrastive objective:

  • The identity-contrastive objective treats other windows of the same subject as positives and windows of different subjects as negatives.It applies NT-Xent at selected encoder depths and weights anchors and depths equally, making intermediate blocks identity-discriminative.Source: 2) Identity-contrastive objective · pp. 2–3
  • Positive pairs are drawn from different videos and batches balance identities, limiting recording-specific cues and preventing any subject from dominating comparisons.A Multi-Similarity Miner retains the most informative comparisons.Source: 2) Identity-contrastive objective · p. 3
  • Training-time temporal masking removes contiguous frame spans so positive-pair views are not trivially matchable.Masking is applied only during training.Source: 2) Identity-contrastive objective · p. 3

D. Reference-Set Matching and Score Fusion

Reference-set matching compares query embeddings with the enrolled POI’s reference embeddings for each branch. It whitens the embedding space using reference variability before scoring similarity.

1) Reference-set matching:

  • Each branch embeds every enrolled reference window, and a query video produces a corresponding set of query embeddings.The reference and query sets are branch-specific.Source: 1) Reference-set matching · p. 3
  • Whitening rescales each direction by inverse reference standard deviation, emphasizing directions where the subject is consistent over genuine videos.This addresses the spread of reference embeddings along variations such as recording conditions and spoken content.Source: 1) Reference-set matching · p. 3
  • The query score is the maximum cosine similarity to a reference embedding in the whitened space.The compared vectors are a reference embedding and a query embedding.Source: 1) Reference-set matching · p. 3

2) Score fusion:

The branch scores are calibrated using leave-one-out reference videos before fusion. Their arithmetic mean requires consistency with the reference set in both the whole-face and mouth descriptions.

2) Score fusion:

  • Leave-one-out scores on reference videos estimate each branch’s mean and standard deviation, expressing a generic query relative to genuine variability.The two calibrated branch scores are combined by arithmetic mean; a query is authentic only if consistent with both descriptions.Source: 2) Score fusion · p. 3

III. EXPERIMENTAL SETUP

The experiments train on real VoxCeleb2 footage and evaluate on four public deepfake datasets spanning major manipulation families. Performance is averaged across POIs using AUC and balanced accuracy.

1) Training data:

  • Training uses VoxCeleb2’s 5,994 identities, 150,480 videos, and 1.13M clips, excluding the 500 FakeAVCeleb identities used for evaluation.Thus, no training identity appears among the evaluation POIs.Source: 1) Training data · p. 3

2) Evaluation data:

  • Evaluation covers DF-TIMIT, FakeAVCeleb, DeepSpeak, and KoDF, spanning identity swapping, reenactment or synthesis, lip-sync, and combinations.Table I groups dataset generation methods into four canonical categories for per-category results.Source: 2) Evaluation data · p. 3, Table I · p. 4

B. Evaluation Metrics

  • AUC measures score-ranking quality regardless of threshold, while balanced accuracy is the maximum balanced accuracy across decision thresholds.Both metrics are computed per POI and averaged over dataset subjects; category-specific results exclude subjects without fakes in that category.Source: B. Evaluation Metrics · pp. 3–4
  • Aggregate values use all fakes for each subject, whereas category-specific values restrict fakes to that manipulation category and leave real videos unchanged.Subjects with no fake in the selected category are excluded.Source: B. Evaluation Metrics · p. 4

C. State-of-the-Art Baselines

The comparison includes general deepfake detectors and POI-specific detectors, alongside the input dimensions and video windows used for MOTIF.

C. State-of-the-Art Baselines

  • The general baselines are RealForensics, LipForensics, FTCN, and Seferbekov, selected for publicly available weights and cross-dataset evaluation.The first three share FaceForensics++ training data; Seferbekov won the DFDC challenge and was trained on that dataset.Source: C. State-of-the-Art Baselines · p. 4
  • POI-Forensics and ID-Reveal are the two POI-specific baselines; POI-Forensics is evaluated video-only, while ID-Reveal is visual-only and also uses 3DMM coefficients.Both were trained on VoxCeleb2, as was MOTIF.Source: C. State-of-the-Art Baselines · p. 4

1) Input:

  • Each frame has 150 coefficient dimensions: 80 shape, 64 expression, three head-pose angles, and three translation components.The coefficients are extracted with 3DDFA-V3.Source: 1) Input · p. 4
  • Videos are processed in 75-frame windows with 50% overlap; the mouth branch uses 968 lip vertices, or 2,904 dimensions per frame, after removing each window’s temporal mean.The windows correspond to three seconds at 25 fps.Source: 1) Input · p. 4

2) Encoders:

The coefficient ablation finds that shape alone nearly matches the full vector, while dynamic readings add bounded gains. The encoder and training setup are held alongside comparisons of coefficient groups and temporal treatments.

2) Encoders:

  • Both branches use a 12-block temporal transformer with hidden size 192 and three attention heads; training uses NT-Xent, identity-balanced batches, and masked window spans.Training selects the checkpoint with lowest contrastive loss on held-out VoxCeleb2 test identities.Source: 2) Encoders · p. 4, 3) Training · p. 4

A. Dissecting the Coefficient Vector

  • The factorial ablation compares shape, expression, and full coefficients in dynamic and clip-averaged static forms, omitting DF-TIMIT because all six configurations reach 100%.The questions are whether coefficient groups are complementary or redundant and whether temporal evolution adds information beyond a time-invariant description.Source: A. Dissecting the Coefficient Vector · p. 4
  • 90.4% overall mean AUC for dynamic shape coefficients is close to 90.6% for the full vector, leaving expression, pose, and translation worth 0.2 points once shape is read over time.Discarding shape costs 0.9 points.Source: A. Dissecting the Coefficient Vector · p. 4, Table II · p. 5
  • Dynamic reading adds 1.0 points overall mean AUC on shape, 0.9 on expression, and 1.6 on their combination, with its largest gain on reenactment.Table II reports AUC and balanced accuracy by dataset and manipulation category for static and dynamic coefficients.Source: A. Dissecting the Coefficient Vector · p. 4, Table II · p. 5
  • The authors conjecture that per-frame fitting lets shape coefficients absorb some expression dynamics, explaining why shape benefits from temporal reading and remains close to expression.The shape coefficients are not constrained to remain consistent across frames, and shape and expression bases deform the same mesh.Source: A. Dissecting the Coefficient Vector · p. 5

B. Exploiting the Reconstructed Surface

The reconstructed mouth surface adds information beyond the global coefficient branch, with the largest gains where that branch is weakest; fusion has the best overall mean AUC among the three configurations.

B. Exploiting the Reconstructed Surface

  • Fusion improves on the global branch by 9.1 points on FakeAVCeleb lip-sync and 3.6 points on KoDF reenactment.The mouth branch alone reaches 84.7% overall mean AUC against 92.9% for the global branch.Source: B. Exploiting the Reconstructed Surface · p. 5
  • For lip-sync from the same generator, the global branch reaches 96.4% on KoDF but 58.3% on FakeAVCeleb, and fusion adds 1.6 versus 9.1 points, respectively.The surface contributes most where the coefficients achieve least.Source: B. Exploiting the Reconstructed Surface · p. 5
  • Fusion never loses more than 2.0 points across the remaining columns and reaches the best overall mean AUC of the three configurations at 93.1%.The authors adopt the fusion for the rest of the article.Source: B. Exploiting the Reconstructed Surface · p. 5

C. Comparison with the State of the Art

MOTIF is competitive with general detectors at high quality and becomes best overall under compression, while outperforming both POI-specific baselines across the benchmark. The conclusions also identify redundancy among 3DMM coefficient groups and additional identity information in dense surfaces.

C. Comparison with the State of the Art

V. CONCLUSIONS

  • MOTIF improves on both state-of-the-art POI detectors in every benchmark cell and at both quality levels.Source: §V · p. 6
  • 3DMM coefficient groups are largely redundant, temporal evolution adds a real but bounded contribution, and dense surfaces carry identity information absent from coefficients.Source: §V · p. 6
Loading 2610.09830v1…