PaperlayerView exact revision on arXiv

Source-linked AI summary

Lightweight Generalized DeepFake Face Detection with WAVIE: Wavelet Augmented Vision Intermediate Embeddings

Arya Pulkit, Aditya Ruhela, Akarshan Kapoor, Arnav Bhavsar

arXiv:2609.14437v1cs.CVcs.AI

TL;DR

Deepfake detectors often generalize poorly to unseen manipulation pipelines, and final-layer representations may miss subtle artifacts. WAVIE combines intermediate CLIP ViT features with low-frequency wavelet refinement; trained on FaceForensics++, it reports strong frame-level AUROC on three unseen datasets.

  • Problem

    Detectors can perform well on familiar manipulations, but final-layer representations may overlook subtle fabrication artifacts that richer multi-scale features could preserve.

    Source: §I · p. 1

  • Method

    WAVIE learns to fuse intermediate CLIP ViT features and applies low-frequency wavelet refinement while preserving high-frequency detail.

    Source: §I · p. 1

  • Results

    0.8518/0.8520 AUROC on Celeb-DF-v1/v2 and 0.8313 on WDF, with results competitive on Celeb-DF-v1 and best among listed methods on Celeb-DF-v2 and WDF.

    Source: §I · p. 1

  • Takeaways & Limitations

    WAVIE remains competitive on FF++ while reducing the performance drop on unseen datasets, supporting its suitability for deployment where manipulation methods are unknown.

    Source: §VI · p. 6

    The evaluation is frame-level and face-centric, trains only on FF++ (c0), and uses fixed three-level db6 wavelets without image-space augmentations, which may limit robustness.

    Source: §VI · p. 6

Abstract

from arXiv · show

Deepfake detection systems often exhibit significant performance degradation when deployed on unseen manipulation methods, limiting their reliability in real-world multimedia environments. This lack of generalization poses critical challenges for misinformation mitigation, digital forensics, and human-centric AI systems. Existing detectors perform well on the forgery methods they are trained on, but their accuracy drops sharply on unseen pipelines. To bridge this generalization gap, we propose WAVIE (Wavelet Augmented Vision Intermediate Embeddings), an end-to-end architecture that combines complementary spatial and frequency cues on top of a frozen CLIP backbone. WAVIE projects intermediate transformer embeddings through a lightweight learnable module, applies a three-level Daubechies-6 (db6) discrete wavelet transform (DWT), refines the low-frequency branch while preserving the high-frequency branch, reconstructs the feature via inverse DWT, and performs classification. Trained only on FaceForensics++, WAVIE achieves AUROC = 0.852 on Celeb-DF-v1, 0.852 on Celeb-DF-v2 and 0.831 on WildDeepFake (WDF) at the frame level, outperforming several state-of-the-art generalization baselines. Extensive ablation studies confirm the importance of both the wavelet module and the intermediate-feature aggregation for cross-dataset performance, highlighting the necessity of jointly leveraging spatial and frequency domains. These results position WAVIE as a strong baseline for deepfake detection in the wild.

I. INTRODUCTION

Deepfake detectors often falter on unseen pipelines, while final-layer CLIP embeddings may miss subtle fabrication artifacts. WAVIE combines intermediate ViT features with low-frequency wavelet refinement and reports strong cross-dataset results with a small trainable module.

I. INTRODUCTION

  • Detectors trained on one generator or dataset often fail on unseen pipelines because they learn dataset-specific artifacts, so strong in-domain accuracy can coexist with poor cross-domain performance.The paper identifies generalizable detection as important amid risks including misinformation, identity theft, and fraud.Source: §I · p. 1
  • Final CLIP embeddings emphasize semantics and may wash out subtle fabrication artifacts, motivating richer representations that preserve fine details and global context.Frozen vision-transformer features have shown strong transfer for synthetic-image detection with minimal training.Source: §I · p. 1
  • WAVIE fuses intermediate ViT features through learned layer-wise aggregation, then applies low-band wavelet refinement while preserving high-frequency detail.Its implementation uses shared projections around the wavelet block and a small fusion tensor and low-band linear head.Source: §I · p. 1
  • 0.8518/0.8520 AUROC on Celeb-DF-v1/v2 and 0.8313 on WDF; WAVIE is competitive with recent methods on Celeb-DF-v1 and best among listed methods on Celeb-DF-v2 and WDF.The evaluation uses an identical training pipeline and approximately 2.14M trainable parameters atop a frozen backbone.Source: §I · p. 1
  • Ablation and interpretability studies examine the roles of intermediate features, wavelet decomposition, and per-layer importance weights.The contribution statement continues across the two cited passages.Source: §I · pp. 1–2

II. RELATED WORK

Earlier detectors used generative fingerprints and artifact cues, while later work explored frequency analysis and pretrained CLIP representations. The cited work notes that dataset bias and reliance on final embeddings remain concerns for cross-domain detection.

A. Deepfake Generation and Early Detection Strategies

  • Frequency-domain methods targeted high-frequency inconsistencies and Face X-ray detected blending artifacts, yet supervised detectors remained sensitive to dataset bias and generalized poorly across domains.The passage cites SRM and Fourier-based representations,,, F3-Net[10], and Face X-ray.Source: A. Deepfake Generation and Early… · p. 2

1) Early Deepfake Detection:

B. CLIP and Vision Transformer (ViT) for Detection

  • Frozen CLIP embeddings can generalize across generators, but final embeddings may emphasize semantics and overlook subtle artifacts, motivating intermediate ViT features.CLIP combines a text encoder with a ViT-based image encoder, [14].Source: B. CLIP and Vision Transformer (ViT)… · p. 2

C. RINE: Leveraging Intermediate Transformer Features

WAVIE combines intermediate transformer features with frequency-aware processing, bringing together information that prior approaches use separately. It refines low-frequency features while preserving high-frequency detail, and is reported to improve generalization to unseen generators.

C. RINE: Leveraging Intermediate Transformer Features

  • RINE aggregates intermediate [CLS] representations with learned layer weights, but operates only in the spatial domain and does not explicitly use frequency information.Its layer-wise weighting combines low-, mid-, and high-level features.Source: C. RINE: Leveraging Intermediate… · p. 2

D. Wavelet Based Deepfake Detection (Wavelet-CLIP)

  • Wavelet-CLIP decomposes the final CLIP embedding into low- and high-frequency components, refining the low-frequency component while preserving the high-frequency component.Unlike WAVIE, it does not leverage intermediate transformer features.Source: D. Wavelet Based Deepfake Detection… · p. 2

III. METHODOLOGY: WAVIE

  • WAVIE refines the low-frequency branch of a three-level db6 wavelet decomposition, preserves high-frequency detail, and reconstructs the fused feature before classification.The method reports that this design improves generalization to unseen deepfake generators with minimal additional overhead.Source: §III · pp. 2–3
  • WAVIE applies its wavelet decomposition to the fused representation, refines only the low-frequency branch, and uses inverse wavelet reconstruction before classification.The low-frequency component represents global structure, while high-frequency components represent fine details.Source: §III · pp. 2–3
  • A frozen CLIP ViT-L/14 supplies per-block [CLS] features, which WAVIE aggregates using learned, per-layer and per-dimension weights.For this backbone, the representation uses 24 blocks and a feature width of 1024 before the unused projection head.Source: §III · pp. 2–3
  • Only lightweight modules are trained; the CLIP backbone and wavelet filters remain fixed, with approximately 2.14M trainable parameters for ViT-L/14.The trained components include shared projections, the low-frequency linear layer, the classifier, and aggregation weights.Source: §III · pp. 3–4

A. Datasets and Preprocessing

The experiments train and validate on FaceForensics++ and evaluate transfer to Celeb-DF and WildDeepFake. Face crops are processed consistently, and the study avoids multi-dataset training to isolate transfer effects.

A. Datasets and Preprocessing

  • The study trains and validates on raw FaceForensics++ c0 and evaluates transfer to Celeb-DF-v1/v2 and WildDeepFake.The authors focus on FF++→Celeb-DF/WDF transfer to isolate intermediate-layer fusion and wavelet-refinement effects without multi-dataset-training confounds.Source: A. Datasets and Preprocessing · p. 4, Table I · p. 4
  • FaceForensics++ training and validation use four manipulation types, with real videos sampled at 1 fps and fake videos at 4 fps to mitigate class imbalance.The resulting frame set contains 131,155 real and 175,465 fake frames, randomly split 80%/20% using a fixed seed.Source: A. Datasets and Preprocessing · p. 4
  • Frames are face-detected, cropped around the largest detected face with a 20-pixel box expansion when in-bounds, and resized to 224×224; frames without a detected face are discarded.All splits use CLIP RGB normalization and no additional augmentations.Source: A. Datasets and Preprocessing · p. 4

B. Network Architecture and Hyperparameters

The detector uses a frozen CLIP ViT-L/14 backbone and trains lightweight modules that aggregate intermediate features, refine a wavelet low-frequency component, and classify the reconstructed representation.

B. Network Architecture and Hyperparameters

  • A frozen CLIP ViT-L/14 backbone provides per-block [CLS] tokens, which are projected and fused by a Trainable Importance Estimator.The backbone remains frozen while the detector modules are trained.Source: B. Network Architecture and… · p. 4
  • A three-level db6 wavelet transforms the fused feature, and a single 137→137 linear layer refines its low-frequency component before classification.The architecture uses 1024-dimensional projected features.Source: B. Network Architecture and… · p. 4
  • The detector has approximately 2M trainable parameters atop the frozen backbone.Source: B. Network Architecture and… · p. 4

C. Training Protocol

Training uses AdamW for 20 epochs with a OneCycleLR schedule and exponential moving averages. A supervised contrastive term complements the classification objective, and the protocol fixes seeds and enables deterministic cuDNN operations.

C. Training Protocol

  • Training runs for 20 epochs with AdamW, learning rate 1×10−3, weight decay 10−2, batch size 32, and automatic mixed precision.OneCycleLR warms up and then decays the learning rate across epochs.Source: C. Training Protocol · p. 4
  • The supervised contrastive term uses same-class positives within a batch, while fixed random seeds and deterministic cuDNN operations support reproducibility.Real and fake frames are sampled at 1 fps and 4 fps, respectively, so no class reweighting is used.Source: C. Training Protocol · p. 4

D. Evaluation Protocol

The evaluation tests frame-level cross-dataset generalization after training on FF++. WAVIE is competitive on Celeb-DF-v1 and leads the listed methods on Celeb-DF-v2 and WDF.

D. Evaluation Protocol

  • Evaluation uses frame-level AUROC, training on FF++ and testing separately on Celeb-DF-v1/v2 and WildDeepFake with the same preprocessing.The authors describe these cross-dataset tests as the most stringent evaluation.Source: D. Evaluation Protocol · p. 4

A. Comparison with State-of-the-Art

B. Ablation Studies

Ablations on Celeb-DF-v1 and v2 assess intermediate-feature fusion, wavelet refinement, and learned layer weighting. Removing fusion or wavelet refinement substantially lowers cross-domain AUROC, while averaging trails WAVIE more modestly.

B. Ablation Studies

  • Removing intermediate-layer fusion drops cross-domain AUROC by nearly 10 points, underscoring the importance of intermediate transformer features.The no-fusion variant feeds the final CLIP ViT embedding directly into the wavelet module.Source: B. Ablation Studies · pp. 5–6, Table III · p. 6
  • Eliminating wavelet refinement further reduces performance, supporting its role in cross-domain robustness.The no-wavelet variant bypasses the DWT and sends the aggregated feature directly to the classifier.Source: B. Ablation Studies · pp. 5–6, Table III · p. 6
  • Uniformly averaging layer features is competitive but consistently trails WAVIE, indicating a modest benefit from learned per-layer weights when wavelet refinement is retained.The average-layer variant averages CLS vectors fetched from encoder blocks.Source: B. Ablation Studies · p. 6, Table III · p. 6

VI. CONCLUSION

The conclusion presents WAVIE as a parameter-efficient framework combining multi-layer transformer representations with frequency-domain refinement for cross-domain deepfake detection. Its evaluation is frame-level and face-centric, trained on FF++ c0 and tested on Celeb-DF and WildDeepFake.

VI. CONCLUSION

  • WAVIE combines multi-layer transformer representations and frequency-domain refinement, remaining competitive on FF++ while reducing the performance drop on unseen datasets.The authors report approximately 0.98 AUROC on FF++ and argue intermediate embeddings preserve fine-grained forgery cues often lost in final-layer representations.Source: §VI · p. 6
  • The evaluation is frame-level and face-centric, trained exclusively on FF++ c0 and tested on Celeb-DF and WildDeepFake.The design uses a fixed three-level db6 DWT without learned filters and does not use image-space augmentations, which may limit robustness.Source: §VI · p. 6
  • The authors identify temporal modeling, learnable wavelet banks, joint spatial-frequency attention, and robustness to compression and post-processing as future directions.Source: §VI · p. 6

Related papers

Loading 2609.14437v1…