PaperlayerView exact revision on arXiv

Source-linked AI summary

DBCF: Dual-Branch Complementary Fusion of Foundation Models for Generalized Deepfake Detection

Fengming Gu, Mingjie He, Zonghui Guo, Jie Zhangb, Shiguang Shan

arXiv:2609.34720v1cs.CV

TL;DR

Deepfake detectors struggle to generalize across evolving domains and unseen manipulations, and single foundation models offer incomplete cue coverage. DBCF combines frozen CLIP and DINOv3 branches through multi-scale spatial alignment, and reports strong cross-dataset and cross-manipulation results. Its two large backbones also bring inference and memory costs.

  • Problem

    Deepfake detectors often struggle to generalize across domains and unseen manipulations, while CLIP's global semantic bias can miss subtle local artifacts.

    Source: §1 · p. 2

  • Method

    DBCF keeps CLIP and DINOv3 backbones frozen and uses multi-scale spatial alignment and cross-feature interaction to combine global context with localized forgery cues.

    Source: §1 · p. 3, §3.3 · p. 9

  • Results

    DBCF achieves the best performance on most evaluated unseen datasets and manipulation types in cross-dataset and cross-manipulation settings.

    Source: §1 · p. 4, §5 · p. 23

  • Takeaways & Limitations

    The evaluations support combining complementary pretrained backbones to learn more comprehensive representations for forgery and spoofing detection.

    Source: §5 · pp. 22–23

    Using two large pretrained backbones increases model size, computation, inference latency, and GPU memory demand; inference takes 56 ms per frame at approximately 17.9 FPS and 6.1 GB peak GPU memory.

    Source: §6 · p. 23

Abstract

from arXiv · show

As image generation and editing technologies have progressed substantially, facial forgeries pose significant challenges to privacy and public safety. Due to limited ability to capture forgery cues, existing small-scale forgery detection models often struggle to generalize across various domains and unseen manipulations. To address this limitation, researchers have turned to large-scale foundation models, which can provide richer representations and better generalization. Nevertheless, relying on a single foundation model alone remains insufficient for effective forgery detection. While models like CLIP offer robust global semantic cues, they lack the capacity to capture detailed local facial features. In contrast, DINO excels at capturing local structural features of faces, but provides weaker global semantic context. To fully utilize the synergies among multiple foundation models, we propose a hierarchical multi-granular framework that integrates complementary pretrained representations. Specifically, a Global Context Branch (GCB) based on CLIP captures holistic semantic cues, while a Fine-grained Cue Branch (FCB) built on DINOv3 captures localized structural irregularities. In addition, we design a feature fusion module that enables parameter-efficient adaptation of the frozen foundation backbones by adaptively extracting and integrating complementary features from the two models. By jointly leveraging global context and fine-grained cues, our method learns more comprehensive forgery representations and achieves strong cross-manipulation performance. Extensive experiments on multiple benchmarks demonstrate the benefit of the proposed design, particularly under cross-dataset and cross-manipulation settings.

1. Introduction

Deepfake detectors remain challenged by evolving manipulations and limited training distributions, while pretrained models offer complementary global and local representations. DBCF combines CLIP and DINOv3 through aligned, frozen-backbone feature fusion and reports strong cross-dataset and cross-manipulation performance.

1. Introduction

  • Detector generalization remains difficult because forgery techniques evolve rapidly and prior approaches remain constrained by training-data scale and manipulation-sample distributions.Prior methods generate diverse manipulated samples or disentangle representations to suppress irrelevant factors such as identity, illumination, and background.Source: §1 · p. 2
  • CLIP-based detectors can miss subtle local artifacts because contrastive pretraining emphasizes global semantic alignment over localized visual inconsistencies.The text notes this limitation for manipulations such as NeuralTexture that do not alter overall semantics.Source: §1 · p. 3
  • CLIP supplies global semantic context, while DINOv3 models local structural patterns and fine-grained visual details.Their complementary representations motivate balancing holistic understanding with sensitivity to subtle manipulation cues.Source: §1 · p. 3
  • DBCF freezes CLIP and DINOv3 backbones and learns representation-level interactions to preserve pretrained priors and reduce the risk of dataset-specific overfitting.The design addresses the tension between preserving global semantics and enhancing sensitivity to fine-grained manipulation cues.Source: §1 · pp. 3–4
  • DBCF achieves strong performance in cross-dataset and cross-manipulation evaluations against approaches using frozen or fully fine-tuned pretrained backbones.The results are presented as evidence for combining complementary pretrained backbones for generalizable detection.Source: §1 · p. 4

2. Related Works

Related work spans face-forgery generation, domain-generalizable detection, complementary cue architectures, and pretrained vision models. Prior studies motivate combining transferable pretrained representations with architectural specialization for global and local cues.

Face Forgery Generation

  • Face-forgery generation includes face swapping, reenactment, and entire-face synthesis, while this paper focuses on manipulation-based face forgery rather than generic image-synthesis detection.Face swapping transfers identity, reenactment transfers motion, and entire-face synthesis generates photorealistic faces.Source: §2 · pp. 4–5

Face Forgery Detection

  • Detection research improves generalization through diverse forgery augmentation, feature disentanglement, real-face modeling, targeted supervision, and cross-domain training.Examples include Face X-ray, SBI, LSDA, disentanglement, and real-face representation learning.Source: §2 · p. 5
  • Hybrid detectors use specialized branches or representations to combine complementary cues, including local/global features in LGDF-Net and RGB/noise maps in Ding et al..The text presents transferable pretrained representations as another direction for complementary cue integration.Source: §2 · pp. 5–6

Vision Foundation Models

  • CLIP-based detectors such as CLIPping and UniFD adapt pretrained vision-language models, while Ffaa [25] and X2-DFD also explore them for deepfake detection.These studies motivate further ways to exploit pretrained visual priors for generalized detection.Source: §2 · p. 6
  • Cross-modal contrastive pretraining supports global semantic consistency and zero-shot transfer, whereas visual self-supervision learns image structure through image-derived supervisory signals.The passage names CLIP as a cross-modal example and MoCo, MAE, and DINO as SSL examples.Source: §2 · p. 6
  • DBCF's parallel GCB and FCB branches are connected by adaptive cross-feature interaction modules to combine global context with fine-grained forgery cues.DINOv3 is used for localized traces such as blending boundaries, texture or illumination mismatches, blurring, and structural distortions.Source: §2 · p. 7

3. Method

DBCF combines CLIP global semantics and DINOv3 local structure through spatially aligned, hierarchical feature interaction. A multi-scale decoder aggregates the resulting features for real/fake prediction.

3.1. Overview

  • DBCF aligns CLIP global-context and DINOv3 fine-grained features, then progressively integrates them with task-adaptive spatial priors for real/fake prediction.The framework comprises GCB, FCB, adaptive interaction modules, and a multi-scale decoder.Source: §3.1 · pp. 7–8

3.2. Global Context Branch

  • The frozen CLIP backbone supplies multi-granular global semantic features from four transformer layers: 5, 12, 18, and 24.Freezing CLIP preserves pretrained semantic biases and mitigates semantic collapse during training.Source: §3.2 · p. 9

3.3. Fine-grained Cue Branch

  • DINOv3 forms the FCB, capturing localized manipulation traces such as blending boundaries, texture or illumination mismatches, blur, and structural distortions.Its selected intermediate features complement CLIP’s global context; resolution-adaptive preprocessing aligns the branches’ token counts.Source: §3.3 · pp. 9–10

3.4. Adaptive Feature Learner

  • The trainable AFL extracts multi-scale spatial priors, projects them into a shared embedding space, and tokenizes three lower-resolution maps for interaction.Its highest-resolution map is reserved for multi-scale decoding.Source: §3.4 · pp. 10–11

3.5. Cross-Feature Interaction

  • CFI uses deformable attention to let multi-scale AFL features interact with spatially aligned GCB–FCB representations, progressively incorporating global and local cues.The four interaction levels correspond to selected transformer layers 5, 12, 18, and 24.Source: §3.5 · pp. 11–12

3.6. Multi-Scale Decoder and Training Objective

  • The decoder combines four resolution levels using top-down fusion, concatenates scale-specific descriptors, and feeds them to a linear classifier trained with cross-entropy.The feature maps span 1/4, 1/8, 1/16, and 1/32 of the input resolution.Source: §3.6 · pp. 12–13

4.1. Settings

The evaluation tests cross-dataset and cross-manipulation generalization with held-out benchmarks, specified AUC and HTER metrics, and a shared implementation protocol.

4.1.1. Datasets

  • Cross-dataset evaluation trains on FF++ c23 and tests on Celeb-DF v2, DFDC, and DFDCP; DF40 covers 40 manipulation techniques for cross-manipulation evaluation.The training subset contains 720 pristine and 2,880 manipulated videos; target evaluations use official test splits without cross-validation or test-time adaptation.Source: §4.1.1 · pp. 13–14

4.1.2. Evaluation Metrics

  • The study reports frame-level and video-level AUC, with video-level scores obtained by averaging frame probabilities; FAS experiments additionally use HTER.Source: §4.1.2 · p. 14

4.1.3. Implementation Details

  • Experiments use CLIP ViT-L/14-336 and DINOv3 ViT-L/16, train for 10 epochs, and sample 8 training frames per video.Cross-dataset tests use 32 frames per video; repeated-run results average three fixed-seed runs and report standard deviation.Source: §4.1.3 · pp. 14–15

4.2. Main Results

DBCF generalizes across unseen datasets and manipulation pipelines, while structured fusion improves on simpler branch combinations. Its attention maps show distinct global and local responses, and its robustness varies by perturbation type.

4.2.1. Cross-dataset evaluation

  • DBCF achieves the highest AUC on DFDC and DFDCP among the compared methods, while some methods score slightly higher on CDF.The comparison evaluates frame-level AUC on unseen datasets; DBCF also combines with SBI augmentation for improvements across multiple datasets.Source: §4.2.1 · pp. 15–16

4.2.2. Cross-manipulation evaluation

  • DBCF exceeds listed baselines on all five DF40 manipulation pipelines and achieves the highest average AUC, with facedancer and simswap proving more challenging.The authors associate these harder cases with weaker, more localized traces from high-fidelity face-swapping pipelines.Source: §4.2.2 · p. 17, Table 3 · p. 17

4.2.3. Ablation Study

  • Across datasets, all components together perform best; multi-granular aggregation and aligned feature interaction improve on branch-only or simple concatenation variants.Simple concatenation does not consistently help and can reduce performance; DBCF raises average frame-level AUC from 79.03% to 88.06% versus naive concatenation, with latency increasing from 33 ms to 56 ms per frame.Source: §4.2.3 · pp. 18–19, Table 4 · p. 19, Table 5 · p. 20

4.2.4. Visualization

  • GCB attention is relatively dispersed over faces and nearby regions, whereas FCB attention is more spatially coherent and locally concentrated within facial regions.The authors interpret this difference as broader contextual responses from GCB and greater sensitivity to localized appearance patterns from FCB.Source: §4.2.4 · p. 20, Figure 3 · p. 20

4.2.5. Robustness Analysis

  • DBCF has the lowest AUC reduction across most tested perturbations; compression causes a moderate drop, while RGB shift and contrast variation leave performance largely unaffected.The analysis includes image compression, RGB shift, and contrast variation; smaller AUC reduction indicates stronger robustness.Source: §4.2.5 · p. 21, Figure 4 · p. 21

4.3. Extensive Experiments on Face Anti-Spoofing (FAS)

In cross-dataset face anti-spoofing, DBCF performs strongly across transfer scenarios, and the dual-branch design combines the strengths of the global and local branches.

4.3. Extensive Experiments on Face Anti-Spoofing (FAS)

  • Across all cross-dataset transfer settings, Ours dual keeps HTER below 5 and reaches AUC up to 99.66, outperforming compared methods in average metrics.The evaluation uses four benchmarks and a standard Leave-One-Out protocol for each source-target combination.Source: §4.3 · pp. 21–22, Table 6 · p. 22
  • The FCB-only branch improves overall HTER stability relative to GCB alone, while the dual-branch model combines their strengths across cross-dataset scenarios.GCB alone ranges from 2.67% HTER on O&M&I →C to 17.14 on O&C&M → I.Source: §4.3 · p. 22

5. Conclusion

DBCF combines CLIP’s global semantic representations with DINO’s fine-grained local features for facial forgery and spoofing detection. Multi-scale spatial alignment supports feature integration, and the reported experiments show strong performance across unseen datasets and manipulation types.

5. Conclusion

  • DBCF combines CLIP’s semantic-level generalization with DINO’s fine-grained local features to capture global and local manipulation cues.The framework targets facial forgery and spoofing detection.Source: §5 · p. 22
  • Cross-dataset and cross-manipulation experiments report the best performance on most evaluated unseen datasets and manipulation types.The authors describe these results as demonstrating generalization capability and robustness under distribution shifts.Source: §5 · p. 23
  • Multi-scale spatial alignment promotes interaction across heterogeneous features at different spatial levels while preserving each branch’s intrinsic strengths.The mechanism is designed to improve integration of global and local information.Source: §5 · p. 23
  • The authors present DBCF as a flexible foundation for open-domain detection, with potential extensions to video-level and multi-modal detection.Source: §5 · p. 23

6. Limitations

DBCF’s two large frozen backbones increase model size, inference cost, latency, and memory use relative to single-backbone alternatives. The framework also does not explicitly identify how cues interact in individual detection decisions.

6. Limitations

  • DBCF’s two frozen large backbones increase model size, computational cost, inference latency, and GPU memory demand compared with single-backbone alternatives.The authors report 56 ms per frame, approximately 17.9 FPS, and peak GPU memory consumption of approximately 6.1 GB under the stated inference setting.Source: §6 · p. 23
  • The reported inference demands may restrict deployment in strict real-time or resource-constrained scenarios.The authors note these costs accompany performance gains in their evaluations.Source: §6 · p. 23
  • DBCF does not explicitly identify how different cues interact in each detection decision; future work includes cue visualization or forgery localization.The authors also propose efficient backbone replacement, model compression, or distillation to improve deployment efficiency.Source: §6 · p. 24

CRediT authorship contribution statement

The statement assigns technical, administrative, and writing contributions across the authors, with supervision and funding acquisition listed for several authors.

Related papers

Loading 2609.34720v1…