PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Shortcut Learning: Hierarchical Spatial Prototype Regularization for Face Forgery Detection

Chao Shuai, Shaojing Fan, Kun Pan, Jinwu Hu, Weichen Lian, Ziqi Yang, Zhongjie Ba, Zhenguang Liu

arXiv:2610.29442v1cs.CV

TL;DR

Deepfake detectors can rely on unstable, dataset-specific cues, limiting generalization to unseen manipulations. The paper proposes HSP, which uses structured spatial prototypes and regularization to model multiple forgery factors. HSP improves generalization, and as a plug-in module raises Effort’s average video-level AUC from 93.5% to 95.3%.

  • Problem

    Deepfake detectors may overfit volatile, dataset-specific cues, and the stability of learned forensic representations across manipulation sources is a key generalization concern.

    Source: §1 · p. 1

  • Method

    HSP models spatial forgery features with learnable prototypes, using orthogonality and second-order assignment balance to structure factor representations and promote their diverse use.

    Source: §1 · p. 2, §4.2 · p. 4

  • Results

    HSP improves Effort’s average video-level AUC from 93.5% to 95.3%, with consistent gains across the reported benchmarks.

    Source: §5.3 · p. 7

  • Takeaways & Limitations

    HSP is presented as a plug-in regularization module for existing detectors, and the conclusion reports state-of-the-art generalization on prominent benchmarks.

    Source: §5.3 · p. 7, §6 · p. 7

    Prototype factors are not guaranteed to align with physically interpretable manipulation traces because the factorization uses representation-level constraints rather than cue-specific supervision.

    Source: G Limitation · p. 14

Abstract

from arXiv · show

Recent advances in generative models have enabled highly realistic face forgeries, yet generalizing deepfake detectors to unseen manipulation pipelines remains a major challenge. One critical yet underexplored issue is that the learned forensic representation itself may fail to remain stable across different manipulation sources. In this paper, we provide new evidence that naively fine-tuned vision models suffer from severe \textit{forensic representation collapse}: their extracted cues are often entangled with unstable, dataset-specific artifacts, resulting in structurally fragile representations and poor cross-domain generalization. Leveraging this insight, we propose \textbf{H}ierarchical \textbf{S}patial \textbf{P}rototype (\textbf{HSP}) regularization, which explicitly models the forgery manifold by learning a structured prototype subspace over spatial features. We further introduce prototype orthogonality and second-order assignment balance, which jointly preserve the geometric independence of latent forgery factors and promote high-effective-rank utilization of the forensic representation, thereby reducing reliance on dataset-specific shortcuts. Empirical results demonstrate that our method substantially improves generalizability, yielding average frame-level AUC gains of 5.6\% and 5.3\% in the cross-dataset and cross-manipulation settings, respectively. Moreover, when used as a plug-in module, HSP boosts the average video-level AUC of existing state-of-the-art detectors to 95.3\% (+1.8\%). These results demonstrate the practical value of HSP as an easy-to-adopt regularization module for enhancing deepfake detectors.

1 Introduction

Deepfake detectors often transfer poorly to unseen manipulations because they rely on volatile, dataset-specific cues. The paper proposes HSP to organize forgery evidence into a structured, multi-factor representation.

1 Introduction

  • Replacing CLIP’s global [CLS] representation with average pooling over spatial patch tokens improves cross-dataset accuracy on Celeb-DF-v2 from 75.4% to 76.9% and DFDCP from 77.8% to 78.6%.This motivates modeling forgery representations as a structured feature space rather than a compact global representation.Source: §1 · p. 1
  • Detectors can overfit dataset-specific cues, while learned representations may concentrate on narrow source-specific forgery directions that fail under unseen manipulations.The paper terms this failure forensic representation collapse.Source: §1 · p. 1
  • HSP is presented as a plug-in module for standard baselines and advanced ViT-based deepfake detectors.The paper motivates the multi-factor view by the diversity and complementarity of forensic traces associated with different forgery processes.Source: §1 · p. 2
  • HSP uses learnable prototypes as a structured basis to softly assign and reconstruct spatial tokens as latent forgery factors.Prototype orthogonality decouples factors; second-order assignment balance promotes diverse prototype use and higher effective rank.Source: §1 · p. 2

2 Related work

Prior deepfake detectors target manipulation artifacts, inconsistencies, and foundation-model adaptation or explainability. HSP is framed as a shift from a monolithic decision boundary toward a structured forgery subspace.

2 Related work

  • Artifact-focused methods mine high-frequency anomalies, fine-grained features, blending artifacts, and simulated forgery patterns.The cited approaches include Face X-Ray and pseudo-fake synthesis methods (Ma et al. 2025).Source: §2 · p. 2
  • Consistency methods use spatial, identity, and temporal cues, while other approaches leverage diffusion guidance or feature disentanglement for manipulation-invariant representations.LipForensics captures temporal mouth-movement irregularities.Source: §2 · p. 2
  • Foundation-model detectors adapt backbones efficiently, disentangle semantic bias, or target open-set generalization and explainability.Examples include Forensics Adapter, Effort (Yan et al. 2024b), and FakeRadar.Source: §2 · p. 2
  • HSP differs by constructing a structured, multidimensional forgery subspace rather than learning a monolithic decision boundary.Figure 1 contrasts entangled real–fake separation with organization of forgery traces into a manifold supported by multiple latent factors.Source: §2 · p. 2, Figure 1 · p. 2

3 Empirical Analysis of Generalization Bottlenecks

Across OOD evaluations, HSP-CLIP maintains cluster-level fake density while improving cluster-label alignment and cross-dataset fake-density consistency. Its representation also requires more principal components and activates broader spatial regions than naively fine-tuned CLIP.

3 Empirical Analysis of Generalization Bottlenecks

  • The diagnostics treat unsupervised feature clusters as temporary proxies for latent forensic factors, and the experiments report that modeling multiple factors improves cross-domain generalizability.The benchmark analysis evaluates detectors trained on FaceForensics++ and tested on Celeb-DF-v2, DFDCP, and DFDC.Source: §3 · pp. 2–3, §3.2 · p. 3

3.1 Feature-Centric Perspective Metrics

  • The analysis uses cluster-level fake density, cluster–label mutual information, and cross-dataset fake-density correlation to characterize representation structure.Higher fake density indicates more forgery-related evidence in a cluster; higher mutual information indicates stronger alignment with forensic labels, and higher correlation indicates more consistent fake-dominated clusters across domains.Source: §3.1 · p. 3

3.2 Empirical Analysis on Deepfake Benchmarks

  • HSP-CLIP preserves high cluster-level fake density while consistently improving cluster-label dependence and cross-dataset fake-density correlation under OOD evaluation.Vanilla CLIP has nearly random scores, while naive fine-tuning raises fake density but its dependence and correlation measures degrade under OOD evaluation.Source: Table 1 · p. 3, §3.2 · p. 4
  • HSP-CLIP requires substantially more principal components than naively fine-tuned CLIP to reach the same explained-variance thresholds and shows broader, more complementary spatial activations.The PCA analysis uses patch embeddings from 5,000 test images and compares cumulative token-wise projection-energy maps.Source: Figure 2 · p. 3, §3.2 · p. 4

4 Methodology

HSP regularization organizes intermediate spatial features around complementary forgery factors while preserving the original representation. Prototype orthogonality and assignment balance encourage diverse, high-effective-rank use of the forgery manifold.

4.1 Prototype-Supported Forgery Manifold

  • Each spatial token is softly assigned to learnable prototypes, and an auxiliary reconstruction loss aligns features with shared forgery factors without replacing the original tokens.The dual-path reconstruction objective uses stop-gradient, with α set to 2 by default.Source: §4.1 · p. 4

4.2 Structured Forgery Manifold Regularization

  • Prototype orthogonality encourages geometrically decoupled forgery factors, while assignment balance distributes feature variation across the centered assignment subspace.Together, these constraints encourage use of all admissible prototype-assignment directions rather than a few source-specific prototypes.Source: §4.2 · pp. 4–5, Figure 3 · p. 5

4.3 Effective Rank of the Forensic Representation

  • The effective-rank proposition states that vanishing orthogonality and balance losses yield a rank-(K −1) spectrum with equal nonzero eigenvalues for reconstructed features.With small reconstruction error, the original centered features approximately inherit this high-effective-rank structure.Source: §4.3 · p. 5

4.4 Integration and Training Objective

  • HSP is a lightweight plug-in inserted at one-quarter and one-half transformer depth, with training combining classification, reconstruction, orthogonality, and assignment-balance losses.The method is evaluated on cross-dataset and unseen-manipulation benchmarks.Source: §4.4 · pp. 5–6, Table 2 · p. 6, Table 3 · p. 6

5 Experiments

HSP-CLIP improves generalization across datasets and manipulation methods, while ablations indicate that its performance depends on where and how prototypes are applied. The experiments also report compatibility with different vision backbones and an existing detector.

5.1 Experimental Setting

  • Experiments train on FaceForensics++ c23 and assess cross-dataset and cross-manipulation generalization, using CLIP ViT-L/14 as the default backbone.The reported measures include video-level AUC and frame-level AUC/ACC.Source: §5.1 · p. 6

5.2 Cross-Paradigm Generalization Ability

  • Frame-level average AUC gains are +5.6% over FSgD (85.2%) cross-dataset and +5.3% over FSgD(87.8%) cross-manipulation; video-level averages are 94.5% and 96.6%, respectively.Cross-dataset video-level AUC surpasses χ2-DFD by +2.6%; cross-manipulation video-level AUC exceeds Effort and VbSaT by +1.8%.Source: §5.2 · p. 6

5.3 Ablation Study

  • HSP placement at L1+L2 achieves the best overall results, while applying it at all three positions provides no further improvement.L1, L2, and L3 mark one-quarter, one-half, and three-quarters of backbone depth; the authors favor shallow-to-intermediate features for fine-grained cues.Source: §5.3 · p. 7, Figure 4 · p. 7
  • Adding assignment balance Lbal after reconstruction Lrec and orthogonality Lorth achieves the best results across all benchmarks.The authors attribute the improvement to balanced prototype utilization promoting a higher-effective-rank representation and cross-domain generalization; Table 4 reports ACC ablations.Source: §5.3 · p. 7, Table 4 · p. 7
  • HSP consistently improves ACC and AUC across backbones and datasets, including ViT-Base, SigLIP, and CLIP.Table 5 covers HSP integrated into different vision foundation models.Source: §5.3 · p. 7, Table 5 · p. 7
  • HSP improves average video-level AUC for Effort from 93.5% to 95.3%, with consistent gains across benchmarks.Figure 4(a) reports video-level AUC for this detector integration.Source: §5.3 · p. 7, Figure 4 · p. 7
  • The best prototype count differs by evaluation: K = 128 for DFDCP and K = 32 for FaceDancer.The authors use K = 128 by default, prioritizing cross-dataset generalization; Figure 5 examines prototype count.Source: §5.3 · p. 7, Figure 5 · p. 7

6 Conclusion

The conclusion identifies forensic representation collapse—overreliance on volatile, dataset-specific shortcuts—as a bottleneck in generalizable deepfake detection, and presents HSP as a prototype-based response. The accompanying diagnostic interpretation distinguishes local fake enrichment, label discrimination, and cross-dataset stability, while noting that no single diagnostic suffices.

6 Conclusion

  • HSP models the forgery manifold with learnable orthogonal prototypes to capture diverse, structurally independent forensic cues and increase effective representation capacity.The conclusion reports state-of-the-art generalization on prominent benchmarks.Source: §6 · p. 7

A Theoretical Interpretation of the Diagnostic Metrics

  • The diagnostic metrics distinguish local fake density, cluster–label dependence, and cross-dataset fake-density correlation, and none is sufficient in isolation.The empirical definition used in the main paper corresponds to the sample estimate of Eq. (15).Source: A Theoretical Interpretation of the… · p. 10

A.1 Local Concentration Measured by H

The four diagnostics sections explain local fake enrichment, global label discrimination, and cross-domain structural stability as complementary properties of learned feature regions. Together, they clarify why a single strong diagnostic can still coexist with a nontransferable representation.

A.1 Local Concentration Measured by H

  • When clustering is label-independent, every nonempty cluster’s fake density equals the global fake prior; under balanced evaluation, clusters above 0.5 are locally fake-enriched.A high local density can still matter little globally when the cluster has little probability mass.Source: §A.1 · p. 10

A.2 Global Discriminability Measured by I

  • Cluster–Label Dependence I is mutual information between cluster assignments and labels; zero dependence means they are statistically independent.I weights cluster-specific fake-density deviations by cluster occurrence probability, capturing global rather than merely local discriminative relevance.Source: §A.2 · p. 10

A.3 Cross-Domain Stability Measured by P

  • Cross-domain stability P correlates aligned source- and target-domain fake-density profiles over the top-M source-selected clusters.A high P reflects preservation of relative organization, even if target fake densities are lower; selecting clusters from the source avoids target-domain information leakage.Source: §A.3 · p. 11

A.4 The Complementarity of Three Diagnostics

  • High H, I, and P capture complementary properties but do not guarantee low target-domain classification error.A source artifact can yield high H and I yet low P if it disappears or reverses its label association after domain shift; high P can also coexist with low I.Source: §A.4 · p. 11

B The Impact of Different M on Diagnostic Metrics

HSP-CLIP maintains higher cluster-level fake density and cross-dataset fake-density correlation than CLIP-FT across target datasets and tested M values. The advantage in correlation is largest at small M, when the most fake-enriched clusters are evaluated.

B The Impact of Different M on Diagnostic Metrics

  • HSP-CLIP achieves higher cluster-level fake density and cross-dataset fake-density correlation P than CLIP-FT across target datasets and tested M values.The comparison uses cluster-level fake density H̄ and cross-dataset fake-density correlation P.Source: B The Impact of Different M on… · p. 11, Table 6 · p. 12
  • The improvement in P is most pronounced at small M and narrows as more clusters are included.The authors attribute the narrowing to added clusters with weaker discriminative relevance or generic visual content.Source: B The Impact of Different M on… · pp. 11–12
  • CLIP-FT concentrates forgery evidence along a few dominant directions, making its most fake-enriched clusters more likely to encode source-specific shortcuts.HSP’s multi-factor forgery-space modeling is associated with greater consistency of dominant forensic regions across domains.Source: B The Impact of Different M on… · pp. 11–12

C Evaluation Benchmarking

The benchmark compares a diverse set of deepfake detectors under a shared training setup. Models trained on FaceForensics++ (c23) are evaluated on multiple unseen datasets and manipulation techniques to assess cross-domain generalization.

C Evaluation Benchmarking

  • All models are trained on FaceForensics++ (c23) and evaluated across unseen datasets and manipulation techniques.The evaluation is intended to assess cross-domain generalization.Source: C Evaluation Benchmarking · p. 12
  • The benchmark includes F3Net, Face X-Ray, RECCE, SBIs, LaV, CFM, IID, UCF, FakeRadar, LSDA, FCG, ProDet, VbSaT, DeepShield, χ2-DFD, FreqDebias, ExposDe, FSgD, RAID, UDD, DFFake, ICT, CDFA, and Effort.Source: C Evaluation Benchmarking · p. 12
  • The compared methods span approaches including parameter-efficient adaptation, feature disentanglement, and plug-and-play video detection.Source: C Evaluation Benchmarking · p. 12

D Robustness to Image Degradations

HSP-CLIP remains stable under five unseen image degradations and across severity levels, while prototype-count and layer-wise weighting experiments reveal setting-dependent design choices. The results favor larger prototype spaces for cross-dataset benchmarks and reduced regularization in the second HSP layer for more consistent performance.

D Robustness to Image Degradations

  • HSP-CLIP maintains consistently high video-level AUC across five unseen perturbations and severity levels, outperforming reproduced baselines with identical augmentation.The perturbations include color saturation, JPEG compression, color contrast, block-wise distortion, and Gaussian blur; competing methods drop especially under severe JPEG compression and blur.Source: D Robustness to Image Degradations · pp. 12–13

E.1 Impact of the Number of Prototypes K

  • Cross-dataset performance generally improves with larger prototype spaces, peaking at K = 128, while cross-manipulation performance peaks at K = 32 and slightly declines beyond it.The tested prototype counts are 8, 16, 32, 64, and 128.Source: §E.1 · p. 13, Table 7 · p. 13

E.2 Impact of Hierarchical Loss Weighting

  • Hierarchical layer-wise weighting yields more consistent performance across most benchmarks than uniform weighting.It halves the second HSP layer’s loss weights; the authors explain that this preserves deeper-layer semantics while structuring fine-grained cues in earlier layers.Source: §E.2 · p. 13, Table 8 · p. 13

F.1 Prototype Diversity Analysis.

The analyses find that HSP prototypes are nearly orthogonal, assignments use a broad range of prototypes and directions, and prototype activations exhibit distinct spatial preferences. Together, these results characterize the learned representation’s diversity and spatial structure.

F.1 Prototype Diversity Analysis.

  • Mean absolute off-diagonal cosine similarities are 1.90 × 10−4 and 4.66 × 10−4 for Modules 1 and 2, respectively, indicating nearly independent prototype directions.The similarity matrices have identity-like structures with diagonal entries of 1.0.Source: §F.1 · pp. 13–14, Figure 7 · p. 14

F.2 Prototype Assignment Analysis.

  • Module 1 and Module 2 achieve assignment perplexities of 125.97 and 126.44 and effective ranks of 95.09 and 103.87, respectively.The perplexities approach the maximum of 128; the effective ranks correspond to 74.8% and 81.8% of the admissible maximum, respectively, with values approaching the theoretical maximum of 127 particularly for Module 2.Source: §F.2 · p. 14

F.3 Visualization of Spatial Prototype Activation Maps.

  • Eight representative prototypes show distinct spatial preferences, including localized eye or mouth artifacts, global facial structures, and blending boundaries.The visualizations use second-module token-to-prototype assignment maps upsampled to original image resolution.Source: §F.3 · p. 14

G Limitation

HSP’s prototype factors are constrained by representation-level objectives, so they are not guaranteed to align with physically interpretable manipulation traces.

G Limitation

  • HSP’s prototype factors are not guaranteed to align with physically interpretable manipulation traces, motivating finer-grained supervision for interpretability and semantic alignment.Suggested directions include manipulation-specific auxiliary objectives or forgery-region localization.Source: G Limitation · p. 14
  • The two HSP modules achieve perplexity close to 128 and effective ranks of 95.09 and 103.87, near the admissible maximum of 127.The figure presents prototype-usage frequencies against a uniform baseline and normalized eigenvalue spectra of centered assignment covariance.Source: Figure 8 · p. 15
  • Across input faces, eight selected prototypes activate distinct, consistent localized regions such as eyes, mouths, or blending boundaries.The figure visualizes bilinearly upsampled token-to-prototype soft assignments.Source: Figure 9 · p. 15

Related papers

Loading 2610.29442v1…