PaperlayerView exact revision on arXiv

Source-linked AI summary

Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection

Zhiyuan Yan, Jiangming Wang, Peng Jin, Ke-Yue Zhang, Chengchun Liu, Shen Chen, Taiping Yao, Shouhong Ding, Baoyuan Wu, Li Yuan

arXiv:2411.15633v4cs.CV

TL;DR

AIGI detectors can struggle to generalize to previously unseen fake-generation methods. Effort uses SVD to preserve pretrained knowledge in one subspace while adapting another to forgery patterns, and reports superiority over other state-of-the-art methods across deepfake and synthetic-image benchmarks. The paper also argues that modeling fakes as derived from real images is important for generalizable detection.

  • Problem

    AIGI detectors can perform well when training and test fake-generation methods are similar, but their performance degrades on previously unseen methods.

    Source: §1 · p. 2

  • Method

    Effort uses SVD to construct orthogonal subspaces, preserving pretrained knowledge while adapting the remaining components to learn forgery patterns.

    Source: §1 · pp. 2–3

  • Results

    Across deepfake and synthetic-image benchmarks, Effort reports significant superiority over other state-of-the-art methods with very little training cost.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper argues that generalizable detection should model fake images as derived from real images, rather than as an independent class.

    Source: §1 · p. 3

    The rank-1 choice is motivated specifically for deepfake detection, where FF++ has only four forgery types and limited pattern diversity.

    Source: How to determine the value of rank in… · p. 9

Abstract

from arXiv · show

AI-generated images (AIGIs), such as natural or face images, have become increasingly important yet challenging. In this paper, we start from a new perspective to excavate the reason behind the failure generalization in AIGI detection, named the \textit{asymmetry phenomenon}, where a naively trained detector tends to favor overfitting to the limited and monotonous fake patterns, causing the feature space to become highly constrained and low-ranked, which is proved seriously limiting the expressivity and generalization. One potential remedy is incorporating the pre-trained knowledge within the vision foundation models (higher-ranked) to expand the feature space, alleviating the model's overfitting to fake. To this end, we employ Singular Value Decomposition (SVD) to decompose the original feature space into \textit{two orthogonal subspaces}. By freezing the principal components and adapting only the remained components, we preserve the pre-trained knowledge while learning fake patterns. Compared to existing full-parameters and LoRA-based tuning methods, we explicitly ensure orthogonality, enabling the higher rank of the whole feature space, effectively minimizing overfitting and enhancing generalization. We finally identify a crucial insight: our method implicitly learns \textit{a vital prior that fakes are actually derived from the real}, indicating a hierarchical relationship rather than independence. Modeling this prior, we believe, is essential for achieving superior generalization. Our codes are publicly available at \href{https://github.com/YZY-stack/Effort-AIGI-Detection}{GitHub}.

1. Introduction

Naively trained detectors overfit limited seen fake patterns, constraining their feature spaces and impairing detection of unseen forgeries. Effort uses orthogonal subspaces to preserve pretrained knowledge while learning forgery patterns, and frames fake images as derived from real ones.

1. Introduction

  • Naively trained detectors quickly overfit seen fake patterns, treating unseen forgeries as real and limiting generalization.The paper attributes this asymmetry to training sets with limited, homogeneous fake types and reports that fake loss can be about 100 times lower than real loss early in training.Source: §1 · pp. 2–3
  • PCA analysis finds that only two principal components capture all information in a naive detector’s feature space, which the authors link to limited generalization.The paper describes the resulting representation as highly constrained and low-ranked.Source: §1 · pp. 2–3
  • Effort uses SVD to form two orthogonal subspaces, freezing principal components to preserve pretrained knowledge while adapting the remaining components to learn forgery patterns.The design aims to expand the constrained feature space without distorting the original pretrained representation.Source: §1 · pp. 2–3
  • Across deepfake and synthetic-image benchmarks, Effort reports significant superiority over other state-of-the-art methods with very little training cost.The paper contrasts its explicit orthogonality with full-parameter and LoRA-based tuning, arguing this maintains a higher-rank feature space and enhances generalization.Source: §1 · p. 2
  • The paper frames fake images as generated from real ones, creating a hierarchy that Effort models by discriminating within semantically aligned subspaces.For example, the model can focus on distinguishing fake dogs from real dogs, reducing model complexity.Source: §1 · p. 3

2. Related Work

The related work reviews generalization in deepfake and synthetic-image detection, where training and testing may differ in forgery methods. It surveys fake-pattern learning, real-distribution learning, and vision-foundation-model approaches.

Generalizable Deepfake Image Detection

  • Deepfake detectors often learn fake patterns through transformations, but the growing diversity of forgery methods makes enumerating patterns for unseen-method generalization unrealistic.Reviewed transformations include frequency processing, blending, reconstruction, and content or identity disentanglement.Source: §2 · p. 3
  • One-class anomaly detection models abnormal samples through reconstruction error, while limited real data and imbalanced facial attributes challenge robust real-image representation learning.Larue et al. (2023) also create pseudo-fake anomalies by blending facial regions; FF++ contains 1,000 real videos.Source: §2 · pp. 3–4

Generalizable Synthetic Image Detection

  • Synthetic-image detectors mine low-level artifacts through RGB augmentation, frequency, gradients, reconstruction, neighboring pixels, and random-mapping features.Examples include dual high-pass filters in BiHPF (Jeong et al., 2022), pretrained-model gradients in LGrad, and up-sampling artifacts in NPR (Tan et al., 2024c).Source: §2 · p. 4
  • Vision-foundation-model approaches include frozen CLIP features with a tuned classifier and image-text contrastive supervision, with reported gains on previously unseen fake methods.UniFD (Ojha et al., 2023) uses a frozen visual encoder and linear layer; LASTED (Wu et al., 2023) uses textual labels.Source: §2 · p. 4

3. Methodology

Effort uses SVD to separate pretrained weights into components that preserve pretrained knowledge and a residual component for learning forgery-related patterns. Orthogonality and singular-value constraints regularize the residual learning while the pretrained components remain fixed.

3. Methodology

  • Effort applies SVD to pretrained weights, keeping a top-r approximation fixed and using the residual component to learn representations specific to fake images.The fixed approximation preserves dominant pretrained knowledge, while only the residual weights are optimized during training.Source: §3 · p. 4
  • The residual component is intended to capture real/fake discrepancies while minimizing changes to the pretrained weight matrix’s real information.Effort applies orthogonality and singular-value constraints to support this goal.Source: §3 · p. 4
  • An orthogonality constraint maintains orthogonality among singular vectors, with concatenated left and right singular-vector matrices used in the constraint.The passage defines the Frobenius norm and identity matrix used in the expression.Source: §3 · pp. 4–5
  • Figure 5 shows a principal-component approximation branch and a residual branch for orthogonal learning of real/fake discriminative features.The approach uses SVD to preserve pretrained knowledge while learning forgery-related patterns.Source: §3 · p. 4, Figure 5 · p. 5
  • The singular-value constraint keeps optimized weights’ singular values consistent with the original weights to control the impact of ΔW and prevent overfitting.The paper relates singular values to the weight matrix’s Frobenius norm.Source: §3 · p. 5
  • Training combines classification and orthogonality-regularization losses, with hyperparameters weighting regularization and application to self-attention linear layers across transformer layers.The algorithm updates ΔW using the total loss and returns the weight matrix as Wr + ΔW.Source: §3 · p. 5

4. Experiment

The experiments evaluate Effort on deepfake and synthetic-image benchmarks, then examine its ablations, compatibility, feature-space behavior, and rank choice. Across the reported settings, results support generalization while preserving pretrained representations with limited training parameters.

4.1. Deepfake Image Detection

  • Effort consistently and largely outperforms competing detectors across tested deepfake scenarios, using 0.19M training parameters versus about 100M for several recent methods.The evaluation trains detectors on FF++ c23 and tests them on other fake data.Source: §4.1 · p. 7

4.2. Synthetic Image Detection

  • On UniversalFakeDetect, Effort reaches 95.19% mAcc and 99.41% mAP, improving over UniFD by 9.27% mAcc and 13.81% mAP.It also improves mAcc by 4.33% over FatFormer without using CLIP’s extra text encoder.Source: Table 3 · p. 6, Table 2 · p. 6, §4.2 · p. 7

4.3. Ablation Study and Analysis

  • AUC rises from 0.859 for the baseline to 0.925 with SVD alone and peaks at 0.941 with SVD plus both constraints.Applying the approach to BEIT-v2 and SigLIP also improves generalization performance.Source: §4.3 · pp. 7–8, Table 4 · p. 7, Table 5 · p. 7
  • Effort retains 159/160 principal components and 99.3% of original variance, while LoRA and full fine-tuning lose 14.4% and 64.4% of variance, respectively.The paper also reports higher generalization results than adapter-based methods across deepfake and synthetic-image detection.Source: Figure 6 · p. 8, §4.3 · p. 8, Table 8 · p. 8
  • Logit visualizations show Xception and fully fine-tuned CLIP collapsing discrimination into a single dimension, whereas Effort learns forgery features in an orthogonal complement to CLIP’s representation.The figure description characterizes Effort as best retaining pretrained knowledge.Source: §4.3 · pp. 8–9, Figure 7 · p. 9

How to determine the value of rank in tuning SVD?

  • The authors choose rank n-r=1 for deepfake detection because the task and training-set forgery patterns are simple, and ranks 1, 4, and 16 perform similarly.They prioritize efficiency and parameter economy, arguing that higher rank offers little additional benefit for binary classification.Source: How to determine the value of rank in… · p. 9

5. Conclusion

The conclusion attributes generalization failures to detectors overfitting seen fake patterns and proposes Effort’s orthogonal decomposition as a remedy. It also presents fake images as derived from real images in a hierarchy and reports advantages on deepfake and synthetic-image benchmarks.

5. Conclusion

  • The paper identifies shortcutting to seen fakes as a source of low-ranked feature spaces, then uses orthogonal subspaces to preserve pretrained knowledge while learning forgery patterns.It reports that experiments on deepfake and synthetic-image benchmarks demonstrate advantages for the method.Source: §5 · p. 9

6. Impact Statements

The impact statement describes the work’s potential positive application in detecting AI-generated images and notes possible misuse to improve deepfake realism. The authors plan access controls to address that risk.

6. Impact Statements

  • The authors identify misuse to enhance deepfake realism as a possible negative outcome and plan measures such as access control.They also urge minimizing negative impacts while leveraging the work’s positive contributions.Source: §6 · p. 9

A. Additional Results and Ablations

A.1. Results on GenImage Benchmark

On GenImage, Effort is evaluated after training on SDv1.4 and outperforms the reported comparison methods on the remaining test set. It also exceeds the second-best video detectors on CDF-v2 and DFDC under video-level AUC.

A.1. Results on GenImage Benchmark

  • 91.1% accuracy on GenImage exceeds UnivFD by 11.6%, FatFormer by 2.2%, and DRCT by 1.6% when SDv1.4 is the training set.The test set comprises the remaining generator models in the benchmark.Source: §A.1 · p. 14

A.2. Comparison with Existing Video Detectors

  • Effort improves over the second-best video-based models by 4.5 points on CDF-v2 and 3.9 points on DFDC in video-level AUC.The comparison includes classical and recent video detectors.Source: §A.2 · p. 14

A.3. Robustness Evaluation

The approach is evaluated against unseen perturbations using video-level AUC across three degradation types and five severity levels. It generally achieves higher results than other methods.

A.3. Robustness Evaluation

  • The approach generally achieves higher video-level AUC than other methods under unseen perturbations.The passage characterizes this as better robustness, without providing numerical values.Source: §A.3 · p. 14
  • Evaluations cover Block-wise distortion, Change contrast, and JPEG compression at five levels each.The model is trained on FF++ (c23), and the evaluations follow prior perturbation methodologies.Source: §A.3 · p. 14
  • The reported measure is video-level AUC across the three perturbation types and their degradation levels.Results are depicted in Fig. 8.Source: §A.3 · p. 14

B. Additional cross-method evaluation

The section evaluates generalization across manipulation methods using DF40 and an additional cross-method protocol. The proposed method is reported to outperform FA-ViT on accuracy in the stated GID evaluations.

B. Additional cross-method evaluation

  • 2.85% on GID-DF and 3.14% on GID-F2F: the method's accuracy exceeds FA-ViT by these amounts, respectively.The comparison is reported for cross-manipulation evaluation.Source: B. Additional cross-method evaluation · p. 14
  • The DF40 evaluation trains on four FF++ (c23) manipulation methods and tests on the other eight DF40 techniques.The section also reports an additional protocol that trains on diverse manipulation types and tests on unknown methods.Source: B. Additional cross-method evaluation · p. 14

B.1. Additional Ablation Studies

The ablations report improved generalization across several pre-trained ViT initializations and architectures, with CLIP and ViT-Large selected as defaults. The section also identifies the evaluation settings for additional benchmark and robustness comparisons.

B.1. Additional Ablation Studies

  • The proposed approach improves generalization across BEIT-v2, CLIP, and SigLIP initializations, with CLIP performing best on both detection tasks.The tasks are deepfake detection and synthetic image detection; CLIP is chosen as the default initialization.Source: §B.1 · p. 14, Table 12 · p. 16, Table 13 · p. 16
  • The Genimage cross-method benchmark reports accuracy when SDv1.4 is the training set and other generative models are test sets.Table 9 includes results from cited papers and reproduced official checkpoints; the supplied passage does not state a winning method.Source: Table 9 · p. 15
  • The robustness figure presents video-level AUC across five degradation levels for three unseen perturbation types.The caption identifies the perturbation setting but does not state comparative outcomes.Source: Figure 8 · p. 15
  • The initial feature-space visualization shows that pre-trained CNN and CLIP models can identify different semantic objects.The comparison is between vanilla CNN and CLIP latent spaces.Source: Figure 9 · p. 15
  • Across ViT-Base-16 and ViT-Large-14, the approach enhances generalization over fully fine-tuning CLIP, and CLIP-Large performs better than CLIP-Base.The experiments cover deepfake and synthetic image detection; ViT-Large is ultimately selected.Source: §B.1 · p. 15
  • Table 10 covers cross-dataset evaluation against existing video detectors using video-level AUC.The caption says results for other detectors are cited from their original papers.Source: Table 10 · p. 16

C. Additional Analysis and Visualizations

Additional visualizations contrast how the approach and vanilla CNN represent seen and unseen fakes, and examine attention maps in CLIP-ViT. The authors report preserved pre-trained knowledge alongside learned forgery patterns and orthogonality between semantic and forgery subspaces.

C.1. Additional t-SNE Visualizations

  • The approach preserves pre-trained knowledge while fitting forgery patterns, whereas vanilla CNN overfits seen fakes and yields a low-ranked feature space.The vanilla CNN also shows greater overlap between real and fake logits, while the overlap is smaller for the proposed approach.Source: §C.1 · pp. 16–17
  • The ViT backbone ablations compare architectures on deepfake detection trained on FF++ (c23) and synthetic detection trained on ProGAN images, tested on 19 generator types.The captions identify the evaluation settings but do not provide outcome values.Source: Table 14 · p. 17, Table 15 · p. 17

C.2. Self-attention Map Visualizations

  • Attention visualizations place semantic information primarily in earlier blocks and show orthogonality between semantic and learned forgery subspaces in the proposed approach.The comparison includes original, fully fine-tuned, LoRA-trained, and orthogonally trained CLIP-ViT models.Source: §C.2 · p. 17

D. Limitation and Future Work

The authors identify grouping all fake methods into one training class as a limitation, and propose incremental learning with a separate SVD branch per fake method as future work. They also discuss broader applications and provide attention visualizations across several datasets.

D. Limitation and Future Work

  • Grouping all fake methods into one class during training may ignore the specificity and generality of different fake methods.This is the limitation the authors identify for their real/fake classifiers.Source: D. Limitation and Future Work · p. 17
  • The authors propose treating each fake method as an SVD branch in an incremental learning framework to preserve orthogonality and avoid severe forgetting.They intend this extension to better address future deepfake types in real-world scenarios.Source: D. Limitation and Future Work · p. 17
  • The authors note potential applications beyond deepfake and synthetic image detection, including face anti-spoofing and anomaly detection.They also hope the work inspires orthogonal modeling strategies.Source: D. Limitation and Future Work · p. 17
  • The facial images used come from publicly available datasets with citations, and the authors plan to release code and checkpoints upon acceptance.The paper states these sources avoid infringement upon personal privacy.Source: D. Limitation and Future Work · p. 17
  • Supplementary attention visualizations pair fake and corresponding real images from CycleGAN, ProGAN, DeepFake, and LDM across 24 CLIP-ViT blocks.Block IDs increase from top to bottom.Source: Figure 11 · p. 18, Figure 12 · p. 19, Figure 13 · p. 20
Loading 2411.15633v4…