PaperlayerView exact revision on arXiv

Source-linked AI summary

Leveraging Representations from Intermediate Encoder-blocks for Synthetic Image Detection

Christos Koutlis, Symeon Papadopoulos

arXiv:2402.19091v2cs.CV

TL;DR

Realistic synthetic images pose risks, while existing foundation-model SID features emphasize high-level semantics rather than potentially useful low-level details. RINE uses intermediate CLIP representations with trainable block weighting, and reports an average +10.6% performance improvement across 20 test datasets.

  • Problem

    Foundation-model features have shown strong SID performance, but last-layer features mainly capture high-level semantics, while intermediate low-level features had not been explored for the task.

    Source: §1 · p. 1

  • Method

    RINE projects intermediate CLIP Transformer-block representations into a learnable forgery-aware space and uses a trainable module to weight each block’s contribution.

    Source: §1 · p. 2

  • Results

    RINE surpasses state-of-the-art performance by an average +10.6% across 20 test datasets.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The reported performance is achieved with one training epoch (about 8 minutes) and limited training data, and the method is robust to common image transformations.

    Source: §6 · p. 11

Abstract

from arXiv · show

The recently developed and publicly available synthetic image generation methods and services make it possible to create extremely realistic imagery on demand, raising great risks for the integrity and safety of online information. State-of-the-art Synthetic Image Detection (SID) research has led to strong evidence on the advantages of feature extraction from foundation models. However, such extracted features mostly encapsulate high-level visual semantics instead of fine-grained details, which are more important for the SID task. On the contrary, shallow layers encode low-level visual information. In this work, we leverage the image representations extracted by intermediate Transformer blocks of CLIP's image-encoder via a lightweight network that maps them to a learnable forgery-aware vector space capable of generalizing exceptionally well. We also employ a trainable module to incorporate the importance of each Transformer block to the final prediction. Our method is compared against the state-of-the-art by evaluating it on 20 test datasets and exhibits an average +10.6% absolute performance improvement. Notably, the best performing models require just a single epoch for training (~8 minutes). Code available at https://github.com/mever-team/rine.

1 Introduction

Realistic synthetic imagery creates societal and economic risks while challenging reliable detection. RINE addresses an unexplored gap in CLIP-based SID by using intermediate encoder representations, and reports improved performance across 20 test datasets.

1 Introduction

  • Highly realistic generated images can deceive people and contribute to societal and economic risks, including fraud and hoaxes, [5], [6].The paper identifies GANs and diffusion models among the generative approaches producing such images, [4].Source: §1 · p. 1
  • Existing SID approaches use image-level or frequency-level features and generator traces, while deep-learning methods have struggled to generalize beyond GANs [8].The cited approaches include image-level methods [10], frequency-level features [11] [12] [13], and trace analysis [14] [15].Source: §1 · p. 1
  • Foundation-model features have performed well in SID [16], but last-layer features capture high-level semantics, while intermediate-layer low-level features had not been explored for the task [15].RINE projects intermediate CLIP representations into a learnable forgery-aware space and uses TIE to weight each block’s contribution.Source: §1 · pp. 1–2
  • RINE surpasses state-of-the-art performance by an average +10.6% across 20 test datasets, using models trained for one epoch (about 8 minutes).The experiments train on ProGAN images from 1, 2, or 4 object classes.Source: §1 · p. 2

2 Related Work

Prior SID work pursued generator fingerprints and varied detection strategies, but generalization to unseen generators remained difficult. Foundation-model features offered a promising direction, and the paper proposes exploring intermediate layers while distinguishing its approach from deep-layer aggregation.

2 Related Work

  • Early fingerprint and manipulation detectors often evaluated on generators also used for training, and such approaches struggled on unseen generators [8].Follow-on strategies included patch predictions, autoencoders, co-occurrence matrices, frequency methods, gradients, and augmentation [10], [8], [24], [11],, [15].Source: §2 · p. 2
  • ProGAN-trained detectors using preprocessing and augmentation generalized to many GAN generators, while perceptual-quality selection helped cross-domain generalization [29].Other cited work exploited frequency traces [26] [11] and frequency-level perturbation maps [27].Source: §2 · p. 2
  • Foundation-model features performed well on both GAN- and diffusion-generated data [16], motivating the hypothesis that intermediate low-level representations could further improve SID.The paper contrasts frozen foundation-model outputs with deep-layer aggregation, which concatenates CNN layer outputs during training [30],.Source: §2 · p. 2
  • Prior work analyzed detector performance drops under social-media transformations such as cropping and compression [15].The paper states that its own method is also effective under these conditions, referring to section 5.6.Source: §2 · p. 2

3 Methodology

RINE uses frozen CLIP intermediate-block representations to capture low-level visual information, then learns projections and block-wise feature importance for synthetic image detection. It trains the resulting classifier with binary cross-entropy and supervised contrastive objectives.

3.1 Representations from Intermediate Encoder-Blocks

  • RINE concatenates CLS tokens from CLIP’s Transformer blocks, keeping CLIP frozen while using the intermediate representations to construct discriminative SID features.The representations are projected and combined into a forgery-aware feature space for classification.Source: §3.1 · p. 3

3.2 Trainable Modules

  • A projection network maps intermediate representations, while the Trainable Importance Estimator weights learned features across Transformer blocks before a second projection and classification head.The importance weights estimate each feature’s relevance at each processing stage.Source: §3.2 · pp. 3–4

3.3 Objective Function

  • The model combines binary cross-entropy for SID classification with supervised contrastive loss, which brings same-target feature vectors closer and separates the rest.A tunable factor ξ combines the two objectives.Source: §3.3 · p. 4

4 Experimental Setup

The experiments train on ProGAN images and evaluate across 20 synthetic-image datasets spanning GAN, translation, face-manipulation, and diffusion sources. They compare against published SID methods using accuracy and average precision under a shared evaluation protocol.

4.1 Datasets

  • Testing spans 20 datasets, including images from GANs, image-translation and face-manipulation systems, and multiple diffusion-model variants.Training uses ProGAN-generated images and corresponding real images, with 1-, 2-, and 4-class training settings.Source: §4.1 · p. 4

4.2 Implementation Details

  • RINE is trained for one epoch with Adam, while hyperparameter searches vary loss weight, projection depth, feature width, and CLIP variant.Training images receive blur and JPEG augmentation, random cropping, and horizontal flips; validation and test images are center-cropped.Source: §4.2 · p. 4, Table 1 · p. 5

4.3 State-of-the-Art Methods

  • The comparison includes ResNet-50, Patch-Forensics, FrePGAN, LGrad, DMID, and CLIP-based UFD as state-of-the-art SID methods.Scores use official checkpoints except FrePGAN, for which the paper reports the results published in its paper [27].Source: §4.3 · p. 5

4.4 Evaluation Protocol

  • Evaluation reports ACC and AP per dataset, uses a 0.5 threshold for accuracy, selects best models by summed ACC+AP, and reports cross-dataset averages.The protocol follows previous works for comparability.Source: §4.4 · p. 5

5 Results

RINE outperforms prior state-of-the-art methods across the evaluation datasets, and ablations identify intermediate representations as its most consequential component.

5.1 Comparative Analysis

5.2 Ablations

  • Removing each of RINE’s three main components reduces performance, with removing intermediate representations causing the largest loss and returning metrics to prior state-of-the-art levels.Source: §5.2 · pp. 5–6

5.3 Hyperparameter Analysis

Hyperparameter experiments favor ViT-L/14 and show limited sensitivity to the contrastive loss factor; extending training does not improve performance.

5.3 Hyperparameter Analysis

  • CLIP ViT-L/14 outperforms ViT-B/32 across the 1-, 2-, and 4-class settings.Source: Figure 3 · p. 6
  • Performance varies little across contrastive-loss factor values; for q=2 and q=4, increasing d′ brings a slight average performance increase.Source: Figure 3 · p. 6

5.4 The Effect of Training Duration

  • Neither average nor maximum performance improves with continued training, and performance drops in the 1- and 4-class settings.Source: §5.4 · p. 6

5.5 The Effect of Training Set Size

Performance remains close when training data is reduced, especially for average precision; tested social-media-style perturbations have varying effects.

5.5 The Effect of Training Set Size

  • Results using 20%, 50%, 80%, and 100% of training data are very close, especially for average precision.The 20% subsets contain 7K, 14K, and 28K images for the 1-, 2-, and 4-class settings, respectively.Source: §5.5 · pp. 7–8

5.6 Robustness to Common Image Perturbations

  • Cropping and compression cause minimal performance loss, while combined perturbations considerably reduce performance to levels similar to unperturbed state-of-the-art results.Source: §5.6 · p. 8

5.7 Importance of Intermediate Stages

The analysis finds that intermediate Transformer blocks often receive greater maximum importance than the last block, supporting the use of intermediate representations. The supplied figures and table also describe evaluation across training-data amounts and image perturbations.

5.7 Importance of Intermediate Stages

  • The block-importance analysis uses the importances estimated by TIE to assess the contribution of each intermediate stage.Source: §5.7 · p. 8
  • Many Transformer blocks have a higher maximum-importance frequency than the last block across all training settings.Figure 7 reports the frequency of each block obtaining maximum importance; for the 1-class and 4-class models n=24 and d′=1024, and for the 2-class model n=24 and d′=128.Source: §5.7 · p. 8, Figure 7 · p. 10
  • Figure 6 reports ACC and AP across 20 evaluation datasets for models trained with 20%, 50%, 80%, or 100% of the training data in 1-, 2-, and 4-class settings.Table 5 is captioned as reporting ACC and AP after common image perturbations; no outcome values are supplied here.Source: Figure 6 · p. 9, Table 5 · p. 10

5.8 The Effect of Training Data Origin on Detecting Diffusion Images

The section evaluates RINE on Synthbuster using training data from ProGAN and a Latent Diffusion Model. It reports that training on diffusion data provides large performance gains.

5.8 The Effect of Training Data Origin on Detecting Diffusion Images

  • Training on diffusion data provides big performance gains on Synthbuster.Source: §5.8 · p. 10
  • The Synthbuster evaluation uses ten-crop validation and averages predictions because of the images’ size.Source: §5.8 · p. 10
  • Table 6 reports RINE performance on Synthbuster after training on ProGAN data from or Latent Diffusion Model data from [15].Source: §5.8 · p. 10, Table 6 · p. 11

6 Conclusions

The conclusion presents RINE, which uses intermediate CLIP encoder-block representations for synthetic image detection, and reports superior state-of-the-art performance. It also highlights short training, limited data needs, and robustness to common image transformations.

6 Conclusions

  • RINE leverages intermediate CLIP encoder-block representations for synthetic image detection and outperforms the state of the art in the reported study.Source: §6 · p. 11
  • The best-performing approach requires one epoch, about 8 minutes of training, and limited training data to achieve maximum performance.Source: §6 · p. 11
  • The conclusion reports that RINE is robust to common image transformations.Source: §6 · p. 11

Appendix

The appendix compares alternative feature-fusion approaches, training-class configurations, and backbones with RINE. The reported results favor RINE over the tested FPN variants and simpler backbone choices.

Appendix

  • FPN variants perform below RINE: CLIP w/ FPN scores 69.6 vs. 91.5 ACC and 97.1 vs. 98.8 AP.CLIP w/ FPN raises GPU memory use from 7GB to 28GB and training time from 8 to 36 minutes per epoch.Source: Appendix · p. 15, Table 7 · p. 15
  • RINE’s 20-class configuration roughly preserves its 4-class performance without further tuning, with 90.7 ACC and 98.1 AP; the authors report no GAN overfitting.The comparison trains RINE on 20 classes and UFD on 4 classes.Source: Appendix · p. 15, Table 7 · p. 15
  • Using ImageNet-pretrained ViT or Wang’s detector as the backbone significantly decreases performance relative to the CLIP-based RINE setup.Source: Appendix · p. 15

Related papers

Loading 2402.19091v2…