PaperlayerView exact revision on arXiv

Source-linked AI summary

Image Copy Detection for Diffusion Models

Wenhao Wang, Yifan Sun, Zhentao Tan, Yi Yang

arXiv:2409.19952v2cs.CV

TL;DR

Existing image copy detection methods overlook diffusion-generated replicas, a challenge relevant to content originality. ICDiff combines a manually labeled replication dataset with PDF-Embedding, and the paper reports replication ratios of 10% to 20% across six well-known diffusion models.

  • Problem

    Current ICD methods trained on hand-crafted image transformations overlook diffusion-generated replicas, which the authors report can confuse these methods.

    Source: §1 · p. 2

  • Method

    ICDiff introduces a manually labeled dataset and PDF-Embedding, which turns replication levels into probability density functions for learning image representations.

    Source: §1 · pp. 2–3

  • Results

    10% to 20% of images generated by six well-known diffusion models replicate contents of a large-scale image gallery.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The authors hope D-Rep and PDF-Embedding will serve as a resource for research on replication in diffusion models and encourage further research.

    Source: §6 · p. 10

    The authors acknowledge that the random variables are discrete, while using the term “PDF” to communicate their intuition and present the method under ideal conditions.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Images produced by diffusion models are increasingly popular in digital artwork and visual marketing. However, such generated images might replicate content from existing ones and pose the challenge of content originality. Existing Image Copy Detection (ICD) models, though accurate in detecting hand-crafted replicas, overlook the challenge from diffusion models. This motivates us to introduce ICDiff, the first ICD specialized for diffusion models. To this end, we construct a Diffusion-Replication (D-Rep) dataset and correspondingly propose a novel deep embedding method. D-Rep uses a state-of-the-art diffusion model (Stable Diffusion V1.5) to generate 40, 000 image-replica pairs, which are manually annotated into 6 replication levels ranging from 0 (no replication) to 5 (total replication). Our method, PDF-Embedding, transforms the replication level of each image-replica pair into a probability density function (PDF) as the supervision signal. The intuition is that the probability of neighboring replication levels should be continuous and smooth. Experimental results show that PDF-Embedding surpasses protocol-driven methods and non-PDF choices on the D-Rep test set. Moreover, by utilizing PDF-Embedding, we find that the replication ratios of well-known diffusion models against an open-source gallery range from 10% to 20%. The project is publicly available at https://icdiff.github.io/.

1 Introduction

Existing image copy detection methods overlook diffusion-generated replicas, motivating ICDiff, a specialized detection task with a new dataset and embedding method. The paper reports replication in 10% to 20% of images from six diffusion models against a large-scale gallery.

  • Current ICD methods trained on hand-crafted transformations can be confused by diffusion-generated replicas, which the authors attribute to a pattern gap.The paper contrasts hand-crafted transformations such as flips, rotations, and crops with diffusion-generated replication.Source: §1 · p. 2, Figure 2 · p. 2
  • ICDiff is introduced as the first image copy detection approach specialized for diffusion-generated replicas.The paper frames the task as identifying content replication caused by diffusion models, without addressing the fair-use or copyright dispute.Source: §1 · p. 2
  • D-Rep contains 40,000 image-replica pairs generated with Stable Diffusion V1.5 and manually labeled across six replication levels, from 0 to 5.The source images are from LAION-Aesthetic V2, and the dataset is split into 36,000 training and 4,000 test pairs.Source: §1 · p. 2
  • PDF-Embedding turns each pair’s replication level into a probability density function and learns image representations using sets of vectors.Its supervision encodes the intuition that probabilities for neighboring replication levels should vary smoothly.Source: §1 · p. 2
  • 10% to 20% of images generated by six well-known diffusion models replicate contents of a large-scale image gallery.The paper presents this finding alongside its proposed method and dataset.Source: §1 · p. 3

2 Related Works

Prior ICD research focuses on copies produced by image transformations, while diffusion-replication research has examined its causes and prevention. ICDiff takes a detection-focused approach using a manually labeled dataset and specialized algorithm.

2.1 Existing Image Copy Detection Methods

  • Existing ICD methods address transformed copies through approaches including transformation-aware training, contrastive learning, and model ensembles.Examples include ASL, SSCD, BoT [17], CNNCL [18], and EfNet [19].Source: §2.1 · p. 3

2.2 Replication in Diffusion Models

  • Prior diffusion-replication work examines training-data mirroring and prevention, whereas ICDiff develops a manually labeled dataset and specialized detection algorithm.The paper distinguishes its approach from work using existing descriptors such as CLIP and SSCD.Source: §2.2 · p. 3

3 Benchmark

The benchmark introduces D-Rep, a dataset of diffusion-generated replicas labeled by replication degree, and protocols for evaluating predictions. It uses PCC and Relative Deviation to assess prediction relationships and normalized error.

3.1 D-Rep Dataset

  • D-Rep pairs real images with diffusion-generated replicas, addressing replication from diffusion models rather than hand-crafted transformations.The dataset construction generates candidate pairs and then manually labels them.Source: §3.1 · p. 3
  • Forty thousand candidate pairs were labeled across six replication levels by ten professional labelers, with votes resolving undecided cases.The levels run from 0 to 5, and the proportions were not predetermined.Source: §3.1 · p. 4

3.2 Evaluation Protocols

  • PCC measures the linear relationship between predicted and ground-truth replication levels, but can remain high despite a consistent global shift.The authors introduce Relative Deviation (RD) to address PCC’s insensitivity to global shifts; RD normalizes deviation against the maximum possible deviation on a 0-to-1 scale.Source: §3.2 · pp. 4–5
  • In the examples, S_i distinguishes predictions that cannot deteriorate further from those that can, while T_i assigns both cases the same value.The two cases have T_i = 0.6, but different S_i values.Source: §3.2 · p. 5

4 Method

PDF-Embedding converts replication levels into probability density functions and learns a set of representative vectors for each image. The method specifies target-distribution properties, image representations, and KL-based training.

4 Method

  • PDF-Embedding converts each replication level into a probability density function and expands the original representative vector into a set of vectors for learning.The method is presented as a way to learn from the converted PDFs.Source: §4 · p. 5

4.1 Converting Level to Probability Density

  • For a normalized level pl, the target distribution is nonnegative, integrates to one, and reaches its maximum at pl; neighboring levels are intended to vary smoothly.The paper notes that the levels are discrete but uses the term PDF to communicate its intuition under ideal conditions.Source: §4.1 · pp. 5–6

4.2 Representing an Image as a Set of Vectors

  • The method represents real and generated images with sets of ViT-derived representative vectors, with N set to the highest replication level.The ViT processes image patch tokens through L layers; the resulting class-token vectors form the representative sets.Source: §4.2 · p. 6
  • Training defines the final loss with KL divergence, which measures disparity between probability distributions.At testing, the normalized replication level is described as discrete within [0, 1].Source: §4.2 · p. 6
  • Table 1 compares publicly available models and PDF-Embedding on D-Rep; qualitative results are directed to Appendix E.Source: Table 1 · p. 7

5 Experiments

The experiments evaluate ICDiff baselines, PDF-Embedding, and diffusion-model replication. PDF-Embedding outperforms protocol-driven and non-PDF alternatives, while evaluated diffusion models show replication ratios of roughly 10%–20%.

5.1 Training Details

  • Training uses a ViT-B/16 backbone pretrained on ImageNet with DeiT, 224 × 224 images, batch size 512, and 25 epochs.Training is distributed over eight A100 GPUs.Source: §5.1 · p. 7

5.2 Challenge from the ICDiff Task

  • GPT-4V Turbo leads public models in PCC at 47.3%, while DINOv2 leads in RD at 32.9%; current ICD models also perform poorly on D-Rep.SSCD records 29.1% PCC and 62.3% RD; BoT records 35.6% PCC and 53.8% RD.Source: §5.2 · p. 7

5.3 The Effectiveness of PDF-Embedding

  • Compared with Enlarging PCC, PDF-Embedding improves PCC by 1.6% and decreases RD by 16.1%; using exponential PDF is better than label smoothing (ϵ = 0.5) by +21.3% PCC and −10.5% RD.Source: §5.3 · p. 8
  • Exponential PDFs benefit PCC, Gaussian PDFs favor RD, and performance remains high with the exponential amplitude A from 0.6 to 1.8.The linear function performs between the convex exponential and concave Gaussian; smoother PDFs outperform steeper ones.Source: §5.3 · pp. 8–9, Figure 5 · p. 8
  • PDF-Embedding generalizes better than zero-shot solutions and other D-Rep-trained methods, with no significant drop from SD1.5 testing to the generalization setting.The evaluation uses 100 manually confirmed highest-level replication pairs per diffusion model because six-level labeling is difficult for those collections.Source: §5.3 · p. 9, Table 3 · p. 9

5.4 Simulated Evaluation of Diffusion Models

  • Replication ratios across the evaluated diffusion models range roughly from 10% to 20%; Midjourney reaches 20.21% and SDXL 10.91%.The evaluation uses LAION-Aesthetics V2 6+ as the gallery and treats levels 4 and 5 as replications.Source: §5.4 · pp. 9–10, Figure 6 · p. 10
  • For a 12-million-image reference dataset, PDF-Embedding takes 2.07×10^-3 seconds for inference and 8.36 × 10^-2 seconds for matching.The paper describes this overhead as negligible compared with image generation, which takes several seconds.Source: §5.4 · p. 10
  • Compared with SSCD-filtered examples, PDF-Embedding retrieves more diverse hard positives, including replicas with substantial visual differences from originals.Examples include a full-body Groot replica versus an original showing only its face, and differently positioned Moai statues.Source: §5.4 · p. 10, Figure 6 · p. 10

6 Conclusion

The conclusion presents D-Rep and PDF-Embedding as resources for studying diffusion-model replication. The paper cautions that the model may produce false positive or negative predictions.

6 Conclusion

  • D-Rep annotates replicas at six levels, and PDF-Embedding smooths those levels into PDFs learned with representative vectors.The conclusion frames the dataset and method as a baseline for research on replication in diffusion models.Source: §6 · p. 10
  • The model may yield false positive or negative predictions, and the paper says its contents are not legal advice.Source: §6 · p. 10

A More examples of the D-Rep Dataset

The appendix provides additional image pairs for each D-Rep replication level.

A More examples of the D-Rep Dataset

B The Instantiations of PDFs

The section instantiates replication-level supervision with Gaussian, linear, and exponential PDFs centered on the normalized level. Their curvature shapes how quickly the distribution changes near that level.

B The Instantiations of PDFs

C The Visualization of What the Network Learns

The authors visualize learned vectors and image-pair distributions to examine what PDF-Embedding learns. The vectors separate different replication levels while retaining similarity between neighboring levels.

C The Visualization of What the Network Learns

  • Different learned vectors have low cosine similarity, while neighboring vectors have relatively high cosine similarity, matching their distinct and similar replication levels.Source: C The Visualization of What the… · p. 17
  • After training, the network produces an appropriate distribution for each image pair, closely imitating its Gaussian supervision in the illustrated example.Source: C The Visualization of What the… · p. 17

D Implement GPT-4V Turbo on our D-Rep Test Dataset

The section compares GPT-4V Turbo and other models with ICDiff on D-Rep. GPT-4V Turbo is prompted to score image-pair similarity, but its performance does not match ICDiff and practical API use is costly.

D Implement GPT-4V Turbo on our D-Rep Test Dataset

E The Similarities Predicted by Other Models

F The Details of Six Diffusion Models

The section describes the sources used to evaluate six diffusion models, provides further replication examples, and identifies two types of detection failures involving common elements and semantic similarity.

F The Details of Six Diffusion Models

  • Evaluation covers three commercial and three open-source diffusion models, with image sources including user-generated Midjourney and New Bing images, DALLE·2 images, and locally generated SDXL, DeepFloyd IF, and GLIDE images.Source: F The Details of Six Diffusion Models · p. 18

G More Replication Examples

H Failure Cases

Loading 2409.19952v2…