PaperlayerView exact revision on arXiv

Source-linked AI summary

A Self-Supervised Descriptor for Image Copy Detection

Ed Pizzi, Sreya Dutta Roy, Sugosh Nagavara Ravindra, Priya Goyal, Matthijs Douze

arXiv:2202.10261v2cs.CVcs.CRcs.LG

TL;DR

Image copy detection must re-identify altered or adversarially edited copies at scale while supporting strict match decisions. SSCD adapts self-supervised contrastive learning with entropy regularization, specialized architecture and augmentations, and descriptor-integrated score normalization. The method significantly improves copy-detection accuracy, with strong results on DISC2021 and transfer to Copydays.

  • Problem

    Image copy detection must identify altered or adversarially edited copies at web scale, where compact descriptors and strict match thresholds are required.

  • Method

    SSCD adapts contrastive learning with entropy regularization, copy-aware architecture and augmentations, and background-based similarity normalization incorporated into the descriptor.

  • Results

    SSCD significantly surpasses baselines on DISC2021 and transfers to state-of-the-art results on Copydays.

  • Takeaways & Limitations

    SSCD provides an efficient copy-detection model with a compact descriptor and calibrated distances that limit verification candidates.

  • Takeaways & Limitations

    SSCD is imprecise for text-only images, may confuse distinct photos of the same scene, and struggles with partial copies; high precision often requires verification.

Abstract

from arXiv · show

Image copy detection is an important task for content moderation. We introduce SSCD, a model that builds on a recent self-supervised contrastive training objective. We adapt this method to the copy detection task by changing the architecture and training objective, including a pooling operator from the instance matching literature, and adapting contrastive learning to augmentations that combine images. Our approach relies on an entropy regularization term, promoting consistent separation between descriptor vectors, and we demonstrate that this significantly improves copy detection accuracy. Our method produces a compact descriptor vector, suitable for real-world web scale applications. Statistical information from a background image distribution can be incorporated into the descriptor. On the recent DISC2021 benchmark, SSCD is shown to outperform both baseline copy detection models and self-supervised architectures designed for image classification by huge margins, in all settings. For example, SSCD out-performs SimCLR descriptors by 48% absolute. Code is available at https://github.com/facebookresearch/sscd-copy-detection.

1. Introduction

Image copy detection supports scalable content moderation but must identify altered copies under strict match decisions and rare positives. SSCD adapts self-supervised contrastive learning with entropy regularization and score normalization for this setting.

  • Motivation: Copy detection can automate repeated moderation decisions by re-identifying images previously reviewed by humans.This avoids manually moderating thousands of copies of viral images.
  • Challenges: Alterations from screenshots or adversarial edits make copied images difficult to re-identify.Users may add content or deliberately modify images to evade moderation.
  • Challenges: At web scale, systems use compact descriptor vectors and approximate nearest-neighbor search for retrieval.The paper focuses on retrieval, the first stage before candidate verification.
  • Challenges: Rare positive pairs and hard match/non-match thresholds make copy detection stricter than ordinary image retrieval.Retrieval ranking alone is insufficient because systems must limit verification candidates using a threshold.
  • SSCD approach: SSCD uses differential entropy regularization to promote uniform embeddings and more comparable distances across embedding regions.The regularization also avoids embedding collapse and can improve ranking metrics without query-consistent thresholds.
  • SSCD approach: SSCD incorporates background-image statistics into the descriptor through score normalization and is presented as a strong single-model baseline.The introduction also states that code and models are planned for release.

2. Related work

The paper situates SSCD among content-based tracing, near-duplicate matching, instance matching, contrastive self-supervision, and entropy-regularized representation learning. It targets strict same-source image copy detection rather than broader semantic or instance similarity.

  • Content tracing: Content tracing methods include metadata-based, watermarking, and content-based approaches; SSCD belongs to the content-based family.Classical tracing datasets often target localized tampering such as splicing, removal, and copy-move edits.
  • Copy definition: This work defines images as copies only when they originate from the same 2D image source.More relaxed near-duplicate definitions may also match nearby video frames.
  • Instance matching: Instance matching recognizes the same 3D object across viewpoint or camera changes and commonly uses specialized pooling and vector normalization.The paper builds on this literature because it addresses complex image matching.
  • Contrastive self-supervision: SimCLR uses transformed copies as a surrogate task, large batches or memory banks, and InfoNCE to bring same-source image copies together.Its learned representations are designed to transfer to tasks such as image classification.
  • Entropy regularization: Differential entropy regularization spreads descriptors across representation space and has been used with contrastive losses for category and instance retrieval.SSCD applies the idea to self-supervised copy detection.

3. Motivation

The motivation experiments combine SimCLR’s contrastive objective with differential entropy regularization. Entropy regularization harms ImageNet classification but improves copy detection by producing clearer separation between matching and non-matching images.

  • 3.1. Preliminaries: SimCLR: SimCLR creates two augmented views per image, compares L2-normalized descriptors with cosine similarity, and uses InfoNCE to favor copies over non-copies.The training setup forms positive pairs within each mini-batch, while inference can use pooled CNN features directly.
  • 3.1. Preliminaries: SimCLR: InfoNCE is a temperature-adjusted softmax cross-entropy averaged over positive descriptor pairs.Each positive pair is compared against descriptors from other images in the batch.
  • 3.2. Entropy regularization: The entropy loss regularizes nearest neighbors from different source images and pushes nearby descriptor vectors apart.Its influence is strongest for nearby vectors and diminishes as distances increase.
  • 3.2. Entropy regularization: The preliminary objective adds λ times the entropy loss to InfoNCE and evaluates the result on classification and copy detection.This isolates the effect of entropy regularization before expanding it into SSCD.
  • 3.2. Entropy regularization: As entropy weight increases, ImageNet linear classification accuracy decreases, whereas copy-detection accuracy increases significantly.The figure compares trends within each task; ImageNet and DISC2021 measures are not directly comparable.
  • 3.3. Experiment: SimCLR and entropy: Entropy regularization increases distances and narrows the negative-neighbor distribution, creating greater separation between matching pairs and typical non-matches.The comparison uses baseline SimCLR versus SimCLR with entropy weight λ = 30, without whitening or similarity normalization.

4. Method

SSCD adapts self-supervised contrastive learning for image copy detection through specialized architecture, augmentations, entropy regularization, and descriptor normalization. Its training handles mixed-image copies, while inference produces compact whitened descriptors and supports background-based similarity normalization.

  • Architecture: SSCD adapts the architecture, data augmentation, pooling, and normalization stages of self-supervised contrastive learning for copy detection.The model uses a ResNet-50 trunk and is designed specifically for the matching task rather than classification transfer.
  • Architecture: GeM pooling with p = 3 replaces average pooling to improve descriptor discriminability for instance retrieval and copy detection.GeM interpolates between average pooling at p = 1 and max-pooling as p approaches infinity.
  • Architecture: GeM improves training only when combined with differential entropy regularization; with vanilla InfoNCE, average pooling trains better.Learning the pooling parameter causes it to grow unbounded and training to become numerically unstable.
  • Data Augmentation: Mixed-image MixUp and CutMix augmentations model partial copies by combining content from two images within a training batch.These augmentations require contrastive losses that accommodate multiple matching images.
  • Training Objective: The mixed-image InfoNCE loss averages positive-match terms per image, while the entropy loss uses the expanded set of matching images.The combined objective is L = L_InfoNCE-mix + λ L_KoLeo.
  • Training Objective: Entropy regularization promotes a uniform embedding distribution, making distances more comparable, preventing collapse, and improving ranking metrics without consistent query thresholds.The method evaluates losses over the global multi-GPU batch to obtain stronger negatives and entropy neighbors.
  • Inference: At inference, SSCD extracts GeM-pooled, linearly projected, L2-normalized features, then applies whitening before cosine or L2 comparison.The whitening matrix is learned on the DISC2021 training set.
  • Similarity Normalization: Similarity normalization subtracts a background-neighbor bias and consistently improves metrics, but adds operational complexity and may hinder detection of content resembling the background distribution.Results are therefore reported both with and without normalization.

5. Experiments

Experiments evaluate SSCD on DISC2021, ImageNet-trained representations, and Copydays/CD10K using retrieval, ranking, and ablation metrics. SSCD consistently benefits from its architectural and training adaptations, with entropy regularization providing especially large gains, while excessive entropy weighting destabilizes training.

  • Evaluation setup: DISC2021 contains 1 million reference images, 50,000 queries, 10,000 true copies, and a disjoint 1 million-image training/background set.Performance is measured with micro average precision (µAP).
  • Evaluation setup: Copydays/CD10K adds 10k YFCC100M distractors and evaluates mAP on strongly transformed copies plus µAP on the full dataset.The standard strong-subset metric is mean average precision (mAP).
  • DISC2021 results: SSCD improves baseline DISC2021 accuracy by 2× to 5× before score normalization, demonstrating benefits from task-specific architecture and training adaptations.The comparison is reported for baseline methods and SSCD in Table 2.
  • Ablations: 39.8 µAP is achieved by SimCLRCD without score normalization versus 13.1 for SimCLR; entropy loss adds 10% absolute µAP, stronger augmentations add 6.2%, and reduced domain shift adds 4.7%.These gains are confirmed after score normalization.
  • Copydays results: SSCD achieves state-of-the-art mAP on Copydays and significantly improves global µAP, indicating better distance calibration, while remaining more compact than baseline descriptors.On high-resolution images, mAP improves but µAP degrades.
  • Ablations: Increasing entropy weight improves global and per-query ranking metrics, but training becomes unstable at λ > 40 as entropy optimization overwhelms the InfoNCE objective.The standard entropy weight is λ = 30.

6. Discussion

Discussion findings connect entropy regularization to fuller descriptor-space use, stronger separation, and improved copy retrieval, while identifying important scope and misuse limitations.

  • Descriptor distribution: Entropy regularization resolves SimCLR’s dimensional collapse, allowing SSCD to use the full descriptor space.SimCLR collapses to approximately 256 dimensions in a 512-dimensional space, whereas SSCD’s regularization avoids this behavior.
  • Descriptor distribution: 47.8 µAP for SSCD with λ = 30 versus 26.8 for an equivalent λ = 0 model demonstrates the accuracy benefit without whitening or normalization.Both entropy regularization and PCA whitening seek a more uniform descriptor distribution, but whitening can distort dimensions with trivial variance.
  • Perceptual prior: Ranking metrics also improve as entropy-loss weight increases, so improved cross-query calibration does not fully explain the regularization benefit.The primary experiments focus on µAP, which requires separation at a fixed threshold.
  • Perceptual prior: SSCD treats a uniform embedding distribution as a perceptual prior that keeps copies close while separating semantically similar images.This differs from classification representations, which group same-class images into dense regions.
  • Visual results: SSCD ignores text overlays and blur or color changes, and recovers sources where SimCLR instead matches low-level texture.The examples attribute these differences to more appropriate training augmentations and copy-detection representations.
  • Limitations: SSCD is limited on text-only images, distinct photos of the same scene, partial composites, and cases requiring high-precision verification.Global descriptors struggle when copied content occupies only a small composite region; local methods may be necessary.
  • Ethical considerations: Publishing copy-detection research may help evasion actors, although the authors believe open research improves the overall balance.The DISC2021 dataset uses identifiable photos of consenting paid actors.
  • Ethical considerations: The technology can scale manual moderation to protect users, but it can also support political censorship.The authors nevertheless judge advancing the technology to be a net benefit.

7. Conclusion

The paper presents SSCD as an effective, compact image copy-detection model whose entropy regularization improves descriptor separation. It reports strong DISC2021 performance, transfer to Copydays, and efficient inference, while providing further ablations and implementation details in the appendices.

  • SSCD adapts contrastive learning through architecture and objective changes for effective image copy detection.
  • Differential entropy regularization dramatically improves copy-detection accuracy by promoting consistent separation of image descriptors.
  • SSCD significantly surpasses baselines on DISC2021 and achieves state-of-the-art results after transfer to Copydays.
  • The method is efficient because it uses a standard trunk, smaller-than-typical retrieval inputs, compact descriptors, and calibrated distances for verification-candidate filtering.
  • The appendices provide additional ablations, Copydays results, embedding-distribution analysis, implementation details, and example matches.

A. Additional ablations

Additional ablations examine training scale, duration, descriptor dimensionality, score normalization, initialization variance, and projected features, showing efficiency and robustness trade-offs.

  • Descriptor dimensionality: Descriptor dimensionality trades retrieval efficiency against accuracy, with 256-dimensional descriptors achieving the highest accuracy when retrieval is constrained to 256 dimensions.
  • Batch size: Larger batch sizes improve final accuracy by making within-batch matching harder, but require more machines and synchronization overhead.
  • Training schedule: Longer training schedules provide no accuracy benefit.
  • Initialization variance: Five random initializations vary by 0.2% µAP and 0.1% µAPS N standard deviation.
  • Similarity normalization: Several similarity-normalization settings perform similarly; n = 2 works best for single-neighbor normalization, while averaging 2–4 neighbors performs similarly.The reported normalization weight is β = 1, and the authors use nend = 3, β = 1 for normalized results.
  • Projected features: At inference, projected features improve accuracy despite producing a significantly more compact code than trunk features.

B. Full Copydays results

The paper supplements its main evaluation with full Copydays results and descriptor-distribution analysis. These materials assess transfer without tuning and show that entropy regularization makes descriptor components use the space more evenly.

  • Full Copydays results: SSCD and SSCDlarge are evaluated on Copydays using prior published preprocessing settings without tuning model-specific parameters.The full results report mAP on the strong subset and µAP on the full dataset.
  • Inference efficiency: At 2242 inference size, ResNet50 provides approximately 4× the throughput of ResNeXt101 or ViT-B/16 and 20× that of ViT-B/8.
  • Ablation context: Table 6 reports DISC21 µAP across batch size, epochs, and dimensionality, with and without score normalization, including reduced 256-dimensional descriptors.
  • Descriptor distribution: Figure 5 compares SSCD, SimCLRCD, and a uniform reference distribution using descriptor principal values.
  • Descriptor distribution: With entropy regularization, descriptor components have similar energy and span less than one order of magnitude, with the maximum 6.6× the minimum.The λ = 0 model fails to use the descriptor space fully.

D. Implementation details

The implementation combines mixed-image augmentations, DINO descriptor details, fixed-size preprocessing, and a spatial matching visualization based on unpooled activations.

  • Augmentations: Mixup and Cutmix combine content from two source images using γ sampled from β(α, α), with α = 2 reducing nearly single-source mixtures.The mixing parameter controls how much content each source contributes.
  • Visualization: Figure 6 displays matching image pairs, with yellow indicating strong matches and blue indicating neutral or negative matches in the central heatmap column.The figure compares corresponding areas across the left and right images.
  • DINO baseline: The DINO descriptor concatenates the CLS token with GeM-pooled patch-token features using p = 4.The DINO DISC evaluation uses a ViT-B/16 trunk.
  • DINO baseline: For DINO DISC evaluation, inputs are resized to 224 × 224 without center cropping, outperforming aspect-ratio-preserving and 288 × 288 preprocessing alternatives.The authors suspect ViT models are less adaptable to rectangular inputs than fully convolutional networks.
  • Visualization: Removing GeM pooling preserves full-resolution activation maps, producing one descriptor per pixel for comparison with a global SSCD descriptor.The resulting strongest activations form a spatial heatmap showing matching areas between image pairs.

F. Retrieved matches

The retrieved-match analysis compares SSCD and SimCLR on DISC2021 using ImageNet-trained models with whitening. SSCD retrieves correct copies substantially more often, while SimCLR frequently follows shared distortion patterns.

  • Evaluation setup: The comparison uses ImageNet-trained SSCD and SimCLR models evaluated with whitening, with SimCLR using trunk features.Score normalization is omitted because it has no effect on top-1 accuracy.
  • Quantitative comparison: Table 10 reports percentage DISC2021 query first-result accuracy for SSCD and SimCLR trained on ImageNet.The table provides the quantitative basis for the first-result comparison.
  • Quantitative comparison: SSCD correctly identifies the copy as the first result 2 × as often as SimCLR on DISC2021.Correct SSCD matches are nearly a superset of SimCLR matches.
  • Qualitative comparison: SimCLR often retrieves images sharing distortion characteristics rather than copies, including angled text, strong diagonals, blur, or grain.Figure 7 presents additional queries that SSCD identifies correctly while SimCLR does not.
Loading 2202.10261v2…