PaperlayerView exact revision on arXiv

Source-linked AI summary

DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data

Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, Phillip Isola

arXiv:2306.09344v3cs.CVcs.LG

TL;DR

Existing perceptual metrics miss mid-level similarities involving layout, pose, and semantic content because they emphasize pixels and patches. The paper builds a synthetic human-judgment dataset, tunes vision-model features into DreamSim, and reports stronger human alignment and downstream performance than prior metrics and vision models.

  • Problem

    Current pixel- and patch-level metrics fail to capture mid-level similarities and differences in image layout, object pose, and semantic content.

  • Method

    The paper collects synthetic human similarity judgments and tunes pretrained vision-model features using efficient adaptation to create DreamSim.

  • Results

    96.16% accuracy was achieved on human judgments, while DreamSim also demonstrated high agreement in retrieval and image-synthesis evaluations.

  • Takeaways & Limitations

    DreamSim generalizes from synthetic training data to real-image applications and combines sensitivity to foreground objects, semantic content, color, and layout.

  • Takeaways & Limitations

    The object-centric synthetic dataset does not capture the full range of human similarity judgments, and its pretrained models may propagate existing biases.

Abstract

from arXiv · show

Current perceptual similarity metrics operate at the level of pixels and patches. These metrics compare images in terms of their low-level colors and textures, but fail to capture mid-level similarities and differences in image layout, object pose, and semantic content. In this paper, we develop a perceptual metric that assesses images holistically. Our first step is to collect a new dataset of human similarity judgments over image pairs that are alike in diverse ways. Critical to this dataset is that judgments are nearly automatic and shared by all observers. To achieve this we use recent text-to-image models to create synthetic pairs that are perturbed along various dimensions. We observe that popular perceptual metrics fall short of explaining our new data, and we introduce a new metric, DreamSim, tuned to better align with human perception. We analyze how our metric is affected by different visual attributes, and find that it focuses heavily on foreground objects and semantic content while also being sensitive to color and layout. Notably, despite being trained on synthetic data, our metric generalizes to real images, giving strong results on retrieval and reconstruction tasks. Furthermore, our metric outperforms both prior learned metrics and recent large vision models on these tasks.

1 Introduction

The paper addresses the gap between low-level perceptual metrics and broad categorical comparisons by introducing a mid-level similarity benchmark and the DreamSim metric. DreamSim is tuned to human judgments using synthetic image triplets and supports downstream image retrieval and synthesis.

  • Motivation: Existing metrics focus on pixels or patches, while image embeddings capture some semantic judgments but leave broader visual similarity unclear.The unresolved scope includes similarities involving higher-level structures beyond low-level color and texture.
  • Dataset and metric: The NIGHTS dataset uses diffusion-generated image triplets with human judgments across appearance, viewpoint, layout, and related variations.Its examples differ qualitatively from prior datasets focused on low-level distortions or category-level changes.
  • Dataset and metric: DreamSim bridges patch-based metrics and categorical comparisons by tuning large vision-model features to align with human similarity judgments.The metric is designed for mid-level similarity rather than only pixel-level or category-level comparison.
  • Results: DreamSim focuses relatively heavily on foreground objects while retaining sensitivity to color and layout compared with modern image embeddings.The paper analyzes which image properties affect the metric’s decisions.
  • Results: 96.16% accuracy was achieved in predicting human judgments on the synthetic triplet dataset.The paper also demonstrates downstream applications to image retrieval and synthesis.

2 Related work

Prior perceptual metrics progress from pointwise and patch-based comparisons to learned deep features and foundation-model embeddings. However, perceptual similarity remains difficult to characterize when images differ structurally or along multiple human-relevant dimensions.

  • Perceptual metrics: Classical metrics such as MSE, PSNR, SSIM, and related patch-based measures struggle with structural ambiguity beyond photometric distortions.These methods are widely used for image quality assessment but are not suited to more complex image-generation tasks.
  • Perceptual metrics: Learning-based perceptual metrics replace pointwise comparisons with distances in deep-feature spaces extracted from pretrained networks.Feature-based metrics were found to outperform classical metrics across architectures and learning paradigms.
  • Perceptual metrics: LPIPS became a predominant perceptual metric while operating on 64×64 patches, despite strong features eventually becoming decorrelated from perceptual similarity.This highlights a tension between feature strength and perceptual alignment.
  • Foundation models as metrics: Foundation models such as CLIP provide image embeddings that support image-to-image distances and many downstream tasks.CLIP maps images and text captions into a shared embedding space.
  • Perceptual tests: Human similarity judgments may emphasize different dimensions depending on the images, and ambiguous image sets can require cognitive rather than automatic decisions.The 2AFC test is used historically to study such similarity and decision-making judgments.
  • Synthetic data: Text-driven generative models have increasingly been used to synthesize images and generate training data for downstream tasks.Stable Diffusion, Imagen, DALL-E 2, and MUSE are examples of such models.

3 Perceptual dataset collection

The authors generate synthetic image triplets with varied mid-level distortions and collect human similarity judgments designed to be automatic, stable, and shared across observers. Iterative unanimity filtering produces a curated 2AFC dataset, while JND judgments provide complementary validation.

  • Generating images with varied distortions: Stable Diffusion generates same-category images whose variations include style, color, pose, perspective, and other details.The category prompts are drawn from labels across ImageNet, CIFAR, Oxford 102 Flower, Food-101, and SUN397.
  • Generating images with varied distortions: The initial collection contains 100,000 triplets, each comprising a reference image and two distortions generated from the same prompt.The resulting images occupy a middle ground between pixel or patch distortions and categorical differences.
  • Human judgment collection: The dataset targets cognitively impenetrable judgments that require little cognition, remain stable, and are shared across humans.It combines 2AFC triplet judgments with complementary JND pair judgments.
  • Human judgment collection: In 2AFC, participants choose which of two distortions is more similar to the reference image.The choice is intended to be instinctive and consistent rather than easily articulated.
  • Human judgment collection: Approximately 20% of participants failed at least one sentinel, so their responses were discarded.Sentinels pair an unchanged reference with an image from a completely different prompt to assess response quality.
  • Human judgment collection: Iterative filtering retains triplets with unanimous votes across ten rounds and at least six judgments.Each retained triplet can receive up to ten votes, subject to sentinel filtering.
  • Dataset curation: 20,019 triplets form the final 2AFC dataset, averaging seven unanimous votes per triplet and using an 80/10/10 train-validation-test split.The dataset is publicly available on the project page.
  • JND validation: The JND study tests whether one image pair is identical and supplies a complementary perceptual-similarity signal.JND measures the boundary at which a distortion becomes just noticeable.

4 Perceptual metric learning

The metric compares candidate feature spaces by cosine distance, then improves human alignment through perceptual-data tuning and feature ensembling. DreamSim combines LoRA-tuned CLIP, OpenCLIP, and DINO representations.

  • 4.1 Embeddings as a distance metric: Candidate embeddings and learned metrics are evaluated by whether cosine-distance comparisons agree with human perceptual judgments.LPIPS and DISTS provide learned metric baselines, while DINO, CLIP, OpenCLIP, and MAE provide transformer-based feature spaces.
  • 4.1 Embeddings as a distance metric: For each triplet, the model selects the candidate image with the smaller distance to the reference, producing a vote against the human-selected label.The training objective maximizes the distance difference between the two candidate images using a hinge loss with margin m = 0.05.
  • 4.1 Embeddings as a distance metric: Agreement is measured on both 2AFC and JND judgments, whose strong correlation indicates that the benchmark captures a reproducible notion of similarity.The two evaluations use separate human-judgment settings.
  • 4.2 Learning an improved metric: LoRA tuning substantially improves alignment over pretrained baselines while updating approximately 0.67% of each model’s parameters.The reported LoRA configuration uses r = 16, dropout p = 0.3, and α = 0.5; LoRA outperformed the MLP-head alternative.

5 Experiments

Experiments show that DreamSim aligns strongly with human judgments and emphasizes color, shading, structure, foreground content, object instances, and people. Its gains transfer to low-level similarity but not uniformly to categorical similarity.

  • 5.1 How well do existing metrics align with human judgments?: 96.16% human agreement is achieved by DreamSim on the benchmark, exceeding prior learned metrics and showing gains from tuning and ensembling.OpenCLIP and DINO outperform LPIPS and DISTS; LoRA tuning and model ensembling further improve performance.
  • 5.2 What image attributes affect similarity decisions?: Ablation results show robustness to orientation but greater sensitivity to changes in color, shading, and structure.These attributes contain critical information for the model’s similarity decisions.
  • 5.2 What image attributes affect similarity decisions?: Removing foreground content harms alignment more than removing background content, indicating that foreground color and texture matter more for similarity recognition.The segmentation ablation replaces removed regions with random uniform noise while preserving their outlines.
  • 5.1 How well do existing metrics align with human judgments?: Strong correlation between 2AFC and JND scores supports the reproducibility of the benchmark’s similarity judgments across independent human studies.The comparison tests whether agreement with one judgment protocol corresponds to agreement with the other.
  • 5.1 How well do existing metrics align with human judgments?: Training on the benchmark improves alignment on BAPPS and IQA datasets but decreases alignment on THINGS categorical variations.The metric remains competitive with low-level metrics on TID2013 and KADID-10k despite not being trained specifically for low-level similarity.
  • 5.2 What image attributes affect similarity decisions?: Compared with LPIPS and DISTS, DreamSim is more sensitive to object-instance and people categories and less sensitive to background “stuff” such as sky and road.Compared with OpenCLIP and DINO, color explains more of DreamSim’s decisions.

6 Applications

DreamSim supports image retrieval, feature inversion, and k-NN classification, showing useful visual and semantic matching across real-image datasets and reconstruction settings.

  • Image retrieval: DreamSim retrieves neighbors with similar appearance and class on ImageNet-R and COCO, including images outside its training domain.
  • Image retrieval: 36.8% of ImageNet-R users preferred DreamSim retrievals, compared with 28.7% for OpenCLIP and 19.5% for DINO.
  • Feature inversion: DreamSim’s fine-tuned embeddings preserve color, shape, semantics, layout, and scene appearance across optimization, weak-prior, and diffusion-prior inversion.
  • k-NN Classification: DreamSim outperforms all baselines on ObjectNet k-NN classification and performs competitively with DINO on ImageNet100.

7 Discussion

The paper broadens perceptual similarity beyond low-level appearance using human-judged synthetic triplets and reports generalization to downstream image tasks. It also acknowledges inherited model biases and an object-centric, human-out-of-domain dataset.

  • The work expands similarity judgments to pose, color, shape, camera angle, and other factors beyond low-level similarity.
  • Synthetic image triplets and a tuned similarity metric are evaluated for generalization and effectiveness against existing metrics.
  • Stable Diffusion, CLIP, OpenCLIP, and DINO may transmit their existing biases into the dataset, model, and downstream decisions.
  • Filtering sensitive categories produces an object-centric dataset in which human-containing content is out of domain.

A Method

The studies collect human judgments through 2AFC, JND, and retrieval interfaces, using practice trials, sentinel checks, and defined voting procedures to control response quality.

  • 2AFC task: In 2AFC, users choose whether Image A or Image B is more similar to a central reference.
  • 2AFC task: Users complete practice, real, and sentinel trials, and responses from workers failing sentinel accuracy are discarded.
  • JND task: JND presents four sequential images and asks whether the first-third and second-fourth pairs are identical.
  • JND task: JND displays each image for 500 ms with 1-second gaps and includes separate different and same trials.
  • Object retrieval: Retrieval users select the most similar image from five candidates, while the study evaluates 200 queries with ten nearest neighbors.
  • Dataset collection: The dataset begins with 100K triplets and advances examples when votes remain unanimous or become inconclusive through sentinel failure.

A.2 Additional Dataset Details

Additional dataset procedures filter generated categories and construct labels from repeated human judgments, including unanimous 2AFC decisions and majority-voted JND pairs.

  • Prompt filtering: Prompt filtering excludes categories prone to malformed generations, leaving human faces in less than 0.5% of the dataset.
  • Prompt filtering: Despite excluding some human categories, DreamSim remains sensitive to the presence of people.
  • 2AFC filtering: Ten filtering rounds retain roughly 20% of original triplets with unanimous votes, while sentinel-failing worker votes are discarded.
  • JND labels: JND triplets target a similarity decision boundary and collect three judgments per pair, yielding six judgments per triplet.
  • Dataset labels: 2AFC labels use unanimous votes, whereas JND labels use majority votes across three trials per pair.

A.3 Model Training.

The model is trained by tuning vision-model backbones on NIGHTS, then evaluated through optimization, Deep Image Prior, and guided-diffusion image-generation procedures.

  • Model Training: Fine-tuned models use NIGHTS with an 80%-10%-10% split, 224 × 224 inputs, Adam, and batch sizes of 512 or 16.Training uses a single RTX 3090 or TITAN RTX GPU, learning rate 3e-4, and zero weight decay.
  • Model Training: Direct optimization updates a randomly initialized image using gradients of embedding distance between random crops and a target image.The procedure compares DINO, OpenCLIP, Ensemble, or DreamSim embeddings without a generator network.
  • Model Training: Deep Image Prior optimizes a trainable U-Net from frozen random noise while comparing its output with the target under different metrics.This introduces a convolutional neural-network prior into the generated image.
  • Model Training: Guided diffusion steers an unconditional ImageNet-trained diffusion model using classifier guidance based on embedding distance to the target.The loss combines spherical distance over 64 random crops with total variation regularization.
  • Model Training: Alignment remains highly correlated between ImageNet and non-ImageNet triplets, despite some backbones being trained only on ImageNet images.The split tests whether ImageNet exposure changes similarity judgments across categories.

B.1 Additional Evaluations

Additional evaluations test robustness across model settings, datasets, retrieval, image quality, and feature dimensionality. DreamSim generally transfers well, while performance depends on whether tasks emphasize appearance or categorical similarity.

  • Model and Dataset Evaluations: LoRA tuning consistently outperforms MLP tuning across experimental runs and model configurations.The supplementary evaluations vary ViT size, patch size, stride, and ensemble seeds.
  • Alternative Datasets: Training on NIGHTS improves alignment on BAPPS but decreases alignment on THINGS, indicating different sensitivity to appearance-level versus categorical variations.BAPPS contains low-level augmentations, whereas THINGS measures categorical variations.
  • Attribute Analysis: 55% alignment with the people category is the strongest reported semantic association, while other categories do not align above chance.Semantic alignment is computed from differences in category area between image pairs.
  • Retrieval and Reconstruction: DreamSim and OpenCLIP best preserve subject content in Sketchy sketch-photo retrieval, while DreamSim also localizes small spatial details.Other metrics often retrieve images matching sketch style or background appearance rather than the depicted subject.
  • Image Quality Assessment: DreamSim remains competitive on IQA, outperforming most low-level metrics and base ViT models, although LPIPS and FSIM perform best overall.IQA performance is measured using Spearman rank correlation with mean opinion scores.
  • Retrieval and Classification: DreamSim outperforms all baselines on ObjectNet k-NN classification and remains competitive with DINO on ImageNet100.DINO performs best on ImageNet100, while DreamSim performs competitively there.

B.2 Additional Visualizations

Visual analyses show that DreamSim differs from prior metrics in which visual changes it treats as important and where it places attention. Its decisions emphasize pose, semantics, color, and foreground objects.

  • Qualitative Comparisons: DreamSim is more sensitive than other metrics to major pose, semantic, and color changes, but less sensitive to granular structural changes when appearance is preserved.Examples include honeycomb and forest pairs where overall appearance remains similar.
  • Metric Differences: Compared with DISTS, DreamSim treats structurally changed pairs as nearby when overall appearance is similar and separates pairs with weaker appearance similarity.Compared with DINO and OpenCLIP, it is more likely to rate similar foreground color or appearance as similar.
  • Qualitative Comparisons: On COCO pairs, DreamSim particularly emphasizes foreground semantic similarity despite those pairs sharing fewer appearance similarities than NIGHTS pairs.Examples include horse and snowboarder image pairs.
  • Attention Maps: The finetuned DINO branch contributes the largest activation to DreamSim’s attention map, making the overall map resemble that branch.The model’s attention better captures entire foreground objects and relevant regions than pretrained backbones.
  • Attention Maps: Fine-tuning shifts DINO attention toward relevant foreground regions, such as the lobster body rather than claws absent from the comparison images.This illustrates how tuning changes which object regions influence similarity decisions.
  • Limitations: The dataset and model remain object-centric, may inherit pretrained-model biases, and do not capture the full range of human similarity judgments.Human-containing content is treated as out-of-domain because categories likely to generate sensitive content were filtered.
Loading 2306.09344v3…