PaperlayerView exact revision on arXiv

Source-linked AI summary

Unsupervised Semantic Correspondence Using Stable Diffusion

Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, Kwang Moo Yi

arXiv:2305.15581v2cs.CV

TL;DR

Semantic correspondence requires finding semantically equivalent image locations without the costly labels needed for supervised training. The paper optimizes Stable Diffusion prompt embeddings to attend to a source location and transfers them across images, outperforming weakly supervised methods across three benchmarks, with a 20.9% relative gain on SPair-71k and parity with strongly supervised state of the art on PF-Willow.

  • Problem

    Semantic correspondence lacks easily verified ground truth, making supervised training and dataset construction difficult.

  • Method

    The method optimizes continuous prompt embeddings for maximum attention at a queried source location, then applies them to target images using fixed Stable Diffusion.

  • Results

    The method outperforms weakly supervised methods on SPair-71k, PF-Willow, and CUB-200, with a 20.9% relative SPair-71k gain and PF-Willow performance matching strongly supervised state of the art.

  • Takeaways & Limitations

    An off-the-shelf diffusion model can provide semantic correspondences without task-specific training or semantic correspondence labels.

  • Takeaways & Limitations

    The method struggles with symmetric objects and requires significant computation, taking 30 seconds to find one prompt for one keypoint on an NVIDIA RTX 3090.

Abstract

from arXiv · show

Text-to-image diffusion models are now capable of generating images that are often indistinguishable from real images. To generate such images, these models must understand the semantics of the objects they are asked to generate. In this work we show that, without any training, one can leverage this semantic knowledge within diffusion models to find semantic correspondences - locations in multiple images that have the same semantic meaning. Specifically, given an image, we optimize the prompt embeddings of these models for maximum attention on the regions of interest. These optimized embeddings capture semantic information about the location, which can then be transferred to another image. By doing so we obtain results on par with the strongly supervised state of the art on the PF-Willow dataset and significantly outperform (20.9% relative for the SPair-71k dataset) any existing weakly or unsupervised method on PF-Willow, CUB-200 and SPair-71k datasets.

1 Introduction

The paper addresses unsupervised semantic correspondence by repurposing Stable Diffusion’s semantic knowledge and attention maps. It optimizes continuous prompt embeddings for source locations, then transfers them to identify semantically corresponding points in target images.

  • Motivation: Semantic correspondence seeks semantically equivalent points across similar or related objects rather than the same physical point.The task supports applications including image registration, object recognition, 3D reconstruction, class-specific alignment, and content transfer.
  • Motivation: Semantic correspondence lacks the easily verifiable ground truth available for geometric matching, making supervised dataset construction difficult.Human labeling is required because automatic verification of semantic correspondences is challenging.
  • Core insight: The method exploits semantic knowledge embedded in text-to-image diffusion models instead of correspondence labels or generic pretrained features.Generating realistic scenes requires models to represent object parts and their expected locations.
  • Method: Continuous prompt embeddings are optimized to maximize cross-attention at a queried source location, then applied to unseen images to locate semantic matches.The embeddings need not correspond to actual words; they only need to produce attention maps highlighting the queried region.
  • Method: Random crops, multiple random initializations, and multi-scale attention reduce overfitting and instability while improving semantic matching.These design choices are presented as critical to achieving the reported state-of-the-art performance.
  • Results: 20.9% relative improvement over the closest weakly supervised baseline on SPair-71k, while matching strongly supervised state of the art on PF-Willow.The method also outperforms weakly and unsupervised baselines across PF-Willow, CUB-200, and SPair-71k.

2 Related work

Prior semantic correspondence methods commonly rely on supervised training, carefully designed weak-supervision frameworks, or pretrained feature representations. This work instead repurposes Stable Diffusion’s attention maps without training a new network.

  • Semantic correspondences: Semantic correspondences enable class-specific alignment that can support generative-model training and content transfer between images.The task focuses on matching locations with the same semantic meaning, such as cat paws.
  • Learning with less supervision: Less-supervised methods include probabilistic student-teacher distillation, semi-supervised losses, and correspondence discovery from pretrained CNN or ViT features.These directions reduce reliance on fully labeled semantic correspondence data through synthetic data, unlabeled pairs, or pretrained representations.
  • Learning with less supervision: Existing approaches either depend on pretrained representation generalization or train a new semantic correspondence network, whereas this method uses Stable Diffusion directly.The method achieves performance similar to strongly supervised approaches without training a new neural network.
  • Diffusion models: Stable Diffusion is a lightweight, open-source latent diffusion model whose text conditioning uses cross-attention mechanisms.Prior work mainly repurposed diffusion models for generative applications such as personalization, text-to-3D, and image editing.
  • Diffusion models: Cross-attention maps in diffusion models can act as pseudo-segmentation for text queries, providing semantic information for correspondence extraction.The paper extends this observation from generative tasks to semantic correspondence without additional training.
  • Diffusion models: Unlike concurrent Stable Diffusion correspondence methods based on deep features, this approach alters attention maps to control semantic matching.The distinction is between using network representations and steering attention toward queried semantics.

3 Method

The method extracts semantic correspondences from frozen Stable Diffusion by optimizing embeddings to localize a source query and transferring them to a target image. It combines multi-layer attention with crop and initialization averaging to improve robustness against overfitting and optimization instability.

  • 3 Method: The method reuses an optimized source embedding on the target image and selects the target location at the attention-map argmax.Stable Diffusion remains frozen; the same denoising step t is used for source optimization and target inference.
  • 3 Method: The method uses the t=8 denoising step of a 50-step diffusion model and the first embedding token to represent the query.The timestep is selected by hyper-parameter tuning; other embedding tokens are optimized as well.
  • 3 Method: Attention maps are formed from U-Net cross-attention and averaged across channels and layers, combining broader early-layer responses with more specific later-layer responses.The method averages selected layers after interpolating their differently sized maps.
  • 3 Method: A source query is represented by a Gaussian focus map, and the embedding is optimized so the source attention reproduces that desired localization.The optimized embedding is obtained by matching the model’s attention map to the Gaussian centered at the query pixel.
  • 3 Method: Qualitative examples organize predictions into successful, mixed, and failure cases, with blue and orange marking correctness under PCK@0.05.Some predictions disagree with human annotations yet remain plausible according to the reported examples.
  • 3 Method: Random crops regularize single-image embedding optimization, while averaging attention across multiple optimization rounds reduces sensitivity to initialization.Crops resize images to 93% and apply uniformly random translations; multiple optimized embeddings are averaged at inference.

4 Results

The method is evaluated on three semantic-correspondence benchmarks, with quantitative, ablation, and qualitative results showing strong performance and cross-class generalization.

  • Evaluation setup: The evaluation uses SPair-71k, PF-Willow, and CUB-200, with PCK thresholds defined relative to object bounding boxes or the overall image according to the benchmark.The test sets contain 12,234, 900, and 1,248 correspondences respectively.
  • Qualitative highlights: A wine-glass occlusion example maps an occluded source point to another wine glass rather than the wine bottle label chosen by the human annotator.This illustrates a plausible semantic interpretation that differs from annotation.
  • Quantitative results: The method outperforms all compared weakly supervised baselines, is on par with strongly supervised PF-Willow results, and achieves a 20.9% relative average PCK@0.1 gap over ASIC.On SPair-71k, results are comparable to VAT except for several symmetry-heavy classes.
  • Ablations: Using multiple diffusion layers, multiple optimized embeddings, and more inference crops improves PF-Willow PCK@0.05, while removing optimization-time crops lowers performance from 53.0 to 45.5.The ablations associate multi-layer matching with multiple receptive fields and show gains from additional embeddings and crops.
  • Cross-class correspondence: Qualitative examples show plausible correspondences across different classes, including sheep–dog, bird–airplane, and chair–horse pairs.The cross-class examples are manually marked when they appear arguably correct.

5 Conclusions

The paper concludes that Stable Diffusion can provide semantic correspondences without task-specific training, while identifying symmetry failures and substantial computation as limitations.

  • Conclusions: The approach significantly outperforms weakly supervised methods on SPair-71k, PF-Willow, and CUB-200, with a 20.9% relative SPair-71k improvement and PF-Willow performance on par with strongly supervised methods.The method optimizes location-specific embeddings despite Stable Diffusion not being trained for correspondence estimation.
  • Conclusions: The study highlights that a text-to-image model can support semantic correspondence estimation and potentially other applications using knowledge encoded during large-scale training.The authors suggest scaling 3D generative-model training with web images without human supervision as an immediate application.
  • Limitations and future work: Symmetric objects are a major failure mode, and finding a single prompt for one keypoint takes 30 seconds on an NVIDIA RTX 3090 GPU.The authors suggest refined correspondence extraction or training a correspondence network to improve robustness and inference speed.

Supplementary Material

The supplementary material provides reproducibility and analysis resources, including per-category SPair-71k results, hyperparameters, architecture details, and additional qualitative results.

  • Supplementary contents: The supplement provides per-category quantitative results for SPair-71k and additional qualitative results across all datasets.
  • Supplementary contents: It also documents experiment hyperparameters and neural-network architecture details.
  • Reproducibility: The authors state that experiment code will be released for complete reproducibility.

A Detailed results for the SPair-71k [14] dataset

Detailed SPair-71k results show broad advantages over weakly supervised methods and a reduced gap to strongly supervised approaches across the dataset’s 18 classes.

  • Per-class results: The method outperforms all weakly supervised methods on 16 of 18 SPair-71k classes.Substantial margins are reported for bike, car, motorcycle, and plant.
  • Per-class results: It reduces the margin to strongly supervised methods and surpasses them on the bike, chair, and motorcycle classes.

B Hyperparameter selection

Hyperparameters were selected through randomized validation runs on a subset of SPair-71k, then fixed using the best-performing run. The selected configuration combines multiple U-Net layers, prompt-optimization settings, noise, steps, and cropping.

  • Hyperparameter selection: Hyperparameters were selected from 50 randomized runs, each using 50 randomly subsampled SPair-71k validation correspondences.The authors note that using the complete validation set might yield a better configuration.
  • Hyperparameter selection: The search randomized U-Net layers, learning rate, sigma radius, noise level, optimization steps, and image crop size.The tested ranges included 7–15 U-Net layers, 100–300 optimization steps, and 50%–100% crop sizes.
  • Hyperparameter selection: The detailed SPair-71k results show the method outperforming weakly supervised baselines and remaining comparable to a strongly supervised baseline.The supplied table caption identifies PCK@0.1 as the evaluation metric.
  • Hyperparameter selection: The selected configuration used U-Net layers 7–10, learning rate 2.37 × 10^-3, sigma radius 27.98, noise level t = 8, 129 steps, and a 93.17% crop.Layers 7–9 provide 16 × 16 attention maps, while layer 10 provides 32 × 32 maps.

C Model architecture

The method uses Stable Diffusion’s latent U-Net to process encoded noisy images across contracting, bottleneck, and expansive layers. Cross-attention connects intermediate image representations to 77-token prompt embeddings.

  • Model architecture: Stable Diffusion encodes a 3×512×512 image into a 4×64×64 latent representation z0, then adds noise to produce zt.This follows the denoising diffusion setup described for the architecture.
  • Model architecture: The U-Net has 16 layers: six contracting layers, one bottleneck, and nine expansive layers spanning resolutions from 64×64 to 8×8.The contracting path reaches 16×16 before the 8×8 bottleneck, while the expansive path returns to 64×64.
  • Model architecture: Cross-attention conditions U-Net intermediate representations on prompt embeddings, using image-derived queries and prompt-derived keys and values.The prompt sequence contains 77 tokens, each with dimensionality 768.

D Additional results

Additional analyses show that performance is concentrated among highly accurate PF-Willow and CUB-200 image pairs, while SPair-71k results are more dispersed. Visual examples also reveal semantically plausible alternatives, ambiguity, symmetry errors, and reflections.

  • Pairwise accuracy distributions: Most PF-Willow and CUB-200 image pairs have high PCK@0.1 correspondence accuracy, whereas SPair-71k image-pair results are more evenly spread.The distribution is measured by the percentage of correspondences within each pair that fall under PCK@0.1.
  • Pairwise accuracy distributions: Incorrect predictions often still select semantically consistent points on the target object but disagree with dataset annotations.Representative image pairs are shown across bins for SPair-71k, PF-Willow, and CUB-200.
  • Attention-map examples: The attention-map visualizations connect source points on the left image to estimated correspondence points on the right target image.The examples cover correct and incorrect maps across all three datasets.
  • Attention-map examples: Attention maps can succeed despite ambiguity, including preferring the correct eye among both eyes and the correct motorcycle tire among two motorcycles.These examples indicate that attention may cover multiple plausible regions while favoring the annotated correspondence.
  • Attention-map examples: Failures include choosing a symmetric bus corner, a car headlight instead of the annotated corner, or a bird’s reflection instead of the bird.A correct CUB-200 example also shows diffuse attention along the bird’s front side rather than a precise point.
Loading 2305.15581v2…