PaperlayerView exact revision on arXiv

Source-linked AI summary

Emergent Correspondence from Image Diffusion

Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, Bharath Hariharan

arXiv:2306.03881v2cs.CV

TL;DR

Image correspondence is important but explicit labels are rarely available. The paper extracts DIFT from pretrained diffusion networks by noising images and using intermediate features for matching. Without additional fine-tuning or supervision, DIFT outperforms weakly supervised and off-the-shelf self-supervised features across semantic, geometric, and temporal correspondence tasks.

  • Problem

    Image correspondence supports major vision tasks, but explicit correspondence labels are rarely available for training.

  • Method

    DIFT adds noise to input images and extracts diffusion U-Net feature maps, which are matched to establish correspondences.

  • Results

    DIFT outperforms weakly supervised methods and other self-supervised features across semantic, geometric, and temporal correspondence tasks without additional fine-tuning or supervision.

  • Takeaways & Limitations

    Correspondence emerges in image diffusion models without explicit supervision, suggesting pretrained diffusion models can serve as self-supervised correspondence learners.

  • Takeaways & Limitations

    The evaluation uses fixed off-the-shelf diffusion and self-supervised models without fine-tuning or retraining on additional data or supervision.

Abstract

from arXiv · show

Finding correspondences between images is a fundamental problem in computer vision. In this paper, we show that correspondence emerges in image diffusion models without any explicit supervision. We propose a simple strategy to extract this implicit knowledge out of diffusion networks as image features, namely DIffusion FeaTures (DIFT), and use them to establish correspondences between real images. Without any additional fine-tuning or supervision on the task-specific data or annotations, DIFT is able to outperform both weakly-supervised methods and competitive off-the-shelf features in identifying semantic, geometric, and temporal correspondences. Particularly for semantic correspondence, DIFT from Stable Diffusion is able to outperform DINO and OpenCLIP by 19 and 14 accuracy points respectively on the challenging SPair-71k benchmark. It even outperforms the state-of-the-art supervised methods on 9 out of 18 categories while remaining on par for the overall performance. Project page: https://diffusionfeatures.github.io

1 Introduction

The paper asks whether accurate image correspondences can be learned without labeled correspondence data and answers affirmatively using features extracted from pretrained diffusion models. DIFT establishes correspondences without fine-tuning or task-specific supervision, including across categories and image modalities.

  • Image correspondence supports 3D reconstruction, tracking, segmentation, and image or video editing, yet explicit correspondence labels are rarely available.
  • Diffusion models may implicitly reason about correspondence because image-to-image translation and editing can preserve pose or context while changing object categories.
  • DIFT extracts feature maps by adding noise to real images before passing them through a diffusion U-Net.
  • DIFT matches pixel locations with cosine-distance nearest-neighbor lookup and remains robust across categories and image modalities.
  • DIFT performs strongly across semantic, geometric, and temporal correspondence tasks without additional fine-tuning or supervision.

2 Related Work

Prior correspondence methods include hand-designed, supervised, and self-supervised features, while diffusion models have mainly been studied for generation and downstream recognition. This paper directly evaluates inherent pretrained diffusion features across visual correspondence tasks.

  • Traditional visual correspondence methods use hand-designed features such as SIFT and SURF, while later approaches learn correspondences under supervision.
  • Supervised correspondence approaches are difficult to scale, motivating methods that use less or no labeled data.
  • Diffusion models are powerful generative models whose intermediate features and attention maps have been explored for controllable generation and visual recognition.
  • This work directly evaluates features inherent to pretrained diffusion models on multiple visual correspondence tasks.

3 Problem Setup

The problem is to find the pixel in one image corresponding to a specified pixel in another. Correspondence may be semantic, geometric, or temporal, and dense feature matching provides a general solution.

  • Given images I1 and I2 and pixel p1 in I1, the goal is to find its corresponding pixel p2 in I2.
  • Semantic correspondence matches pixels with similar meanings, geometric correspondence matches the same object across viewpoints, and temporal correspondence tracks deforming objects over time.
  • The standard approach extracts dense feature maps for both images and matches the feature vector at p1 to locations in I2.
  • The matching location is selected by minimizing a distance between feature vectors, using cosine distance by default.

4 Diffusion Features (DIFT)

DIFT uses intermediate diffusion-network activations as dense features, extracting them from appropriately noised images. Time step and layer determine the balance between semantic and low-level information, while noise averaging improves stability.

  • 4.1 Image Diffusion Model: Diffusion models transform a Normal distribution toward an arbitrary data distribution, with image diffusion operating over 2D image space.
  • 4.1 Image Diffusion Model: During the forward diffusion process, Gaussian noise of varying magnitudes is added to clean images to produce noisy images.
  • 4.1 Image Diffusion Model: A U-Net takes noisy images and a diffusion time step as input, predicts noise, and reverses the process iteratively to generate cleaner images.
  • 4.1 Image Diffusion Model: For generated images, DIFT extracts intermediate-layer feature maps at a chosen backward-process time step to establish correspondences.
  • 4.2 Extracting Diffusion Features on Real Images: For real images, DIFT first adds noise at time step t before extracting feature maps with the diffusion U-Net.
  • 4.2 Extracting Diffusion Features on Real Images: Larger t and earlier layers tend to produce more semantic features, whereas smaller t and later layers emphasize low-level details.
  • 4.2 Extracting Diffusion Features on Real Images: Averaging features from multiple noisy versions improves representation stability against random input noise.

5 Semantic Correspondence

DIFT identifies semantic correspondences across challenging image variations and categories using diffusion-model features, with robust performance across evaluation settings. On SPair-71k, it outperforms weakly supervised and contrastive self-supervised baselines without additional supervision.

  • Model Variants and Baselines: DIFT extracts features from Stable Diffusion and ADM, while DINO and OpenCLIP provide self-supervised baselines using comparable training data.Neither DIFT nor the comparison features are fine-tuned or retrained with additional data or supervision.
  • Qualitative Results: DIFT establishes semantic correspondences under clustered scenes, viewpoint changes, occlusions, and instance-level appearance changes.Qualitative visualizations report correct matches in these challenging conditions.
  • Benchmark Evaluation: Without supervision, DIFTsd and DIFTadm outperform previous weakly supervised methods and self-supervised techniques on SPair-71k, while DIFTsd achieves the best results on PF-WILLOW and CUB.The evaluation reports PCK using per-image and per-point variants across the listed benchmarks.
  • Cross-category Correspondence: DIFT identifies reasonable semantic correspondences across different object categories, including shared parts such as airplane and bus wheels.Cross-category retrieval uses nearest DIFT features for image patches from different categories.
  • Sensitivity to Time Step: DIFTsd maintains high PCK per point across a wide range of time steps t and outperforms other off-the-shelf self-supervised features on SPair-71k.The reported robustness concerns the choice of diffusion time step for semantic correspondence.
  • Applications: DIFT also supports edit propagation across different instances, categories, and domains without correspondence supervision.The qualitative comparison reports more accurate edit propagation than OpenCLIP.

6 Other Correspondence Tasks

DIFT extends beyond semantic matching to geometric and temporal correspondence without task-specific training or correspondence supervision. It performs competitively on homography estimation and strongly on video label propagation.

  • 6.1 Geometric Correspondence: DIFT matches methods using explicit geometric supervision on HPatches homography estimation.The evaluation uses SuperPoint keypoints, mutual nearest-neighbor matches, and corner correctness at multiple pixel thresholds.
  • 6.1 Geometric Correspondence: DIFT produces robust sparse matches under both viewpoint and illumination changes on HPatches.The visualizations remove outliers with cv2.findHomography().
  • 6.2 Temporal Correspondence: DIFT supports video object segmentation and pose tracking despite never being trained or fine-tuned on video data.The evaluated tasks are DAVIS-2017 instance segmentation and JHMDB keypoint estimation.
  • 6.2 Temporal Correspondence: DIFTadm outperforms all compared self-supervised methods on DAVIS-2017 and JHMDB temporal correspondence benchmarks.It also surpasses models specifically trained on video data by a significant margin.
  • 6.2 Temporal Correspondence: DIFTadm produces sharper segmentation boundaries than DINO, including when objects are occluded.The qualitative comparison covers scenes with single or multiple objects.

7 Conclusion

The paper concludes that image diffusion models contain emergent correspondence knowledge that DIFT can expose without explicit supervision. Across semantic, geometric, and temporal tasks, DIFT outperforms weakly supervised and off-the-shelf self-supervised features and remains competitive with supervised semantic methods.

  • 7 Conclusion: Correspondence emerges from image diffusion models without explicit supervision.The conclusion presents DIFT as a feature extractor for this implicit knowledge.
  • 7 Conclusion: DIFT outperforms weakly supervised methods and off-the-shelf self-supervised features across semantic, geometric, and temporal correspondence.The conclusion frames this as the result of extensive experiments.
  • 7 Conclusion: DIFT remains on par with state-of-the-art supervised methods on semantic correspondence.The conclusion reports this alongside the broader cross-task comparison.

A Societal Impact

DIFT can use publicly available diffusion models, but the dominant Stable Diffusion training data introduces documented dataset-related concerns that may affect correspondence accuracy across image types.

  • A Societal Impact: Stable Diffusion is predominantly trained on LAION, whose racial bias and stereotype issues are inherited by diffusion models.The authors note that these issues might lead to differing correspondence accuracies for different kinds of images.
  • A Societal Impact: Training a diffusion model on a curated dataset could avoid these issues, but its computational cost limits academic development.The authors encourage more carefully trained diffusion models.

B Discussion

The discussion examines how diffusion time, feature extraction choices, inversion, and adaptation affect DIFT’s correspondence behavior and practical use. It also locates the correspondence signal inside the diffusion U-Net and notes data-related scope constraints.

  • Time Step: Larger diffusion time steps remove details while preserving semantic structure until excessive noise distorts object structure.This motivates smaller t for detail-dependent correspondences and larger t for semantic correspondence.
  • Efficiency: DIFT extraction has similar runtime to competing self-supervised features when using one inference step.On one NVIDIA A6000, DIFTsd takes 203 ms versus OpenCLIP’s 231 ms for a 768×768 image.
  • Efficiency: Averaging features from random-noise samples improves stability and performance but increases runtime.Removing this optimization causes small SPair-71k PCK decreases: DIFTsd 59.5→57.9 and DIFTadm 52.0→51.1.
  • Diffusion Inversion: DDIM inversion yields similar correspondence results but makes inference several times slower.The authors leave whether inversion can improve correspondence for future work.
  • Representation Location: DIFTsd’s correspondence signal emerges inside the U-Net rather than the Stable Diffusion VAE encoder.The encoder performs lower by an order of magnitude, and the authors associate DIFTsd’s correspondence with diffusion-based training.
  • Task-Specific Adaptation: Task-specific fine-tuning could further improve DIFT correspondence performance, but the paper avoids it to isolate off-the-shelf representation quality.The authors leave better downstream adaptation to future work.

C Implementation Details

The implementation evaluates DIFT variants across correspondence tasks using task-specific diffusion time steps, U-Net layers, resolutions, prompts, and matching procedures. It also documents a benchmark exclusion caused by PF-PASCAL train/test duplication.

  • Configuration: DIFT extracts features from U-Net upsampling blocks, with feature-map size and dimension determined by the selected layer and input resolution.ADM has 18 upsampling blocks and Stable Diffusion has 4 under their respective block definitions.
  • Configuration: DIFTadm uses t = 101 and n = 4 at 512×512, while DIFTsd uses t = 261 and n = 1 at 768×768 for semantic correspondence.These settings are shared across SPair-71k, PF-WILLOW, CUB, and the associated visualizations.
  • Configuration: On HPatches, DIFTadm uses t = 41 and n = 11, whereas DIFTsd uses t = 0 and n = 2, with both models receiving 768×768 images.The resulting feature maps have different spatial resolutions and channel dimensions.
  • Evaluation: Homography estimation reports the best result among cosine or L2 matching and RANSAC or LMEDS homography fitting.This selection is applied to every method listed in the comparison table.
  • Evaluation: The authors omit PF-PASCAL because 302 of 506 test images also occur in its training set, making benchmark comparisons less convincing.They report competitive performance against PWarpC and large gains over other off-the-shelf self-supervised features in the additional evaluation.
  • Evaluation: DIFT achieves competitive performance on HPatches despite lacking explicit geometric correspondence supervision.The evaluation uses Mean Matching Accuracy, averaging correct mutual-nearest-neighbor matches below specified pixel-error thresholds.

D.3 Analysis on Hyper-parameters

The analysis studies how diffusion time step and U-Net layer affect correspondence quality. Performance changes smoothly across time steps, while multiple U-Net layer choices remain effective.

  • Time step: A larger diffusion time step yields more semantic information and fewer low-level details, whereas a smaller step preserves more low-level detail.The preferred balance depends on the correspondence task.
  • Time step: HPatches homography accuracy and DAVIS video-label propagation maintain competitive performance across a broad range of time steps.Both curves vary smoothly rather than changing abruptly.
  • U-Net layer: On SPair-71k, a fine-grained sweep across Stable Diffusion’s 15 upsampling layers shows that multiple layer indices produce good PCK performance.The transition from block indices 0/1/2/3 to layer indices 3/7/11/14 is evaluated explicitly.

E Additional Qualitative Results

Additional qualitative results show DIFT’s correspondence behavior across semantic, geometric, temporal, instance, category, and domain changes. They also identify failure cases under ambiguous keypoint semantics and extreme appearance or viewpoint changes.

  • Semantic correspondence: PCA visualizations show that the same object parts share similar DIFT embeddings across segmented instance pairs.The first three PCA components are mapped to color channels to visualize the embeddings.
  • Temporal correspondence: DIFT maintains stable and competitive DAVIS video-label propagation across a wide range of diffusion time steps.Additional DAVIS examples show instance masks that closely follow object silhouettes.
  • Cross-domain correspondence: DIFT identifies correspondences across instances, categories, and image domains, including photo-to-logo and photo-to-sketch examples.The qualitative results use per-pixel cosine-distance heatmaps alongside predicted points.
  • Semantic correspondence: On SPair-71k, DIFT works under occlusion, viewpoint change, and intra-class appearance variation, and retrieves reasonable cross-category object-part correspondences.Nearest-neighbor queries include semantically matching parts with different overall appearances, such as bird and dog legs.
  • Failure cases: DIFT fails when keypoint semantics are ambiguous or source-target appearance and viewpoint changes are too dramatic.These conditions are illustrated as semantic-correspondence failure cases on SPair-71k.
  • Image editing: DIFT supports edit propagation across categories and domains, producing reliable correspondences even with a large domain gap.The examples propagate edits from source images to target images.
  • Geometric correspondence: On HPatches, DIFT works under large viewpoint and challenging illumination changes despite no explicit geometry supervision.Sparse feature matching results are shown after removing outliers.
Loading 2306.03881v2…