Source-linked AI summary
A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, Ming-Hsuan Yang
TL;DR
The paper addresses the limited understanding of how text-to-image diffusion features relate across different images and objects. It analyzes Stable Diffusion and DINOv2 features, aligns and fuses them for zero-shot semantic correspondence, and reports significant gains across benchmark datasets while enabling instance swapping.
Problem
How Stable Diffusion features relate across multiple images and objects for semantic correspondence remains less understood than their use in single-image tasks.
Method
The paper aligns and fuses Stable Diffusion and DINOv2 features, then evaluates correspondence zero-shot using nearest neighbors.
Results
+13% on SPair-71k, +15% on PF-Pascal, and +5% on TSS are reported gains for fused features over previous methods.
Takeaways & Limitations
Stable Diffusion and DINOv2 provide complementary correspondence properties, and their fused representation also supports instance swapping.
Takeaways & Limitations
The fused features have relatively low resolution, limiting precise matches particularly for dense correspondence, while Stable Diffusion increases computation cost over DINO-only features.
Abstract
from arXiv · showhide
Text-to-image diffusion models have made significant advances in generating and editing high-quality images. As a result, numerous approaches have explored the ability of diffusion model features to understand and process single images for downstream tasks, e.g., classification, semantic segmentation, and stylization. However, significantly less is known about what these features reveal across multiple, different images and objects. In this work, we exploit Stable Diffusion (SD) features for semantic and dense correspondence and discover that with simple post-processing, SD features can perform quantitatively similar to SOTA representations. Interestingly, the qualitative analysis reveals that SD features have very different properties compared to existing representation learning features, such as the recently released DINOv2: while DINOv2 provides sparse but accurate matches, SD features provide high-quality spatial information but sometimes inaccurate semantic matches. We demonstrate that a simple fusion of these two features works surprisingly well, and a zero-shot evaluation using nearest neighbors on these fused features provides a significant performance gain over state-of-the-art methods on benchmark datasets, e.g., SPair-71k, PF-Pascal, and TSS. We also show that these correspondences can enable interesting applications such as instance swapping in two images.
1 Introduction
The paper investigates how Stable Diffusion features relate across different images and objects for semantic correspondence. It finds that SD and DINOv2 offer complementary strengths, and their fusion improves zero-shot correspondence performance.
- Research gap: Stable Diffusion features are less understood across multiple images and objects than for single-image tasks, motivating semantic correspondence analysis.Semantic correspondence connects similar pixels across images containing objects with different appearances, viewpoints, or deformations.
- Application: The approach supports pixel-level instance swapping followed by Stable-Diffusion refinement to produce a plausible swapped instance.Figure 1 also depicts matching across substantial category, shape, and pose differences.
- Feature behavior: SD features encode spatial layout and produce smooth correspondences, but their pixel-level semantic matches can be inaccurate.For example, a bus front may match another bus back.
- Feature behavior: DINOv2 produces sparse but accurate matches and therefore complements SD’s higher spatial information.The paper reports that DINOv2 outperforms DINOv1 for correspondence but requires different post-processing.
- Approach: A simple strategy aligns and fuses SD and DINOv2 features, evaluating them zero-shot with nearest neighbors and no correspondence-specific training.The fused representation combines the distinct strengths of the two feature types.
- Results: +13% on SPair-71k, +15% on PF-Pascal, and +5% on TSS are reported gains for fused features over previous methods.The paper also presents instance swapping using the resulting high-quality correspondences.
2 Related work
Related work covers learned feature descriptors for dense correspondence, semantic correspondence methods, and diffusion models applied to image understanding. These strands motivate examining Stable Diffusion features for semantic correspondence.
- Feature descriptors: Deep models learn dense-correspondence features robust to photometric and geometric changes such as rotation, scaling, and perspective transformation.Prior work largely focuses on outdoor image matching under rigid transformations.
- Semantic correspondence: Semantic correspondence estimates dense matches between same-class objects despite different appearance, viewpoint, or non-rigid deformation.Conventional pipelines include feature extraction, cost-volume construction, and displacement-field or transformation regression.
- Diffusion models: Diffusion models achieve state-of-the-art image generation and have also been applied to segmentation, detection, and monocular depth estimation.This broader vision-task use motivates analyzing their internal representations.
3 Semantic correspondences via Stable Diffusion and Vision Transformer
The paper analyzes Stable Diffusion and DINO features for semantic correspondence, showing that their different strengths—spatial coherence and semantic accuracy—can be combined effectively. A simple fusion strategy and carefully selected decoder features support strong correspondence performance.
- Stable Diffusion features: Stable Diffusion features are extracted from a latent diffusion pipeline and analyzed across U-Net decoder layers for correspondence.The process encodes an image, adds timestep-dependent Gaussian noise, and applies the denoising U-Net with an implicit captioner.
- Stable Diffusion features: Early decoder layers emphasize coarse semantics and structure, whereas later layers capture detailed texture and appearance.The analysis compares early, intermediate, and final decoder layers using PCA and clustering.
- Stable Diffusion features: Decoder-only features outperform combinations with encoder skip features and single-sub-layer features.The extraction location within the decoder layer materially affects performance.
- DINO features: DINOv2 improves substantially over DINOv1, with its final-layer token features providing the best correspondence performance for DINOv2.This contrasts with DINOv1, whose strongest performance was previously associated with earlier-layer key features.
- Comparing feature properties: DINO produces sparse, accurate matches, while Stable Diffusion supplies smoother spatial correspondence, especially for textureless or spatially challenging inputs.The complementary behavior appears across identical instances, object masks, intra-class variation, and semantic-flow comparisons.
- Feature fusion: The fusion strategy independently normalizes and concatenates SD and DINO features, with α = 0.5 empirically balancing their complementary strengths.The fused features improve challenging SPair-71k cases while retaining SD’s spatial smoothness and reducing correspondence noise.
4 Experiments and analysis
The experiments evaluate zero-shot and supervised correspondence, showing that fused Stable Diffusion and DINOv2 features improve benchmark performance and support instance swapping, while resolution and computation cost remain limitations.
- Sparse correspondence: The zero-shot fused representation achieves a leading average PCK of 64.0 on SPair-71k, improving over DINOv2 by 8.4 points.The comparison is from 55.6 to 64.0.
- Sparse correspondence: On PF-Pascal, the fusion approach achieves the highest average PCK across all thresholds among unsupervised methods.PF-Pascal contains lower variation in appearance, pose, and shape between paired instances.
- Dense correspondence: On TSS, Fuse-ViT-B/14 outperforms DINOv2-ViT-B/14 by 7.7 points among unsupervised nearest-neighbor methods.TSS contains less challenging examples with low appearance, viewpoint, and deformation variation.
- Instance swapping: Instance swapping combines dense correspondence with DDIM-based refinement to produce visually coherent images balancing spatial smoothness and fine details.The process upsamples features, swaps pixels by nearest-neighbor patches, then refines the result through inversion and denoising.
- Instance swapping: The fused features improve reference faithfulness and perceptual quality, with higher CLIP and quality scores and lower FID scores.
- Limitations: The relatively low fused-feature resolution limits precise dense matches, while Stable Diffusion increases computation cost compared with DINO-only correspondences.The limitation is particularly relevant for dense correspondence tasks such as TSS.
- Feature complementarity: In most settings, SD and DINOv2 succeed exclusively in 20–30% of cases, indicating substantial non-redundant information.The analysis uses error distributions at three PCK levels on SPair-71k and PF-Pascal.
5 Discussion on feature behavior
The paper offers tentative explanations for why Stable Diffusion and DINOv2 features differ, while emphasizing that the causes remain unresolved.
- The authors suggest that DINO’s self-supervised augmentations may induce invariance to spatial information, unlike Stable Diffusion’s text-to-image training.This explanation is presented as difficult to validate because of resource limitations.
- Architecturally, DINOv2’s patch-based ViT may prioritize local structures less than Stable Diffusion’s convolutional UNet.The proposed contrast links patch processing in DINOv2 with greater detail retention in SD’s convolutional layers.
- The underlying causes of the feature differences remain an open direction for future exploration.
6 Conclusion
The paper studies Stable Diffusion representations for semantic correspondence and shows that combining them with DINO-ViT features improves correspondence performance.
- The paper investigates Stable Diffusion’s internal representations for semantic correspondence and their relationship to self-supervised DINO-ViT features.
- The proposed strategy aligns and fuses Stable Diffusion and DINO-ViT features to combine their complementary properties.
- The fused features significantly enhance performance on SPair-71k and outperform existing methods.