Source-linked AI summary
Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence
Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, Trevor Darrell
TL;DR
Diffusion-model features are distributed across layers and timesteps, making useful descriptors difficult to extract. Diffusion Hyperfeatures aggregate these features into per-pixel descriptors for real and synthetic images, achieving superior semantic keypoint correspondence performance on SPair-71k and transferring to synthetic data.
Problem
Diffusion-model feature maps vary across network layers and diffusion timesteps, creating a challenge for extracting useful descriptors for downstream tasks.
Method
A lightweight aggregation network combines intermediate diffusion feature maps into per-pixel Diffusion Hyperfeatures extracted through inversion for real images and generation for synthetic images.
Results
The method achieves the best reported result among the compared methods, reaching 72.56 PCK@0.1img on SPair-71k and 82.29 PCK@0.1img on CUB.
Takeaways & Limitations
The learned aggregation retains diffusion features' utility for semantic correspondence across real benchmark images and synthetic images with unseen objects and compositions.
Abstract
from arXiv · showhide
Diffusion models have been shown to be capable of generating high-quality images, suggesting that they could contain meaningful internal representations. Unfortunately, the feature maps that encode a diffusion model's internal information are spread not only over layers of the network, but also over diffusion timesteps, making it challenging to extract useful descriptors. We propose Diffusion Hyperfeatures, a framework for consolidating multi-scale and multi-timestep feature maps into per-pixel feature descriptors that can be used for downstream tasks. These descriptors can be extracted for both synthetic and real images using the generation and inversion processes. We evaluate the utility of our Diffusion Hyperfeatures on the task of semantic keypoint correspondence: our method achieves superior performance on the SPair-71k real image benchmark. We also demonstrate that our method is flexible and transferable: our feature aggregation network trained on the inversion features of real image pairs can be used on the generation features of synthetic image pairs with unseen objects and compositions. Our code is available at https://diffusion-hyperfeatures.github.io.
1 Introduction
Diffusion Hyperfeatures consolidate diffusion-model features that vary across network layers and timesteps into per-pixel descriptors for semantic correspondence. The framework extracts features through inversion for real images and generation for synthetic images, then evaluates them on real-image correspondence and synthetic transfer.
- Diffusion features vary across both network layers and timesteps, complicating the extraction of useful descriptors.
- Diffusion Hyperfeatures consolidate all intermediate feature maps into a single per-pixel descriptor map using a lightweight aggregation network.
- Real images use inversion features, whereas synthetic images use generation features before correspondence is found by nearest-neighbor search.
- The method is evaluated on semantic keypoint correspondence with real SPair-71k images and on synthetic images generated with unseen objects and compositions.
2 Related Work
Prior work uses deep and generative-model representations for semantic correspondence and related vision tasks, but diffusion features remain comparatively under-explored. Diffusion models add feature variation over time to the multi-scale representations already used by hypercolumn methods.
- Hypercolumn Features: Hypercolumns collect activations corresponding to a pixel across network layers to combine coarse and fine information for precise localization.
- Hypercolumn Features: Semantic correspondence benefits from representations that are robust to illumination and viewpoint changes while retaining precise localization.
- Deep Features for Semantic Correspondence: Diffusion features are under-explored for semantic correspondence and may contain enhanced semantic representations from image-text training.
- Diffusion Model Representations: Prior diffusion-model applications typically use features from a single layer or a hand-selected subset of layers and timesteps.
3 Diffusion Hyperfeatures
The method caches diffusion features across the extraction process and aggregates their variation over layers, scales, and timesteps into descriptors. It supports generation for synthetic images and continuous-chain inversion for real images, followed by task-specific correspondence matching.
- Diffusion Process Extraction: For real images, continuous-chain inversion produces features more faithful to original content than independently noising and denoising at a selected timestep.
- Diffusion Process Extraction: Diffusion generation progressively denoises images, while inversion progressively noises clean images; both processes produce intermediate UNet feature maps across timesteps and layers.
- Diffusion Process Extraction: Generation features can preserve shared latent representations between synthetic images throughout the process, including early steps with nearly pure-noise inputs.
- Diffusion Hyperfeatures Aggregation: The aggregation network standardizes feature maps with bottleneck layers and combines them using learned layer-timestep mixing weights.
- Diffusion Hyperfeatures Aggregation: Semantic correspondence training compares descriptor-map points with cosine similarity and uses labeled keypoints with a symmetric cross-entropy loss.
4 Experiments
Experiments evaluate Diffusion Hyperfeatures for semantic keypoint correspondence on real benchmarks, ablate layers and timesteps, and test transfer across model variants and synthetic out-of-domain images.
- Semantic Keypoint Matching on Real Images: A single SD-Layer-4 feature map beats DINO and DHPF by at least 4% in PCK@0.1img, while naive concatenation of all maps degrades the improvement.The authors attribute the degradation to unequal feature granularity across maps.
- Semantic Keypoint Matching on Real Images: 14% higher PCK@0.1img than SD-Layer-4 results from aggregating feature maps across layers and timesteps.The result supports complementary information across layers and information distributed over multiple timesteps.
- Ablations: Early diffusion timesteps provide the most useful information, and aggregating all layers with one-step inversion still trails the full multi-timestep method.The authors interpret timestep selection as controlling the amount of image frequency detail represented.
- Ablations: Soft mixing of many feature maps outperforms pruning to the single highest-weight map, with Ours-Pruned 6% better than SD-Layer-Pruned on SPair-71k.The highest-weight raw map is Layer 5, Timestep 10 for Ours-SDv1-5.
- Transfer on Synthetic Images: The aggregation network transfers from inversion features of real images to generation features of synthetic images with out-of-domain categories, outperforming DINO and SD-Layer-4.Qualitative examples show improved relative placement of fine-grained parts such as Eiffel Tower rungs.
5 Conclusion
Diffusion Hyperfeatures consolidate information distributed across diffusion time and network space into descriptors that support semantic correspondence. The method performs strongly on real benchmarks and transfers to challenging synthetic images and dense warping applications.
- Diffusion Hyperfeatures distill information distributed across diffusion time and space into a single descriptor map.
- The interpretable aggregation network identifies useful layers and timesteps for the underlying model and task.
- The method outperforms supervised hypercolumns and self-supervised descriptors by a large margin on real-image semantic correspondence.
- Despite tuning on few real images with limited categories, the method retains strong correspondence performance on challenging synthetic images, especially versus raw feature maps.
- The same feature maps used to generate synthetic images could support pseudo-labeled semantic keypoint datasets for image-to-image translation or 3D reconstruction.
Supplementary Material
The supplementary results examine computational trade-offs, feature behavior across Stable Diffusion variants and timesteps, cross-dataset transfer, and applications to synthetic correspondence and dense warping. They show that aggregation can preserve useful semantic and fine-grained information across varied settings.
- Computational Resources: Using the first 10 diffusion timesteps gives a 4% PCK@0.1img improvement over DINOv2 with an almost 2x faster inference process.The full method uses all diffusion features and takes 6.62s.
- Stable Diffusion Model Variant: Stable Diffusion layer behavior is similar within model variants but differs substantially between SDv1 and SDv2.SDv1 Layer 4 is strongest, whereas SDv2 Layers 5 and 6 are stronger for semantic correspondence.
- Inversion versus Generation: Late inversion timesteps t = 25 to 50 are generally more informative than generation features, with a margin of as much as 5% PCK@0.1img.
- Cross-Dataset Evaluation: On PF-PASCAL and PF-WILLOW, the method outperforms DINOv2 by 2% and 3% PCK@0.1img respectively.
- Synthetic Images: An aggregation network tuned on real-image inversion features generalizes to generation features and unseen synthetic object categories.The transfer reverses timestep ordering when operating on generation features.
- Dense Warping: Sparse-keypoint supervision also supports dense backward warps and video-frame forward splats that preserve fine-grained textures and propagate edits.Examples include preserving animal textures and propagating object masks or semantic edits.