PaperlayerView exact revision on arXiv

Source-linked AI summary

Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondence

Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, Trevor Darrell

arXiv:2305.14334v2cs.CV

TL;DR

Diffusion-model representations are distributed across network layers and timesteps, complicating the extraction of useful descriptors. A lightweight aggregation network consolidates diffusion feature maps across layers and timesteps into a single per-pixel descriptor map. Diffusion Hyperfeatures achieves 72.56 PCK@0.1img on SPair-71k, besting DINOv2 and CATS++, and generalizes to out-of-domain synthetic images.

  • Problem

    Diffusion-model representations are distributed across network layers and timesteps, complicating the extraction of useful descriptors.

    Source: §1 · p. 1, §3 · p. 3

  • Method

    A lightweight aggregation network consolidates diffusion feature maps across layers and timesteps into a single per-pixel descriptor map.

    Source: §1 · p. 1, §3 · p. 3

  • Results

    Diffusion Hyperfeatures achieves 72.56 PCK@0.1img on SPair-71k, besting DINOv2 and CATS++, and generalizes to out-of-domain synthetic images.

    Source: §1 · p. 2, §4.1 · p. 6

  • Takeaways & Limitations

    Learned mixing weights provide a task- and model-specific way to identify useful diffusion layers and timesteps.

    Source: §1 · p. 1, §5 · p. 9

    The aggregator is tuned on a small set of real images with limited categories.

    Source: §5 · p. 9

Abstract

from arXiv · show

Diffusion models have been shown to be capable of generating high-quality images, suggesting that they could contain meaningful internal representations. Unfortunately, the feature maps that encode a diffusion model's internal information are spread not only over layers of the network, but also over diffusion timesteps, making it challenging to extract useful descriptors. We propose Diffusion Hyperfeatures, a framework for consolidating multi-scale and multi-timestep feature maps into per-pixel feature descriptors that can be used for downstream tasks. These descriptors can be extracted for both synthetic and real images using the generation and inversion processes. We evaluate the utility of our Diffusion Hyperfeatures on the task of semantic keypoint correspondence: our method achieves superior performance on the SPair-71k real image benchmark. We also demonstrate that our method is flexible and transferable: our feature aggregation network trained on the inversion features of real image pairs can be used on the generation features of synthetic image pairs with unseen objects and compositions. Our code is available at https://diffusion-hyperfeatures.github.io.

1 Introduction

Diffusion models contain potentially useful representations spread across layers and timesteps, but selecting features by hand is task-specific and can leave information unused. Diffusion Hyperfeatures aggregate intermediate features into per-pixel descriptors for semantic correspondence, with evaluation on real benchmark images and synthetic images.

Introduction

  • Prior diffusion-feature methods hand-select layers and timesteps for each task, a laborious process that can discard useful information distributed elsewhere in the diffusion process.The introduction notes that semantic correspondence may favor middle layers, while textural content may appear in later layers.Source: §1 · p. 1
  • Diffusion Hyperfeatures consolidate intermediate feature maps across scale and time into per-pixel descriptors using a feature aggregation network that learns task-relevant mixing weights.The network takes the collection of intermediate diffusion features as input and produces a single descriptor map.Source: §1 · p. 1, Figure 1 · p. 2
  • For real images, descriptors come from diffusion inversion; for synthetic images, they come from generation, and image-pair correspondences are found by nearest-neighbor search.The paper evaluates semantic keypoint correspondence on real SPair-71k images and tests whether an aggregator trained on real-image features generalizes to synthetic out-of-domain data.Source: Figure 1 · p. 2, §1 · p. 2

2 Related Work

Diffusion Hyperfeatures revisit hypercolumns by using diffusion models’ features across both layers and timesteps, and differ from earlier diffusion-representation methods through learned aggregation and inversion features for real images.

Hypercolumn Features

  • Hypercolumns combine activations at a pixel across network layers, supporting precise localization by reasoning at coarse and fine scales.Prior work applied them to keypoint detection, segmentation, and semantic correspondence; diffusion models add feature variation over time alongside scale.Source: §2 · p. 2

Deep Features for Semantic Correspondence

  • Prior correspondence work uses generative models for aligned-image supervision [34] [43] and DINO descriptors [2]; raw diffusion features remain under-explored.DINO descriptors have also supported segmentation [14], relative camera pose estimation [10], and dense visual alignment [31]. The authors suggest diffusion features may offer enhanced semantics because they are trained on image-text pairs.Source: §2 · pp. 2–3

Diffusion Model Representations

  • Earlier diffusion-representation methods use selected layers or timesteps, while Diffusion Hyperfeatures learns to weight all features into fixed-size descriptor maps.Plug-and-Play [43] uses a single-layer feature during editing; FeatureNeRF [49] distills that layer, while DDPMSeg and ODISE aggregate selected features for segmentation.Source: §2 · p. 3
  • Unlike prior methods that extract representations through generation, Diffusion Hyperfeatures uses inversion to obtain higher-quality features at the same timesteps for real images.The learned aggregator builds on Xu et al..Source: §2 · p. 3

3 Diffusion Hyperfeatures

Diffusion Hyperfeatures aggregate diffusion-model feature maps across layers and timesteps into per-pixel descriptors for semantic correspondence. The framework extracts features from both synthetic and real images and learns how to combine them for the task.

3 Diffusion Hyperfeatures

  • The method caches UNet features across diffusion steps, aggregates them into descriptor maps, and evaluates correspondences by nearest-neighbor matching against ground-truth annotations [28].A lightweight aggregation network learns the relative importance of feature maps for semantic correspondence.Source: §3 · p. 3

3.1 Diffusion Process Extraction

  • Generation features change across layers and timesteps, ranging from coarse shared semantic parts to refined parts and high-frequency gradients, motivating extraction across both axes.Shared cat structure appears even in early steps, while feature content evolves as generation progresses.Source: §3.1 · p. 4
  • Inversion features capture real-image structure and edges along a continuous chain, whereas generation features on real images can lose content or hallucinate details at noisy timesteps.The passages attribute inversion features’ greater trustworthiness to each step using the previous model output; continuous inversion also makes them interchangeable with generation features through the same aggregator.Source: §3.1 · pp. 4–5, Figure 3 · p. 4

3.2 Diffusion Hyperfeatures Aggregation

  • The aggregation network upsamples each feature map, standardizes its channels through a bottleneck, and sums maps using learned mixing weights.Bottleneck layers are shared across timesteps, while each layer–timestep combination has a unique mixing weight; task-specific supervision tunes both.Source: §3.2 · p. 5
  • For semantic correspondence, training supervises cosine similarities between all point pairs with labeled keypoints and symmetric cross entropy; inference upsamples descriptors before nearest-neighbor matching.Only a subsample of diffusion timesteps is aggregated to conserve memory.Source: §3.2 · p. 5

4 Experiments

Diffusion Hyperfeatures achieve the best reported real-image keypoint results on SPair-71k and CUB, and the aggregation network also transfers to synthetic-image generation features and unseen categories.

4.1 Semantic Keypoint Matching on Real Images

  • 72.56 PCK@0.1img on SPair-71k and 82.29 PCK@0.1img on CUB are the best results, and Diffusion Hyperfeatures improve PCK@0.1img by 14% over SD-Layer-4.The method outperforms the competitive DINOv2 and CATS++ baselines on both datasets.Source: Table 1 · p. 6, §4.1 · p. 6
  • SD-Layer-4 and DINOv2 confuse fine-grained parts, while Diffusion Hyperfeatures correctly distinguish car lights and bird-neck fronts from backs.The baselines often capture broad semantic regions but confuse visually similar subparts.Source: §4.1 · pp. 6–7, Figure 4 · p. 7

4.2 Ablations

  • One-step aggregation lags behind the full method, which benefits from complementary features across timesteps, especially early steps that provide useful image-frequency detail.The most useful correspondence information is concentrated in early diffusion timesteps, when the image is relatively clean but still noisy.Source: §4.2 · p. 7, Figure 5 · p. 8
  • Pruning to the highest-weight feature map performs comparably to SD-Layer-4, but its bottleneck-refined version is 6% better on SPair-71k and the full method is 9% ahead of Ours-Pruned.The selected raw map is Layer 5 at Timestep 10; the gap between the pruned and full methods indicates that soft mixing multiple maps matters.Source: Table 1 · p. 6, §4.2 · p. 7, Figure 5 · p. 8
  • Across SDv1-5 and SDv2-1, useful features concentrate in Layers 4–9 and Timesteps 5–15, while the aggregation network adjusts its selections to each variant.Selection shifts from Layers 4–5 and Timestep 10 in SDv1-5 to Layers 5–7 and Timestep 5 in SDv2-1.Source: §4.2 · p. 8, Figure 5 · p. 8

4.3 Transfer on Synthetic Images

  • The network tuned on real-image inversion features transfers to synthetic-image generation features and unseen categories, producing more reliable fine-grained correspondences than DINO and SD-Layer-4.For example, it correctly matches Eiffel Tower rungs in a triangle formation where DINO collapses predictions to the middle rung and SD-Layer-4 to the left rung.Source: §4.3 · p. 8, Figure 6 · p. 9

5 Conclusion

Diffusion Hyperfeatures consolidate diffusion features across time and space into descriptor maps, with interpretable aggregation that analyzes useful layers and timesteps. The method outperforms descriptor baselines on real-image correspondence and performs strongly on challenging synthetic images despite limited real-image tuning.

5 Conclusion

  • Diffusion Hyperfeatures outperform supervised hypercolumns and self-supervised descriptors by a large margin on semantic keypoint correspondence with real images.Source: §5 · p. 9
  • Despite tuning on few real images with limited categories, the method predicts strong semantic correspondences in challenging synthetic images, especially versus raw feature maps.Source: §5 · p. 9
  • The framework distills diffusion information distributed across time and space into a single descriptor map, while interpretable aggregation identifies useful layers and timesteps by model and task.Source: §5 · p. 9
  • Correspondences from the same feature maps used to generate synthetic images could enable pseudo-labeled keypoint datasets for image-to-image translation or 3D reconstruction.Source: §5 · p. 9

Supplementary Material

The supplementary material reports efficiency and benchmark comparisons, analyzes how diffusion features vary across models and timesteps, and demonstrates correspondence transfer and dense warping applications.

6.1 Computational Resources

  • The 10-timestep version improves PCK@0.1img by 4% over DINOv2 with almost 2x faster inference; the full method takes 6.62s.The aggregation network takes one day to train on one Nvidia Titan RTX GPU, and inference can run on a Nvidia T4 in Google Colab.Source: §6.1 · p. 12, Table 2 · p. 12

6.2 Stable Diffusion Model Variant

  • SDv1’s Layer 4 is strongest, whereas SDv2 favors Layers 5–6; Diffusion Hyperfeatures aggregates features effectively for both variants.Simple concatenation performs reasonably for SDv1 but poorly for SDv2, whose Layer 4 features strongly encode image position.Source: §6.2 · pp. 12–13, Figure 7 · p. 13, Figure 8 · p. 13

6.3 Inversion vs. Generation Features

  • At late timesteps, inversion features improve semantic keypoint matching by as much as 5% PCK@0.1img over generation features, while remaining comparable earlier.The comparison uses raw SDv1-5 Layer 4 features; generation features come from independently noising and denoising at each timestep, while inversion uses one continuous chain.Source: Figure 9 · p. 13, §6.3 · p. 14

6.4 Additional Evaluation Datasets

  • On PF-PASCAL and PF-WILLOW, Diffusion Hyperfeatures outperforms DINOv2 by 2% and 3% PCK@0.1img, respectively.These additional real-image evaluations complement the paper’s main SPair-71k and CUB evaluations.Source: §6.4 · p. 14, Table 3 · p. 14

6.5 DINO Features Aggregation

  • At 224-pixel input resolution, Stable Diffusion aggregation reaches 72.56% PCK@0.1img, versus 54.69% for DINO and 68.37% for DINOv2.The trained Stable Diffusion feature aggregation network performs best among the compared methods.Source: §6.5 · p. 14, Table 4 · p. 14

6.6 Semantic Keypoint Matching

  • The method handles real-image correspondence cases with large viewpoint changes and occlusions, and produces correspondences for synthetic images outside the training domain.For synthetic images, the aggregation network tuned on real SPair-71k images transfers to varied prompts, including an astronaut riding a horse and a raccoon playing chess.Source: §6.6 · p. 15, Figure 10 · p. 16, Figure 11 · p. 17

6.7 Dense Warping

  • Sparse-keypoint-trained Diffusion Hyperfeatures support dense warping and video mask or edit propagation while preserving fine-grained textures.The demonstrations include real and synthetic image warps and propagation across video frames, retaining details such as a cow’s red spots and lines on a bear’s vest.Source: §6.7 · p. 15, Figure 12 · p. 18, Figure 13 · p. 19

Related papers

Loading 2305.14334v2…