PaperlayerView exact revision on arXiv

Source-linked AI summary

Readout Guidance: Learning Control from Diffusion Features

Grace Luo, Trevor Darrell, Oliver Wang, Dan B Goldman, Aleksander Holynski

arXiv:2312.02150v2cs.CV

TL;DR

Readout Guidance addresses the difficulty of adding arbitrary control to diffusion models without large annotated training efforts. It learns lightweight readout heads from frozen diffusion features and back-propagates target-matching signals during sampling. The framework supports spatial and relational controls, while using substantially fewer resources than methods such as ControlNet.

  • Problem

    Arbitrary user control for diffusion outputs commonly requires substantial training on large annotated datasets, making custom control cumbersome.

    Source: §1 · p. 1

  • Method

    Readout Guidance trains lightweight readout heads on frozen diffusion features and uses their target-matching gradients to guide sampling at every timestep.

    Source: §1 · p. 1, §4.1 · p. 4

  • Results

    The method supports drag manipulation, identity-consistent generation, and spatially aligned control within one framework, using 49MB of added parameters versus 1.4GB for ControlNet.

    Source: §1 · p. 2, §6 · p. 8

  • Takeaways & Limitations

    Readout Guidance provides a general control recipe applicable to vanilla and adapter-augmented diffusion models with small-scale training data.

    Source: §6 · p. 8

    Sampling-time guidance generally requires more memory and runtime and can sometimes produce cartoonish or unrealistic images satisfying the readout constraints.

    Source: §5.1 · p. 8

Abstract

from arXiv · show

We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity. Furthermore, by comparing the readout estimates to a user-defined target, and back-propagating the gradient through the readout head, these estimates can be used to guide the sampling process. Compared to prior methods for conditional generation, Readout Guidance requires significantly fewer added parameters and training samples, and offers a convenient and simple recipe for reproducing different forms of conditional control under a single framework, with a single architecture and sampling procedure. We showcase these benefits in the applications of drag-based manipulation, identity-consistent generation, and spatially aligned control. Project page: https://readout-guidance.github.io.

1. Introduction

Readout Guidance addresses arbitrary user control by training lightweight readout heads on frozen diffusion features and using their predictions to guide sampling. It supports diverse controls while reducing training data, time, and added parameters relative to prior conditional methods.

  • Readout Guidance trains small readout heads on frozen diffusion features to extract image properties and guide generation toward user-defined constraints.The heads can use intermediate features to predict properties, then back-propagate a comparison between predictions and targets during sampling.Source: §1 · pp. 1–2
  • The framework handles spatial properties such as pose, depth, and edges, alongside higher-order properties including correspondence, appearance similarity, and shared identity.Source: §1 · pp. 1–2
  • The method demonstrates drag-based manipulation, identity-consistent generation, and spatially aligned control within one guidance framework.Source: §1 · p. 2
  • Readout Guidance uses as few as 100 supervised pairs, a few hours of training, and 49MB of added parameters, compared with ControlNet’s 200k pairs, more than a week, and 1.4GB.Source: §1 · p. 2

2. Related Work

Prior work adapts diffusion models through conditional networks, sampling-time guidance, or feature decoders for specific properties. Readout Guidance instead learns general guidance functions from diffusion features and applies them across spatially aligned and relational controls.

  • Conditional models such as ControlNets, Adapters, and LoRAs modulate frozen diffusion features using control signals, whereas Readout Guidance steers sampling from the model’s features.Source: §2 · p. 2
  • Sampling-time guidance has used classifiers, score differences, predicted clean images, or hand-designed feature functions; Readout Guidance learns this guidance function from diffusion features.Source: §2 · p. 2
  • Diffusion features support tasks including segmentation, depth, pose, and semantic correspondence, but prior decoders typically extract representations at a single timestep.Source: §2 · p. 2
  • Readout Guidance extends these representations across the entire diffusion process and applies learned appearance and correspondence constraints without hand-designed operators or per-example fine-tuning.Source: §2 · p. 2

3. Readout Guidance

Readout Guidance replaces classifier-style guidance with learned readouts that predict properties from frozen diffusion features and optimize distances to references during sampling. The same recipe supports aligned and relative constraints and can combine multiple guidance functions.

3.1. Background

  • Classifier guidance conditions diffusion sampling by nudging each step toward a criterion, while Readout Guidance trains small heads on frozen U-Net decoder features.Source: §3.1 · p. 3
  • The readout architecture is shared across task classes, with task-specific input data and losses rather than bespoke model designs.Source: §3.1 · p. 3

3.2. A Revised Guidance Recipe

  • Readout Guidance replaces class probabilities with continuous regressors and minimizes a distance between a reference r and predicted readout r̂ = fψ(x_t).Source: §3.2 · pp. 3–4
  • For relative constraints, the reference can come from a separate diffusion process with a reference noisy image, rather than directly from a user input.Source: §3.2 · p. 4
  • The distance function can encode arbitrary spatial relationships, and multiple readout heads can be combined with text guidance or with one another.Source: §3.2 · p. 4

4. Readout Heads

Readout heads aggregate frozen decoder features at each diffusion timestep and specialize either in spatially aligned outputs or relative relationships between images. They are trained for pose, depth, edges, correspondence, appearance similarity, and identity.

4.1. Common Architecture

  • Heads project and upsample frozen U-Net decoder features, mix layers, add timestep conditioning, and query readout estimates at every sampling step.Source: §4.1 · p. 4

4.2. Spatially Aligned Heads

  • Spatial heads convert features into RGB readouts and train with per-pixel losses for pose, depth, and edges using PascalVOC pseudo-labels from OpenPose, MiDaS, and HED.Source: §4.2 · pp. 4–5

4.3. Relative Heads

  • The correspondence head learns video-based features whose matched points have high similarity, using tracked points and a symmetric cross-entropy contrastive loss.Source: §4.3 · p. 5
  • The appearance head uses video triplets and a margin-0.5 hinge loss to preserve color, texture, and identity while allowing pose or camera changes.Source: §4.3 · p. 5
  • An identity-specific variant retains the appearance-head architecture and training scheme while using CelebA-HQ face crops and hard negatives from different identities.Source: §4.3 · p. 5

5. Results

Readout Guidance supports drag-based manipulation, appearance and identity preservation, spatial control, and refinement of existing conditional controls. It remains competitive with substantially more data, while sampling-time guidance increases computational cost and can produce unrealistic images.

5. Results

  • Readout Guidance enables drag-based manipulation on real images without the additional foreground mask required by DragDiffusion.Its appearance-similarity and correspondence heads operate using reference features seeded by DDIM inversion.Source: Figure 5 · p. 6, §5 · p. 6
  • Large out-of-plane motions can be handled while preserving background appearance, because correspondence features are more orientation-agnostic and appearance guidance protects the foreground edit context.The method fully rotates a tiger’s head where DragDiffusion slightly translates its nose upward.Source: §5 · p. 6
  • Appearance guidance generates reference-image variations with identity largely preserved while pose changes, and specialized identity guidance transfers a person’s identity into different prompted contexts.Varying guidance strength explores appearance from shared color and texture to shared object identity and proportion.Source: §5 · p. 6, Figure 6 · p. 7, Figure 7 · p. 7
  • Readout Guidance complements ControlNet and T2I-Adapter by correcting difficult pose-control mistakes, including last-mile pose artifacts.The same guidance head trained on vanilla diffusion features can be applied to models augmented with these adapters.Source: §5 · p. 7, Figure 8 · p. 8
  • Training on 100 to 8.5k PascalVOC images supports pose control, with more data generally improving quality and the 8.5k model performing comparably to T2I-Adapter trained on 3M images.Combining T2I-Adapter with Readout Guidance improves the base conditional model by 30% PCK at α = 0.05, or 2.3x in PCK.Source: §5 · pp. 7–8, Figure 9 · p. 8

5.1. Limitations

  • Sampling-time guidance generally requires more memory and runtime, and can sometimes produce cartoonish or unrealistic images that satisfy the readout constraints.These limitations arise despite using the same number of sampling steps as baselines.Source: §5.1 · p. 8

6. Conclusion

Readout Guidance controls sampling in pre-trained diffusion models using lightweight readout heads trained on diffusion features, requiring small-scale training data and supporting diverse controls.

  • Readout Guidance controls sampling with readout heads trained on diffusion features, requiring only small-scale training data.Because it is a guidance technique, it works with vanilla diffusion models and fine-tuned conditional models.Source: §6 · p. 8

8. Readout Guidance

The method applies timestep-aware readout guidance with task-specific hyperparameters, a shared lightweight architecture, and normalized gradient updates. Its implementation adds measurable runtime and memory costs, while excessive guidance can produce implausible images.

8.1. Guidance Hyperparameters

  • Guidance applies over a selected timestep span, with early steps shaping structure and late steps shaping detail and photorealism.For appearance and identity tasks, starting later preserves structural diversity; ending before the final timestep avoids competition with photorealism.Source: §8.1 · p. 12

8.2. Update Rule

  • Normalized gradients produce the highest-fidelity results because their update step is agnostic to gradient scale, unlike unnormalized variants that can show contrast and saturation artifacts.Timestep rescaling has negligible effect, while timestep conditioning of readout heads is important for avoiding improbable outputs.Source: Figure 11 · p. 12, §8.2 · p. 13

8.3. Sampling Compute

  • Readout Guidance increases pose-control sampling time and memory versus text-only sampling, with larger costs for drag-based manipulation and SDXL.The reported implementation retains features, gradients, and target signals in memory and is not optimized for memory usage.Source: §8.3 · p. 13

8.4. Training Compute

  • Readout heads train in minutes to a few hours on an Nvidia A100, depending on the head and diffusion model.Reported training times range from 8 minutes for appearance similarity on SDv1-5 to 2 hours 40 minutes for depth on SDXL.Source: §8.4 · p. 13

8.5. Readout Head Architecture

  • All readout heads share an architecture that standardizes decoder features, aggregates them across layers, and optionally converts them into RGB outputs.Timestep conditioning is added during feature standardization, while spatially aligned heads use convolutions and bounded Tanh outputs.Source: §8.5 · pp. 13–14, Figure 12 · p. 14

8.6. Improbable Images

  • Excessive guidance can satisfy pose controls while producing cartoonish colors, surreal imagery, or textureless scenes outside the natural-image manifold.Lowering guidance weight, stopping guidance earlier, or changing the initial seed can resolve these artifacts.Source: §8.6 · p. 14

9. Relative Control

For identity-consistent generation, Readout Guidance constrains a face-specific identity signal while separately preserving the surrounding appearance. Removing background appearance guidance increases background drift, although identity may become closer.

  • Face-region identity guidance is paired with whole-image appearance guidance toward the original trajectory to preserve the background.The identity head is trained on face photographs, and an off-the-shelf face detector determines where the identity constraint applies.Source: §9.1 · p. 14
  • Removing appearance similarity guidance makes the background drift more in appearance but often matches identity more closely.The full method combines identity consistency and appearance similarity heads.Source: §9.1 · p. 14
  • IP-Adapter [67] can borrow a reference identity but cannot inject that identity into a particular sampled image.The comparison uses an adapter conditioned on the CLIP image embedding of a reference image.Source: §9.1 · p. 15

10. Spatially Aligned Control

Readout Guidance provides informative intermediate readouts throughout diffusion, including early structural predictions, and supports pose control with preserved image quality. Qualitative examples also reveal occasional hallucinated details and imperfect depth smoothness.

10.1. Readout Evolution

  • Readouts reveal composition early: a foreground figure appears at the first sampling step and a coarse silhouette within the first 10%.The corresponding denoising states are less interpretable than clean-image predictions or feature-derived readouts.Source: §10.1 · p. 15
  • Readouts remain informative across the entire diffusion process, including early noisy timesteps, supporting their use for guidance.Source: §10.1 · p. 16

10.2. Quantitative Comparison

  • 13% and 30% PCK improvements at α = 0.05 are obtained when the pose head guides ControlNet [70] and T2IAdapter [40], respectively.The pose head trained on the vanilla base model transfers to fine-tuned conditional models because they share the frozen base model’s feature space.Source: §10.2 · p. 16
  • Readout Guidance slightly improves image quality over the base model in CMMD comparisons, while more training data yields progressively better image quality.The comparisons include No Control vs. RG8.5k and T2IAdapter vs. T2IAdapter + RG8.5k.Source: §10.2 · p. 16

10.3. Additional Examples

  • Qualitative examples show reliable foreground-background depth separation but occasional extra pose details and nonuniform depth within objects.Performance improves on more difficult poses when training uses all 8.5k images, while simpler poses perform similarly across variants.Source: §10.3 · p. 16

11. Text Prompts

This section documents the prompts and supplementary visual examples used to demonstrate Readout Guidance across pose, depth, edge, refinement, and limited-data control.

Text prompts

  • The section lists text prompts for demonstrations involving portraits, animals, people, sports, and other generated scenes.The supplied prompts include examples such as a woman with the Eiffel Tower, a squirrel with a hamburger, and people playing sports.Source: §11 · p. 17

Readout evolution

  • Readout Evolution shows noisy states, predicted clean images, and pose, depth, and edge readouts across diffusion, with composition largely determined by t = 10.The readouts become crisper and more refined over the diffusion process.Source: Figure 15 · p. 18

Pose, depth, and edge control

Control refinement

  • Readout Guidance is also shown combined with ControlNet and T2IAdapter for control refinement.Separate supplementary figures provide examples for each combination.Source: Figure 22 · p. 25, Figure 23 · p. 26

Limited data

  • Pose-head training is illustrated with varying amounts of data, including 100 images, 1k images, and 8.5k images.The supplementary labels compare these three training-data settings.Source: Figure 24 · p. 27, §11 · p. 27
Loading 2312.02150v2…