PaperlayerView exact revision on arXiv

Source-linked AI summary

Sketch-Guided Text-to-Image Diffusion Models

Andrey Voynov, Kfir Aberman, Daniel Cohen-Or

arXiv:2211.13752v1cs.CVcs.GRcs.LG

TL;DR

Pretrained text-to-image diffusion models lack intuitive spatial control. The paper introduces a per-pixel latent guidance predictor that guides inference with spatial maps using lightweight training, and demonstrates diverse out-of-domain sketch-guided synthesis and other spatially guided applications.

  • Problem

    Pretrained text-to-image diffusion models lack control handles for guiding the spatial properties of synthesized images.

  • Method

    A small per-pixel MLP maps noisy-image latent features from the diffusion model’s core network to spatial maps and guides inference through a propagated spatial-map loss.

  • Results

    The method demonstrates sketch-guided text-to-image synthesis across diverse domains, including free-hand drawings, and supports saliency-guided inpainting and horizon control.

  • Takeaways & Limitations

    The approach accepts varied sketch styles while providing intuitive spatial control over outputs that retain text-prompt semantics.

  • Takeaways & Limitations

    The technique struggles with complex or cluttered sketches, stroke-style effects, and conflicts between random initialization and the input sketch.

Abstract

from arXiv · show

Text-to-Image models have introduced a remarkable leap in the evolution of machine learning, demonstrating high-quality synthesis of images from a given text-prompt. However, these powerful pretrained models still lack control handles that can guide spatial properties of the synthesized images. In this work, we introduce a universal approach to guide a pretrained text-to-image diffusion model, with a spatial map from another domain (e.g., sketch) during inference time. Unlike previous works, our method does not require to train a dedicated model or a specialized encoder for the task. Our key idea is to train a Latent Guidance Predictor (LGP) - a small, per-pixel, Multi-Layer Perceptron (MLP) that maps latent features of noisy images to spatial maps, where the deep features are extracted from the core Denoising Diffusion Probabilistic Model (DDPM) network. The LGP is trained only on a few thousand images and constitutes a differential guiding map predictor, over which the loss is computed and propagated back to push the intermediate images to agree with the spatial map. The per-pixel training offers flexibility and locality which allows the technique to perform well on out-of-domain sketches, including free-hand style drawings. We take a particular focus on the sketch-to-image translation task, revealing a robust and expressive way to generate images that follow the guidance of a sketch of arbitrary style or domain. Project page: sketch-guided-diffusion.github.io

1. Introduction

The paper addresses the missing spatial control in pretrained text-to-image diffusion models by introducing a lightweight latent guidance predictor. Its per-pixel, domain-oblivious design supports diverse out-of-domain sketches and other spatially guided tasks.

  • Pretrained text-to-image diffusion models provide semantic guidance but lack intuitive control over synthesized images’ spatial properties.
  • A dedicated encoder baseline performs well in-domain but struggles with out-of-domain free-hand sketches.
  • The method trains a small MLP to map noisy-image latent features from the diffusion model’s core network to spatial maps.
  • The predictor is trained independently at each latent pixel, requiring only a few thousand images rather than dedicated image-to-image model training.
  • Per-pixel training makes the predictor domain-oblivious, enabling free-hand sketches and diverse outputs that follow both text semantics and sketch layout.
  • The approach demonstrates sketch-guided synthesis across various domains, including free-hand drawings, and extends to saliency-guided inpainting and horizon control.

2. Related Work

Prior image-translation and diffusion approaches commonly train task-specific models or encoders. This work instead uses a pretrained text-to-image diffusion model as a universal generative prior guided by spatial maps from other domains.

  • Image-to-image translation methods commonly use conditional GANs and task-specific datasets and models, although some efforts seek unified models.
  • Sketch-to-photo methods include augmentation-based GANs, joint image-sketch representations, and VQ-GAN embedding alignment.
  • Diffusion-based image translation has also used models trained independently from scratch for each task.
  • The paper builds on text-to-image diffusion advances by guiding a pretrained model with spatial maps from different domains.
  • Related inference methods guide diffusion with noisy reference images or partially noised guiding images in the RGB domain.
  • Compared with specialized per-task encoders, this method trains a small per-pixel MLP on only a few thousand images and generalizes beyond the training domain.

3. Method

The method guides a pretrained text-to-image diffusion model with a per-pixel latent edge predictor trained from noisy image features, then uses its gradients during denoising to align generated edges with a sketch. The predictor requires only a few thousand training images, while guidance is applied during selected reverse-diffusion steps.

  • Latent Edge Predictor: The method uses an MLP edge predictor over internal U-net activations to guide a pretrained text-to-image diffusion model toward a reference sketch.The predictor operates per pixel on concatenated features extracted from selected intermediate layers of the diffusion model.
  • Latent Edge Predictor: Training pairs images, edge maps, and captions, encodes the images and edges, adds scheduled Gaussian noise, and maps noisy latent features to encoded edge maps.The input is z_t = α_t · E(x) + μ_t · ξ, with noise scheduling matched to the diffusion process.
  • Latent Edge Predictor: The predictor is applied independently at each latent pixel, enabling local edge prediction and training with a few thousand images in about one hour on a single A100 GPU.Its per-pixel design avoids modeling the whole image and supports domain-oblivious operation on guiding maps.
  • Sketch-Guided Text-to-Image Synthesis: During inference, features from each denoising step are passed through the predictor to estimate latent edges and compute their similarity to the target sketch.The resulting loss is differentiated with respect to the noisy latent input.
  • Sketch-Guided Text-to-Image Synthesis: The normalized anti-gradient modifies the next denoising sample, with the guidance strength controlled by α, pushing intermediate samples toward the target edges.Edge guidance is applied only from t = T through S > 1, commonly with S = 0.5T, because later steps typically do not alter geometric layout.

4. Experiments

The experiments evaluate sketch-guided text-to-image synthesis against diffusion and image-translation baselines, then analyze guidance parameters and stroke-style robustness. Results cover varied domains, including free-hand sketches, while exposing trade-offs between texture, realism, and sketch fidelity.

  • Sketch-guided synthesis: The framework converts sketches into images under text prompts across varied domains, seeds, and captions.Figure 4 presents sketch-to-image galleries demonstrating the framework’s intended use.
  • Comparisons: SDEdit struggles to add texture at low starting t, while increasing t significantly reduces fidelity to the input sketch.The comparison uses the prompt “A photograph of a bike made of wood”.
  • Comparisons: pix2pix performs well on sketches within its training domain but fails on out-of-domain hand-drawn sketches.This limitation reflects its paired-image training setup.
  • Comparisons: PITI handles realistic sketches but struggles with out-of-domain free-hand sketches, while the proposed method provides significantly more color and style variability.The comparison highlights differences in both domain robustness and output diversity.
  • Ablations and parameter tuning: The edge-guidance scale β controls a trade-off between image realism and alignment with the target sketch’s edges.Small β favors realistic details and textures, whereas larger β favors edge alignment but produces less realistic, piece-wise smooth results.
  • Ablations and parameter tuning: Per-pixel latent edge prediction makes outputs insensitive to stroke style: identical sketch geometry preserves shape while colors and textures vary.The observed variation is attributed to stroke style affecting the inner synthesis process.

5. Applications

The framework extends beyond sketch-to-image synthesis to saliency-guided image editing and other spatially guided tasks. Its applications include controlling saliency, background inpainting, and the balance between realism and edge fidelity.

  • Applications: The generic framework applies to spatially guided tasks beyond sketch-to-image translation, including saliency-guided inpainting and horizon control.The paper presents sketch-to-image as one application of a broader spatial-guidance approach.
  • Saliency guidance: Saliency maps guide whether generated regions receive high or low saliency, enabling targeted image enhancement and background inpainting.The latent guidance predictor is trained to predict the original downsampled saliency map.
  • Edge fidelity: The edge-guidance scale β provides user control over the realism–edge-fidelity trade-off.Lower β favors realism, while higher β favors edge alignment and less realistic piece-wise smooth outputs.
  • Saliency guidance: For background inpainting, low saliency in a masked bird-body region removes the body and fills the region with background, unlike semantic continuation from the visible tail.High saliency guidance can instead illuminate a marked region, and guidance is sufficient for the first 20% of steps.
  • Stroke ablation: The method preserves sketch geometry across stroke styles while varying output colors and textures.This supports the stated insensitivity of the local latent edge predictor to stroke style.

6. Conclusions

The paper presents spatial-map guidance for pretrained text-to-image diffusion models using lightweight per-pixel prediction, with flexibility across sketch styles and several demonstrated applications. It also identifies sensitivity to stroke style, scene complexity, initialization, and seed choice.

  • The technique guides a pretrained text-to-image diffusion model with a spatial map, focusing on sketches and extending to other spatially guided tasks.The paper demonstrates saliency-guided inpainting and horizon control in addition to sketch guidance.
  • Per-pixel training of a lightweight MLP uses small data and avoids binding the predictor to one global sketching style.The predictor behaves more like a differential edge-detector than a whole-image model.
  • The method accepts varied sketch styles and provides semantic control through text while generating diverse outputs from the same spatial guidance.The approach combines a spatial sketch map with text conditioning.
  • The technique remains vulnerable to local stroke style, cluttered or ambiguous scenes, and conflicts between stochastic initialization and the input sketch.These conditions can reduce agreement between the generated output and the sketch.

A. Societal Impact

The work is presented as a tool for converting simple sketches into detailed images, with potential to reduce the time required for artistic creation and broaden access to creative production.

  • The method could significantly speed artistic creation because simple sketches take minutes while detailed colored pictures normally take hours.The authors frame the system as assisting artists by handling technical parts rather than replacing them.

B. Ablation studies

Ablation studies support using deep diffusion-network features and per-pixel prediction rather than intermediate image states, while exposing failure modes for alternative guidance inputs.

  • Training on only dogs from ImageNet still produced reasonably good edge prediction, supporting the predictor’s out-of-domain behavior.The experiment used an extremely tiny, single-domain training set.
  • An edge predictor operating on noisy intermediate states commonly collapsed to empty maps, unlike the proposed predictor using deep diffusion features.The authors attribute the difficulty to noisy-image edge prediction approaching the complexity of DDPM denoising.
  • Low guidance weight failed to match the sketch, whereas high guidance scale produced nearly adversarial images.This illustrates a narrow useful range for guidance strength in the tested alternative.
  • Guidance through intermediate denoised-image predictions remained unsuccessful despite using noise-free predicted images as the edge-predictor input.The alternative computes edges from the decoded z0 prediction and backpropagates their similarity to target edges.

C. Additional tests

Additional tests show that text and spatial guidance jointly shape generation, stochastic sampling preserves output diversity, and soft spatial labels can interpolate scene attributes. Guidance also adds substantial sampling cost.

  • Relevant prompts improve plausibility and detail, while conflicting or unsupported objects can confuse sketch alignment.An artistic prefix increased sketch alignment, whereas a conflicting dog prompt still produced a dog consistent with the sketch.
  • A single sketch and prompt can generate a variety of samples because DDPM sampling is stochastic.The variation is illustrated across different random seeds.
  • Interpolating day and night label probabilities produces images whose scene appearance also interpolates between night and day.The method uses a soft [0, 1] grayscale spatial map and trains the per-pixel predictor with 500 day and 500 night images.
  • Applying guidance during the first half of reverse diffusion steps increases relative sampling time by approximately 80%.The overhead mainly comes from backpropagating the edge-prediction loss from inner features to the input image.

D. Models details and data

The experiments use Stable Diffusion checkpoints with non-deterministic sampling and 250 reverse diffusion steps, drawing examples from Sketchy, edge2shoes, ImageNet, and author-provided sketches. Figures examine textual conditioning, edge guidance, and generalization across training domains.

  • Datasets: Synthetic quantitative results are based on Sketchy, while real results use ImageNet with class names as prompts.Additional samples come from edge2shoes and author-provided sketches.
  • Experimental setup: Experiments use Stable Diffusion v1-4-original except for inpainting, which uses Stable Diffusion inpainting.All experiments sample non-deterministically with 250 reverse diffusion steps.
  • Evaluations: Figure 18 examines textual conditioning and includes a failure case when the prompt does not match the sketch.Blue text identifies the subject presented in the sketch.
  • Evaluations: Figure 19 presents edge-guided samples from Sketchy using class names with the prefix “A photo of”.The listed examples span objects and animals, including lizard, rabbit, car, turtle, and airplane.
  • Evaluations: Figure 20 compares guidance from a predictor trained on varied ImageNet samples with one trained only on different dog classes.The comparison tests whether narrowly trained guidance generalizes across image domains.
Loading 2211.13752v1…