PaperlayerView exact revision on arXiv

Source-linked AI summary

Controllable Generation with Text-to-Image Diffusion Models: A Survey

Pu Cao, Feng Zhou, Qing Song, Lu Yang

arXiv:2403.04279v2cs.CV

TL;DR

Text conditioning alone cannot fully satisfy the varied requirements of controllable text-to-image generation. This survey reviews the field and identifies two fundamental mechanisms for introducing novel conditions: conditional score prediction and condition-guided score estimation.

  • Problem

    Text-based conditions cannot fully satisfy diverse user requirements, motivating research into precise control of text-to-image diffusion models.

    Source: §1 · p. 1

  • Method

    The survey reviews foundations and prior work, organizing controllable-generation methods by specific, multiple, and universal conditions and examining their control mechanisms.

    Source: §1 · p. 2

  • Results

    The survey identifies two fundamental paradigms for introducing novel conditions: conditional score prediction and condition-guided score estimation.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The condition-based taxonomy offers a clearer lens on methods’ core theoretical principles and category-specific challenges.

    Source: §1 · p. 2

    Existing controllable diffusion models are often tailored to specific conditions or tasks, leaving unified control across spatial, semantic, and multimodal inputs an open direction.

    Source: §7 · p. 14

Abstract

from arXiv · show

In the rapidly advancing realm of visual generation, diffusion models have revolutionized the landscape, marking a significant shift in capabilities with their impressive text-guided generative functions. However, relying solely on text for conditioning these models does not fully cater to the varied and complex requirements of different applications and scenarios. Acknowledging this shortfall, a variety of studies aim to control pre-trained text-to-image (T2I) models to support novel conditions. In this survey, we undertake a thorough review of the literature on controllable generation with T2I diffusion models, covering both the theoretical foundations and practical advancements in this domain. Our review begins with a brief introduction to the basics of denoising diffusion probabilistic models (DDPMs) and widely used T2I diffusion models. We then reveal the controlling mechanisms of diffusion models, theoretically analyzing how novel conditions are introduced into the denoising process for conditional generation. Additionally, we offer a detailed overview of research in this area, organizing it into distinct categories from the condition perspective: generation with specific conditions, generation with multiple conditions, and universal controllable generation. For an exhaustive list of the controllable generation literature surveyed, please refer to our curated repository at https://github.com/PRIV-Creation/Awesome-Controllable-T2I-Diffusion-Models.

1 INTRODUCTION

Text-to-image diffusion models generate high-quality images from natural-language descriptions, but text alone cannot express some nuanced visual requirements. This survey reviews controllable generation methods, organizing them by condition type and analyzing their foundations and applications.

Introduction

  • Text prompts alone may not convey requirements such as depicting an unseen person or a distinct art style, motivating conditions beyond text.The survey notes that open-source T2I models have facilitated exploration of these additional conditions.Source: §1 · pp. 1–2
  • Earlier surveys often treated control of T2I diffusion models briefly or focused on other modalities, leaving novel-condition integration and impact under-analyzed.The paper is organized around model foundations and taxonomy, novel-condition introduction, multi-condition integration, universal controllable generation, and applications.Source: §1 · p. 2
  • The survey reviews T2I diffusion theory and widely adopted models, examines prior methods technically, and discusses their applications across contexts and related tasks.It also presents controllable generation applications across diverse generative tasks.Source: §1 · p. 2
  • The survey proposes a condition-based taxonomy covering novel condition introduction, multi-condition integration, and universal controllable generation.It uses the taxonomy to explain core theoretical principles and challenges in each category.Source: §1 · p. 2
  • It organizes methods around two paradigms for incorporating novel conditions: conditional score prediction and condition-guided score estimation.The survey reviews methods under these paradigms and highlights their features, contributions, and comparative advantages.Source: §1 · p. 2

2 PRELIMINARIES

The preliminaries explain diffusion and flow-based generation, outline major text-to-image architectures and controllable tasks, summarize evaluation dimensions, and introduce a three-part taxonomy for controllable generation.

2.1 Denoising Diffusion Probabilistic Models

  • DDPMs add Gaussian noise over timesteps, then learn reverse transitions that progressively denoise from noise to data, with neural networks such as U-Nets predicting the noise.Generation samples an initial noise vector and repeatedly applies learned reverse transition kernels.Source: §2.1 · p. 2

2.2 Flow Matching and Rectified Flow

  • Flow Matching learns a continuous velocity field and generates by deterministic ODE integration, while Rectified Flow uses straight-line interpolation to yield smoother flows and faster integration.The survey notes that flow matching underpins systems including Stable Diffusion 3 [28] and Flux.Source: §2.1 · p. 2, §2.2 · p. 3

2.3 Text-to-Image Diffusion Models

  • T2I models condition generation on text through U-Net cross-attention or transformer backbones, with MMDiT enabling bidirectional interaction between text and image tokens.LDMs diffuse in compressed latent space; Imagen [15] found that scaling the text encoder improved quality more than enlarging the diffusion model.Source: §2.3 · p. 3

2.4 Controllable Generation Tasks

  • Controllable tasks add signals beyond text, including spatial layouts, poses, depth, segmentation, exemplar concepts, and viewpoint or multi-view information.View-conditioned methods target geometric consistency and support novel-view synthesis, 3D-aware generation, and panoramic rendering.Source: §2.4 · p. 3
  • Other controllable tasks address complex-prompt alignment and multilingual limits, adapt through task-specific image–text examples, use brain or sound inputs, and render legible text in images.Recent text-conditioned strategies aim to improve alignment and expand linguistic scope; in-context methods adapt to new tasks with minimal supervision.Source: §2.4 · pp. 3, 5

2.5 Evaluation of Controllable T2I Generation

  • Controllable T2I evaluation considers image quality and conditional alignment, using distribution, aesthetic, preference, similarity, feature-discrepancy, or detection metrics.Examples include IS [53] and FID [52] for image quality, CLIP-based similarity [34], [57] for text alignment, and mAP for layout-conditioned generation.Source: §2.5 · p. 5

2.6 Taxonomy

  • The taxonomy separates novel-condition methods, multi-condition control, and condition-agnostic frameworks; their challenges are injecting conditions without quality loss, faithfully fusing signals, and handling diverse inputs.Novel-condition methods are grouped by conditional score prediction or condition-guided score estimation; multi-condition strategies include joint training, continual learning, weight fusion, attention integration, and guidance composition.Source: §2.6 · p. 5

3 CONTROL TEXT-TO-IMAGE DIFFUSION MODELS WITH NOVEL CONDITIONS

This section surveys mechanisms and methods for controlling pretrained text-to-image diffusion models with novel conditions, including tuning- and adapter-based score prediction, training-free prediction, and gradient guidance. It covers applications such as audio and text rendering, spatial and attribute control, reference-based generation, high-resolution synthesis, and semantic alignment.

3 CONTROL TEXT-TO-IMAGE DIFFUSION MODELS WITH NOVEL CONDITIONS

  • Novel conditions enter through conditional score prediction, which directly predicts the joint text-and-condition score, or condition-guided estimation, which injects a likelihood-derived gradient into sampling.The latter adjusts the original text-conditioned score with a scale parameter.Source: §3 · p. 6

3.1 Conditional Score Prediction

  • Conditional score prediction methods are grouped as tuning-based, adapter-based, or training-free according to how they empower pretrained denoisers to predict conditioned latents.Source: §3.1 · p. 6

3.1.1 Tuning-based Conditional Score Prediction

  • Tuning-based methods adapt text embeddings or model parameters to specific conditions, often from few-shot data, while addressing concept disentanglement and preservation of pretrained knowledge.Textual Inversion learns new token embeddings, DreamBooth also updates UNet parameters with prior-preservation loss, and Custom Diffusion focuses updates on cross-attention key and value projections. Perfusion, SVDiff, and OFT use distinct strategies to reduce overfitting and retain generality.Source: §3.1.1 · pp. 6–7

3.1.2 Adapter-based Conditional Score Prediction

  • Adapter-based prediction adds an encoder that maps novel conditions to features; its encoder and parameters are learned offline and remain fixed at inference.Adapters interface with the backbone through token concatenation, cross-attention keys and values, or feature-wise modulation, depending on the condition family.Source: §3.1.2 · pp. 7–8
  • Spatial adapters support controls such as geometry and layout: ControlNet uses a U-Net-mirroring branch with zero convolutions, while later controllers target multiple control types and structured placement.GLIGEN injects grounded language through gated layers, and UniControl uses task-aware condition encoding and task instructions to regulate feature injection.Source: §3.1.2 · p. 8
  • Adapter methods also cover image personalization, view-conditioned and long-text generation, in-context tasks, and brain-guided image reconstruction.Open-world encoders extract subject-level conditions; camera parameters or panorama-aware denoising provide view control, while other methods condition on long-form text, visual context, fMRI, or EEG.Source: §3.1.2 · pp. 8–9
  • Audio adapters map sound into text or latent spaces, while language-model encoders and added layout or glyph conditions address text rendering.GlueGen aligns multimodal encoders with Stable Diffusion [48], and AAI turns sound into a plug-and-play token [236]. T5-based encoders are used by eDiffI [17] and Imagen [15]; text-rendering approaches include DeepFloyd IF, GlyphControl [244], TextDiffuser [238], and training-free Diff-Text [49], while TextDiffuser contributes the 10-million-pair MARIO-10M dataset.Source: §3.1.2 · pp. 9–10

3.1.3 Training-free Conditional Score Prediction

  • Training-free attention-map adjustment controls object placement, attributes, and textual concept representation by changing token-to-image correspondences.Paint-withwords uses correspondence segmentation maps to rectify word cross-attention [17]; DenseDiffusion adds regularization for layout precision and flexibility [247], and soft refinement improves adjacent interactions [251]. Region-based diffusion [256] and Structure Diffusion [257] target attribute binding.Source: §3.1.3 · p. 10
  • Feature injection adds reference-image information to self- or cross-attention for object placement, appearance, and style control.Methods inject object features during inversion [259], share attention with a style reference [265], replace principal components using reference features [261], or transfer cross-attention features using Earth Movers Distance [266].Source: §3.1.3 · p. 10
  • Other training-free approaches reweight diffusion features, localize editable attributes, coordinate high-resolution crops, propagate boundary cues, or select better initial noise.FreeU reweights skip-connection and backbone features [268], while mechanistic localization identifies relevant UNet layers [269]. MultiDiffusion fuses crop denoising directions [270], Progressive Boundary Complement improves position propagation [271], and InitNo ranks sampled noise using attention-based scores [272].Source: §3.1.3 · p. 10

3.2 Condition-guided Score Estimation

  • Condition-guided score estimation computes losses between predicted conditions and supplied conditions, then uses their gradients to guide denoising.LGP predicts sketches from intermediate UNet features and guides generation by sketch similarity [274]; Universal Guidance and FreeDom use image-space predictors and one-step denoised images to calculate guidance gradients.Source: §3.2 · p. 11
  • Layout and segmentation guidance can derive losses directly from attention maps, avoiding additional trained predictors, while other losses enforce textual alignment and attribute binding.BoxDiff applies inner-box, outer-box, and corner constraints; ZestGuide aligns cross-attention segmentation with masks, VersaGen uses object-token attention, and VODiff models visibility order. Attend-and-Excite, EBAMA, and SynGen [254] target semantic coverage, attribute binding, and syntactic bindings.Source: §3.2 · p. 11

4 CONTROLLABLE GENERATION WITH MULTIPLE CONDITIONS

Multi-condition generation creates images from multiple simultaneous conditions. The survey organizes methods into joint training, continual learning, weight fusion, attention-based integration, and guidance composition.

4.1 Joint Training

  • Jointly trained multi-condition methods focus on multi-condition encoders and training strategies; Composer projects conditions into uniform-dimensional embeddings matching noisy-latent spatial size and jointly trains image generation from representations.Composer independently drops each condition with probability 0.5, drops all conditions with probability 0.1, and retains all with probability 0.1. SVDiff and FastComposer constrain cross-attention so text embeddings are effective only in corresponding areas.Source: §4.1 · p. 11

4.2 Continual Learning

  • Continual-learning approaches address catastrophic forgetting through regularization, memory, distillation, and concept-binding mechanisms.C-LoRA regulates new LoRA updates using past weight deltas; L2DM preserves prior and personalized concepts with memory and distillation modules; STAMINA adds forgetting and sparsity regularization, while ConceptGuard combines shift embeddings, binding prompts, and memory preservation.Source: §4.2 · p. 12

4.3 Weight Fusion

  • Weight-fusion methods merge personalized model weights or embeddings while aiming to preserve each condition’s controllability.Cones fine-tunes concept neurons; Custom Diffusion merges key and value matrices, Mix-of-Show uses gradient fusion, ZipLoRA fuses subject and style LoRA weights, and Po et al. encourage orthogonal residual weights.Source: §4.3 · p. 12

4.4 Attention-based Integration

  • Attention-based integration controls where conditions appear by modulating attention maps, using layouts or regional prompts with masks.Cones2 edits cross-attention with predefined layouts and timestep-dependent edit intensity; Mix-of-Show combines global and regional prompts through cross-attention and predefined masks.Source: §4.4 · p. 12

4.5 Guidance Composition

  • Guidance composition combines independent denoising results for multiple conditions using condition-specific weights and spatial masks.Decompose and Realign derives masks from cross-attention, Face-diffuser fuses text-driven and subject-augmented diffusion results, and Cao et al. combine domain guidance from a null-text UNet with control guidance from the original diffusion prior.Source: §4.5 · p. 12

5 UNIVERSAL CONTROLLABLE TEXT-TO-IMAGE GENERATION

The survey groups universal controllable generation into conditional score prediction and condition-guided score estimation, both intended to accommodate arbitrary conditions.

5.1 Universal Conditional Score Prediction Framework

  • Universal conditional score prediction encodes arbitrary conditions and uses them to predict noise at each timestep, adapting generation to diverse inputs.It integrates conditional information directly into the generative model.Source: §5.1 · p. 13
  • DiffBlender [310] routes image, spatial, and non-spatial conditions through different modules, while Emu2 uses a 37-billion-parameter multimodal model for task-agnostic in-context learning.DiffBlender injects image conditions in ResNet blocks, spatial conditions through local self-attention, and non-spatial conditions through global self-attention and cross-attention.Source: §5.1 · p. 13

5.2 Universal Condition-Guided Score Estimation

  • Universal Guidance uses a reconstructed clean image to obtain feedback from an off-the-shelf predictor, supporting conditions such as text, style, segmentation, identity, and bounding boxes.The survey identifies obtaining condition-specific guidance from the latent during denoising as the central challenge for this approach family.Source: §5.2 · p. 13
  • FreeDom combines off-the-shelf predictors with time-independent energy functions and a time-travel resampling strategy to address condition misalignment on large domains such as ImageNet.Its strategy takes the intermediate result z_t back j steps to z_(t+j), then resamples it at timestep t.Source: §5.2 · p. 13
  • Symplectic Adjoint Guidance (SAG) [319] estimates the clean image with n function calls, then applies the symplectic adjoint method to obtain gradients accurately.This two-stage procedure differs from one-step condition-related estimation used by the previously mentioned approaches.Source: §5.2 · p. 13

6 APPLICATIONS

The section surveys how controllable text-to-image diffusion models apply novel conditions to tasks ranging from image editing and completion to composition, 3D generation, and video-based world models.

6.1 Image Manipulation

  • SINE builds its fine-tuning prompt from the source image and uses adapter-based classifier-free guidance; Choi et al. [328] use reference-image elements, while Zhou et al. [329] introduce dual-objective LQR to mitigate error accumulation in multi-turn editing.Source: §6.1 · p. 13

6.2 Image Completion and Inpainting

  • DreamInpainter [322] and Realfill [323] use reference images to personalize masked-area filling and support realistic, coherent image completion; Uni-inpaint jointly controls filling with text, strokes, and exemplar images.Source: §6.2 · p. 13

6.3 Image Composition

  • ObjectStitch controls viewpoint, geometry, lighting, and shadow; DreamCom [331] preserves foreground-object details, while ControlCom [332] uses a task indicator to unify four composition-related tasks.These methods address composition stages such as color harmonization, geometry correction, and shadow generation.Source: §6.3 · pp. 13–14

6.4 Text/Image-to-3D Generation

  • Score Distillation Sampling (SDS), introduced by DreamFusion, transfers text-to-image control to 3D generation; DreamBooth3D [334] personalizes from a few subject images, while adapted ControlNet methods, [336], [336], use spatial signals.Examples of spatial signals include depth maps and sketches.Source: §6.4 · p. 14

6.5 World Model

  • Camera-controlled video methods, –[343] condition generation on camera parameters; training-free WorldForge [346] adjusts a projected static point cloud along camera trajectories to guide video generation.ReCamMaster [343] conditions DiT blocks by frame dimension, while AC3D uses lightweight 128-dimensional DiT-SX blocks.Source: §6.5 · p. 14

7 CONCLUSION

The survey reviews the foundations, mechanisms, methods, and applications of controllable text-to-image diffusion generation. It concludes with open directions for unified cross-modal control and diffusion-based world models.

Conclusion

  • The survey introduces diffusion and text-to-image model foundations, explains how novel conditions enter generation, and organizes prior methods by theory, technical advances, and solution strategies.It also discusses practical applications and the potential of controllable generation.Source: §7 · p. 14

Future directions

  • Despite remarkable progress, future work could unify spatial, semantic, and multimodal controls and extend controllable generation beyond text-to-image to audio, video, and 3D.The proposed goal is a flexible, generalizable framework that supports diverse conditions within one generative system.Source: §7 · pp. 14–15
  • Future studies could use diffusion-based control mechanisms to build video world models that simulate camera-controllable environments with temporal–spatial consistency.The survey positions such world models as a potential link between generative modeling, embodied AI, and interactive virtual environments.Source: §7 · p. 15
Loading 2403.04279v2…