Source-linked AI summary
InstanceDiffusion: Instance-level Control for Image Generation
Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, Ishan Misra
TL;DR
Text-to-image diffusion models lack precise control over individual instances. InstanceDiffusion combines per-instance language and flexible location conditions through unified modeling and three architectural changes, outperforming specialized prior methods across conditions while leaving small-object quality and texture binding as challenges.
Problem
Text-to-image diffusion models produce high-quality images but lack precise control over individual instances, including their locations and attributes.
Method
InstanceDiffusion conditions generation on a global caption plus per-instance captions and locations, using UniFusion, ScaleU, and a Multi-instance Sampler.
Results
InstanceDiffusion significantly outperforms specialized state-of-the-art models across location conditions, including a 20.4% APbox 50 increase over GLIGEN for boxes and a 25.4% IoU boost over DenseDiffusion for masks.
Takeaways & Limitations
The approach enables precise instance-level control with boxes, masks, scribbles, points, and per-instance attribute prompts.
Takeaways & Limitations
Small objects have lower generation quality than larger ones, and texture binding remains challenging across tested methods, including InstanceDiffusion.
Abstract
from arXiv · showhide
Text-to-image diffusion models produce high quality images but do not offer control over individual instances in the image. We introduce InstanceDiffusion that adds precise instance-level control to text-to-image diffusion models. InstanceDiffusion supports free-form language conditions per instance and allows flexible ways to specify instance locations such as simple single points, scribbles, bounding boxes or intricate instance segmentation masks, and combinations thereof. We propose three major changes to text-to-image models that enable precise instance-level control. Our UniFusion block enables instance-level conditions for text-to-image models, the ScaleU block improves image fidelity, and our Multi-instance Sampler improves generations for multiple instances. InstanceDiffusion significantly surpasses specialized state-of-the-art models for each location condition. Notably, on the COCO dataset, we outperform previous state-of-the-art by 20.4% AP$_{50}^\text{box}$ for box inputs, and 25.4% IoU for mask inputs.
1. Introduction
InstanceDiffusion targets precise, flexible control over multiple instances by combining per-instance locations and text attributes. Its unified modeling and specialized design choices outperform prior methods across location conditions and improve attribute adherence.
- Problem and scope: Instance-conditioned generation specifies each instance with a location and an instance-level text prompt, supporting boxes, masks, points, scribbles, and mixed formats.Per-instance prompts control attributes such as color and texture, while location precision can vary across instances.
- Model design: Unified modeling parameterizes and fuses different location formats with a simpler architecture while exploiting their shared structure to improve performance.The approach avoids separate architectures and strategies for each location format.
- Results: 20.4% increase in APbox 50 over GLIGEN is achieved for bounding-box inputs on COCO val.This evaluation compares InstanceDiffusion with GLIGEN under bounding-box conditioning.
- Results: 25.4% boost in IoU over DenseDiffusion and 36.2% gain in APmask 50 over ControlNet are obtained for mask-based inputs.These results are reported on COCO evaluations for mask-based conditioning.
- Results: The method improves instance attribute adherence, including a 25.2 point gain in color accuracy and a 9.2 point improvement in texture accuracy over GLIGEN.These gains are reported for instance-level text-prompt attributes.
- Contributions: The paper contributes UniFusion, ScaleU, and a Multi-instance Sampler, plus an instance-captioned dataset and benchmarks for location-grounded generation.UniFusion projects conditions into a shared feature space, ScaleU recalibrates UNet features, and the sampler reduces multi-instance condition leakage and confusion.
2. Related Work
Prior work adds spatial conditioning to pretrained diffusion models to provide finer-grained control over generated images.
- Diffusion models: Image diffusion models generate images through iterative denoising from an initial random noise map.Latent diffusion models perform this process in a VAE latent space for computational efficiency.
- Spatial controls: Spatially controlled image generation introduces conditioning signals that guide the image synthesis process.Examples include semantic segmentation masks and other spatial controls added to large pretrained diffusion models.
- Prior methods: Make-a-Scene, SpaText, GLIGEN, and ControlNet provide finer-grained spatial control by allowing additional images or semantic masks as inputs.These methods extend large pretrained diffusion models with spatial conditioning mechanisms.
3. Instance Diffusion
InstanceDiffusion adds unified, instance-level conditioning to text-to-image diffusion, allowing per-instance captions and flexible location inputs. Its design combines UniFusion, ScaleU, and Multi-instance Sampler components to improve control, fidelity, and multi-instance generation.
- Problem definition: InstanceDiffusion conditions generation on a global caption plus per-instance captions and locations, generalizing open-set grounded text-to-image control.Each instance location may use a point, scribble, bounding box, or mask, while captions specify attributes such as color and texture.
- Location parameterization: Location formats are unified by converting masks, boxes, scribbles, and points into point sets, enabling one parameterization for diverse inputs.Masks use sampled interior and boundary points; boxes use corner points; scribbles use uniformly sampled points along the curve.
- UniFusion block: UniFusion projects instance captions and location formats into a shared feature space and fuses them with visual tokens from the frozen diffusion backbone.It is inserted between self-attention and cross-attention layers and supports multiple location formats per instance.
- Instance-masked attention: Masked self-attention prevents information leakage between instance conditions, addressing errors such as one instance’s color bleeding into another.Format-aware modeling treats location formats independently, while the mask restricts interactions across instances.
- ScaleU block: ScaleU dynamically recalibrates main and skip-connection features with learnable channel-wise scaling, increasing parameters by less than 0.01% overall.It scales main features directly and adjusts low-frequency skip features in the Fourier domain.
- Multi-instance Sampler: Multi-instance Sampler performs separate denoising for each instance during inference to further reduce conditioning leakage and improve generation quality and fidelity.The separate denoising operation runs for fewer than 10% of the overall steps.
4. Experiments
Experiments evaluate InstanceDiffusion across location formats, attribute binding, human preference, generalization, and component ablations. The method consistently outperforms prior work, while combining location formats and specialized components improves control and quality.
- Experimental setup: InstanceDiffusion supports boxes, masks, points, and scribbles, using the same model parameters across location inputs.Table 1 evaluates YOLO AP and AR for location adherence and FID for image quality on COCO val.
- Attribute binding: InstanceDiffusion improves attribute binding and human preference over prior work for instance-level captions.Evaluations measure color and texture accuracy, local CLIP alignment, and human judgments of adherence to locations and captions.
- Single location format at inference: InstanceDiffusion outperforms prior work across location conditions and evaluation metrics for object location and image quality.For box inputs, it achieves APbox 38.8 and ARbox 52.9, exceeding prior state of the art by +19.2 and +21.8, respectively.
- Multiple location formats at inference: Using all location formats together provides the best performance and more precise control over instance locations.The result supports unified modeling of the location formats at inference.
- Ablation study: Removing ScaleU substantially reduces APmask 50 and Acccolor, while removing generated instance captions causes a 17% drop in Acccolor.The ablation identifies ScaleU as important for location adherence and image fidelity, and captions as important for attribute binding.
- Ablation study: Multi-instance Sampler improves image quality and attribute binding, and transfers to prior location-conditioned models with notable gains.Its benefits are reported through lower FID, improved color accuracy, and stronger attribute binding for GLIGEN.
5. Conclusions, Limitations and Future Work
InstanceDiffusion enables precise instance-level control across masks, boxes, scribbles, and points while improving compliance with instance attributes. Small-object quality and texture binding remain challenges across the evaluated methods.
- Conclusions: InstanceDiffusion supports precise instance-level control across masks, boxes, scribbles, and points while outperforming prior work on instance-attribute compliance.The conclusion identifies flexible location formats and attribute adherence as the method’s central outcomes.
- Limitations and future work: Generation quality for small objects is noticeably lower than for larger objects, and texture binding remains challenging across tested methods.The authors identify improving instance conditioning in these cases as future work.
A1. Preliminary
Preliminary diffusion models generate images by iteratively denoising latent representations conditioned on text. Latent diffusion performs this process in VAE latent space and decodes the final latent into an image.
- Diffusion models: Diffusion models learn text-to-image generation through iterative denoising from an initial random noise map.Latent diffusion models perform the process in VAE latent space for computational efficiency.
- Training objective: A denoising autoencoder predicts the noise added to a noised latent vector, conditioned on the text prompt.The training objective samples denoising time steps uniformly and uses CLIP-processed text tokens.
- Inference: During inference, DDIM iteratively denoises a standard-normal latent zT to z0, which the VAE decoder converts into an image.The inference process begins from zT sampled from N(0, 1).
A2. Ablation Study
The ablations examine UniFusion, ScaleU, Multi-instance Sampler, and hybrid location inputs, identifying design choices that improve conditioning or fidelity while exposing sampler limitations.
- UniFusion: Fourier frequency bandwidth improves UniFusion performance until a plateau beyond 16, while MLP dimension 3072 balances efficacy and model size.Increasing the dimension from 3072 to 4096 yields no further improvement; multiple tokens for different location types achieve optimal performance.
- ScaleU: ScaleU re-calibrates main and skip-connected UNet features before concatenation, while SE-ScaleU performs comparably but requires additional MLP parameters.ScaleU is therefore selected as the default lightweight design.
- Multi-instance Sampler: The default instance-latent averaging sampler is compared with crop-and-paste sampling, whose overlap handling can blur and diminish pixels.Crop-and-paste also requires bounding boxes or masks because point and scribble inputs do not specify instance size or shape.
- Hybrid location inputs: Multiple location formats provide the best performance and more precise instance localization, including 9.9% APmask and 16.3% PiM gains.Boxes can provide points through their centers, and masks can provide boxes and central points without extra user input.
- UniFusion: Increasing UniFusion’s active timesteps progressively improves adherence to instance conditions.The module is evaluated across increasing timestep proportions, with stronger location adherence at longer activation durations.
A3. Model Training
Model training initializes InstanceDiffusion from a pretrained text-to-image model and trains added instance-conditioning parameters while keeping the original layers frozen.
- A4. Model Training: The model follows GLIGEN’s setup, freezes the pretrained text-to-image model’s layers, and trains learnable instance-conditioning parameters.Training uses batch size 512 for 100K steps with Adam and an exponentially averaged model.
A4. Applications and Qualitative Results
InstanceDiffusion supports iterative editing and varied spatial compositions, preserving existing scene content while adding, replacing, repositioning, or resizing instances.
- Iterative Image Generation: InstanceDiffusion supports multi-round generation while maintaining previously generated objects and the overall scene.Users can place additional objects in specified locations during subsequent rounds.
- Iterative Image Generation: Qualitative examples show adding new instances, replacing one instance with another, repositioning an instance, and adjusting its size.The examples include adding flowers and a donut, replacing a donut with a lighted candle, and moving the candle.
- Hierarchical location conditioning: Hierarchical location conditioning lets part and subpart orientations alter the pose of the whole object.The demonstrated compositions produce right-, left-, and front-facing orientations.
- Flexible location inputs: Point and scribble inputs enable image-generation cases that previous layout-conditioned text-to-image models did not support.Additional examples also demonstrate point and bounding-box inputs alongside standard text-to-image comparisons.