Source-linked AI summary
BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusion
Jinheng Xie, Yuexiang Li, Yawen Huang, Haozhe Liu, Wentian Zhang, Yefeng Zheng, Mike Zheng Shou
TL;DR
Existing spatially controlled image synthesis often requires costly paired layout-image data or remains limited to closed-set categories. BoxDiff uses box or scribble conditions to impose three spatial constraints during diffusion denoising, and experiments show photorealistic synthesis that follows those conditions while retaining broad visual concepts.
Problem
Existing spatially conditioned image synthesis requires considerable paired layout-image data and is restricted to closed-set categories, limiting its open-world applicability.
Method
BoxDiff is a training-free method that applies Inner-Box, Outer-Box, and Corner Constraints to cross-attentions during diffusion denoising using box or scribble conditions.
Results
Extensive experiments demonstrate that BoxDiff synthesizes photorealistic images following the given spatial conditions.
Takeaways & Limitations
BoxDiff controls where objects and contexts are synthesized while retaining diffusion models’ ability to represent diverse visual concepts.
Takeaways & Limitations
Spatial constraints are applied only at 16 × 16 cross-attention resolution, which may restrict control precision.
Abstract
from arXiv · showhide
Recent text-to-image diffusion models have demonstrated an astonishing capacity to generate high-quality images. However, researchers mainly studied the way of synthesizing images with only text prompts. While some works have explored using other modalities as conditions, considerable paired data, e.g., box/mask-image pairs, and fine-tuning time are required for nurturing models. As such paired data is time-consuming and labor-intensive to acquire and restricted to a closed set, this potentially becomes the bottleneck for applications in an open world. This paper focuses on the simplest form of user-provided conditions, e.g., box or scribble. To mitigate the aforementioned problem, we propose a training-free method to control objects and contexts in the synthesized images adhering to the given spatial conditions. Specifically, three spatial constraints, i.e., Inner-Box, Outer-Box, and Corner Constraints, are designed and seamlessly integrated into the denoising step of diffusion models, requiring no additional training and massive annotated layout data. Extensive experimental results demonstrate that the proposed constraints can control what and where to present in the images while retaining the ability of Diffusion models to synthesize with high fidelity and diverse concept coverage. The code is publicly available at https://github.com/showlab/BoxDiff.
1. Introduction
BoxDiff addresses the limited spatial control and annotation burden of existing text-to-image and layout-to-image methods with a training-free approach using simple user-provided spatial conditions. It applies three constraints during denoising to control object and context placement while preserving Stable Diffusion’s concept coverage and image fidelity.
- Existing text-to-image models primarily use text or class labels, limiting fine-grained control over object and context location or scale.
- Prior layout-to-image methods require labor-intensive paired box, skeleton, or mask-image data and are restricted to closed-set categories.
- BoxDiff uses simple box or scribble conditions and requires no additional training or massive paired layout-image data.
- Inner-Box, Outer-Box, and Corner Constraints guide cross-attention maps at each denoising timestep to control synthesized object and context locations and scales.
- Extensive experiments show that BoxDiff synthesizes photorealistic images following supplied spatial conditions while retaining diverse visual concepts.
2. Related Work
Related work spans diffusion-based image generation, text-to-image synthesis, and supervised layout-to-image models. BoxDiff is positioned as a way to use spatial constraints while extending beyond closed-set visual concepts and avoiding costly annotation.
- Diffusion Models: Diffusion models generate images by iteratively removing noise from Gaussian noise, with class-conditional and image-guided synthesis also explored.
- Text-to-Image Models: Large-scale Internet image-text pairs have enabled models such as DALL-E, Imagen, and Stable Diffusion to synthesize higher-quality and more diverse images.
- Layout-to-Images Models: Traditional layout-to-image methods synthesize images from bounding boxes but generally rely on training and validation with time-consuming, labor-intensive annotation.
- Layout-to-Images Models: BoxDiff differs by supporting a wide range of visual concepts rather than a limited closed set.
3. Preliminaries: Stable Diffusion
Stable Diffusion encodes images into a latent space and uses a UNet denoiser with attention-based text conditioning. During inference, iterative latent denoising produces a final latent that is decoded into an image.
- Stable Diffusion uses an encoder E and decoder D to map images into latent representations and reconstruct images from those latents.
- A UNet denoiser with self-attention and cross-attention predicts the noise added to a noised latent conditioned on text tokens.
- Inference samples a latent zT from a standard normal distribution, iteratively removes noise to obtain z0, and decodes z0 into an image.
4. Methodology
BoxDiff converts user-provided boxes or scribbles into spatial masks and constrains Stable Diffusion cross-attention during denoising. Inner-Box, Outer-Box, and Corner Constraints guide object location and scale without additional training.
- Cross-Modal Attention: Cross-attention maps link text tokens to intermediate denoiser features, with high-response regions corresponding perceptually to synthesized objects.This correspondence motivates spatial guidance over cross-attention rather than direct image supervision.
- Cross-Modal Attention: Box conditions are transformed into binary spatial masks that supervise target-token cross-attention during denoising.The method uses masks derived from user-provided top-left and bottom-right coordinates.
- Box-Constrained Diffusion: Inner-Box maximizes selected high-response attention inside target masks, while Outer-Box minimizes attention outside them.The constraints are complementary: Inner-Box encourages content within the region, and Outer-Box suppresses content beyond it.
- Box-Constrained Diffusion: Corner Constraint addresses undersized objects by constraining projected attention at the mask corners along both spatial axes.It is designed for weak box or scribble conditions that lack exact object boundaries.
- Box-Constrained Diffusion: The combined loss updates the current latent with a decaying step size, steering attention toward the specified location and box-consistent scale.The overall constraint is L = LIB + LOB + LCC, and the latent is updated using the gradient of L.
- Box-Constrained Diffusion: BoxDiff requires no additional training or massive paired layout-image data and can also operate with scribbles and as a plug-and-play diffusion component.The paper notes compatibility with multiple diffusion models, including GLIGEN.
5. Experiments
The experiments evaluate BoxDiff on a separately collected dataset using object-detection and text-image similarity metrics. Evaluation focuses on conditional synthesis precision and semantic correctness.
- Datasets: The evaluation dataset contains images with no intersection with COCO or VG, with objects detected using YOLOv4.The authors introduce this dataset because direct comparison with fully supervised layout-to-image methods would be unfair.
- Evaluation Metrics: YOLO score metrics AP, AP50, and AP75 measure conditional synthesis precision, while T2I-Sim measures semantic correctness.YOLOv4 detects object boxes and predicts classification scores on synthesized images.
“Aurora, reindeer, meadow, and lake” (the same text prompt as Figure 1)
BoxDiff is evaluated under fixed spatial conditioning and through ablations of its constraints and sampling choices. The results show improved spatial adherence, while sampling balances control against text consistency and image quality.
- Visualization Results: Figure 5 contains multiple samples synthesized with fixed spatial conditioning inputs.The comparison is across samples sharing the same spatial conditioning.
- Ablation Studies: LIB alone achieves 0.3516 T2I-Sim and 9.8 AP, but synthesized objects are mostly inconsistent with the conditional spatial input.Combining LIB and LOB raises the YOLO score to 20.2 AP.
- Ablation Studies: 22.3 AP is achieved when inner-box, outer-box, and corner constraints are combined, compared with 20.2 AP without the corner constraint.The corner constraint helps make synthesized object scales consistent with the bounding boxes.
- Ablation Studies: Removing topk(·) improves AP but decreases T2I-Sim, semantic consistency with the text prompts, and image quality.Applying constraints to all cross-attention elements can therefore impair synthesis quality.
- Ablation Studies: Figure 6 presents ablations of constraint combinations alongside visual comparisons with [27] and [10].The left and right panels address constraint effects and model comparisons, respectively.
- Ablation Studies: Removing sample(·) significantly degrades T2I-Sim and lowers AP by impairing consistency with the conditional input.The method therefore retains both topk(·) and sample(·).
- Ablation Studies: Sampling all pixels improves adherence precision but degrades synthetic-content quality relative to topk(·).Random pixel sampling can activate background regions and significantly degrade both AP and T2I-Sim.
- Visualization Results: Fixed spatial inputs produce animal and accessory locations and scales consistent with their conditional boxes.The same pattern is also observed for mountains, aurora, castles, and hats.
“A castle in the middle of a calm lake”
BoxDiff varies spatial conditioning to control synthesized content and object placement. It also outperforms compared fully supervised layout-to-image methods on the reported spatial and semantic metrics.
- Visualization Results: BoxDiff varies castle location, scale, and surrounding content while preserving the lake in its fixed conditional region.The lake remains at the bottom, while castle placement changes according to the varying box.
- Quantitative Results: BoxDiff significantly outperforms LostGAN, LAMA, and TwFA in YOLO score and achieves the best T2I-Sim.The comparison uses newly collected spatial conditions.
- Quantitative Results: Integrating BoxDiff further improves GLIGEN as a plug-and-play component.The paper reports this as evidence that BoxDiff can improve existing models.
6. Conclusion and Discussion
BoxDiff applies spatial constraints during diffusion denoising without additional training to control where objects and contexts are synthesized. Its control is limited by applying constraints only at the 16 × 16 cross-attention scale.
- Conclusion and Discussion: BoxDiff controls object synthesis spatially by applying constraints during diffusion-model denoising without additional training.The approach controls where objects and contexts are synthesized.
- Conclusion and Discussion: Applying constraints only at the 16 × 16 cross-attention scale may restrict control precision.The paper suggests extension to additional bounding-box conditions for downstream tasks.
7. Appendix
The appendix reports BoxDiff’s limitations, implementation settings, ablations, and additional visualizations, including experiments with scribble conditions and unusual prompts.
- Limitations and Discussion: BoxDiff may produce unrealistic images or violate spatial conditions for infrequent object combinations, uncommon object locations, or unusual scenes.Examples include “car” with “basin,” a giraffe flying in the sky, and a mountain underneath the water.
- Implementation Details: The experiments use Stable Diffusion v1.4 with 50 denoising steps, guidance scale 7.5, and 512 × 512 image resolution.The implementation also uses a 3 × 3 Gaussian kernel with σ = 0.5.
- Evaluation Setup: The evaluation layout contains 4,274 valid bounding boxes across 9 animal and 18 object categories collected for conditional image synthesis.The candidate categories are intended for comparison on newly collected layouts without intersection with COCO and VG.
- Ablation Studies: Selecting the highest-response elements in topk(·) gives the best T2I-Sim and AP among the tested sampling methods.The ablation interprets higher cross-attention response inside a box as indicating greater probability that the object appears there.
- Ablation Studies: Selecting 80% of mask and non-mask elements yields the best T2I-Sim and relatively higher AP, while L = 6 achieves the best T2I-Sim and AP.Increasing L from 6 to 14 keeps AP relatively stable but decreases T2I-Sim, indicating that additional cross-attention constraints can reduce image quality.
- Scribble as Conditions: Scribble conditions provide more pixel information about object content and motivate the additional objectness constraint LOC for controlling content or direction.The appendix presents further synthetic samples using scribble conditions.