Source-linked AI summary
ReCo: Region-Controlled Text-to-Image Generation
Zhengyuan Yang, Jianfeng Wang, Zhe Gan, Linjie Li, Kevin Lin, Chenfei Wu, Nan Duan, Zicheng Liu, Ce Liu, Michael Zeng, Lijuan Wang
TL;DR
Large-scale text-to-image models offer high image fidelity but limited control over content in specific regions described with free-form text. ReCo adds quantized coordinate tokens to regional descriptions and fine-tunes a pre-trained T2I model, improving region control and image quality while supporting complex scene specifications.
Problem
Text-to-image models have limited controllability for precisely specifying free-form content in specific image regions.
Method
ReCo augments a pre-trained T2I model with four quantized position tokens per region followed by an open-ended regional description.
Results
ReCo improves region control and image quality over positional-word baselines, with COCO SceneFID of 6.51 and FID improving from 10.44 to 7.36.
Takeaways & Limitations
ReCo provides a flexible interface for controlling object count, spatial relationships, and regional attributes through free-form regional text.
Takeaways & Limitations
ReCo can produce lower-quality images for overly challenging inputs, its COCO-trained style is biased toward COCO, and it inherits possible biases from large-scale pre-trained T2I models.
Abstract
from arXiv · showhide
Recently, large-scale text-to-image (T2I) models have shown impressive performance in generating high-fidelity images, but with limited controllability, e.g., precisely specifying the content in a specific region with a free-form text description. In this paper, we propose an effective technique for such regional control in T2I generation. We augment T2I models' inputs with an extra set of position tokens, which represent the quantized spatial coordinates. Each region is specified by four position tokens to represent the top-left and bottom-right corners, followed by an open-ended natural language regional description. Then, we fine-tune a pre-trained T2I model with such new input interface. Our model, dubbed as ReCo (Region-Controlled T2I), enables the region control for arbitrary objects described by open-ended regional texts rather than by object labels from a constrained category set. Empirically, ReCo achieves better image quality than the T2I model strengthened by positional words (FID: 8.82->7.36, SceneFID: 15.54->6.51 on COCO), together with objects being more accurately placed, amounting to a 20.40% region classification accuracy improvement on COCO. Furthermore, we demonstrate that ReCo can better control the object count, spatial relationship, and region attributes such as color/size, with the free-form regional description. Human evaluation on PaintSkill shows that ReCo is +19.28% and +17.21% more accurate in generating images with correct object count and spatial relationship than the T2I model.
1. Introduction
ReCo addresses the limited regional controllability of text-to-image models by combining spatial position tokens with open-ended regional descriptions. It improves regional alignment and image quality while supporting detailed scene controls.
- Motivation: Text-only T2I queries provide limited control over content in specific regions, especially for long, complicated, or unusual scenes.Positional words can be ambiguous, causing users to rely on time-consuming prompt engineering.
- Motivation: Layout-to-image models use bounding boxes and constrained object labels, but struggle with free-form regional text and combined spatial-text conditions.ReCo instead combines text and box referring in a unified interface.
- Method: ReCo extends pre-trained T2I models with position tokens that specify spatial coordinates alongside open-ended regional descriptions.Coordinates are quantized into bins and represented by four tokens for each region.
- Capabilities: ReCo supports control of object count, spatial relationships, view, type, and regional attributes such as color and size.These controls are especially useful for long, detailed, or uncommon scene descriptions.
- Results: ReCo improves region control and image quality over T2I models using positional words, including object classification accuracy from 42.02% to 62.42% on COCO.Its detector averaged precision also rises from 2.3 to 32.0.
2. Related Work
Prior work covers text-to-image and layout-to-image generation, but ReCo focuses on extending pre-trained T2I models to combine open-ended descriptions with precise localization. Its target is improved controllability for complex scenes without training a separate model from scratch.
- Text-to-image generation: Large-scale T2I models generate high-fidelity images but remain difficult to control precisely through text alone.ReCo builds on these pre-trained models to introduce position-token conditioning.
- Layout-to-image generation: Layout-to-image models condition generation on complete layouts of bounding boxes and paired object labels.Their major limitation is dependence on constrained object vocabularies rather than open-ended regional text.
- Unified conditioning: Prior approaches unify text and localization conditions by separately encoding image descriptions and layout information or using localized traces.These approaches motivate combining complementary referring mechanisms in one interface.
- Complex scenes: ReCo extends pre-trained T2I models instead of training another conditional generation model parallel to T2I and layout-to-image.The design targets flexible input and better handling of unusual scenes and complex queries.
3. ReCo Model
ReCo augments Stable Diffusion with a unified vocabulary of text and spatial position tokens, enabling open-ended regional descriptions at specified image locations. The model is fine-tuned to preserve the pre-trained T2I capability while improving regional control and handling of detailed queries.
- Model overview: ReCo combines text words and position tokens in one input vocabulary to provide both free-form description and precise spatial control.It is implemented by extending Stable Diffusion.
- Preliminaries: Stable Diffusion uses an auto-encoder, a U-Net for noise estimation, and a CLIP ViT-L/14 text encoder.The latent diffusion process operates on encoded image representations and conditions denoising on text embeddings.
- Fine-tuning: ReCo introduces a position-token embedding matrix and fine-tunes the diffusion model and text encoder with the latent diffusion objective.The design minimizes newly introduced parameters while adapting the pre-trained model to spatial inputs.
- ReCo input sequence: Each region is represented by four discrete position tokens for its top-left and bottom-right coordinates, followed by an open-ended regional description.The image description comes first, and users can specify any number of regions.
- Behavior: Position tokens help ReCo place objects in arbitrary regions and reduce neglected or incorrect details in long, attribute-rich input sequences.The reported benefit is strongest for complicated queries and uncommon scenes.
4. Experiments
ReCo is evaluated across COCO, PaintSkill, and LVIS using automatic and human metrics for region control, image quality, and skill correctness. Results show stronger placement, detailed regional control, challenging-scene generation, and open-vocabulary handling, with limitations under unusually difficult queries and COCO-style bias.
- Experiment Settings: The evaluation covers COCO, PaintSkill, and LVIS, using region-control accuracy, image-quality metrics, and human judgments where automatic metrics are ineffective.COCO uses a 30,000-caption validation subset; PaintSkill and LVIS test generalization without further fine-tuning.
- COCO Results: 32.0 AP and 62.42% object classification accuracy show substantially stronger COCO region control for ReCo than ReCoPosition Word.ReCoPosition Word obtains 2.3 AP and 42.02% classification accuracy.
- COCO Results: FID improves from 10.44 to 7.36, while SceneFID reaches 6.51, indicating gains in whole-image quality and region-level placement fidelity.SceneFID evaluates the quality of generated objects together with their positioning in queried regions.
- Qualitative Results: ReCo more reliably handles counting, complex relationships, unusual camera views, and detailed regional attributes, including reduced attribute leakage in long queries.Qualitative COCO examples include five birds, a top-down cat view, and separately controlled clock and shirt colors.
- PaintSkill Results: 87.38% and 82.08% of PaintSkill images achieve correct object count and spatial relationship, respectively, exceeding ReCoPosition Word by 19.28% and 17.21%.Region-control accuracy on the count and skill subsets reaches 63.40% and 67.30%, respectively.
- Analysis: ReCo supports open-vocabulary regional descriptions on LVIS, whereas the closed-vocabulary ReCoOD Label variant performs better on matched COCO categories but degrades on LVIS.ReCoOD Label also produces worse FID than ReCo by 1.72 on COCO and 5.33 on LVIS.
- Limitations: ReCo may produce lower-quality images for overly challenging queries, and COCO fine-tuning biases its generated image style toward COCO.The authors suggest fine-tuning on a small subset of the original pre-training data as a possible mitigation.
5. Conclusion
ReCo extends pre-trained text-to-image models with position tokens for precise open-ended regional descriptions and improves region control and image generation quality.
- ReCo extends a pre-trained T2I model with position tokens that specify open-ended regional descriptions on arbitrary image regions.The resulting interface supports challenging queries with detailed regional attributes or unusual scenes.
- ReCo improves both region control accuracy and image generation quality in experiments.
A. ReCo with LAION data
ReCo fine-tuned on LAION data preserves Stable Diffusion’s open-vocabulary capabilities while adding region-controlled generation, with strong qualitative and quantitative performance.
- Training setup: ReCoLAION applies the same ReCo fine-tuning procedure to a small LAION subset used by the pre-trained Stable Diffusion model.
- Qualitative results: ReCoCOCO and ReCoLAION both show strong region-controlled generation capabilities, while ReCoLAION achieves better image aesthetic scores from high-aesthetic LAION fine-tuning data.
- Qualitative results: ReCoLAION preserves Stable Diffusion’s understanding of celebrities, art styles, and open-vocabulary descriptions while adding region-controlled T2I generation.
- Quantitative results: ReCoCOCO achieves better region control accuracy and image FID than ReCoLAION because COCO boxes and images are used during its training and evaluation.
- Quantitative results: ReCoLAION and ReCoCOCO both outperform the original Stable Diffusion model in region control accuracy and image generation quality.
B. Position Token Cross-Attention
ReCo’s position tokens help localize specified regions through cross-attention, progressively focusing on box corners or edges and helping text tokens focus on corresponding areas.
- Four position tokens per region gradually localize the specified area by attending to the box region’s corner or edge positions.The analysis averages attention maps across all diffusion steps and U-Net blocks.
- Position tokens help text tokens localize and focus on the specified region.