PaperlayerView exact revision on arXiv

Source-linked AI summary

OverLay++: Dense-Overlap Layout-to-Image Generation Dataset

Shivansh Aggarwal, Shresth Grover, Divyansh Srivastava, Haiyang Xu, Bingnan Li, Xiang Zhang, Ethan J. Armand, Chuan Li, Jianwen Xie, Zhuowen Tu

arXiv:2610.09071v1cs.CV

TL;DR

Layout-to-Image methods still struggle with complex overlapping scenes, and existing training datasets have limited dense interactions and detailed local captions. OverLay++ is a large-scale dataset built around dense, overlap-aware layouts and rich captions. Methods trained on it improve consistently across benchmarks and converge faster than corresponding baselines.

  • Problem

    Existing Layout-to-Image datasets have few dense, overlapping interactions and often lack detailed local captions, while methods struggle with complex scenes.

    Source: §1 · p. 2

  • Method

    OverLay++ constructs a large-scale dataset with dense, overlapping object annotations and rich scene-level and object-level captions.

    Source: §1 · p. 2, §3.1 · p. 5

  • Results

    Across multiple benchmarks, Layout-to-Image methods trained on OverLay++ show consistent improvements and faster convergence than corresponding baselines trained on existing datasets.

    Source: §5 · p. 9

  • Takeaways & Limitations

    The authors conclude that dense, overlap-aware, caption-rich supervision is important for controllable image generation.

    Source: §5 · p. 9

    The dataset may inherit model biases and annotation noise, and may underrepresent unusual object interactions, transparent objects, and rare categories.

    Source: §5 · p. 9

Abstract

from arXiv · show

Layout-to-Image generation has made substantial progress in spatial and object-level control. However, existing methods still struggle with complex scenes containing many overlapping and interacting objects. We argue that training data is a particular bottleneck: existing datasets lack examples with dense, complex object interactions. To address this gap, we introduce OverLay++, a large-scale Layout-to-Image dataset with structurally complex scenes. OverLay++ contains approximately 500K images with an average of 6.6 objects per image, exceeding existing datasets by 1.67 times in annotation density. Beyond annotation density, OverLay++ provides rich semantic detail with object captions over six times longer than in current datasets. Our dataset generation pipeline is simple and produces dense, overlapping object annotations with rich per-object captions. Across multiple benchmarks, state-of-the-art Layout-to-Image methods trained on the OverLay++ dataset show consistent improvement and faster convergence, demonstrating the importance of dense, overlap-aware, and caption-rich supervision for controllable image generation.

1 Introduction

Layout-to-Image generation offers spatial control, but methods still struggle with dense overlapping scenes, and existing datasets often lack both such interactions and detailed object captions. OverLay++ targets this gap with a dense, caption-rich dataset and reports improved benchmark performance and faster convergence for trained methods.

1 Introduction

  • Existing methods struggle particularly with overlapping objects, while current datasets contain few dense layouts and local captions often omit object attributes needed to control color, texture, and interactions.Recent data-collection efforts improve object grounding and annotation detail but do not specifically target dense, overlapping scenes [Zhang et al., 2025b] [Zhang et al., 2025a].Source: §1 · p. 2
  • OverLay++ training improves state-of-the-art Layout-to-Image methods across multiple benchmarks and supports faster convergence than corresponding baselines trained on existing datasets.The authors highlight dense, overlap-aware, and caption-rich supervision as important to controllable image generation.Source: §1 · pp. 2–3
  • The dataset pipeline prompts MLLMs to propose overlapping objects and applies per-object grounded validation to improve annotation accuracy and localization.The resulting images are described as having dense layouts, substantial overlap, and rich captions.Source: §1 · pp. 2–3
  • OverLay++ contains approximately 500K images averaging 6.6 objects each, with short and long captions at both global and per-object levels.Its short captions are reported as 6× longer than those in existing datasets.Source: §1 · pp. 2–3

2 Related Work

Layout-to-Image research has moved from inference-time controls and U-Net-based conditioning toward training-based methods using more capable DiT backbones, yet dense overlapping scenes remain challenging. Existing datasets also provide limited structural complexity and object-level detail; OverLay++ addresses these gaps with overlapping proposals and grounded validation.

2.1 Layout-to-Image Generation

  • Training-free methods modify inference, such as attention maps or classifier-free guidance, while training-based methods incorporate layouts into the model; the latter generally outperform the former in complex scenes [Li et al., 2025].Training-free approaches retain off-the-shelf models [Lee et al., 2024, Chen et al., 2025, Bar-Tal et al., 2023, Kim et al., 2023, Chefer et al., 2023, Xie et al., 2023, Gani et al., 2023, Yang et al., 2024, Chen et al., 2024c].Source: §2.1 · p. 3
  • Early training-based methods added layout adapters, masks, gated attention, or ControlNet-like modules to U-Nets, while recent methods use multimodal DiTs such as SD3 and FLUX [Li et al., 2023a, Wang et al., 2024a, Zhang et al., 2023] [Esser et al., 2024, Black Forest Labs, 2024].Despite increased modeling capacity and image realism, recent methods still struggle with numerous overlapping objects.Source: §2.1 · p. 3

2.2 Training Datasets for Layout-to-Image Generation

  • Object-detection and grounding datasets often provide class labels rather than descriptive local captions, limiting fine-grained object descriptions for controllable generation [Shao et al., 2019, Lin et al., 2014].Later datasets improve captioning and regional descriptions, but retain limited structural complexity in dense scenes [Zhang et al., 2025a] [Zhang et al., 2025b].Source: §2.2 · pp. 3–4
  • InstanceDiffusion, SiamLayout, and EliGen use object extraction, localization, and captioning pipelines, while OverLay++ explicitly prompts overlapping object proposals and validates detections per object [Wang et al., 2024a] [Zhang et al., 2025a] [Zhang et al., 2025b].Figure 2 depicts image filtering, object detection, scene filtering, and hierarchical scene- and object-level captioning.Source: §2.2 · p. 4, Figure 2 · p. 4

3 Dataset Curation Pipeline

OverLay++ is designed to address sparse layouts, weak local spatial interactions, and limited object-level language supervision. Its curation pipeline filters images, detects and validates objects, and supplies hierarchical captions in long and short forms.

3.1 Motivation and Design Goals

  • Prior datasets typically contain 1.78–3.93 objects per image, with low nearest-neighbor IoU and relatively large normalized nearest-neighbor distance, while often lacking systematic object-level appearance descriptions.The design goal is denser layouts, stronger local interactions, and richer multi-level language supervision; Table 1 compares dataset statistics, with caption lengths measured in words.Source: §3.1 · pp. 4–5, Table 1 · p. 5
  • The pipeline starts from the LAION-Aesthetics V2 4.75 subset of 956M images and uses image filtering, object detection, scene filtering, and captioning.These stages are designed to improve object density, spatial interaction, and annotation granularity.Source: §3.1 · p. 5

3.2 Image Filtering

  • Images are retained when their aesthetic score is at least 5.98 and both dimensions are at least 1024 pixels, then deterministically center-cropped to 1024 × 1024.This filtering step yields 850,900 images.Source: §3.2 · p. 5

3.3 Object Detection

  • Qwen3-VL-32B detects salient objects and returns category labels, normalized bounding boxes, and detailed visual-attribute captions; images with fewer than five valid detections are discarded.The detection prompt uses bounding boxes in [x1, y1, x2, y2] format normalized to a 1024-pixel coordinate space.Source: §3.3 · p. 5

3.4 Scene Filtering

  • Each candidate object is rechecked by cropping its image region and asking Qwen3-VL-32B whether the described object is present; negative detections are removed and images with fewer than five remaining objects are discarded.499,249 images remain after this validation and filtering step.Source: §3.4 · p. 5

3.5 Captioning

  • Captioning provides holistic scene descriptions and object descriptions tied to bounding boxes, with long and short versions at both levels.Short captions are prompted to use fewer than 20 words; long object captions describe appearance, style, and visual characteristics, while short object captions capture identity and salient attributes.Source: §3.5 · p. 6

4 Evaluation

The evaluation compares OverLay++ training with baseline datasets across LayoutSAM-Eval, OverLayBench, and DenseLayout, using two L2I methods. OverLay++ training improves reported benchmark results, and the caption-granularity ablation favors short local captions with long global captions.

4.1 Implementation Details

  • The evaluation trains EliGen-SD3 from pretrained SD3 on each dataset and fine-tunes SiamLayout-SD3 on OverLay++, then compares results across three benchmarks.EliGen-SD3 runs use 10K steps; SiamLayout-SD3 is fine-tuned for 60K steps because its training code is unavailable.Source: §4.1 · p. 6

4.2 LayoutSAM Benchmark

  • On LayoutSAM-Eval, EliGen-SD3 (Ours) achieves state-of-the-art region-wise quality with gains up to 8.5%, while SiamLayout-SD3 (Ours) has the best FID and second-best IS.The qualitative comparison reports more coherent, detailed OverLay++ images and identifies missing objects, missing attributes, and poor dense-layout handling as common prior-dataset failures.Source: §4.2 · p. 6, Table 2 · p. 7, Figure 3 · p. 7

4.3 OverLayBench Benchmark

  • Across OverLayBench tiers, EliGen-SD3 (Ours) improves spatial alignment over the original EliGen-SD3 and leads on several metrics.It achieves best SRE (91.83%) and SRR (91.06%) on Simple; best mIoU (62.40), O-mIoU (36.66), and SRE (89.45%) on Regular; and best mIoU (57.90), O-mIoU (30.99), and SRE (83.30%) on Complex.Source: Table 3 · p. 8, §4.3 · p. 8

4.4 DenseLayout Benchmark

  • On DenseLayout, EliGen-SD3 (Ours) raises mIoU from 45.79 to 46.92 and improves Color by 7.86%, Texture by 7.33%, and Shape by 7.7%.SiamLayout-SD3 (Ours) also improves over its baseline on all evaluation metrics.Source: §4.4 · pp. 8–9, Table 4 · p. 9

4.5 Caption Granularity

  • The Short-Long caption setting performs best overall, with short local captions supporting object grounding and long global captions providing scene-level context.The authors note that CLIP's 77-token local-prompt limit may truncate longer object descriptions; stronger local encoders or longer context windows may let long local captions help.Source: §4.5 · p. 9

5 Conclusion, Limitations, and Broader Impact

OverLay++ provides dense, overlap-aware, caption-rich supervision, and trained Layout-to-Image methods show consistent improvement and faster convergence across multiple benchmarks. The dataset may retain curation-related biases and annotation noise, underrepresent difficult object cases, and carry open-source-model safety risks.

5 Conclusion, Limitations, and Broader Impact

  • Across multiple benchmarks, Layout-to-Image methods trained on OverLay++ show consistent improvements and faster convergence.The conclusion highlights dense, overlap-aware supervision as important for controllable image generation.Source: §5 · p. 9
  • OverLay++ contains approximately 500K images with 6.6 objects per image and 1.67× existing datasets’ annotation density.It provides short and long captions at global and per-object levels.Source: §5 · p. 9
  • Because curation uses MLLMs for captioning and grounding, OverLay++ may inherit model biases and annotation noise.It may also underrepresent unusual interactions, transparent objects, and rare categories.Source: §5 · p. 9
  • Short local captions with long global captions achieve the best region-level alignment in the OverLayBench granularity ablation.The caption identifies Table 5 as an ablation study on caption granularity.Source: Table 5 · p. 10
  • The dataset may inherit safety risks present in open-source generative models, and further misuse research is needed for safe deployment.The conclusion frames this as a need for further research, not a measured outcome.Source: §5 · p. 10

A.1 Prompts

The curation prompts use Qwen3-VL-32B to solicit dense overlapping object detections, check candidate objects against image crops, and generate captions at global and local levels and at short and long granularities.

A.1 Prompts

  • The pipeline prompts Qwen3-VL-32B for detection, scene filtering, and hierarchical caption generation.Its prompts are designed to encourage dense, overlap-aware annotations while reducing hallucinations through per-object validation.Source: §A.1 · p. 16
  • The detection prompt requests object categories, bounding boxes, and detailed descriptions of each object’s visual attributes.It requires overlapping boxes, minimum box dimensions of 51 pixels, and coordinates in a 1024 × 1024 image space.Source: §A.1 · p. 16
  • For validation, the prompt asks whether a described object is present in a cropped image region, with a binary yes-or-no response.The object description is supplied in the form “Object: {category}. {local_prompt}.”Source: §A.1 · p. 17
  • The scene-caption prompts request an overall image description or short global caption, with short captions constrained to fewer than 20 words.The short-caption instructions request key objects and setting and return caption text without additional explanation.Source: §A.1 · p. 17
  • The local-caption prompt returns a short description for each detection, preserving detection order and including color and other visual attributes.Each short local prompt must be fewer than 20 words and is returned in a detections JSON structure.Source: §A.1 · p. 17

A.2 Long Local Captions with a Long-Context Text Encoder

With long local captions and EliGen-FLUX’s longer-context T5 path, results improve over the baseline across nearly all reported metrics. The authors suggest this is consistent with a longer-context path making more caption detail usable.

A.2 Long Local Captions with a Long-Context Text Encoder

  • Long local captions average 89.70 words, while the CLIP path has a 77-token limit, making truncation a plausible explanation for degraded long-prompt performance.The authors explicitly describe this as a hypothesis, not a direct measurement of truncation’s effect.Source: §A.2 · p. 17
  • SRE improves by up to 11.54% on the Complex split, and the long-caption model improves over EliGen-FLUX on nearly every OverLayBench metric.It also improves FID; the sole reported exception is a marginal CLIPG drop on the Regular split.Source: §A.2 · p. 17, Table 6 · p. 18
  • On LayoutSAM-Eval, the model improves over EliGen-FLUX on all region-wise metrics and on FID and IS.The reported region-wise metrics are Spatial, Color, Texture, and Shape.Source: §A.2 · p. 17, Table 7 · p. 18
  • The authors fine-tune released EliGen-FLUX with long local captions for 5K steps using batch size 8; its local prompts use a longer-context T5 path.They interpret the results as consistent with that path being able to use more caption detail.Source: §A.2 · p. 17

A.3 Global Alignment with Long-CLIP

The global prompts exceed CLIP’s 77-token input limit, so the authors reevaluate global text–image alignment with Long-CLIP. OverLay++-trained EliGen-SD3 scores slightly higher than its baseline on all three splits.

A.3 Global Alignment with Long-CLIP

  • The OverLay++-trained EliGen-SD3 model has slightly higher Long-CLIPG scores than its baseline on all three splits.This comparison changes the evaluation encoder, not the generation model’s local-prompt encoder.Source: §A.3 · p. 18
  • Global prompts average 79.66, 86.81, and 82.91 words on the Simple, Regular, and Complex splits, respectively, exceeding CLIP’s 77-token input limit.The authors use Long-CLIP, which supports up to 248 tokens, to reevaluate global alignment.Source: §A.3 · p. 18
  • The Long-CLIP comparison evaluates global text–image alignment rather than testing a change to the generation model’s local-prompt encoder.Source: §A.3 · p. 18

A.4 Source of the Improvements

OverLay++ training improves local instruction-following metrics over EliGen in the reported ablation, including with shorter local captions. Longer captions improve some local metrics but show mixed results elsewhere, while the main training runs do not isolate which dataset properties account for the gains.

A.4 Source of the Improvements

  • Removing 50K high-nearest-neighbor-IoU images lowers mean IoU from 0.31 to 0.27, near EliGen’s 0.26, but differing dataset composition and object counts prevent clean isolation.The authors note that the comparisons do not isolate either factor perfectly.Source: §A.4 · p. 18, Table 9 · p. 19
  • Despite shorter local captions, OverLay++ training beats EliGen on all five local instruction-following metrics across Simple, Regular, and Complex splits.The comparison uses 6.68 versus 2.56 objects per layout, and the dataset settings also differ in other properties.Source: §A.4 · p. 18, Table 9 · p. 19
  • Longer local captions (15.89 vs. 1.88 words; 6.59 vs. 6.68 objects/layout) improve mIoU, O-mIoU, and SRE across splits, while SRR gains vary and CLIPL falls throughout.Both settings use 2K training steps.Source: §A.4 · p. 18, Table 9 · p. 19
  • In the 2K-step longer-caption setting, CLIPG decreases and FID worsens; on Regular, FID rises from 50.99 to 59.65, with the cause of global-metric changes unestablished.Longer runs and a FLUX ablation report better global metrics but do not isolate which change is responsible.Source: §A.4 · p. 19
  • Main 5K- and 10K-step runs improve region-wise metrics and FID over EliGen at both checkpoints, while IS is comparable; these runs do not isolate object count, overlap, or caption detail.These are separate from the 2K-step overlap-matched ablation.Source: §A.4 · p. 19

A.5 Human Evaluation of Local Layout and Instruction Following

The appendix reports a human yes/no evaluation of local layout and instruction following, alongside an account of OverLay++’s open-domain object categories and semantic composition. Human ratings favor OverLay++ over the baseline, and the dataset spans many semantic groups with a long tail of less frequent categories.

A.5 Human Evaluation of Local Layout and Instruction Following

  • The human evaluation compares EliGen models trained on the baseline dataset and OverLay++.The evaluation targets local layout and instruction-following quality.Source: §A.5 · p. 19
  • The study samples 60 layouts across three splits, covering 780 boxes rated by 12 raters at an average of 2.37 ratings per box.Raters answer yes or no on whether each generated region satisfies its box and local prompt.Source: §A.5 · p. 19
  • OverLay++ receives more “yes” ratings than the baseline overall (+11.80 percentage points) and in Simple (+9.30), Regular (+13.10), and Complex (+11.80).These human judgments complement the automatic benchmarks in the main text.Source: §A.5 · pp. 19–20, Table 11 · p. 20

A.6 Dataset Composition

  • OverLay++ uses free-form object categories generated during detection rather than choosing labels from a predefined class list.The passage contrasts this with existing datasets adapted from object detection or grounding datasets.Source: §A.6 · p. 20
  • The reported composition ranges from furniture, clothing, and natural scenery to a long tail including musical instruments, weapons, and vehicles.The categories are based on the zero-shot classification of short local prompts.Source: §A.6 · p. 20
  • The authors zero-shot classify short local prompts into 26 broad semantic groups and report each group’s box count and share of all boxes.Table 12 presents the composition by semantic group.Source: Table 12 · p. 21
Loading 2610.09071v1…