OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps

1 Introduction

• Overlapping boxes with semantically similar categories can produce object blending, spatial ambiguity, and visual distortion in generated images.The paper identifies these as failure modes of existing layout-to-image frameworks.Source: §1 · p. 2
• OverLayScore quantifies layout difficulty by summing pairwise box IoUs weighted by the semantic similarity of instance annotations.The paper reports that generation quality degrades as OverLayScore rises, and calls high-score layouts complex.Source: §1 · p. 2
• OverLayBench targets complex layouts and instance relationships with rich captions, improved box grounding, quality relationship annotations, and a balanced difficulty distribution.The paper reports that existing benchmarks are biased toward simpler layouts, limiting evaluation in challenging scenarios.Source: §1 · p. 2
• CreatiLayout-AM fine-tunes CreatiLayout with amodal-mask supervision, which the authors report helps mitigate occlusion artifacts in complex layouts.The proposed baseline aligns diffusion-model attention maps with amodal masks.Source: §1 · pp. 2–3

2.1 Layout-to-Image Generation

• Trained L2I methods inject layout conditioning into text-to-image models; newer DiT-based approaches improve image fidelity, while training-free methods use diffusion guidance but often treat layout as a soft constraint.U-Net-based methods commonly add layers or manipulate attention maps and masks; the paper notes their quality can be constrained by backbone capacity.Source: §2.1 · p. 3
• Overlapping boxes, especially those assigned semantically similar categories, remain a challenge for generating coherent, distinguishable objects.Reported failure modes include confusion, ambiguity, visual distortion, and artifacts.Source: §2.1 · p. 3

2.2 Layout-to-Image Benchmarks

• COCO and its variants lack detailed image and instance-level captions, while HiCo-7k and LayoutSAM add annotations but rely on GroundingDINO box extraction.The paper notes HiCo-7k includes human curation, whereas LayoutSAM is generated without human intervention.Source: §2.2 · p. 3
• OverLayBench uses Qwen2.5-VL-32B and human curation, and includes more complex layouts for a more balanced difficulty distribution.The authors report that Qwen2.5-VL-32B surpasses GroundingDINO in benchmark performance and their qualitative evaluation.Source: §2.2 · pp. 3–4

3 OverLayScore

• As bounding-box IoU rises, generation quality deteriorates; at equal IoU, higher semantic similarity further degrades output quality.These trends are reported for a simplified two-object setting.Source: §3 · p. 4
• OverLayScore sums pairwise bounding-box IoUs weighted by the CLIP-based cosine similarity of the corresponding instance captions.The weighted sum captures spatial-semantic entanglement among overlapping object pairs.Source: §3 · p. 4
• Across 100 sampled layouts at each of three difficulty levels, GLIGEN, InstanceDiffusion, and CreatiLayout performance consistently declines as OverLayScore increases.The evaluation uses COCO layouts with 2–10 objects.Source: §3 · p. 5
• Most COCO, HiCo, and LayoutSAM samples fall in the low-difficulty regime, limiting evaluation of complex overlapping scenarios.The analysis identifies a strong imbalance in these existing benchmarks.Source: §3 · p. 5

4 OverLayBench

• OverLayBench selects images across a broad range of OverLayScore values to evaluate complex, spatially and semantically overlapping layouts.It addresses existing benchmarks’ skew toward low-OverLayScore samples.Source: §4 · p. 5

4.1 Dataset Curation

• The curation pipeline generates reference images, uses a vision-language model to extract image and instance descriptions and relationships, then applies human filtering and difficulty balancing.These are the pipeline’s three stages.Source: §4.1 · p. 5

4.1.1 Reference Image Generation

• About 86,000 candidate images are generated with Flux from captions extracted from real-world COCO images.The captions are extracted from the COCO training set, and the generated images are paired with their captions.Source: §4.1.1 · pp. 5–6

4.1.2 Image Grounding and Captioning

• VLM processing refines global captions, grounds instances, filters for overlapping pairs, and extracts pairwise spatial-semantic relationships.Retained images contain 1–10 valid overlapping pairs; each pair has IoU above 5% and intersection area above 1% of the image.Source: §4.1.2 · p. 6

4.1.3 Scoring and Curation

• Manual verification checks boxes, global and instance captions, and relationship descriptions; curation retains 2,052 simple, 1,000 regular, and 1,000 complex layouts.Invalid cases are discarded during validation.Source: §4.1.3 · p. 6

4.2 Benchmark Metrics

• O-mIoU measures fidelity within overlap regions, while SRR measures whether predicted object-pair relationships match ground truth; evaluation also adopts mIoU, CLIP, SRE, and FID.O-mIoU and SRR are designed for overlapping scenarios.Source: §4.2 · p. 6

• Amodal masks provide complete object shape information under occlusion, and a custom L2I training set is used to test this approach.Source: §5 · p. 6
• CreatiLayout-AM fine-tunes CreatiLayout with two additional losses that align model attention maps with ground-truth amodal masks.Source: §5 · p. 7
• The final objective combines the original denoising loss with weighted token-level and pixel-level alignment losses.Source: §5 · p. 7

6.1 Implementation Details

• Data curation used Flux-1-dev, Qwen models, and RealVisXL; CreatiLayout-AM training ran 3,500 steps on eight RTX A6000 GPUs, batch size 16, at learning rate 10^-5.Source: §6.1 · p. 9

6.2 Quantitative Results

• As difficulty increases, all models' spatial metrics decline, particularly O-mIoU, while DiT-based models show more stable visual quality and stronger semantic alignment.Source: §6.2 · p. 9

6.3 CreatiLayout-AM Comparison

• O-mIoU rose +15.90% on Simple and +5.42% on Regular for CreatiLayout-AM versus CreatiLayout; Complex performance stayed competitive with slight mIoU and CLIP declines.Source: §6.3 · p. 9

6.4 Qualitative Analysis

• Figure 6 presents state-of-the-art model examples across overlap complexity, with row icons identifying the top three performers.Source: §6.4 · p. 9

7 Conclusion and Future Works

• CreatiLayout-AM outperforms existing methods under OverLayScore, while amodal-mask supervision mitigates artifacts in complex layouts.The conclusion describes this as initial evidence that explicit mask-level guidance improves generation quality under high-overlap conditions.Source: §7 · p. 9
• OverLayScore measures layout difficulty using spatial overlap and semantic similarity, which the paper reports correlate with degraded generation quality.OverLayBench is presented as a balanced benchmark spanning layout difficulty, with high-fidelity images and dense captions for assessing instance interactions.Source: §7 · p. 9

A.1.1 Prompts Construction

• The appendix describes Qwen-based prompts for image captioning, grounding distinct individual objects, and extracting relationships only for valid overlapping pairs when explicitly specified.Unspecified relationships are returned as None rather than invented.Source: §A.1.1 · p. 15, Grounding · p. 15, Relationship Extraction · p. 15

A.1.2 Web-UI Construction

• Human auditing uses a custom Web-UI to display image captions, bounding boxes, instance captions, and relationship captions for annotators to verify.The stated purpose is to uphold data quality.Source: §A.1.2 · p. 15

A.2.1 EliGen-AM

• For EliGen-AM, the instance attention map is approximated by averaging attention maps over the tokens in that instance’s local description.EliGen lacks CreatiLayout’s explicit layout tokens, making direct extraction of an entity-specific attention map non-trivial; the approximation replaces the map in the training losses.Source: §A.2.1 · p. 17

A.3 Inference and Evaluation

• Three images are generated per method and layout using fixed random seeds 20251202, 20251203, and 20251204 for comparison.The passage states this is to ensure a fair comparison.Source: §A.3 · p. 17
• Standard mIoU matches ground-truth and predicted boxes with the Hungarian algorithm, while O-mIoU measures mIoU in the cropped intersection region of related instances.The authors argue O-mIoU better captures rendering fidelity and realism in densely overlapping scenarios.Source: §A.3 · p. 17

Object Detection

• Qwen-2.5-VL-32B performs object detection for mIoU and question answering for instance-level SRE and relationship-level SRR.The appendix supplies prompts for these evaluation tasks.Source: §A.3 · p. 17

A.4 Baseline Clarification

• The evaluation uses HiCo-SD1.5 because its SDXL version was unavailable, FLUX versions of 3DIS and DreamRender, and DepthAnything v2 maps for DreamRender.DreamRender receives the extracted depth maps together with the layout.Source: §A.4 · p. 18

B Quantitative Results Analysis on Training-Free Methods

• On Complex, RegionalPrompting leads mIoU (28.35%), O-mIoU (9.05%), SRE (53.56%), and CLIPLocal (25.29%), with FID (49.41) described as competitive.The comparison covers training-free methods on OverLayBench; Table 4 identifies best and second-best methods and distinguishes U-Net- and DiT-based methods.Source: B Quantitative Results Analysis on… · p. 18, Table 4 · p. 19
• On Regular, RegionalPrompting leads mIoU (32.72%), O-mIoU (12.29%), SRE (63.74%), and CLIPLocal (25.82), and has the lowest FID (43.13), despite SRR (67.08%).The passage states that SRR is a slight drop, without specifying a comparison value.Source: B Quantitative Results Analysis on… · p. 18

C Error Pattern Analysis of Existing Methods

• Five failure classes are incorrect object number, object fusion, object distortion, incorrect category, and bounding-box misalignment.Object fusion is described as overlapping instances becoming one merged or entangled object; misalignment includes overflow or failure to fill the designated box.Source: C Error Pattern Analysis of Existing… · p. 19

D Comparison Between GroundingDINO and Qwen

• In the Figure 10 example, GroundingDINO v1.0 Liu et al. [2023] produces false positives, including a shark classified as a butterfly, while Qwen Bai et al. [2023] detects more accurately.The passage characterizes Qwen’s detection as more accurate and robust in this comparison.Source: D Comparison Between GroundingDINO and… · p. 19

E User Study

• 55.2%, 51.9%, and 46.8% were CreatiLayout-AM’s winning rates on Simple, Regular, and Complex settings, respectively, excluding “No Preference” cases.The study compared 60 image pairs; Table 5 presents results across the three difficulty levels.Source: E User Study · p. 19, Table 5 · p. 19
• 15 participants compared CreatiLayout and CreatiLayout-AM over 60 image pairs across three difficulty levels.The settings were Simple, Regular, and Complex.Source: E User Study · p. 19
• The authors characterize the results as showing moderate preference for CreatiLayout-AM in simpler scenarios.Its reported winning rate declined from 55.2% on Simple settings to 46.8% on Complex settings.Source: E User Study · p. 19