Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Models

Introduction

• Current benchmarks often use short, information-sparse prompts and coarse classification or yes-or-no evaluation, leaving higher-order spatial reasoning and interaction insufficiently tested.The paper frames long, information-dense prompts and fine-grained evaluation as necessary for a more comprehensive assessment of spatial intelligence.Source: §1 · pp. 1–2
• SpatialGenEval contains 1,230 prompts across 25 real-world scenes, with each prompt covering 10 spatial sub-domains and paired with 10 multiple-choice question-answer pairs.Its coverage spans four domains—spatial foundation, perception, reasoning, and interaction—and abilities from object position and layout to occlusion and causality.Source: §1 · p. 2
• SpatialT2I comprises 15,400 text-image pairs whose prompts are refined for better image alignment while preserving information density.The pairs are generated from 1,100 retained prompts using 14 open-source T2I models, then evaluated with Qwen2.5-VL-72B before Gemini 2.5 Pro refines the prompts.Source: §1 · p. 2
• Across 23 state-of-the-art models, spatial reasoning remains a universal bottleneck: accuracy falls on higher-order tasks such as relative positioning, occlusion, and causality despite strong basic object composition.The authors identify higher-order spatial understanding as a primary barrier to current T2I capabilities.Source: §1 · p. 2
• +4.2% on Stable Diffusion-XL, +5.7% on UniWorld-V1, and +4.4% on OmniGen2 after fine-tuning with SpatialT2I.The paper presents these consistent gains as support for a data-centric path to improving spatial intelligence.Source: §1 · pp. 2–3

2.1 KEY PRINCIPLES •

• The benchmark uses dense prompts with interdependent spatial constraints and 10 multiple-choice questions per prompt, while preventing answer leakage and allowing evaluators to choose “E: None.”Evaluators are not given the text-to-image prompt, and humans check questions for leakage; the None option handles cases where no choice matches the generated image.Source: §2.1 · p. 3

2.2 FOCUSED ASPECTS

• SpatialGenEval organizes spatial intelligence into four levels: foundation, perception, reasoning, and interaction, covering object correctness, spatial arrangements, higher-order relations, and dynamic events.Foundation tests object completeness and attribute binding; perception covers position, orientation, and group layout.Source: §2.2 · p. 4
• Spatial reasoning tests comparative properties, fine-grained proximity, and occlusion, probing quantitative relations, object boundaries, and 3D-aware depth and layering.Examples include relative size or length, objects touching or far apart, and one object partially or fully obscuring another.Source: §2.2 · pp. 4–5
• Spatial interaction assesses motion and physical causality, requiring images to depict specific moments in action and visually consistent cause-effect relationships.Examples include a dog jumping over a log and a rock hitting water and causing ripples.Source: §2.2 · p. 5

2.3.1 INFORMATION-DENSE & SPATIAL-AWARE PROMPT GENERATION

• Human-reviewed Gemini 2.5 Pro prompts integrate all 10 spatial constraints into approximately 60 words, yielding 1,230 prompts across 25 real-world scenes.Reviewers improve fluency, correct logical impossibilities, and replace ambiguous or unusual words with common alternatives.Source: §2.3.1 · pp. 5–6

2.3.2 OMNI-DIMENSIONAL QAS GENERATION

• For each prompt, Gemini 2.5 Pro generates 10 multiple-choice QA pairs, each targeting one spatial sub-domain, with a prompt-grounded answer and three plausible distractors.Human validation removes questions that reveal their answers, and each question receives an “E: None” option for cases where no choice matches the image.Source: §2.3.2 · p. 6

3.1 SETUP

• The setup tests 23 open- and closed-source models with Qwen2.5-VL-72B as the primary zero-shot multiple-choice evaluator and uses five-round voting, requiring four correct votes.Each image receives 10 sub-domain questions; evaluation takes about 1.8 seconds per image, or about 40 minutes for 1,230 images on 8×H20 GPUs.Source: §3.1 · p. 6

3.2 MAIN RESULTS OF SPATIALGENEVAL BENCHMARK

• 62.7% for Seed Dream 4.0 versus 60.6% for Qwen-Image shows a narrowing open/closed-source gap, but top scores remain around the 60-point threshold.The leaderboard evaluates 23 open- and closed-source generative models.Source: Table 2 · p. 7, §3.2 · p. 7
• Above 70.0% on object and attribute generation contrasts with comparison and occlusion scores often below 30%, near the 20% random baseline.Models can render objects correctly but struggle to organize them according to specific rules, including relative size and physical layering.Source: §3.2 · p. 7
• 60.6% for Qwen-Image, 56.5–58.5% for FLUX.1, and 54.0–54.6% for SD-3 compare with 28.5% for CLIP-only SD-1.5, linking stronger text encoders to better scores.The results suggest that understanding complex, information-dense prompts is important for spatial generation.Source: §3.2 · p. 7
• Relational Reasoning errors often exceed 35%, Visual Perception errors reach 28.5% in complex Nature scenes, and Motion Interaction errors typically stay below 18%.Basic Composition errors are generally low; the analysis identifies relational logic as the principal barrier to advanced spatial intelligence.Source: Figure 5 · p. 8, §3.2 · p. 8
• SpatialGenEval model rankings align closely with four other benchmark meta-rankings, while GPT-4o and Qwen2.5-VL-72B produce similar rankings and scores.These findings support the benchmark’s ranking consistency and indicate that relative results are robust to judge choice.Source: Table 3 · p. 8, §3.2 · p. 8, Table 4 · p. 8
• 60.6% for 20B Qwen-Image versus 54.0% for 8B SD-3.5-L illustrates a scale trend, while 57.0% for 7B Bagel is comparable to 58.5% for 12B FLUX.1-krea.The authors identify model scale and unified architectures that combine understanding and generation as concurrent pathways to improved spatial intelligence.Source: §3.2 · p. 8
• Near 80% balanced accuracy, the three MLLM judges align with human judgments, with Gemini-2.5-Pro performing best and alignment lower on Spatial Reasoning.Five annotators judged 200 Qwen-Image images without access to the original prompts.Source: Table 5 · p. 8, §3.2 · pp. 8–9

Additional SFT data construction

• SpatialT2I is a separate dataset built by generating prompts and QAs, then rewriting prompts to match images while preserving information density and spatial dimensions.It contains 15,400 image-text pairs from 22 scenes; 130 Design-scene prompts were excluded for low image quality.Source: §4 · p. 9

Training details and SFT results

• Fine-tuned UniWorld-V1, OmniGen2, and Stable Diffusion-XL consistently improve spatial abilities on SpatialGenEval; OmniGen2 is selected as the final model.Qualitative comparisons show competitive results and more realistic spatial effects.Source: §4 · pp. 9–10

Ablation study of SpatialT2I and trend on data scaling

• Across SD-XL and OmniGen2, all three 1,100-pair subsets improve performance, higher-scoring subsets contribute more, and performance rises as training data scales from 0% to 100%.The subsets are Unipic-v2 (54.3), Bagel (57.0), and Qwen-Image (60.6), progressively added in the scaling experiment.Source: §4 · p. 10

Text-to-Image models

• Diffusion is the dominant T2I paradigm; transformer backbones and powerful LLM text encoders extend its generation and semantic capabilities (Rombach et al., 2022; Peebles & Xie, 2023; Hu et al., 2024b).The passage describes diffusion models as efficient and globally coherent, and says diverse approaches have enabled high-fidelity generation with basic compositional elements.Source: §5 · p. 10
• Autoregressive models generate images token by token for strong visual-token dependencies and finer compositional control, while unified multimodal architectures combine visual understanding and generation (Wu et al., 2025b; Han et al., 2025; NextStep Team et al., 2025; Deng et al., 2025; Chen et al., 2025b; Xie et al., 2024; Lin et al., 2025).Source: §5 · p. 10

Text-to-Image benchmarks

• T2I benchmarks have shifted from testing object presence and attributes to assessing multi-object relationships and higher-order capabilities, including world knowledge, physical plausibility, reasoning, and risks (Ghosh et al., 2023; Huang et al., 2023; Wei et al., 2025; Hu et al., 2024a; Chang et al., 2025; Chatterjee et al., 2024; Niu et al., 2025; Meng et al., 2024; Chen et al., 2025a; Lee et al., 2023).The passage characterizes this evolution as a shift from object-level fidelity toward scene-level compositional logic.Source: §5 · p. 10

• Evaluations reveal a stark disparity between basic object generation and advanced spatial tasks, identifying spatial reasoning as the primary bottleneck.Source: §6 · p. 10
• SpatialGenEval systematically evaluates T2I spatial intelligence using a hierarchical framework and information-dense prompts in complex real-world scenarios.Source: §6 · p. 10
• The effectiveness of the supervised fine-tuning dataset SpatialT2I validates a data-centric approach as a practical path toward resolving these shortcomings.Source: §6 · p. 10

ETHICS STATEMENT

• MLLM-generated prompts and question-answer pairs are reviewed by human experts for logic, neutrality, and absence of harmful or personally identifiable information.The benchmark uses common real-world scenes to support broad applicability.Source: ETHICS STATEMENT · p. 11
• The research aims to transparently identify current AI limitations and foster more capable, reliable text-to-image generative models.Source: ETHICS STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• The paper documents benchmark construction, experimental setup, and dataset creation, and makes the benchmark, dataset, and evaluation code publicly available for verification and future work.Benchmark construction is covered in Section 2 and Appendices A.2 and A.4; the experimental setup in Section 3; and dataset creation in Section 4 and Appendix A.5.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 BENCHMARK STATISTICS AND ANALYSIS

• SpatialGenEval covers five primary scene groups and 25 sub-scenes, balancing real-world application breadth with a manageable benchmark size.The groups are nature, indoor, outdoor, human, and design.Source: §A.1 · p. 16
• 32.5% of scenes are outdoor, 28.5% nature, 16.3% indoor, 12.2% human, and 10.6% design, spanning distinct spatial challenges.These include dense public layouts, large-scale organic environments, constrained indoor spaces, human interactions, and conceptual composition.Source: §A.1 · p. 16
• The benchmark organizes four domains and ten sub-domains as a progression from object identification through properties, arrangements, and relationships to event dynamics and causes.This structure is intended to make the benchmark a diagnostic tool.Source: §A.1 · pp. 16–17
• The evaluation selects 23 representative T2I models spanning diffusion, autoregressive, and unified architectures, including open-source checkpoints and closed-source API systems.This dual coverage enables comparison across community-driven and commercial systems.Source: §A.1 · p. 17
• Spatial Foundation questions test scene awareness beyond a single object, while Attribute questions probe combinations of properties rather than isolated attributes.Source: §A.1 · p. 17
• Spatial Perception questions require determining position or orientation from another object’s viewpoint, probing relative and absolute arrangements beyond a global perspective.Source: §A.1 · p. 17
• Spatial Reasoning probes quantitative comparison, occlusion, and physical-distance proximity, while Spatial Interaction asks about causal outcomes of multi-object dynamics.Source: §A.1 · p. 17

A.2 HUMAN ANNOTATION INTERFACE

• Five expert annotators complete over 168 person-hours of two-stage refinement after calibrating with a detailed guidebook; Figure 9 shows the annotation interface.Source: §A.2 · p. 18
• Prompt refinement improves clarity, fluency, and logical consistency by simplifying uncommon vocabulary and checking coverage of all 10 spatial sub-domains.Source: §A.2 · p. 18
• QA refinement aligns each question with a specific spatial sub-domain, removes answer leakage, corrects phrasing, and independently validates each pair with Qwen2.5-VL (Bai et al., 2025).For example, annotators replace “in the prompt” with “in the scene” or “in the image”.Source: §A.2 · p. 18

A.3 ADDITIONAL CLOSED-SOURCE EVALUATION

• GPT-4o-250306 rankings are highly consistent with the primary Qwen2.5-VL-72B evaluation, supporting the robustness of the conclusions across evaluators.Table 10 presents the GPT-4o leaderboard.Source: §A.3 · p. 18, Table 10 · p. 19

A.4 META INSTRUCTION OF SPATIALGENEVAL CONSTRUCTION

• SpatialGenEval construction uses human-overseen MLLM stages to generate long, spatially dense prompts, create 10 corresponding QA pairs per prompt, and evaluate images using visually grounded answers.The evaluation instruction prohibits external knowledge.Source: §A.4 · pp. 18–20

A.5 META INSTRUCTION OF SPATIALT2I CONSTRUCTION

• SpatialT2I rewriting compares ground-truth answers with image-based predictions, treats the image as the source of truth for mismatches, and synthesizes a faithful prompt; a separate rewriting instruction integrates all 10 QA facts.The SpatialT2I process uses the generated image as visual ground truth, while the separate prompt-rewriting process builds a clear paragraph from the confirmed scene facts.Source: §A.5 · p. 21, §A.6 · p. 22

A.7 IMPACT OF PROMPT REWRITING TO IMPROVE SPATIAL INTELLIGENCE

• All models benefit from prompt rewriting; SD-3.5-L gains +2.3%, while explicit-relation gains include +4.5% in Comparison for OmniGen2 and +4.7% in Position for SD-3.5-L.The authors identify prompt decomposition and textual ambiguity as important bottlenecks.Source: §A.7 · pp. 22–23, Table 11 · p. 22
• Rewriting offers little benefit for Occlusion and Orientation, which the authors attribute to generators’ 3D and physical reasoning limitations rather than text comprehension.They suggest specialized fine-tuning or unified-based design as approaches beyond prompt engineering.Source: §A.7 · p. 23

A.8 THE USE OF LLMS

• Gemini 2.5 Pro handles SpatialGenEval data generation and prompt rewriting; Qwen2.5-VL-72B is the primary evaluator, while GPT-4o is the secondary closed-source evaluator and rewrites SpatialT2I prompts.The T2I model carries the main spatial-intelligence burden, while MLLMs perform simple visual checks from direct visual evidence. The study reports exceptionally high correlation between MLLM and human evaluations.Source: §A.8 · p. 23, §A.9.3 · p. 24

A.9.2 DISCUSSION ABOUT THE POTENTIAL BIAS FROM MLLM PREDICTIONS

• Without image input, overall accuracy falls to 16.9%, below 19.8% random-guess accuracy, while Spatial Interaction is 28.1% versus 20.0%, suggesting bias has minimal impact.Source: §A.9.2 · p. 24

A.10 LIMITATIONS

• The authors identify labor-intensive human-in-the-loop scaling and the limited scope of 10 spatial sub-domains, and propose extensions to other generative tasks, video, object-level settings, and post-training strategies.Future strategies include curriculum learning and reinforcement learning from MLLM feedback; further exploration could address fluid dynamics, complex deformations, and multi-agent predictive interactions.Source: §A.10 · p. 25, §A.11 · p. 25

A.12 BROADER IMPACT

• Improved spatial intelligence could give artists, architects, and designers more precise control over object placement and scene layout, and advance robotics and embodied AI (Duan et al., 2022).Robotics and embodied agents must understand and interact with the physical world.Source: §A.12 · p. 25