PaperlayerView exact revision on arXiv

Source-linked AI summary

Everything in Its Place: Benchmarking Spatial Intelligence of Text-to-Image Models

Zengbin Wang, Xuecai Hu, Yong Wang, Feng Xiong, Man Zhang, Xiangxiang Chu

arXiv:2601.20354v2cs.CV

TL;DR

T2I models generate high-fidelity images but remain weak at complex spatial perception, reasoning, and interaction, which existing sparse-prompt benchmarks undermeasure. The paper introduces SpatialGenEval and SpatialT2I, finding higher-order spatial reasoning to be a bottleneck while fine-tuning yields consistent gains.

  • Problem

    Existing T2I models and benchmarks inadequately address complex spatial relationships because current evaluations often use short or information-sparse prompts.

  • Method

    The paper constructs SpatialGenEval with 1,230 dense prompts across 25 scenes, 10 spatial sub-domains, and corresponding multiple-choice QA pairs, plus the SpatialT2I dataset.

  • Results

    Higher-order spatial reasoning is a universal bottleneck across 23 evaluated models, while SpatialT2I fine-tuning yields +4.2% on Stable Diffusion-XL, +5.7% on UniWorld-V1, and +4.4% on OmniGen2.

  • Takeaways & Limitations

    The results support information-dense spatial evaluation and a data-centric fine-tuning approach for improving spatial abilities in T2I models.

  • Takeaways & Limitations

    The human-in-the-loop construction is labor-intensive for scaling, and the 10-domain framework does not cover all nuanced or dynamic spatial phenomena.

Abstract

from arXiv · show

Text-to-image (T2I) models have achieved remarkable success in generating high-fidelity images, but they often fail in handling complex spatial relationships, e.g., spatial perception, reasoning, or interaction. These critical aspects are largely overlooked by current benchmarks due to their short or information-sparse prompt design. In this paper, we introduce SpatialGenEval, a new benchmark designed to systematically evaluate the spatial intelligence of T2I models, covering two key aspects: (1) SpatialGenEval involves 1,230 long, information-dense prompts across 25 real-world scenes. Each prompt integrates 10 spatial sub-domains and corresponding 10 multi-choice question-answer pairs, ranging from object position and layout to occlusion and causality. Our extensive evaluation of 21 state-of-the-art models reveals that higher-order spatial reasoning remains a primary bottleneck. (2) To demonstrate that the utility of our information-dense design goes beyond simple evaluation, we also construct the SpatialT2I dataset. It contains 15,400 text-image pairs with rewritten prompts to ensure image consistency while preserving information density. Fine-tuned results on current foundation models (i.e., Stable Diffusion-XL, Uniworld-V1, OmniGen2) yield consistent performance gains (+4.2%, +5.7%, +4.4%) and more realistic effects in spatial relations, highlighting a data-centric paradigm to achieve spatial intelligence in T2I models.

1 INTRODUCTION

SpatialGenEval addresses the undermeasurement of complex spatial intelligence in T2I models with long, information-dense prompts and fine-grained evaluations. Across evaluated models, higher-order spatial reasoning remains a bottleneck, while SpatialT2I fine-tuning produces consistent gains.

  • Current T2I models often misplace or misorient objects and fail to depict relative comparisons or causal interactions in complex scenes.
  • Short, information-sparse prompts and coarse metrics in existing benchmarks mainly test object presence, attributes, or simple binary relations.
  • SpatialGenEval contains 1,230 information-dense prompts across 25 real-world scenes, each covering 10 spatial sub-domains and paired with 10 multiple-choice QA pairs.
  • 23 state-of-the-art models show a universal bottleneck in higher-order spatial reasoning, including relative positioning, occlusion, and causality.
  • +4.2% on Stable Diffusion-XL, +5.7% on UniWorld-V1, and +4.4% on OmniGen2 are reported after fine-tuning with SpatialT2I.

2 SPATIALGENEVAL BENCHMARK

SpatialGenEval organizes spatial intelligence into 10 sub-domains and constructs dense prompts and corresponding QA pairs through automated generation plus human validation. Its pipeline spans scene selection, constraint integration, image generation, and fine-grained evaluation.

  • The benchmark uses a hierarchical framework covering spatial foundation, perception, reasoning, and interaction through 10 spatial sub-domains.
  • SpatialGenEval is designed around long prompts containing multiple interdependent spatial constraints rather than short object-attribute or simple-relation prompts.
  • Spatial reasoning includes quantitative comparison, fine-grained proximity, and higher-order relationships beyond direct spatial perception.
  • Spatial interaction evaluates dynamic motion and explicit physical causality, such as jumping, impacts, ripples, and hammering.
  • Gemini 2.5 Pro combines one of 25 curated real-world scenes with all 10 sub-domain definitions to generate a single integrated prompt.
  • Human experts revise generated prompts and QA pairs to correct logical impossibilities, awkward phrasing, ambiguity, and answer leakage, producing validated benchmark materials.

3 EXPERIMENTAL RESULTS

SpatialGenEval evaluates 23 T2I models with multi-domain spatial questions and finds that higher-order spatial reasoning, especially relational logic, remains the central weakness despite progress in basic composition.

  • 3.1 SETUP: 23 open-source and closed-source T2I models are evaluated using generated images and ten questions spanning the benchmark’s spatial sub-domains.The primary evaluator is Qwen2.5-VL-72B, with five answer options and five-round voting used to stabilize evaluation.
  • 3.2 MAIN RESULTS OF SPATIALGENEVAL BENCHMARK: Top models exceed 70.0% on spatial foundation tasks, but performance drops on complex perception, reasoning, and interaction tasks.This gap indicates stronger object and attribute generation than organization under specific spatial constraints.
  • 3.2 MAIN RESULTS OF SPATIALGENEVAL BENCHMARK: Qwen-Image reaches 60.6%, narrowing the gap with Seed Dream 4.0 at 62.7%, but overall performance remains near the 60-point passing threshold.The results describe open-source models as catching up while complex spatial intelligence remains difficult.
  • 3.2 MAIN RESULTS OF SPATIALGENEVAL BENCHMARK: Scale and architecture both correlate with stronger spatial performance: 20B Qwen-Image scores 60.6% versus 54.0% for 8B SD-3.5-L, while 7B Bagel scores 57.0%.Bagel’s score is comparable to the 12B FLUX.1-krea at 58.5%, illustrating parameter efficiency in unified models.
  • 3.2 MAIN RESULTS OF SPATIALGENEVAL BENCHMARK: Relational reasoning is the primary failure point, with error rates often exceeding 35%, while visual-perception errors reach 28.5% in complex nature scenes.Motion interaction is less frequently erroneous, typically remaining below 18%.
  • 3.2 MAIN RESULTS OF SPATIALGENEVAL BENCHMARK: Model rankings on SpatialGenEval closely align with four other major benchmarks, and GPT-4o and Qwen2.5-VL-72B produce similar rankings and scores.These comparisons support consistency across benchmarks and evaluators.

4 SUPERVISED FINE-TUNING (SFT)

SpatialT2I extends the benchmark’s information-dense design into supervised fine-tuning data, using rewritten prompts aligned with generated images. Fine-tuning consistently improves spatial abilities, with higher-quality subsets and more data producing larger gains.

  • 4 SUPERVISED FINE-TUNING (SFT): SpatialT2I is created as a separate supervised fine-tuning dataset with no overlap with SpatialGenEval.Its construction has two stages: generating prompts and omni-dimensional questions, then rewriting prompts to match images while preserving information density.
  • 4 SUPERVISED FINE-TUNING (SFT): The dataset uses 14 top-performing T2I models and MLLM-rewritten prompts to improve text-image consistency while preserving spatial information.The source models have average SpatialGenEval scores above 50%.
  • 4 SUPERVISED FINE-TUNING (SFT): Fine-tuned UniWorld-V1, OmniGen2, and Stable Diffusion-XL consistently achieve better spatial abilities on SpatialGenEval.Qualitative comparisons also report competitive images and more realistic spatial effects.
  • 4 SUPERVISED FINE-TUNING (SFT): All three SpatialT2I subsets improve fine-tuning performance, and higher-scoring subsets contribute more significantly for both diffusion and non-diffusion models.The subsets are Unipic-v2, Bagel, and Qwen-Image, with source scores of 54.3, 57.0, and 60.6, respectively.
  • 4 SUPERVISED FINE-TUNING (SFT): Performance consistently improves as training data increases from 0% to 100% by progressively adding higher-scoring subsets.The result suggests that scaling information-dense, spatial-aware data is promising.

5 RELATED WORK

Related work covers T2I architectures and benchmark evolution from foundational semantic alignment toward longer prompts and relational understanding.

  • Text-to-Image benchmarks: SpatialGenEval’s ablation study examines how different SpatialT2I subsets and progressively larger training sets affect fine-tuning performance.The study is presented as a data-scaling analysis alongside the broader benchmark and model literature.
  • Text-to-Image models: Diffusion models are the dominant T2I paradigm, with transformer backbones and powerful LLM text encoders improving generation and semantic comprehension.The literature traces architectural development from earlier approaches toward diffusion-based systems.
  • Text-to-Image benchmarks: T2I benchmarks include foundational tests of object presence and attribute binding, alongside newer evaluations of complex instruction following and relational understanding.The newer category uses longer prompts and question-answering formats for multi-object relations.

6 CONCLUSION

The paper introduces SpatialGenEval to expose the gap between basic object generation and advanced spatial tasks, identifying spatial reasoning as the primary bottleneck. It also presents SpatialT2I fine-tuning as a practical data-centric path toward improving these shortcomings.

  • 6 CONCLUSION: SpatialGenEval systematically evaluates T2I spatial intelligence with information-dense prompts designed for real-world complexity.The benchmark uses a hierarchical framework to assess spatial capabilities beyond basic generation.
  • 6 CONCLUSION: The evaluation reveals a stark disparity between basic object generation and advanced spatial tasks, with spatial reasoning as the primary bottleneck.The conclusion identifies this pattern as the central weakness revealed by current-model evaluation.
  • 6 CONCLUSION: SpatialT2I demonstrates the effectiveness of supervised fine-tuning as a practical data-centric path toward resolving these shortcomings.This conclusion is based on the paper’s reported fine-tuning results.

ETHICS STATEMENT

SpatialGenEval is introduced as a benchmark for evaluating the spatial intelligence of text-to-image models, with construction overseen by large multimodal models and human experts.

  • Human experts review every generated prompt and question-answer pair for logicality, neutrality, and freedom from harmful or personally identifiable information.

REPRODUCIBILITY STATEMENT

The paper supports reproducibility by providing detailed methodology and publicly releasing its benchmark, dataset, and evaluation resources.

  • The paper provides methodological details covering benchmark construction, experimental setup, and SpatialT2I dataset creation.
  • The complete benchmark, dataset, and evaluation code are publicly available.

A.1 BENCHMARK STATISTICS AND ANALYSIS

SpatialGenEval organizes diverse real-world scenes and spatial domains into a structured benchmark designed to diagnose increasingly complex spatial capabilities. Its question design extends beyond simple object checks to test viewpoint-dependent perception and deeper scene understanding.

  • Scene selection and diversity: SpatialGenEval covers 5 primary scenes and 25 sub-scenes representing real-world applications where spatial intelligence is important.The scene set balances broad coverage with a manageable benchmark size.
  • Scene selection and diversity: Outdoor scenes comprise 32.5% of the benchmark and include dense public environments challenging object-layout, crowd-dynamics, and large-scale functional understanding.
  • Domain selection and diversity: The framework selects 4 domains and 10 sub-domains to progress from object identification and static arrangements toward relationships, event dynamics, and causes.This progression is intended to make the benchmark diagnostic.
  • T2I model selection and diversity: The benchmark samples 23 representative T2I models across diffusion, autoregressive, and unified architectural paradigms.
  • Question design: Questions test comprehensive scene awareness, multi-attribute combinations, and viewpoint-dependent object position or orientation rather than isolated global checks.

A.2 HUMAN ANNOTATION INTERFACE

SpatialGenEval uses a two-stage human refinement process to improve prompt and question-answer quality, supported by expert calibration and substantial annotation effort.

  • Five expert annotators spend over 168 person-hours refining SpatialGenEval through two stages.Annotators first complete a calibration phase using a detailed guidebook to support high inter-annotator agreement.
  • Prompt refinement: Prompt refinement improves clarity, fluency, and logical consistency while ensuring coverage of all 10 spatial sub-domains.
  • QA refinement: QA refinement aligns each question with a specific spatial sub-domain, removes answer leakage, and corrects phrasing inaccuracies.

A.3 ADDITIONAL CLOSED-SOURCE EVALUATION

The paper validates its main evaluation with GPT-4o and describes semi-automated benchmark and dataset construction, prompt rewriting, bias checks, and acknowledged limitations and future directions.

  • Additional Closed-Source Evaluation: GPT-4o produces model rankings highly consistent with the primary Qwen2.5-VL-72B evaluation, supporting evaluator-robust conclusions.The secondary evaluation is presented in Table 10.
  • Construction Pipeline: The benchmark and dataset use a semi-automated pipeline that combines multimodal models with human oversight.The pipeline includes prompt generation, question-answer generation, image evaluation, and prompt refinement stages.
  • Prompt Rewriting: Prompt rewriting improves spatial performance overall, with larger gains for explicit Position, Comparison, and Layout relations.Reported gains include +4.5% in Comparison for OmniGen2 and +4.7% in Position for SD-3.5-L.
  • Prompt Rewriting: Rewriting has minimal impact on Occlusion and Orientation, indicating that these challenges involve visual reasoning beyond textual clarification.The paper points to specialized fine-tuning or unified-based design as possible directions.
  • Evaluation Robustness: The evaluation includes safeguards against prediction bias, and image-free testing yields 16.9% accuracy versus 19.8% random-guess accuracy.The image-free result supports the visual grounding of the questions.
  • Limitations and Future Work: The benchmark is labor-intensive to scale and abstracts only part of real-world spatial complexity, leaving fluid dynamics, deformations, and predictive interactions for future work.Future extensions include other generative capabilities, text-to-video, more objects, curriculum learning, and MLLM-feedback reinforcement learning.
Loading 2601.20354v2…