PaperlayerView exact revision on arXiv

Source-linked AI summary

Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text

Xu Wang, Kaixiang Yao, Miao Pan, Xiaohe Zhou, Xuanyu Liu, Wenqi Zhang, Xuhong Zhang

arXiv:2607.21072v1cs.CV

TL;DR

Existing spatial benchmarks usually require coordinates, labels, or text, creating an answer-interface mismatch for image-generation models. ProVisE enables protocol-constrained pixel-space evaluation and finds complementary strengths: image generators compete on directly visualizable answers, while text-output VLMs lead in compositional spatial reasoning.

  • Problem

    Existing spatial benchmarks typically require coordinates, option labels, or textual descriptions, despite many spatial judgments being naturally expressed through visual marking or drawing.

  • Method

    ProVisE elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original task metrics, with an Agentic builder for new benchmarks.

  • Results

    Image-generation models are most competitive when answers can be externalized directly in pixel space, while text-output VLMs remain stronger in compositional spatial reasoning.

  • Takeaways & Limitations

    The findings support a division of strengths rather than a single winner: pixel-space expression and text-based compositional reasoning excel in different spatial tasks.

  • Takeaways & Limitations

    Using one shared protocol may favor visual representations executable by GPT Image 2 and disadvantage weaker or differently trained generators.

Abstract

from arXiv · show

Spatial intelligence is essential for agents to move from static semantic understanding toward interacting with the physical world. Many spatial tasks are grounded in continuous visual scenes, where locations, regions, and paths are more naturally expressed by pointing, marking, or drawing than by reporting precise coordinates or discrete textual symbols. Yet existing spatial reasoning benchmarks usually require coordinates, options, or text, creating an answer-interface mismatch for image-generation models. This makes it difficult to evaluate image-generation models under the same task semantics as text-output VLMs, despite their ability to externalize spatial judgments directly in pixel space. We propose ProVisE (Protocolized Visual Evaluation), a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original metrics. ProVisE also includes an Agentic builder that constructs and validates task-specific protocols for new benchmarks. We further introduce SpatialGen-Bench, a curated diagnostic benchmark of 470 samples across 14 spatial subtasks, four capability levels, and diverse answer forms. We evaluate representative text-output VLMs and image-generation models in a unified setting and validate Agentic protocol construction on six external spatial benchmarks. Results show that image-generation models are competitive when spatial answers can be externalized directly in pixel space, while text-output VLMs retain a clear advantage in compositional spatial reasoning. These findings reveal complementary strengths of pixel-space expression and text-based reasoning and establish a metric-compatible testbed for studying spatial cognition in image-generation models.

1 Introduction

The introduction identifies an answer-interface mismatch that limits shared spatial evaluation, then presents ProVisE and SpatialGen-Bench as a metric-compatible framework and diagnostic suite. It motivates a unified comparison showing complementary strengths between pixel-space image generation and text-based spatial reasoning.

  • Problem: Spatial benchmarks typically require coordinates, option labels, or textual descriptions, forcing continuous visual judgments into symbolic outputs and hindering evaluation of image-generation models.For regions, paths, depths, and orientations, required outputs include bounding-box coordinates, per-pixel depth values, and direction labels.
  • Problem: Auxiliary VLM judges introduce dependence on judge reliability and uncertainty while making alignment with original text-model metrics difficult.This undermines controlled comparisons between image-generation models and text-output VLMs under shared task semantics.
  • Framework: ProVisE elicits protocol-constrained visual answers and parses them into structured predictions compatible with original task metrics.The framework is benchmark-agnostic and supports image-generation models while preserving shared task semantics for comparison with text-output VLMs.
  • Findings: Image-generation models are most competitive when spatial answers can be externalized in pixel space, whereas text-output VLMs remain stronger in compositional spatial reasoning.The introduction frames these as complementary strengths revealed by systematic cross-model evaluation.
  • Benchmark: SpatialGen-Bench provides a curated diagnostic benchmark with 470 samples across 14 subtasks, four spatial capability levels, and diverse answer forms.The suite is designed to extend spatial evaluation beyond a single benchmark or model family.

2 Protocolized Visual Evaluation

ProVisE adapts textual spatial benchmarks to image-generation models by replacing only the response interface with fixed visual protocols while preserving original task semantics and metrics. Its Agentic builder constructs and validates task-specific protocols, whose parsed outputs are scored in the same answer space as text-model responses.

  • Visual protocols: A visual protocol pairs a guidance prompt with a parser, answer format, invalid-output rules, and a mapping to the original task metric.These elements are fixed before evaluation, avoiding subjective post-hoc interpretation of generated responses.
  • Visual protocols: The protocol pool supports instance and point marking, direction grids, relative-depth maps, region masks, state matching, trajectory drawing, and evidence rendering.Each visual answer format is paired with a corresponding parser and operational invalid-output boundaries.
  • Protocol routing: Protocol routing selects one configuration per task, reusing validated protocols or having the Agentic builder construct task-specific protocols from benchmark contracts.The accepted configuration is shared by all evaluated models, separating protocol design from model comparison.
  • Agentic construction: Three construction routes—Reuse, Build, and Fallback—produce the same declarative artifact containing a generation prompt, parser configuration, answer contract, and invalid-output rules.The benchmark is first normalized into task-level contracts without changing task semantics or scoring rules.
  • Validation and evaluation: Smoke validation tests visual generation, stable parsing, and original-metric compatibility before accepted protocols are applied unchanged to every target model.A failed protocol may be revised once before rejection or transition to Fallback.
  • Validation and evaluation: Parsed visual responses are converted into benchmark answer spaces such as labels, counts, masks, candidate states, or trajectories and scored with the original task metric.Text-output VLM responses are directly normalized, whereas image-generation outputs are parsed after protocol-guided generation.

3 SpatialGen-Bench

SpatialGen-Bench organizes heterogeneous spatial tasks into four progressive capability levels, spanning perception through action-oriented interaction. Its 14 subtasks preserve source-task semantics and metrics while supporting answer forms from discrete choices and counts to regions, states, and trajectories.

  • Answer interfaces: Heterogeneous answer contracts range from discrete choices and counts to points, masks, candidate states, and continuous trajectories.This interface diversity supplies the answer forms required by ProVisE while preserving source-task semantics and metrics.
  • Capability hierarchy: Four progressive levels organize the benchmark from spatial perception to understanding, reasoning, and interaction.The hierarchy advances from basic attribute recognition through viewpoint and layout comprehension, complex inference, and action-oriented decisions.
  • Spatial perception: 145 samples cover perception through counting, depth estimation, orientation judgment, and object-size comparison.These tasks are grounded in direct visual evidence and represent the foundation of the capability hierarchy.
  • Spatial understanding: 140 samples cover understanding through relationship verification, perspective judgment, mental modeling, and spatial grounding.These tasks require scene-level representation, viewpoint adjustment, layout inference, and localization of referred targets.
  • Spatial reasoning: 90 samples cover reasoning through multi-hop relations, spatial state prediction, and geometric feasibility.The tasks assess compositional relations, physically plausible state changes, and fit or collision constraints.
  • Spatial interaction: 95 samples cover interaction through affordance grounding, navigation, and trajectory planning.These tasks require deciding where interaction is possible, which state to move toward, and how to reach a target.

4 Experiments

Experiments evaluate 31 model-interface systems on SpatialGen-Bench with frozen, shared visual protocols and native text answer spaces. Results show complementary interface strengths, substantial human gaps, successful cross-benchmark transfer, and meaningful sensitivity to parser choice and externalization failures.

  • Experimental setup: 31 model-interface systems are evaluated: 20 Text Answering and 11 Visual Answering systems, with dual-interface models treated separately.Visual protocols are frozen before evaluation, shared across visual models without model-specific tuning, and invalid or unparseable outputs receive zero.
  • Main results: 61.04 and 54.49 are the Overall scores of GPT-5.4 and GPT Image 2, respectively, versus 87.79 for the Human reference.The largest capability-level gap is 38.57 points in Understanding, with widest task-level gaps on Perspective (51.43), Relationship (31.43), and Mental Modeling (28.57).
  • Interface comparison: Visual answering gains 18.85 points on Depth and 6.74 on Relationship in the model-group comparison, while SenseNova gains 50.00 and 5.71 on those tasks.Visual answers are strongest when the target remains a pixel-level spatial state, whereas text leads on Size, Feasibility, Prediction, and Counting and by 17.63 points at the Reasoning level.
  • Complementary capabilities: GPT Image 2 solves 71 items missed by GPT-5.4, recovering 37.0% of its diagnostic failures, while SenseNova’s visual mode recovers 30.6% by solving 78 text-mode misses.These paired outcomes show that visual successes are not simply a subset of textual successes, despite SenseNova’s lower aggregate score.
  • Transfer and robustness: The Agentic builder enables metric-compatible GPT Image 2 evaluation across all six external benchmarks; text leads four columns, while GPT Image 2 leads SAT and RoboAfford.Parser substitution changes rankings and scores: universal-parser rank correlations versus ProVisE are 0.60, 0.31, and 0.60, with mean absolute score changes of 5.87, 5.63, and 5.26 points.
  • Failure analysis and limitations: 88.03% of non-correct outcomes are valid but incorrect predictions, compared with 8.46% protocol noncompliance, 3.45% parser failure, and 0.06% generation failure.Agentic construction uses GPT-5.4 and GPT Image 2 backends, improving quality and comparability but potentially favoring representations executable by GPT Image 2 and disadvantaging weaker or differently trained generators.

5 Conclusion … B Dataset Construction and Quality Control

The paper concludes that ProVisE and SpatialGen-Bench enable metric-compatible evaluation of spatial cognition through visual answers, while documenting related work, benchmark construction, and supporting analyses in the appendices.

  • 5 Conclusion: ProVisE converts protocol-guided visual responses into structured predictions scored with original task metrics.Its Agentic builder adapts the process to new benchmarks.
  • 5 Conclusion: SpatialGen-Bench supports shared comparisons between text-answering and visual-answering models across diverse spatial tasks.The conclusion characterizes the results as a division of strengths rather than a single winner.
  • 5 Conclusion: Image-generation models are most competitive when spatial answers can be represented directly in pixels, including depth fields or grounded locations.This conclusion identifies complementary strengths rather than one universally superior model type.
  • Appendix Overview: Appendix A reviews spatial evaluation and generative-model evaluation.The appendix provides supporting context for the paper’s evaluation framework.
  • Appendix Overview: Appendix B documents data sources, schema, and quality control, while Appendix C specifies parser rules, Agentic construction, and validation analyses.These appendices provide implementation details for the benchmark and evaluation pipeline.
  • Appendix Overview: Appendix D presents representative visual-only successes.Appendix E reports runtime settings, scoring, complete results, and uncertainty.
  • A Related Work: Related work spans single-image spatial tests, broader spatial taxonomies, and planning or embodied reasoning suites.Examples include VSR, BLINK, ViewSpatial, SpatialBench, OmniSpatial, SpatialScore, and action-oriented benchmarks.
  • B Dataset Construction and Quality Control: The dataset-construction appendix focuses on source provenance, task mapping, unified representation, and quality control of the released SpatialGen-Bench split.It complements the main text’s coverage of benchmark scale, capability hierarchy, and task coverage.

B.1 Data Sources, Task Mapping, and Licenses … C.3 Agentic Construction Details and Coverage

The paper builds SpatialGen-Bench from provenance-preserving spatial tasks and standardizes their released representation without changing task semantics. ProVisE protocols enforce spatial evidence and executable parsing, while Agentic construction validates coverage across external benchmarks.

  • B.1 Data Sources, Task Mapping, and Licenses: SpatialGen-Bench maps public source tasks by objective across object perception, scene understanding, multi-step reasoning, and action-oriented interaction.Object-facing direction is treated as perception, whereas viewpoint selection is treated as understanding because it requires reasoning.
  • B.1 Data Sources, Task Mapping, and Licenses: Samples retain source provenance and official usage terms, while “N/A” indicates no explicit license field was identified rather than unrestricted use.The documentation records provenance without reinterpreting rights for upstream media assets.
  • B.2 Unified Instance Format: Released JSONL rows normalize shared fields, paths, identifiers, and labels while preserving each source task’s answer contract, including comparisons, boolean verification, and ordered paths.The schema standardizes transport rather than redefining task semantics; evaluator-only masks and richer metadata remain separately represented.
  • B.3 Data Integrity and Review Coverage: Every retained record passes deterministic integrity checks and manual inclusion review covering identifiers, media, schemas, task inputs, visual evidence, annotations, and scoring consistency.The checks span the entire released split.
  • C Protocol Design and Parsing: ProVisE protocols require task-grounded spatial evidence, minimal scene intervention, an executable parser with invalid-output rules, and no answer shortcuts.The builder rejects option letters, corner codes, verdict symbols, or prose-only outputs when they encode answers without spatial evidence.
  • C.1 Protocol Design: SpatialGen-Bench uses fixed-slot protocols for Mental Modeling and Multi-hop, plus a trajectory-guided Prediction protocol whose drawn path and destination are parsed by a fixed auxiliary VLM.The evaluated benchmark protocols are distinct from Agentic routes for external benchmarks.
  • C.2 Parser Rules and Thresholds: Parser specifications distinguish reusable, instantiated, and Agentic rules, explicitly naming CLIP, OCR, VLM assistance, thresholds, decoded answer types, and invalid-output boundaries.The parser inventory covers color, components, differencing, geometry, points, masks, paths, measurements, OCR, choice mapping, and fixed CLIP similarity.
  • C.3 Agentic Construction Details and Coverage: 23 task units pass smoke validation: four Reuse routes, 11 Build routes, and eight generated-image-only Fallback routes use validated metric-compatible parsing.Smoke acceptance requires generation, parsing, repeated-parse agreement, recoverable spatial evidence, and metric-compatible predictions; correctness is diagnostic, not an acceptance gate.

C.4 Cross-Benchmark Evaluation Details

The cross-benchmark evaluation uses deterministic, task-balanced archived outputs and matched text-baseline IDs, while reporting protocol executability separately from answer correctness. OmniSpatial is an explicit scope exception because its archived pilot and current construction coverage contain different task-unit counts.

  • Evaluation setup: 50 archived GPT Image 2 record IDs per benchmark were selected by deterministic task-balanced allocation, with each text baseline evaluated on the same IDs.This aligns visual and text-baseline comparisons on matched samples.
  • Protocol execution: 95.00% of visual outputs parsed successfully: deterministic or fixed-auxiliary routes achieved 97.40%, while Fallback achieved 86.96%.These correspond to 285/300, 225/231, and 60/69 outputs, respectively.
  • Protocol execution: Six parser failures and nine protocol-noncompliant responses were retained as zeros, so the reported rates measure protocol executability rather than answer correctness.The remaining outputs were not treated as correctness-specific failures.
  • Scope limitations: OmniSpatial is the only scope exception: its archived pilot covers four task units, while current construction coverage includes a fifth restored unit.Unmatched full-split text scores are omitted because their sample composition differs from the visual subsets.

C.5 Depth Parser Route Ablation

The formal Depth parser samples generated maps at benchmark-provided A/B locations using local grayscale evidence, replacing a mixed coordinate/VLM route. It achieves higher parse rate and benchmark accuracy, while 5 × 5 remains the fixed default after limited kernel sensitivity.

  • Parser route: The formal parser samples a 5 × 5 local grayscale mean at normalized benchmark A/B locations and applies the fixed brighter-is-closer convention without a VLM call.It records both grayscale measurements from the generated depth map.
  • Parser route: The coordinate-sampling route yields higher parse rate and benchmark accuracy than the prior mixed coordinate/VLM route on the same saved outputs.The mixed route uses cached coordinates when available and otherwise asks a VLM to judge depth directly.
  • Kernel sensitivity: 1.21 points: aggregate accuracy varies only 1.21 points across 1 × 1 to 11 × 11 kernels, so the existing 5 × 5 default is retained.The choice is not made retrospectively.
  • Validity: Two FLUX.2 [klein] outputs are invalid because zero-valued neighborhoods at both locations provide no distinguishable depth evidence.The invalid outputs are included in the stated parser analysis as lacking usable evidence.
  • Validation: 100% agreement between recorded grayscale evidence and parser predictions is an implementation check rather than an independent accuracy result.This agreement verifies parser consistency but does not establish benchmark performance.

C.6 Universal VLM Parser Sensitivity Details · C.7 Failure Attribution Details · C.8 Parsed-Output Sensitivity

The appendices test parser sensitivity, formalize failure attribution, and show that filtering invalid parses changes rankings because models retain different scorable subsets. These diagnostics preserve fixed-denominator comparability while clarifying how availability, response validity, and metric-consumable outputs affect results.

  • C.6 Universal VLM Parser Sensitivity Details: Aggregation required exact per-task ID matches, rejected duplicate or missing IDs and invalid records, and reproduced ProVisE task scores within 0.005 points before masking.Unavailable images and invalid responses remained explicit zero-scored records.
  • C.6 Universal VLM Parser Sensitivity Details: Excluding Depth preserved parser sensitivity at Spearman 0.71/0.43/0.43 and mean absolute shifts of 6.40/5.59/5.74 points.Depth is the only input exception because archived caches pair the source marker image with the generated depth map for direct A/B questioning.
  • C.6 Universal VLM Parser Sensitivity Details: 2,820 expected model–record pairs yielded 2,784 cached artifacts, while 36 unavailable records received zero under all four parser columns.The unavailable records comprised 34 multihop, one perspective, and one size record, concentrated in GPT Image 2 (31) and Seedream 4.5 (5).
  • C.6 Universal VLM Parser Sensitivity Details: Conditional on available images, valid-parse rates were 99.54%, 98.49%, and 91.96% for Qwen3-VL-8B, Qwen2.5-VL-72B, and Llama 4 Scout.Their valid/invalid/unavailable counts were 2,771/13/36, 2,742/42/36, and 2,560/224/36; score and rank changes persisted beyond response-format failures.
  • C.7 Failure Attribution Details: Failure attribution prioritized generation failure, protocol noncompliance, parser failure, and then correct or incorrect prediction using cached outputs only.Invalid or ambiguous Fallback outputs count as noncompliance, unsuccessful parses as parser failure, and exact local Depth ties as noncompliance.
  • C.7 Failure Attribution Details: 32 records required conservative correctness inference using precision ≥0.5 for Affordance, DFD < 0.4 for Trajectory, and normalized score ≥0.999 elsewhere.These thresholds affect attribution only, not native benchmark scores.
  • C.8 Parsed-Output Sensitivity: Conditioning on metric-consumable outputs kept GPT Image 2 and Nano Banana 2 first and second, while SenseNova-Vision-7B-MoT moved 7→3 and OmniGen-v1 9→6.JoyAI-Image moved 5→7 and FLUX.2 [klein] 9B moved 6→8 because models retain different subsets.
  • C.8 Parsed-Output Sensitivity: Scorable-only scores are diagnostic rather than alternative leaderboard scores because denominators differ by model and task; tasks with no scorable prediction remain zero.Invalid outputs are zero-scored in the official fixed-denominator results and omitted only from the conditional diagnostic.

D Qualitative Interface Complementarity · E Experimental Settings and Scoring

The qualitative cases show visual answers passing unchanged benchmark metrics where text baselines fail, across depth, relationships, trajectories, and additional spatial tasks. The evaluation settings define the post-protocol interface, runtime, prompts, and score aggregation used for comparison.

  • D Qualitative Interface Complementarity: Figures 8–10 pair incorrect text-baseline responses with visual responses that pass the unchanged task metric.The examples cover depth, relationship, and trajectory tasks, with complete inputs and responses shown without cropping or aspect-ratio normalization.
  • D Qualitative Interface Complementarity: 7 of 11 visual systems pass the trajectory example under the DFD criterion.The task requires a five-point normalized-image-coordinate polyline, and JoyAI-Image produces a path that passes the criterion.
  • D Qualitative Interface Complementarity: Additional examples cover Size, Spatial Grounding, Geometric Feasibility, and Orientation, including a direction-grid protocol.These examples use visual outlines, normalized points, scene placement, and deterministic grid parsing as answer interfaces.
  • E Experimental Settings and Scoring: The evaluation interface is specified after the benchmark split and visual protocols are fixed.It covers text-output prompts, model runtime settings, and score aggregation.
  • E Experimental Settings and Scoring: Protocol constraints and released generation-prompt locations are documented in Appendix C.This documentation supplements the interface specification for the experimental evaluation.

E.1 Human Evaluation … E.5 Statistical Uncertainty

The evaluation uses matched human and model answer protocols, unified parsing and task-specific scoring, macro-aggregated benchmark metrics, and bootstrap intervals that quantify sample-selection uncertainty for fixed outputs.

  • E.1 Human Evaluation: Human references cover the full benchmark and use the same task-specific metrics and macro-aggregation as model outputs.This produces the Human row in result tables and remains separate from the source-data quality audit.
  • E.2 Text-Output VLM Prompt Formats: Text-output VLMs answer in each benchmark’s original answer space using response formats matched to the target task.Formats include integers, A/B choices, direction labels, true/false decisions, option labels, normalized points, and required structured text.
  • E.3 Model and Scoring Settings: Each visual system produces one protocol-constrained image per sample without correctness retries or manual selection.Outputs are resized to source resolution when needed, parsed automatically, and marked invalid when missing, unreadable, or unparseable.
  • E.4 Score Aggregation: Parsed predictions become per-sample scores si ∈[0, 1], with exact-match accuracy for ten listed spatial tasks and task-specific scores for the remainder.All reported numbers are percentages.
  • E.4 Score Aggregation: Affordance grounding uses precision over predicted actionable support inside the annotated region, while trajectory planning succeeds when Discrete Fréchet Distance is below 0.4.Image-output masks or rasterized text-output points are compared with the target region; trajectory scoring is binary after thresholding.
  • E.4 Score Aggregation: Task scores average over samples, capability scores macro-average subtasks, and the overall score macro-averages all 14 subtasks.This prevents larger source tasks from dominating the final benchmark score.
  • E.5 Statistical Uncertainty: 10,000 bootstrap replicates resample examples within every task and report percentile 95% confidence intervals for fixed cached outputs.The intervals capture sample-selection uncertainty but not generation stochasticity or provider-side model updates.

E.6 Complete Task-Level Results

Table 19 presents complete task-level SpatialGen-Bench results, grouped by capability, while capability and overall aggregates are reported in Table 2.

  • Complete task-level results: Table 19 reports complete task-level SpatialGen-Bench results, with tasks grouped by capability.The table reports results in percentages; capability and overall aggregates appear in Table 2.
  • Task abbreviations: The task abbreviations cover counting, depth, orientation, relationship, perspective, and spatial grounding.Cnt. denotes Counting; Dep. Depth; Ori. Orientation; Rel. Relationship; Pers. Perspective; and Ground. Spatial Grounding.
  • Task abbreviations: Additional task abbreviations cover multi-hop, prediction, geometric feasibility, affordance, navigation, and trajectory.M-Hop denotes Multi-hop; Pred. Prediction; Feas. Geometric Feasibility; Aff. Affordance; Nav. Navigation; and Traj. Trajectory.
Loading 2607.21072v1…