Source-linked AI summary
StateSight: Benchmarking Latent Spatial-State Reconstruction in Vision-Language Models
Michelle Lin
TL;DR
StateSight addresses the difficulty of isolating latent spatial-state reconstruction from the other capabilities bundled into broad multimodal evaluations. It introduces procedurally generated tasks and deterministic intermediate states, finding that models often produce format-valid answers without recovering the required spatial representation. The benchmark is deliberately narrow and evaluates only two proprietary models alongside a convenience-sample human baseline.
Problem
Broad multimodal benchmarks mix perception, OCR, language modeling, domain knowledge, and reasoning, making latent spatial-state reconstruction difficult to isolate.
Method
StateSight uses procedurally generated cube-net, occluded cube-tower, and 4-neighbor connected-component tasks with renderer-aligned oracle labels, and StateSight-Steps adds deterministic intermediate visual states.
Results
GPT-5.5 outperformed Claude Sonnet 5 across all three direct tasks, while both remained below humans and produced zero format errors in final direct runs.
Takeaways & Limitations
The results show that format-valid responses can mask failures to recover the spatial representation required for verifiable visual inference.
Takeaways & Limitations
The evaluation is limited to two proprietary models, a convenience-sample human baseline, and three narrow generated task families, restricting conclusions about natural images.
Abstract
from arXiv · showhide
Vision-language models are increasingly used for multimodal question answering, yet their ability to reconstruct latent spatial structure from a single image remains difficult to isolate. Broad benchmarks often combine perception, optical character recognition, domain knowledge, linguistic priors, and reasoning in the same evaluation. We introduce StateSight, a procedurally generated benchmark for cube-net opposite-face reasoning, occluded cube-tower counting, and 4-neighbor connected-component counting. Each task family contains 300 single-image prompts with deterministic oracle labels and exact-match scoring. OpenAI GPT-5.5, using the API model identifier gpt-5.5, achieved 59.3%, 33.3%, and 28.3% accuracy across the three tasks, while Claude Sonnet 5 achieved 53.3%, 18.7%, and 7.3%. All final direct runs had zero format errors. A 30-participant human baseline on 60 items exceeded both models on every task, with mean accuracies of 80.8%, 68.8%, and 64.3%. Visible-derivation analysis identified recurring errors in image-state reconstruction and reasoning procedure. We also introduce StateSight-Steps, a companion dataset of 900 interleaved image-text examples and 3,600 deterministic intermediate visual states. The results show that format-valid responses can mask failures to recover the spatial structure required for verifiable visual inference.
1 Introduction
StateSight isolates latent spatial-state reconstruction with controlled visual tasks and exact oracle labels. Its results indicate that validly formatted answers can still reflect failures to recover the required spatial structure.
- Broad multimodal benchmarks combine perception, OCR, language modeling, domain knowledge, and reasoning, obscuring whether models recover intended visual structure.
- Diagnostic benchmarks motivate testing whether vision-language models can reconstruct latent spatial states from clean visual prompts.
- StateSight evaluates cube-net opposite-face reasoning, hidden cube occupancy, and 4-neighbor connected-component counting using generator-derived oracle labels.
- The benchmark contains 900 items, and GPT-5.5 outperformed Claude Sonnet 5 across all three direct tasks with zero format errors.Both models declined substantially on hidden occupancy and connected-component counting.
- Thirty human participants exceeded both models on all three tasks under comparable answer-only conditions.Human performance was not perfect on towers or grids.
- Visible derivations revealed recurring perception, reconstruction, and reasoning-procedure errors, while StateSight-Steps supplies deterministic intermediate visual states for future supervision studies.
- The evaluation provides controlled evidence that models can produce valid answers while failing to recover the spatial representation required by the task.
2 Related Works
Related work has developed diagnostic evaluations for visual and spatial reasoning, but StateSight narrows the problem to generator-controlled latent spatial reconstruction. StateSight-Steps extends this approach with machine-checkable intermediate visual states.
- CLEVR and GQA use generated or structured scenes to expose weaknesses that broad visual question-answering benchmarks can hide.
- StateSight fixes rendering style, answer type, target relation, and oracle state through a complete data-generating process.
- Winoground, BLINK, and Perception Test report compositional, perception, and multimodal video-understanding gaps between models and human performance.
- Unlike broader spatial benchmarks, StateSight focuses on three generated tasks rather than the full range of human spatial cognition.
- Prior work motivates visual intermediate representations because some spatial transformations are difficult to represent through language alone.
- StateSight-Steps provides task-metadata-generated intermediate images aligned with text, making visual states machine-checkable for supervision experiments.
3 Methodologies
StateSight uses a programmatic pipeline that pairs rendered images with deterministic latent-state oracles across three spatial reconstruction tasks. The evaluation combines exact-match model testing, visible-derivation coding, and a human interpretability baseline.
- Benchmark pipeline: Each task defines a latent state space, renderer, answer oracle, prompt template, and parser, keeping rendered images and labels aligned by construction.The renderer and oracle share one latent state; parsing normalizes returned labels or counts without awarding partial credit.
- Evaluation: Direct evaluation used exact-match accuracy over 300 outcomes per task, while model differences used paired bootstrap resampling and human summaries used participant-level task accuracies.Human-model gaps are descriptive because humans completed 20 items per task versus 300 for direct model evaluation.
- Task design: The benchmark contains cube-net folding, hidden cube occupancy, and 4-neighbor connected-component counting tasks.Cube nets require inferring opposite faces after folding; towers require summing hidden supporting cubes; grids count edge-connected gray regions, excluding diagonal contact.
- Dataset: 900 items are divided evenly across the three task families, with 300 examples per task and deterministic exact-match labels.The dataset is generated by custom task-specific renderers and supports reproducible regeneration and controlled analysis.
- Intermediate supervision: StateSight-Steps provides 900 examples and 3,600 deterministic intermediate images showing coordinates, folded directions, height maps, layer slices, masks, component maps, and final groupings.These traces are intended for future prompting, supervision, or fine-tuning experiments; the study does not claim that training on them improves performance.
- Diagnostic analysis: Visible-derivation testing used 30 matched examples per task and requested an answer, derivation, and explanation, with conservative coding of response-content errors.The largest coded category was incorrect image reconstruction, followed by incorrect reasoning procedure; these categories were not treated as causal evidence about internal processes.
4 Conclusions and Future Work
StateSight’s models remained below the human baseline despite format-valid responses, with the largest deficits on hidden occupancy and connected-component topology. Future work should broaden task families and external validation while recognizing the benchmark’s narrow generated scope.
- Format-valid model responses remained below the 30-participant human baseline, especially on hidden cube occupancy and connected-component topology.Visible derivations also showed that plausible explanations can accompany an incorrect reconstruction of the image state.
- The evaluation is limited to two proprietary models, a convenience-sample human baseline, and three deliberately narrow generated task families.The generated setting improves experimental control but limits conclusions about natural images.
- Future work should broaden model coverage, add new spatial relations, and test StateSight-Steps through prompting and training ablations.Proposed extensions include path structure, symmetry completion, layered alignment, rotations, clutter, and more complex occlusion.
- External validation against natural-image spatial benchmarks could clarify which StateSight failures persist beyond controlled rendering.Suggested comparisons include SpatialSense, Visual Spatial Reasoning, What’s Up, and SugarCrepe.
- GPT–Claude gaps were +6.0 points for cube nets, +14.7 for cube towers, and +21.0 for connected grids.Paired intervals excluded zero on all three tasks, supporting the observed ordering.