PaperlayerView exact revision on arXiv

Source-linked AI summary

pySpatial: Generating 3D Visual Programs for Zero-Shot Spatial Reasoning

Zhanpeng Luo, Ce Zhang, Silong Yong, Cunxi Dai, Qianwei Wang, Haoxi Ran, Guanya Shi, Katia Sycara, Yaqi Xie

arXiv:2603.00905v1cs.CV

TL;DR

MLLMs struggle with 3D spatial reasoning from limited visual observations. pySpatial addresses this gap by using zero-shot Python visual programs to compose spatial tools and generate explorable 3D evidence, outperforming strong baselines and supporting successful robot navigation.

  • Problem

    MLLMs still struggle with 3D spatial reasoning across limited views, where explicit 3D supervision is sparse and costly.

  • Method

    pySpatial uses an MLLM to generate Python programs that invoke spatial tools such as reconstruction, camera operations, and novel-view synthesis.

  • Results

    pySpatial consistently outperforms strong MLLM baselines on MINDCUBE and OMNI3D-BENCH, including a 12.94% improvement over GPT-4.1-mini on MINDCUBE.

  • Takeaways & Limitations

    Generated visual programs support interpretable spatial reasoning and route plans that enable a quadrupedal robot to traverse complex environments.

  • Takeaways & Limitations

    Among analyzed MINDCUBE failures, 13% stem from 3D reconstruction limitations and 20% from final MLLM reasoning.

Abstract

from arXiv · show

Multi-modal Large Language Models (MLLMs) have demonstrated strong capabilities in general-purpose perception and reasoning, but they still struggle with tasks that require spatial understanding of the 3D world. To address this, we introduce pySpatial, a visual programming framework that equips MLLMs with the ability to interface with spatial tools via Python code generation. Given an image sequence and a natural-language query, the model composes function calls to spatial tools including 3D reconstruction, camera-pose recovery, novel-view rendering, etc. These operations convert raw 2D inputs into an explorable 3D scene, enabling MLLMs to reason explicitly over structured spatial representations. Notably, pySpatial requires no gradient-based fine-tuning and operates in a fully zero-shot setting. Experimental evaluations on the challenging MindCube and Omni3D-Bench benchmarks demonstrate that our framework pySpatial consistently surpasses strong MLLM baselines; for instance, it outperforms GPT-4.1-mini by 12.94% on MindCube. Furthermore, we conduct real-world indoor navigation experiments where the robot can successfully traverse complex environments using route plans generated by pySpatial, highlighting the practical effectiveness of our approach.

1 INTRODUCTION

MLLMs remain weak at 3D spatial reasoning from limited views, motivating pySpatial, a zero-shot visual-programming framework that composes spatial tools. Evaluations and navigation experiments show strong benchmark performance and practical robot traversal.

  • MLLMs struggle with 3D spatial reasoning, from relative positions and depth to egocentric motion and multi-view relations.
  • Limited views require models to reason across perspectives, while explicit 3D supervision remains sparse and costly.
  • pySpatial uses MLLMs as Python code-generation agents to compose spatial tools through a defined API in a fully zero-shot setting.
  • 12.94% improvement over GPT-4.1-mini on MINDCUBE accompanies consistent gains over strong MLLM baselines.
  • The framework generates executable and interpretable visual programs and route plans that enable a quadrupedal robot to traverse complex environments.

2 RELATED WORK

Prior work spans spatial reasoning, 3D reconstruction, and modular visual reasoning. These directions motivate combining multimodal models with structured geometric representations and composable vision tools.

  • MLLM spatial limitations matter for applications including robotic manipulation and embodied navigation.
  • Classical 3D reconstruction often uses multistage, time-consuming optimization, whereas newer feed-forward methods directly predict pixel-aligned 3D point maps.
  • Modular visual reasoning composes vision specialists or predefined toolchains with LLMs to solve complex multimodal problems.

3 METHOD

pySpatial turns image sequences and spatial queries into executable Python programs that orchestrate reconstruction, camera-motion interpretation, and novel-view synthesis. The resulting outputs provide visual evidence for final MLLM responses.

  • Problem formulation: The framework takes a partially observed image sequence and spatial query, then produces an answer through visual-program execution.
  • Problem formulation: Feed-forward reconstruction estimates depth, camera intrinsics, and extrinsics to form a world-coordinate point cloud for downstream reasoning.
  • Spatial tools and API: The API exposes reconstruction, camera-motion description, and novel-view synthesis as callable tools for visual programs.
  • Spatial tools and API: CUT3R supplies metric-scale reconstruction, while VGGT is used when relative distances in normalized units are sufficient.
  • Spatial tools and API: Camera description converts pose changes into interpretable egocentric motion using camera centers, frame transformations, and yaw angles.
  • Spatial tools and API: Novel-view synthesis rasterizes a point cloud from a camera pose and supports high-level actions such as rotation and turning around.
  • 3D visual programming: A code agent synthesizes Python programs that invoke public spatial-tool APIs, while an interpreter executes them without exposing internal implementation details.
  • 3D visual programming: Program execution produces text, images, or rendered-view sequences that provide explicit visual evidence before the final MLLM response.

4 EXPERIMENTS

The experiments evaluate pySpatial across multi-view and single-view benchmarks, qualitative reasoning tasks, real-world navigation, efficiency, and failure sources. Across these settings, the framework combines executable 3D visual programs with spatial tools and generally outperforms prior baselines.

  • Experimental settings: The evaluation covers MINDCUBE, MINDCUBE-1k, OMNI3D-BENCH, qualitative spatial reasoning, robot navigation, efficiency, and failure analysis.Baselines include open-weight and proprietary MLLMs, specialized spatial models, and prior visual programming approaches.
  • Quantitative results: 58.56% overall accuracy on MINDCUBE exceeds DeepSeek-VL2-Small by 10.94% and GPT-4.1-mini by 12.94%.On the Among category, pySpatial reaches 60.54%, while no baseline exceeds 50%.
  • Quantitative results: 16.5% higher performance than VLM-3R on MINDCUBE is achieved despite pySpatial operating entirely zero-shot.VLM-3R uses CUT3R as a 3D encoder and is fine-tuned on synthetic spatial reasoning data.
  • Quantitative results: 21.9% improvement over VADAR on MINDCUBE-1k and gains of 3.8% over VADAR and 17.5% over ViperGPT on OMNI3D-BENCH demonstrate broad benchmark improvements.PySpatial also sets a new overall state-of-the-art on OMNI3D-BENCH and surpasses GPT-4o on the total score.
  • Qualitative results: Qualitative examples show executable, interpretable programs operating in reconstructed explorable 3D scenes, including camera transformations and richer control flow.Generated comments expose the reasoning process for verification, debugging, and modification; additional examples use for-loops, conditionals, and lambda-style compositions.
  • Real-world robot navigation: In a 50 m^2 two-room laboratory, pySpatial guides a quadrupedal robot through doorways and correct turns to a mushroom toy using limited views.GPT-4.1 struggles with relative direction and metric distance, whereas pySpatial outputs rotations and translations aligned with execution.
  • Failure analysis: Among 39% of analyzed failure cases, 6% come from incorrect visual programs, 20% from final reasoning, and 13% from 3D reconstruction limitations.The analysis suggests that advances in reconstruction and code-generation models could further improve performance.
  • Efficiency: 2.17 hours for 1,050 MINDCUBE-1k queries corresponds to 7.45 seconds per query, compared with 17.25 seconds for VADAR.The reported average includes code generation, program execution, and answer generation on a single GPU using one thread.

5 CONCLUSION

pySpatial enhances MLLM spatial reasoning through zero-shot Python code generation, converting image sequences into explorable 3D scenes. It outperforms strong baselines and supports successful robot navigation in complex environments.

  • pySpatial is a visual programming framework that enhances MLLM spatial reasoning through zero-shot Python code generation.
  • The framework composes functions such as 3D reconstruction and novel-view synthesis to convert 2D image sequences into explorable 3D scenes.
  • 12.94% is the reported maximum gain over GPT-4.1-mini on MINDCUBE.
  • Robots successfully traversed complex environments using route plans generated by pySpatial in real-world indoor navigation experiments.

REPRODUCIBILITY STATEMENT

The authors support reproducibility by open-sourcing the spatial tools and making the evaluated benchmark datasets publicly available. They also provide experimental and implementation details and publicly release the code.

  • All spatial tools used in the work are open-sourced, and the evaluated benchmark datasets are publicly available.
  • Detailed experimental setup and implementation details are provided in Section 4 and the appendix.
  • The pySpatial code is publicly available through the project repository.

Explicit 3D Spatial Reasoning via Program Generation

The appendix organizes supplementary material for pySpatial across experimental results, API specifications, agent prompts, and large-language-model disclosures.

  • Section A provides additional experimental results.
  • Section B presents the API specification for pySpatial.
  • Section C describes the prompt implementation for the agent.
  • Section D discloses the use of large language models.

A.1 RESULTS ON MMSI-BENCH

On MMSI-Bench, pySpatial improves overall performance by 6.4% on average, with results reported in Table A1.

  • 6.4% is pySpatial's reported average improvement in overall MMSI-Bench performance.
  • Table A1 reports the MMSI-Bench evaluation results for pySpatial based on GPT-4o.

A.2 MORE ABLATION STUDIES

Additional ablations show that pySpatial’s gains persist across code agents and 3D reconstruction backbones, while a small number of in-context examples further improves performance. The framework reaches its strongest reported results with GPT-4o, Pi3, and four examples in the evaluated comparisons.

  • Code Agents: 62.67% overall accuracy is achieved with GPT-4o, while Qwen3-Coder and DeepSeek-v3 reach 62.10% and 61.05%, respectively.GPT-4o provides the most balanced improvements across task types, whereas Qwen3-Coder performs best on Around.
  • 3D Reconstruction Backbones: PySpatial remains effective across VGGT, Pi3, and CUT3R, with Pi3 achieving the best overall performance at 63.33%.All three reconstruction backbones substantially improve over the GPT-4o baseline; Pi3 shows notable gains in Rotation and Among.
  • In-Context Learning Examples for Code Agents: Using 0 examples raises overall accuracy to 53.62% from the 42.29% base GPT-4o result.This demonstrates that pySpatial provides a strong boost even without demonstration guidance.
  • In-Context Learning Examples for Code Agents: 62.67% overall accuracy is reached with 2 in-context examples, increasing to 63.14% with 4 examples.Four examples produce the best reported performance, particularly improving the Rotation and Around categories.

A.3 ADDITIONAL QUALITATIVE RESULTS

Additional MINDCUBE examples show that pySpatial can execute expressive, multi-step 3D visual programs beyond linear program flows. Its control-flow constructs support viewpoint exploration, geometric filtering, and adaptive reasoning.

  • A.3 ADDITIONAL QUALITATIVE RESULTS: PySpatial supports for-loops, conditionals, and lambda-style functional compositions in executable 3D programs.These structures extend the linear program flow shown earlier and enable complex multi-step 3D operations.
  • A.3 ADDITIONAL QUALITATIVE RESULTS: The programs can iteratively explore candidate viewpoints and selectively apply geometric predicates.These operations provide structured ways to search and filter spatial possibilities.
  • A.3 ADDITIONAL QUALITATIVE RESULTS: PySpatial can adapt its reasoning based on intermediate results during multi-step 3D operations.

A.4 ADDITIONAL REAL-WORLD EXPERIMENTS

Additional qualitative results examine pySpatial in MINDCUBE and in a challenging dynamic real-world scene. The dynamic-scene example involves temporally inconsistent views and moving objects, yet the reconstruction preserves stable structural cues for spatial reasoning.

  • A.4 ADDITIONAL REAL-WORLD EXPERIMENTS: The additional real-world example uses views captured at different times while a person moves through the environment.This creates temporal inconsistency and dynamic elements in the scene.
  • A.4 ADDITIONAL REAL-WORLD EXPERIMENTS: The reconstruction module integrates the multi-view observations and preserves stable structural cues needed for accurate spatial reasoning.
  • A.4 ADDITIONAL REAL-WORLD EXPERIMENTS: The MINDCUBE examples show pySpatial reasoning within reconstructed, explorable 3D scenes.They also illustrate executable 3D visual programs with control flow for diverse spatial reasoning tasks.

B IMPLEMENTATION FOR pySpatial

pySpatial provides a Python-facing interface for loading scenes, reconstructing 3D structure, manipulating camera viewpoints, generating visual clues, and answering spatial questions. An agent generates and executes API programs that connect these operations into a reasoning workflow.

  • Scene and input handling: The Scene class stores the question, scene identifier, original path, loaded images, reconstruction, generated code, and visual clue.It accepts either a string path or a list of image paths and loads the corresponding image data.
  • Scene and input handling: The image loader handles demo directories, regular image directories, single image files, and lists of image paths.Demo directories may load images from a color/ subdirectory, while supported extensions include PNG, JPG, and JPEG.
  • 3D reconstruction: The reconstruction interface converts scene images into a Reconstruction object containing point clouds, camera extrinsics, intrinsics, colors, and the raw tool result.The implementation supports demo-directory reconstruction and reconstruction directly from image paths, with tensor outputs converted to NumPy arrays when needed.
  • Spatial tools: The API exposes camera-motion description, novel-view synthesis, and viewpoint operations including rotation, forward and backward movement, and 180-degree turns.Novel-view synthesis takes a reconstruction and a new camera pose and can return an image object or save a rendered image.
  • Agent workflow: The Agent generates Python code from a scene, extracts code from the language-model response, executes it, and passes the resulting visual clue to an answer function.The public API includes code generation, response parsing, program execution, and final answer generation, while GPT-based prompts instruct the model to use the spatial interface.
  • Agent workflow: The prompt explicitly asks the model to use the API's 3D reconstruction and spatial exploration capabilities when solving otherwise difficult visual-language spatial problems.The API supplies a 3D point cloud and camera parameters, and the prompt asks the model to reason about why a particular program should be written.
Loading 2603.00905v1…