PaperlayerView exact revision on arXiv

Source-linked AI summary

Geometrically-Constrained Agent for Spatial Reasoning

Zeren Chen, Xiaoya Lu, Zhijie Zheng, Pengrui Li, Lehan He, Yijin Zhou, Jing Shao, Bohan Zhuang, Lu Sheng

arXiv:2511.22659v1cs.AIcs.CV

TL;DR

VLMs’ semantic representations can omit or distort the geometry needed for spatial reasoning, and existing methods leave gaps in learned spatial logic or planning. GCA formalizes each query as a geometric task constraint before tool-based computation. It reports state-of-the-art results across challenging benchmarks, with average gains of ∼37% across several foundation VLMs.

  • Problem

    VLMs’ lossy semantic space does not reliably preserve the geometric details required for high-fidelity spatial reasoning.

    Source: §1 · p. 2

  • Method

    GCA has the VLM formalize an ambiguous query as a constraint defining its reference frame and objective, then execute tool calls within that constraint.

    Source: §1 · p. 2, §3.2.1 · p. 4

  • Results

    Across several foundation VLMs, GCA reports average gains of ∼37% and state-of-the-art results across challenging spatial reasoning benchmarks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    GCA presents a training-free, geometrically constrained pathway for spatial reasoning on multiple challenging benchmarks.

    Source: §5 · p. 8

    The current toolbox is primarily designed for image-based spatial reasoning; extending it to temporal reasoning and motion tracking is future work.

    Source: §5 · p. 9

Abstract

from arXiv · show

Vision Language Models (VLMs) exhibit a fundamental semantic-to-geometric gap in spatial reasoning: they excel at qualitative semantic inference but their reasoning operates within a lossy semantic space, misaligned with high-fidelity geometry. Current paradigms fail to bridge this gap. Training-based methods suffer from an ``oracle paradox,'' learning flawed spatial logic from imperfect oracles. Tool-integrated methods constrain the final computation but critically leave the VLM's planning process unconstrained, resulting in geometrically flawed plans. In this work, we propose Geometrically-Constrained Agent (GCA), a training-free agentic paradigm that resolves this gap by introducing a formal task constraint. Specifically, we strategically decouples the VLM's role into two stages. First, acting as a semantic analyst, the VLM translates the user's ambiguous query into the formal, verifiable task constraint, which defines the reference frame and objective. Second, acting as a task solver, the VLM generates and executes tool calls strictly within the deterministic bounds defined by the constraint. This geometrically-constrained reasoning strategy successfully resolve the semantic-to-geometric gap, yielding a robust and verifiable reasoning pathway for spatial reasoning. Comprehensive experiments demonstrate that GCA achieves SOTA performance on multiple spatial reasoning benchmarks, surpassing existing training-based and tool-integrated methods by ~27%. Please see our homepage at https://gca-spatial-reasoning.github.io.

1. Introduction

VLMs lose geometric detail when translating visual information into semantic space, while existing approaches leave weaknesses in either their learned spatial logic or their planning. GCA formalizes the task before constraining geometric computation, and reports state-of-the-art results across challenging benchmarks.

1. Introduction

  • VLMs’ lossy translation of visual information into semantic space omits or distorts fine-grained geometry needed for high-fidelity spatial reasoning.A model may understand spatial commonsense yet fail to determine an object’s orientation or imagine an egocentric perspective.Source: §1 · p. 1
  • Training-based methods risk learning flawed spatial logic from imperfect oracles, while tool-integrated methods constrain computation but leave VLM planning unconstrained.An unconstrained plan can adopt the camera’s viewpoint for a query about a user sitting on a sofa, compromising the problem definition before tool use.Source: §1 · pp. 1–2
  • GCA is a training-free agentic paradigm that formalizes an ambiguous query into a verifiable constraint defining the reference frame and objective.The VLM then executes tool calls strictly within the constraint’s deterministic bounds.Source: §1 · p. 2
  • Across several foundation VLMs, GCA reports average gains of ∼37% and establishes state-of-the-art results across challenging spatial reasoning benchmarks.The paper describes the results as extensive and the approach as effective and generalizable.Source: §1 · p. 2

2. Related Work

Prior spatial-reasoning approaches include specialized end-to-end training and tool-integrated execution, while constraint-guided methods translate language into formal representations. The related-work discussion distinguishes these approaches from GCA’s focus on geometrically grounded task constraints.

2. Related Work

  • Specialized end-to-end spatial training incorporates geometric priors but relies on high-quality datasets, which can be hindered by flawed-oracle data generation [8] [22] [30] [33] [39] [48] [50] [52].Examples include explicit 3D structural features [48] and depth maps.Source: §2 · p. 2
  • Tool-integrated systems such as SpatialAgent [49] and TIGeR [12] translate queries into iterative tool executions, but unconstrained planning can conflate what to solve with how to solve it.These methods offload deterministic geometric computation to external modules [12] [25] [29] [49] [62].Source: §2 · pp. 2–3
  • Constraint-guided reasoning translates natural language into formal representations, including LogicLM’s task-specific formal logic [34], LLM+P’s PDDL [26], and ReKep’s relational keypoint constraints [16].The approaches are presented as applications of constraint-guided reasoning in neuro-symbolic reasoning and planning [13] [34] [40] [58] [3] [16] [26] [35] [56].Source: §2 · p. 3

3. Methodology

GCA replaces unconstrained spatial reasoning with a two-stage process built around a formal task constraint. The VLM formalizes the query and visual input, then iteratively acquires geometric data and computes an answer under that constraint.

3. Methodology

  • GCA is a training-free agentic paradigm centered on a formal task constraint that governs constrained computation.The methodology presents this constraint as a deterministic bridge between semantics and geometry.Source: §3 · p. 3

3.1. Geometrically-Constrained Spatial Reasoning

  • The two-stage process replaces a generic agent policy that can leave planning unconstrained, despite using external tools for final computation.In Fformalize, the VLM establishes what to solve; in Fcompute, it executes tool calls under the resulting constraint.Source: §3.1 · p. 4

3.2.1. Constraint for Spatial Reasoning

  • Ctask contains a non-negotiable reference-frame constraint CR and an objective constraint CO, defining the coordinate system and what is measured within it.The reference frame is anchored to scene geometry to resolve ambiguities such as defaulting to the camera’s viewpoint.Source: §3.2.1 · p. 4
  • Spatial queries use a 3D Cartesian frame with an origin and three orthogonal vectors, anchored to object-, camera-, or direction-based geometry.The objective specifies what is measured relative to that frame; for example, a toaster can define the frame for measuring a chair’s positional relationship to it.Source: §3.2.1 · p. 4

3.2.2. Automated Formalization via VLM

  • The VLM automatically generates Ctask = (CR, CO) through qualitative interpretation guided by formal definitions, before computation begins.This geometry-grounded constraint serves as the contract for Fcompute.Source: §3.2.2 · pp. 4–5

3.3.1. Tool Integration and Code Generation

  • Fcompute uses an iterative, closed-loop process for data acquisition, ambiguity resolution, and augmented computation under the immutable Ctask.The VLM binds tool outputs to constraint symbols, resolves ambiguous detections, then retrieves fixed geometric formulas for code generation.Source: §3.3.1 · p. 5

3.3.2. Toolbox

  • The toolbox combines 3D reconstruction and 2D detection or segmentation with Python-based computation and utilities that connect 2D boxes to 3D point clouds.The Python tool runs generated code in a sandbox; reconstruction supplies a unified 3D scene representation.Source: §3.3.2 · p. 5

3.4. Discussion

  • GCA addresses flawed planning by formalizing the task before tool use and constrains subsequent execution to a geometrically sound Ctask.The discussion distinguishes the formalization stage’s role in avoiding direct reasoning within lossy semantic space from Fcompute’s role in binding high-precision computation to the constraint.Source: §3.4 · p. 5

4. Experiments

Across several spatial reasoning benchmarks, GCA averages 64.8% accuracy and outperforms foundation, training-based, and agentic baselines. Experiments examine benchmark performance, formalization, generality across VLMs, component contributions, and failure modes.

4.2. Main Results

  • Across benchmarks, GCA averages 64.8% accuracy, beating Gemini-2.5-Pro by 12%, SpatialLadder by 27%, and TIGeR by 38%.Table 1 reports overall accuracy across benchmarks; the text identifies the strongest foundation, training-based, and agentic counterparts.Source: Table 1 · p. 6, §4.2 · p. 6

4.1. Experimental Setup

  • The evaluation covers single- and multiple-image benchmarks, reporting overall and subcategory accuracy against foundation VLMs, training-based methods, and tool-integrated agents.The primary experiments use Qwen3-VL-Thinking; other VLMs are evaluated in ablations.Source: §4.1 · pp. 5–6

4.2. Main Results

  • On MMSI-Bench, GCA reaches 47.6% accuracy, a 28% relative gain over Gemini-2.5-Pro, and often beats SpatialLadder out of domain by ∼20 points.SpatialLadder reaches 70.3% on the same-source SPBench, while GCA consistently outperforms it on out-of-domain benchmarks.Source: §4.2 · pp. 6–7

4.3.1. Formalization Analysis

  • The formalization ablation finds that prompted hints yield only negligible improvement, while formal Ctask substantially boosts performance; GCA scores 47.6% versus 49.5% with human-annotated formalization.The comparison includes CoT-only, prompted and unprompted unconstrained tools, and an annotated-constraint oracle; formalization accuracy is approximately 70%.Source: §4.3.1 · p. 7, Figure 4 · p. 7, §4.3 · p. 7

4.3.2. Generalizability Across VLMs

  • Across tested VLMs, GCA improves over CoT-only baselines; Gemini-2.5-Pro gains 49% to reach 55.0% on MMSI-Bench, while GPT-4o gains 19%.Figure 5 reports an average 37% relative performance improvement across tested foundation VLMs; the text says gain magnitude appears to correlate with agentic proficiency and baseline spatial reasoning.Source: §4.3.2 · p. 7, Figure 5 · p. 7

4.3.3. Component Contribution

  • Tool integration (+4.2 points), KACG (+1.9), and visual feedback (+1.4) yield +7.5 points over CoT-only; Fformalize adds another +7.5 points.Table 2 studies the contribution of each component; its caption defines KACG and Feedback.Source: §4.3.3 · pp. 7–8, Table 2 · p. 8

4.4. Error Attribution and Failure Cases

  • Fformalize accounts for 30% of errors, while the remaining 70% occur in Fcompute, including perception failures (∼24%), Python Tool errors (25%), and other issues (21%).Formalization errors often involve complex semantics, multi-image ambiguity, or ignored implications; compute failures include reconstruction, coordinate transformations, tool parameter passing, and turn-budget exhaustion.Source: §4.4 · p. 8, Figure 6 · p. 8

5. Conclusion

GCA uses a formal task constraint to turn ambiguous spatial queries into deterministic mathematical problems, and the paper reports new state-of-the-art results on multiple challenging benchmarks. The current toolbox is primarily designed for image-based spatial reasoning, with temporal reasoning and motion tracking identified as future extensions.

5. Conclusion

  • GCA establishes a new state-of-the-art on multiple challenging spatial reasoning benchmarks.The conclusion describes the approach as a robust and generalizable pathway for spatial reasoning.Source: §5 · p. 8
  • A formal task constraint transforms ambiguous spatial queries into deterministic mathematical problems, avoiding reasoning about geometric details in the VLM’s lossy semantic space.Source: §5 · p. 8
  • The toolbox is primarily designed for image-based spatial reasoning; temporal reasoning and motion tracking are future directions.Source: §5 · p. 9

A. Spatial Task Constraint

GCA’s formal task constraint is a deterministic bridge between semantic analysis and constrained task solving. It consists of a reference-frame constraint and an objective constraint.

A. Spatial Task Constraint

A.1. Universality of Ctask in Spatial Tasks

Ctask is a general principle for spatial tasks whose main geometric ambiguity may lie in the reference frame or the objective. The VLM formalizes that ambiguity into a deterministic constraint for a solver.

A.1. Universality of Ctask in Spatial Tasks

  • For spatial understanding tasks, the objective is often explicit, so the main ambiguity is grounding the reference frame, such as what “you” refers to.GCA formalizes reference frames as object-based, camera-based, or direction-based.Source: §A.1 · p. 12
  • In manipulation tasks, the frame is often simple while the objective can require multi-stage affordances, contact points, and trajectories.ReKep represents objectives as relational keypoint cost functions for an inverse-kinematics solver; EmbodiedCoder uses executable programs to parameterize geometry and produce waypoints.Source: §A.1 · p. 12
  • Ctask generalizes across tasks by having the VLM formalize whichever ambiguity—reference frame or objective—matters into a deterministic constraint for a solver.Source: §A.1 · pp. 11–12

A.2. Generalizability of R Definintion

GCA defines object-based, camera-based, and direction-based reference frames, which cover most static spatial queries in existing benchmarks. Dynamic frames and abstract regions remain challenging cases for the current implementation.

A.2. Generalizability of R Definintion

  • Three reference-frame types—object-based, camera-based, and direction-based—cover the vast majority of static spatial queries in existing benchmarks.Source: §A.2 · p. 12
  • Video tasks with changing agent pose require extending the reference-frame constraint to a time-dependent CR(t).Source: §A.2 · p. 12
  • Abstract-region directions may rely on camera-view proxies that can introduce cumulative errors and lack generalizability.The paper suggests that future work could use two image points to define the abstract direction’s corresponding 3D vector.Source: §A.2 · p. 12

B. More Ablation Studies

The ablations find the reference-frame constraint more consequential than the objective constraint, and show that a textual constraint alone does little without tool integration. Across ten identical evaluation runs, GCA’s results are stable.

Importance of CR and CO in Spatial Reasoning

  • Removing the reference-frame constraint causes a 6.6-point drop, compared with a 1.2-point drop when the objective constraint is removed.The authors attribute the difference to reference frames resolving the core “from where” ambiguity, while objectives are often simple enough to infer from the query.Source: B. More Ablation Studies · p. 13

Ctask without Tool Integration

  • A textual task constraint used with chain-of-thought alone yields only a negligible 0.9-point improvement over GCA.The authors say the VLM still cannot bypass flawed spatial imagination and high-precision computation in its lossy semantic space.Source: B. More Ablation Studies · p. 13

Stability and Robustness Analysis

  • Across 10 independent MMSI-Bench runs with Qwen3-VL-Thinking, GCA achieves a mean accuracy of 47.6 ± 0.3.The authors attribute the low standard deviation to formalizing a deterministic constraint before computation, which limits subsequent stochasticity and ambiguity.Source: B. More Ablation Studies · p. 13

C. More Implementation Details

The implementation combines visual foundation models behind tool APIs with a modular agent framework. A single VLM handles formalization, tool orchestration, and code generation.

C.1. Visual Foundation Models

  • GCA uses VGGT for 3D reconstruction, MoGe-2 for metric scale, GroundingDINO for object detection, SAM-2 for segmentation, and Orient Anything for object orientation.These foundation models are wrapped behind abstract tool interfaces rather than provided directly to the agent.Source: §C.1 · p. 13

C.2. Tool Interfaces

  • Eight exposed APIs provide reconstruction, detection, 3D point projection, object pose prediction, scale estimation, OCR, motion analysis, and sandboxed code execution.The code tool generates and runs Python using relevant geometric formulas injected into its prompt.Source: §C.2 · pp. 13–14

C.3. Agentic Framework

  • Ray and LangGraph manage tool services and the two-stage reasoning flow, while one VLM serves as semantic analyst, tool orchestrator, and coder.The tool suite runs as independent Ray Serve actors; coder sampling uses temperature 0.0, while the analyst and orchestrator use temperature 0.6 and top-p 0.95.Source: §C.3 · p. 14

D. Evaluation Benchmark Details

The evaluation covers multiple spatial reasoning benchmarks, with examples illustrating the prompts and qualitative cases used by GCA. The benchmark descriptions focus on visual and geometric spatial tasks.

D.1. MMSI-Bench

  • MMSI-Bench evaluates multi-image spatial reasoning across positional relationships, attributes, motion, and multi-step reasoning.The categories include relations among objects, cameras, and semantic regions, as well as object or camera movement.Source: §D.1 · p. 14

D.2. MindCube-tiny

  • MindCube-tiny tests spatial mental modeling from limited 2D images through rotation, around, and among tasks.The tasks probe reconstruction from partial views, inference from novel viewpoints, and spatial arrangement under occlusion.Source: §D.2 · pp. 14–15

D.3. OmniSpatial

  • The evaluation uses OmniSpatial’s perspective-taking and dynamic-reasoning categories, excluding its spatial-interaction and complex-logic categories.The excluded categories focus more on abstract or symbolic reasoning than high-fidelity geometric perception.Source: §D.3 · p. 15

D.4. SPBench

  • SPBench covers single-image distance, size, and direction questions, plus multiple-view questions about positions and object counts.The multiple-view subset concerns objects in overlapping scenes.Source: §D.4 · p. 15

D.5. CV-Bench

  • CV-Bench includes 2D positional relationships and counting, as well as 3D depth and distance comparisons.Figure 7 shows in-context examples used to formalize reference frames; its output format follows the prompt in Table 4.Source: §D.5 · p. 15, Figure 7 · p. 15

E. Prompts Used in GCA

  • The paper provides prompts for task formalization, tool orchestration, and knowledge-augmented code generation.The prompt materials include examples for reference-frame formalization and fixed formulas selected according to the input variables.Source: E. Prompts Used in GCA · p. 15

F. Qualitative Case Study

  • The qualitative case studies cover multi-view object counting, direction- and object-based reference frames, camera rotation, object movement, and metric-scale estimation.The cases are presented in Figures 8–13.Source: F. Qualitative Case Study · p. 15

G. Broader Impacts

The broader-impact discussion connects GCA to robotics and AR/VR, emphasizing traceable formal constraints and possible process-level supervision. It also warns that dependence on visual foundation models can transmit or amplify perception biases.

G. Broader Impacts

  • Robotics and AR/VR applications depend on translating ambiguous human instructions into precise geometric actions.The paper gives “sit on the sofa” as an example of an ambiguous instruction.Source: G. Broader Impacts · pp. 15–16
  • GCA’s explicit, human-readable task constraint can be checked before high-stakes actions, and its structured outputs may help supervise end-to-end spatial VLM training.The paper presents this verify-then-execute capability as important for safety in real-world applications.Source: G. Broader Impacts · p. 16
  • Poor visual-foundation-model performance on objects, scenes, or lighting can cause GCA to inherit and amplify biases, producing incorrect real-world outcomes.Source: G. Broader Impacts · p. 16

Related papers

Loading 2511.22659v1…