PaperlayerView exact revision on arXiv

Source-linked AI summary

SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors

Chenyang Ma, Kai Lu, Ta-Ying Cheng, Niki Trigoni, Andrew Markham

arXiv:2403.13438v5cs.CV

TL;DR

Spatial reasoning VLMs are mainly evaluated on spatial VQA, leaving higher-level 3D-aware tasks less explored. SpatialPIN uses progressive prompting and 3D foundation-model priors without training, and the paper reports performance across spatial VQA and robotics applications, including trajectory planning.

  • Problem

    Spatial reasoning-enhanced VLMs are mostly tested on spatial VQA, while higher-level 3D-aware tasks such as scene comparisons and trajectory planning remain underexplored.

    Source: §1 · p. 1

  • Method

    SpatialPIN progressively prompts and interacts with 2D/3D foundation models to build explicit 3D scene understanding in a zero-shot, training-free manner.

    Source: §1 · pp. 1–2

  • Results

    SpatialPIN significantly outperforms state-of-the-art VLMs trained on extensive spatial VQA datasets and performs across spatial VQA and robotics tasks.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper demonstrates applications of its spatial reasoning approach in spatial VQA and traditional and novel robotics tasks, including trajectory planning.

    Source: §1 · p. 1, §5 · p. 9

    Inference takes approximately 20 seconds per image, with 3D object reconstruction and closed-source VLM API calls as bottlenecks; this process runs once per image.

    Source: §5 · p. 9

Abstract

from arXiv · show

Current state-of-the-art spatial reasoning-enhanced VLMs are trained to excel at spatial visual question answering (VQA). However, we believe that higher-level 3D-aware tasks, such as articulating dynamic scene changes and motion planning, require a fundamental and explicit 3D understanding beyond current spatial VQA datasets. In this work, we present SpatialPIN, a framework designed to enhance the spatial reasoning capabilities of VLMs through prompting and interacting with priors from multiple 3D foundation models in a zero-shot, training-free manner. Extensive experiments demonstrate that our spatial reasoning-imbued VLM performs well on various forms of spatial VQA and can extend to help in various downstream robotics tasks such as pick and stack and trajectory planning.

1 Introduction

SpatialPIN addresses the limits of spatial-VQA-focused evaluation and training by combining VLMs with 3D foundation-model priors in a zero-shot, training-free framework. The paper reports strong results across spatial VQA and robotics tasks, including trajectory planning.

1 Introduction

  • SpatialPIN targets a gap in prior work: spatial reasoning VLMs are mostly tested on VQA, while higher-level tasks such as scene comparison and trajectory planning remain underexplored.The paper notes that prior spatial-VQA training may yield surface-level associations and may not generalize beyond dataset distributions.Source: §1 · p. 1
  • The framework combines progressive prompting and interactions with 2D/3D foundation models to construct an explicit 3D scene representation without training.The authors describe the approach as zero-shot and training-free.Source: §1 · pp. 1–2
  • The paper presents SpatialPIN as a step toward generalizing spatial reasoning from VQA to 3D-aware tasks such as trajectory planning.Its reported applications include spatial VQA and both classic and novel robotics tasks.Source: §1 · p. 2
  • SpatialPIN significantly outperforms state-of-the-art VLMs trained on extensive spatial VQA datasets across the paper’s broad evaluation.The study spans relative positions and orientations, fine-grained object properties and dynamic changes, and robotics tasks with full 3D trajectories.Source: §1 · p. 2

2 Related Work

Related work covers VLM grounding and spatial understanding, positioning SpatialPIN as an effort to combine foundation-model knowledge for fuller 3D scene understanding.

VLM Grounding

  • Socratic Model combines foundation models for downstream tasks but focuses on 2D pixel-space image understanding, while SpatialPIN targets challenges in combining information for 3D scene understanding.The authors identify full-scene understanding in 3D as an unresolved challenge they aim to address.Source: §2 · p. 2

VLM Spatial Understanding

  • SpatialVLM fine-tunes on 3D-VQA datasets to improve 3D understanding precision, but the paper argues that 3D-VQA covers only part of an image’s 3D understanding and may yield suboptimal out-of-distribution performance.SpatialPIN instead proposes using multiple 3D foundation models through prompting and interaction with RGB inputs.Source: §2 · p. 2

3 Method

SpatialPIN progressively builds spatial understanding from an RGB image through 2D scene decomposition, coarse 3D reasoning, and partial scene reconstruction. It then supplies summarized 3D information to a VLM for downstream tasks, including collision-free path generation.

3 Method

  • The pipeline progresses from scene decomposition through comprehension and reconstruction, then prompts a VLM with summarized 3D information for downstream tasks.It is designed to provide explicit scene understanding and support unforeseen tasks without training.Source: §3 · p. 3

3.1 2D Image Scene Understanding

  • VLM descriptions of object appearance and 2D location guide segmentation; repainting and re-segmentation produce occlusion-free masks for individual objects.The descriptions also distinguish items with similar or identical categories and appearances.Source: §3.1 · p. 3

3.2 Coarse 3D Scene Understanding

  • Metric depth and camera intrinsics estimated from perspective fields determine scene dimensions, which the VLM uses with the image to infer camera shot angle.The authors motivate perspective canonicalization as a way to resolve orientation ambiguities caused by camera perspective.Source: §3.2 · p. 4

3.3 Fine-Grained 3D Scene Understanding

  • The method partially reconstructs the scene with full 3D foreground objects and a planar background, then summarizes object poses, sizes, axes, and spatial relationships for VLM prompting.Object principal axes are derived from minimal oriented bounding boxes, and the reconstructed 3D information is grounded in the canonicalized perspective.Source: §3.3 · pp. 4–5

3.4 Combining External Tools for Downstream Tasks

  • With the partially reconstructed scene, the framework can use RRT* to generate collision-free paths based on task specifications.The paper presents this as an application combined with task-specific prompting.Source: §3.4 · p. 5

4 Experiments

The experiments evaluate spatial VQA, robotics tasks, and framework modules, using existing and handcrafted datasets to assess higher-level spatial awareness.

4 Experiments

  • The experiments address general spatial reasoning, new applications, and module effectiveness, combining four existing datasets with two handcrafted datasets.Implementations use off-the-shelf segmentation, repainting, 3D reconstruction, camera-intrinsic, and depth-estimation models.Source: §4 · p. 5, §4.1 · p. 5

4.1 Spatial Visual Question Answering

  • On IaOR-VQA, fine-grained 3D information substantially improves qualitative and quantitative spatial reasoning across the evaluated VLMs.Tables 1 and 2 report the qualitative and quantitative IaOR-VQA comparisons.Source: §4.1 · p. 6, Table 1 · p. 6, Table 2 · p. 6
  • Quantitative IaOR-VQA performs suboptimally compared with the more challenging quantitative IrSD-VQA; the authors also identify camera/world-frame confusion in quantitative IrSD-VQA.Tables 3 and 4 report results for IaAD-VQA and IrSD-VQA.Source: §4.1 · pp. 6–7, Table 3 · p. 7, Table 4 · p. 7

4.2 Robotics Pick and Stack

  • In ManiSkill scenes, precise 3D information improves pick-and-stack success, and VLM reasoning about physics and object properties further enhances performance.The setup uses 50 scenes with YCB rigid and articulated objects and ground-truth camera matrices and depth.Source: §4.2 · p. 7, Table 5 · p. 8

4.3 Discovering and Planning for Robotics Tasks from a Single Image

  • From a single image of an unknown scene, the system proposes tasks and plans their execution using axes-constrained 3D motion with waypoints.The evaluation dataset combines 38 self-captured images and 13 NOCS scenes, with 278 tasks or planned trajectories.Source: §4.3 · pp. 7–8, Figure 5 · p. 8
  • Video-LLaVA reports a 36.3% false-positive rate on deliberately misaligned task-video pairs, which the authors subtract from raw accuracy when assessing machine understanding.Table 7 reports machine-understanding results for 278 task executions.Source: §4.3 · pp. 8–9, Table 7 · p. 9

4.4 Ablation Study

  • The ablations report module effectiveness and note that off-the-shelf mesh-free and mesh-based pose-and-size methods may struggle to generalize to novel objects in diverse scenes.The overall-design comparisons include ShAPO, SAM-6D with One-2-3-45++, and SpatialVLM’s data-generation backbone.Source: §4.4 · p. 9

5 Discussion and Conclusion

SpatialPIN uses prompting and 3D priors to enhance VLM spatial reasoning without training, with applications in spatial VQA and robotics. Its inference takes about 20 seconds per image, chiefly due to reconstruction and closed-source VLM API calls.

5 Discussion and Conclusion

  • SpatialPIN enhances VLM spatial reasoning by prompting and interacting with 3D priors in a zero-shot, training-free manner.The paper presents this as a step toward more generalized spatial reasoning, demonstrated in spatial VQA and robotics tasks.Source: §5 · p. 9
  • The framework is demonstrated on varied spatial VQA and both traditional and novel robotics tasks.The paper names these applications as evidence of a step toward more generalized spatial reasoning.Source: §5 · p. 9
  • Inference takes about 20 seconds per image, with 3D object reconstruction and closed-source VLM API calls as bottlenecks.The authors say this process runs only once per image and expect speed to improve with future 3D foundation models.Source: §5 · p. 9

Appendix for SpatialPIN

The appendix collects technical and implementation details, added experiments, visualizations, and prompt information for SpatialPIN.

Appendix for SpatialPIN

  • The appendix provides additional technical details about partial 3D scene reconstruction.Source: A Overview · p. 15
  • It adds details, templates, and visualizations for the two proposed forms of SpatialVQA.Source: A Overview · p. 15
  • It describes the single-image robotics application, including task proposal, axes-constrained waypoint planning, and trajectory generation and smoothing.Source: A Overview · p. 15
  • Further robotics experiments cover dataset statistics, qualitative demonstrations, human understanding, and task diversity.Source: A Overview · p. 15
  • The appendix also supplies prompt details for VLMs.Source: A Overview · p. 15

B Partial 3D Scene Reconstruction Details

The reconstruction details describe assumptions and procedures for estimating scene geometry, canonical object poses, template matches, and object scales.

B Partial 3D Scene Reconstruction Details

  • The method treats predicted normalized depth as perpendicular distance to the camera plane, rather than distance along a straight line to the camera lens.Source: B Partial 3D Scene Reconstruction… · p. 15
  • Given estimated vertical field of view, camera focal length is computed from image height as f = H / (2 tan(θ_v / 2)).H is image height in pixels.Source: B Partial 3D Scene Reconstruction… · p. 15
  • For each reconstructed object, the method uses its reconstruction camera pose to restore canonical poses across images.One-2-3-45++ provides the camera pose; rendered object templates matched against masks provide an alternative for models without it.Source: B Partial 3D Scene Reconstruction… · p. 15
  • Rendered templates are compared with occlusion-free object masks using cropped-mask area and Hu-moment contour similarity, and the best match minimizes the score.Templates sample camera positions around the object on an icosphere for coverage from different angles.Source: B Partial 3D Scene Reconstruction… · p. 15
  • Object scales are refined after scene integration to reflect depth variation, using rendered-mask contour lengths relative to occlusion-free masks.Source: B Partial 3D Scene Reconstruction… · p. 16

C.1 Spatial Visual Question Answering

The SpatialVQA material describes questions about object orientation and movement, with annotation procedures for intra-image angular differences and inter-image spatial dynamics.

C.1 Spatial Visual Question Answering

  • Intra-image angular-discrepancy questions use YCBInEOAT ground-truth 6D poses and principal axes of objects resting on the table or ground to annotate qualitative and quantitative angle comparisons.The listed questions include object inclination, relative angles, and whether objects are upright.Source: §C.1 · pp. 16–17
  • Inter-image spatial-dynamics annotations manually measure object location and angle changes between two photos, and tripod-angle changes track camera shot-angle changes.Question examples ask about object movement and rotation, camera changes, and changes in relative position or orientation.Source: §C.1 · p. 17
  • The appendix illustrates example input images for all forms of spatial VQA in Figure 6.Source: §C.1 · p. 18

C.2 Discovering and Planning for Robotics Tasks from a Single Image

From a single RGB scene image, the framework proposes tasks and plans their execution as full 3D trajectories. Its waypoint planner constrains motion to the manipulated object’s principal axes, while task proposals are shaped by object focus and feasibility constraints.

C.2 Discovering and Planning for Robotics Tasks from a Single Image

  • The single-image pipeline combines task proposal with axes-constrained motion planning to produce full 3D execution trajectories.The pipeline is presented for scenes with unknown environments and objects.Source: §C.2 · p. 18
  • Task proposals use role-play and sequential object focus, with interaction tasks for multiple detected objects and single-object tasks when only one is detected.For multiple objects, the prompt requests two interaction tasks and one task focused on the manipulating object.Source: §C.2 · p. 18
  • Proposals are guided by practical object affordances and potential interactions, while excluding disassembly, functionality tests, and imaginary objects.Source: §C.2 · p. 18
  • The dataset contains 51 scenes—13 from NOCS and 38 captured from varied perspectives—and qualitative examples show geometrically aware trajectories aligned with task descriptions.Source: Figure 8 · p. 19, Figure 9 · p. 20
  • The waypoint planner defines axial rotation and target-relative pitch, yaw, and roll as rotation options.Source: §C.2 · p. 19
  • Translation waypoints specify centimeters along the target’s principal axes or along the directional vector between two reference objects.For target-axis translation, [0, 0, 0] cm denotes the target object's center.Source: §C.2 · p. 19
  • Because VLMs lack direct 3D-coordinate and low-level-action generation, the method translates natural-language instructions into precise waypoints.The paper describes the four manipulation types as simple and comprehensive across a broad spectrum of manipulation tasks.Source: §C.2 · p. 19
  • For task-diversity comparison with RoboGen and prior reinforcement-learning benchmarks, the authors sample 106 proposed tasks.Source: Table 10 · p. 20

Experiment on Human Understanding

Human descriptions show high alignment with VLM-proposed task descriptions, and participants appear to understand task executions more accurately than machines. The section also reports that the proposed tasks are more diverse than those in compared benchmarks.

Experiment on Human Understanding

  • Human descriptions show high alignment with VLM-proposed task descriptions, and people appear to understand executions more accurately than machines.The authors hypothesize that this discrepancy reflects limitations of current video-understanding models and humans’ prior experience.Source: Experiment on Human Understanding · p. 20, Table 9 · p. 20
  • The authors report that their method generates the most diverse tasks among the compared reinforcement-learning benchmarks, using Self-BLEU and embedding similarity, where lower scores mean greater diversity.Source: Experiment on Human Understanding · p. 20

C.3 Trajectory Generation Implementation Details

This section describes how VLM waypoints are converted into collision-aware, smoothed trajectories and how structured prompts guide task discovery, spatial interpretation, and fine-grained motion planning. It illustrates the prompting setup on a screwdriver-penetration task with object-relative translations and rotations.

C.3 Trajectory Generation Implementation Details

  • RRT* plans paths through point-cloud-derived voxel obstacles, whose sizes are expanded to account for the manipulated object’s dimensions.Source: §C.3 · p. 21
  • When RRT* detects waypoint collisions, Gaussian sampling adjusts waypoints under geometric rules for target-relative translations and motion along object-to-object directions.Type 1 sampling follows selected axes according to [dx, dy, dz], with vertical samples constrained by zsampled·dz > 1; type 2 samples along the connecting vector with Dsampled·dD > 1.Source: §C.3 · p. 21
  • Trajectory rotations are linearly interpolated, while translations use cubic-spline interpolation to produce smooth, natural motion.Source: §C.3 · p. 21

D Prompt Details

  • The task-discovery prompt asks a single robot hand to identify objects and propose interaction or rotation tasks, while excluding disassembly, cleaning, imaginary objects, and object articulation.It permits affordance-based assumptions and specifies that the manipulated object moves while the interacting object remains static.Source: D Prompt Details · pp. 21–22
  • A coarse-understanding prompt combines visual cues with scene size and object dimension bounds to classify the camera angle as horizontal, top-down, or bottom-up.Source: D Prompt Details · p. 22
  • Fine-grained motion prompts use object centers, local axes, sizes, and spatial relations to specify self-rotation, target-relative rotation, target-relative translation, or directional translation.Rotations can use exact degrees or fixed_towards/fixed_back orientations; translations are specified in centimeters.Source: D Prompt Details · pp. 22–23
  • For screwdriver penetration, the prompt specifies using a screwdriver to penetrate an avocado, with screw_idx manipulating and avocado_idx interacting.Source: D Prompt Details · p. 24
  • The trajectory moves screw_idx to [-5, -5, 0] cm relative to avocado_idx’s local [x, y, z] axes, then rotates it around yaw by [fixed_towards].Source: D Prompt Details · p. 24
  • It then rotates screw_idx relative to avocado_idx around the roll axis by [360] degrees.Source: D Prompt Details · p. 24

Related papers

Loading 2403.13438v5…