Source-linked AI summary
GraFT: A Training-Free Framework for Spatial Reasoning in Multimodal Large Language Models via 3D Scene Graphs
Junqing Du, Fernando Ropero, Erkin Turkoz, Yanfeng Zhang, Lu Liu
TL;DR
MLLMs remain unreliable at 3D spatial reasoning, while common remedies require costly supervision or backbone-specific 3D encoders. GraFT instead uses a compact 3D scene graph to provide task-matched geometric, allocentric, and egocentric evidence without training. It improves frozen backbones across VSI-Bench and ScanQA, surpassing broad baseline groups and several fine-tuned spatial models.
Problem
MLLMs struggle with precise geometry, egocentric-to-allocentric viewpoint changes, and fine-grained visual-attribute grounding, while common solutions require costly supervision or backbone-specific encoders.
Method
GraFT operates as a training-free reasoning layer over a 3D scene graph, using symbolic tools, task-conditioned BEV rendering, and geometry-guided egocentric frame retrieval.
Results
GraFT lifts three frozen open-source backbones by 37% to 65% on VSI-Bench, reaches 51.4 on its average, and improves every ScanQA caption metric under the same backbone.
Takeaways & Limitations
A frozen MLLM paired with a compact 3D scene graph can provide strong spatial reasoning without spatial supervision or a backbone-specific encoder.
Takeaways & Limitations
Accuracy is bounded by the quality of the scene graph.
Abstract
from arXiv · showhide
3D spatial reasoning underpins understanding and acting in the physical world, yet it remains unreliable in current multimodal large language models (MLLMs). These models falter at precise geometric measurement, at transforming between egocentric and allocentric viewpoints, and at grounding fine-grained appearance. The most common remedies fine-tune the model on large-scale curated spatial-reasoning datasets or attach dedicated encoders for 3D geometry, which typically couples the solution to costly supervision and a specific backbone. We instead introduce GraFT, a training-free framework that supplies the missing 3D structure through a compact, easily maintained 3D scene graph (3DSG). From this 3DSG, GraFT provides three spatial reasoning capabilities: (1) deterministic geometry through symbolic tools, (2) allocentric layout through a bird's-eye-view (BEV) rendering, and (3) visual-attribute grounding through task-relevant egocentric frames. On ScanQA, GraFT improves every metric over the same-backbone baseline, raising CIDEr by 27%. On VSI-Bench, GraFT improves frozen MLLMs by up to 65%, surpassing every proprietary and general-purpose open-source baseline, and several prominent fine-tuned spatial models.
1 Introduction
GraFT addresses persistent MLLM weaknesses in 3D spatial reasoning with a training-free 3D scene graph that supplies task-appropriate evidence. It combines symbolic geometry, task-conditioned BEV layouts, and geometry-guided egocentric views, improving results across ScanQA and VSI-Bench.
- MLLMs remain unreliable at interpreting 3D sizes, distances, and spatial relations despite their growing use as interfaces for spatial tasks.
- Existing approaches rely on costly supervised or reinforcement-learning training, specialized 3D encoders, or fixed spatial priors tied to a training distribution and backbone.
- GraFT uses a compact, incrementally maintainable 3D scene graph to match each task with symbolic tools, BEV rendering, or egocentric frame selection.
- GraFT supplies exact geometry through symbolic tools, global layout through task-conditioned BEV views, and visual-attribute evidence through geometry-ranked egocentric frames.
- 51.4 on VSI-Bench is GraFT’s best average, exceeding every evaluated proprietary and general-purpose open-source baseline and several fine-tuned spatial models.
- 58.0 to 73.6 on CIDEr is the largest ScanQA caption gain from selected views under the same backbone, without updating parameters.
2 Related Work
Prior work addresses MLLM spatial reasoning through training, specialized 3D encoders, structured visual prompts, or external tools. GraFT differs by selecting task-conditioned evidence from one reusable 3D scene graph and amortizing perception across questions.
- Spatial-reasoning benchmarks document a persistent gap between MLLMs’ general visual competence and their spatial understanding.
- Training-based methods inject metric supervision or reinforcement learning, while other systems encode 3D geometry directly into models or consume explicit 3D inputs through dedicated encoders.
- Structured visual prompting reformats scenes into 2D inputs, but existing schemes commonly apply one fixed rendering to every question.
- GraFT renders BEVs on demand, conditions views on each task, and performs deterministic geometry over a graph constructed once for all questions.
- Tool-augmented systems let MLLMs invoke estimators for depth, pose, or bounding boxes, typically rerunning perception for each question.
- 3D scene graphs organize semantics, geometry, and viewpoints and are widely used in robotics and embodied AI for navigation, embodied QA, and object search.
3 The GraFT Framework
GraFT is a training-free reasoning layer around a frozen MLLM that uses a 3D scene graph to select task-appropriate geometric, allocentric, or egocentric evidence. Its modules provide deterministic geometry, selective BEV layout rendering, and geometry-guided frame retrieval for visual attributes.
- Framework overview: GraFT supplies a frozen MLLM with three task-specific capabilities: symbolic geometric tools, task-conditioned BEV rendering, and geometry-guided egocentric view retrieval.The framework selects among these capabilities according to the spatial task.
- Symbolic module: 3D scene-graph boxes determine geometric quantities directly, allowing counting, size, distance, proximity, and room tools to run deterministically without backbone computation.The box geometry fixes object position, size, and orientation, so derived quantities inherit upstream perception accuracy.
- BEV module: GraFT renders only task-relevant objects as proportionally accurate box footprints, then optionally rotates the scene and adds an anchor-to-target arrow for allocentric reasoning.The task sets the viewpoint and heading, aligning the facing direction to the top of the image.
- Egocentric module: A frame scores highest when the queried object is visible, near, centered, and fully covered; distance, off-center placement, or partial framing lowers the product score.In the illustrated comparison, view 1 leads because it is close, centered, and fully covered, while view 2 is penalized for partial coverage.
- Egocentric module: For visual-attribute questions, GraFT scores posed frames using visibility, proximity, centrality, and coverage, then forwards the highest-scoring non-duplicate views.Visibility gates frames based on projected box geometry, depth, and camera-facing angle; the score combines the remaining terms multiplicatively.
4 Experiments
GraFT is evaluated across symbolic metric estimation, directional BEV reasoning, egocentric ScanQA retrieval, and comparisons with prior systems on VSI-Bench. Across these experiments, task-conditioned evidence and deterministic tools improve frozen-backbone spatial reasoning without training.
- Experimental setup: The experiments cover symbolic tools on five VSI-Bench metric subtasks, BEV rendering on directional subtasks, egocentric retrieval on ScanQA, and full-system comparisons.The studies use ground-truth and realistic-perception settings where specified, with fixed backbones for capability isolation.
- 4.2 Symbolic Tools for Metric Estimation: GraFT’s deterministic functions outperform serialized-graph reasoning in every tested setting and reduce the performance gap between GPT-o3 and Qwen2.5-VL-7B.On GPT-o3, gains over Serialized Graph are +5.9 points under ground-truth perception and +1.5 under realistic perception; on Qwen2.5-VL-7B, they are +36.9 and +16.7.
- 4.3 BEV Rendering for Egocentric-to-Allocentric Reasoning: +35.2 and +15.8 are GraFT’s gains over the Video input baseline under ground-truth and realistic perception on directional VSI-Bench subtasks.GraFT also exceeds prior BEV methods, with selectivity attributed to rendering only query-relevant objects.
- 4.4 Egocentric Frame Retrieval on ScanQA: CIDEr rises from 58.0 to 73.6 and BLEU-1 from 22.2 to 34.2 over uniform sampling on the same Qwen2.5-VL-7B backbone.GraFT ranks first across all five ScanQA metrics, and selected frames can outperform uniform eight-frame sampling even with one frame.
- 4.5 GraFT versus Prior Work: GraFT gains +9.3 and +2.7 average points over Struct2D under ground-truth and realistic perception in GPT-o3 comparisons, and +6.8 with Qwen2.5-VL-7B under realistic perception.The Qwen2.5-VL-7B comparison covers all seven subtasks Struct2D reports, despite Struct2D being fine-tuned.
- 4.5 GraFT versus Prior Work: On complete VSI-Bench, GraFT reaches 51.4 on the official average, exceeds every general-purpose open-source baseline and several fine-tuned spatial models, and does so without training.A 3B GraFT model scores 47.5, surpassing the strongest 72B general-purpose baseline at 40.9.
5 Conclusion
GraFT gives frozen MLLMs 3D spatial reasoning through one 3DSG and three task-specific evidence pathways. It substantially improves VSI-Bench performance, while scene-graph quality remains the main limitation.
- GraFT combines symbolic geometric tools, selective BEV rendering, and geometry-ranked egocentric frames for task-specific spatial reasoning.These capabilities target metric estimation, allocentric layout, and visual-attribute grounding, respectively.
- 37% to 65% gains on VSI-Bench average performance lift three open-source backbones, reaching up to 51.4 and surpassing several fine-tuned spatial models.Even a 3B backbone overtakes the strongest 72B baseline.
- GraFT’s accuracy is bounded by the quality of its scene graph.The shared, maintainable graph also supports extending the framework with additional modules.
A Implementation Details
GraFT constructs and uses a 3DSG as the interface for spatial reasoning, rendering query-conditioned BEVs for direction and route-planning questions. The BEV aligns the observer’s heading upward and pairs geometric marks with textual coordinates and steps.
- Scene representation: The 3DSG is built from annotations under ground-truth perception or reconstructed detections under realistic perception, with camera poses and intrinsics retained for egocentric reasoning.Both settings convert scenes into the same graph structure consumed by the reasoning modules.
- BEV for Relative Direction: For relative direction, GraFT renders one query-conditioned, heading-aligned orthographic BEV on the world XY plane.The fixed 640 × 480 view labels relevant objects, draws an anchor-to-target arrow, and provides observer-frame coordinates in a textual key.
- BEV for Route Planning: Route planning resolves landmarks, aligns the initial heading, draws numbered scene marks, and supplies an ordered forward-step list with coordinates.The backbone determines each turn from the fixed BEV and route instructions.
A.4 Egocentric Frame Retrieval
GraFT retrieves egocentric frames by scoring visibility, proximity, centrality, and coverage for queried objects, then forwarding diverse high-scoring views. The same visibility score supports chronological appearance-order reasoning.
- Frame scoring: For visual-attribute questions, GraFT forwards the top-k posed frames for the queried object using a visibility-gated geometric score.The score combines inverse distance, image centrality, and the fraction of visible box corners; k=2 on ScanQA.
- Frame filtering: Fixed depth, corner, viewing-angle, and temporal de-duplication thresholds control which frames pass and prevent nearly identical consecutive views.A candidate within 15 positions of a selected frame is skipped in the chronological frame list.
- Defaults and algorithm: The egocentric module’s default parameters are fixed across scenes and tasks.Its algorithm ranks frames by visibility count and score, selects a diverse top-k set, and falls back to uniformly spaced frames when none pass.
- Appearance order: For VSI-Bench appearance order, GraFT scans frames chronologically and orders objects by the first frame whose visibility score exceeds the threshold.This temporal rule replaces ranking for that task.
B.1 Single-Module Ablation
The single-module ablation shows that each GraFT module performs best on its intended VSI-Bench task family, while routing questions to the appropriate module yields the strongest overall score.
- Module specialization: Tools leads metric subtasks, BEV leads relative direction at 42.8, and Ego leads appearance order at 46.8.Each module was applied to all eight VSI-Bench subtasks under realistic perception.
- Full-system routing: 50.0 for full GraFT exceeds the 45-level performance of every single module through per-question module selection.The full system routes each question to the capability best matched to its task family.
B.2 Egocentric Frame Budget
The ScanQA ablation varies the number of geometry-ranked egocentric frames and finds k=2 a strong default. Geometric selection contributes more than simply increasing the frame count.
- CIDEr rises from 69.2 to 73.6 to 75.9 as the retrieved-frame budget increases from k=1 to k=2 to k=3.Scores increase steadily on most metrics across k ∈ {1, 2, 3}.
- k=2 recovers most of the gain and achieves the best BLEU-4, motivating it as GraFT’s default frame budget.The main text uses k=2 on ScanQA.
- The ablation attributes the gain to the geometric selection rule rather than to the number of retrieved frames.