GeoLift: Grounding Video Spatial Reasoning in Explicit 3D Maps

Xinglin Li; TingTing Long; Jingzhi Zhou; Jiajing Chen; Jian Yang; Jin Xie

Source-linked AI summary

GeoLift: Grounding Video Spatial Reasoning in Explicit 3D Maps

Xinglin Li, TingTing Long, Jingzhi Zhou, Jiajing Chen, Jian Yang, Jin Xie

arXiv:2610.33558v1Oct 24, 2026cs.CV

TL;DR

Video and multiview models can recognize objects in frames yet struggle to integrate spatial evidence across viewpoints and steps. GeoLift builds a shared metric 3D map from frozen model grounding, improving results across VSI-Bench and MindCube.

Problem
Models can ground objects in individual frames, but integrating partial observations across viewpoints for reliable scene-level spatial reasoning remains challenging.
Source: §1 · p. 1
Method
GeoLift combines frozen MLLM grounding with reconstructed metric 3D geometry, using geometric superpoints to associate observations across views for explicit spatial computation.
Source: §1 · pp. 1–2
Results
GeoLift improves results across VSI-Bench and MindCube; its VSI-Bench category macro-average rises from 55.4% to 64.1%.
Source: §6 · p. 9
Takeaways & Limitations
The results indicate that connecting visual groundings through a shared 3D map can support complex spatial reasoning without task-specific training.
Source: §1 · p. 2
Performance depends on grounding and reconstructed spatial references, whose residual errors can propagate to relations, metric estimates, and route planning.
Source: §5 · p. 9

Abstract

from arXiv · show

Video spatial reasoning requires multimodal large language models (MLLMs) to understand how objects are arranged and related across viewpoints. While these models can recognize objects in individual frames, integrating these observations to solve complex spatial tasks remains challenging. We examine route planning as a representative task and observe that existing models become substantially less accurate as the number of required turns increases, with several models achieving accuracy close to random guessing on routes requiring three or more turns. This finding suggests that recognizing landmarks alone is insufficient: models must also connect observations across viewpoints and track how landmarks are arranged in a shared space. Motivated by this observation, we introduce GeoLift, a training-free framework that uses feed-forward 3D reconstruction to map objects detected in individual frames into a shared metric 3D scene. GeoLift combines a frozen MLLM for 2D grounding with pretrained geometry models for scene reconstruction. It then uses shared geometric regions, called superpoints, to associate observations of the same object across views without a learned matcher. The reconstructed 3D map enables explicit computation of spatial relations and multistep routes, and provides geometric evidence for estimating distances and sizes. With all model weights frozen, GeoLift improves Qwen3-VL-4B's average score on VSI-Bench from 55.4% to 64.1%, including an increase in route-planning accuracy from 30.9% to 53.6%, and achieves 59.8% accuracy on MindCube. These results show that connecting frame-level object grounding through a shared 3D map can improve complex spatial reasoning without task-specific training.

1 Introduction

Video spatial reasoning requires connecting partial observations across viewpoints, not merely recognizing objects in individual frames. GeoLift uses a shared metric 3D map to support explicit spatial computation, improving benchmark results with frozen model weights.

Spatial reasoning requires stable object relations across viewpoints, abilities relevant to navigation, manipulation, and physical interaction.Video frames provide partial views, and objects in separate frames may never appear together.Source: §1 · p. 1
Existing models can approach random guessing on routes requiring more turns, motivating a consistent spatial reference for composing relations across observations.Source: §1 · pp. 1–2
GeoLift maps frame-level 2D groundings into a shared 3D space, using geometric superpoints to associate observations across viewpoints for explicit spatial computation.The framework combines frozen MLLM grounding with reconstructed metric geometry and uses no benchmark-specific parameter updates.Source: §1 · pp. 1–2
With all weights frozen, Qwen3-VL-4B raises VSI-Bench average from 55.4% to 64.1% and route-planning accuracy from 30.9% to 53.6%; MindCube accuracy rises from 23.4% to 59.8%.Source: §1 · p. 2

2 Related Work

The related work frames video and multiview benchmarks as tests of coherent spatial reasoning across observations. It contrasts learned spatial interfaces with training-free methods, positioning GeoLift as an explicit metric 3D workspace.

2.1 Video and Multiview Spatial Reasoning Benchmarks

VSI-Bench and MindCube test whether models maintain coherent spatial state across video or limited multiview observations for spatial and multistep reasoning.Source: §2.1 · p. 3

2.2 Learning-Based Spatial Reasoning

Learning-based approaches use spatial supervision, learned geometric fusion, or structured representations, but generally require learning to align geometric cues with MLLM reasoning.Source: §2.2 · p. 3

2.3 Training-Free Spatial Reasoning

Training-free methods compose frozen MLLMs with inference-time tools; GeoLift instead uses a shared metric 3D workspace for explicit spatial computation.Source: §2.3 · p. 3

3 Method

GeoLift addresses the difficulty of combining partial views into a consistent spatial representation, using frozen grounding and reconstructed 3D geometry. Its pipeline grounds objects, builds and associates a shared map, then uses that map for spatial reasoning.

3.1 Problem Setup and Motivation

47.4% accuracy on routes requiring three or more turns, 23.7 percentage points above direct Qwen3-VL-4B, while the task remains challenging.Existing models approach random guessing on these routes: Qwen3-VL-4B reaches 23.7%, and GS-Reasoner and Cambrian-S with 32 frames reach 26.3%.Source: §3.1 · p. 4, Table 1 · p. 4

3.2 GeoLift Overview

GeoLift uses frozen MLLM grounding, metric 3D reconstruction, and superpoint agreement to create a view-consistent map for explicit spatial computation.The map supports both direct spatial operations and geometric hints alongside the original video; all neural components remain frozen.Source: §3.2 · p. 5, Figure 3 · p. 5

3.3 Grounding

A frozen video MLLM locates query-relevant objects in sampled frames, refines their boxes, and discards low-confidence or invalid observations before mapping.For missing targets it searches more densely; sparse-view inputs are grounded independently, and observations retain frame or view, box, and confidence information.Source: §3.3 · p. 5

3.4 Reconstruction

VGGT-Omega reconstructs shared camera geometry, while MoGe-2 depth estimates calibrate its unknown scale to metric units.Grounded boxes are filtered around their median depth, back-projected, and transformed into a common world coordinate system to form a metric point cloud.Source: §3.4 · pp. 5–6

3.5 Association

Geometric superpoints provide cross-view association evidence despite loose boxes and background points that can make direct aggregation unreliable.Observations vote for superpoints; weak votes are removed, adjacent regions are dilated, and widely shared regions are downweighted. Matches combine cosine similarity and an extent-aware center-distance constraint; union-find merges are rejected if they cause excessive center drift.Source: §3.5 · p. 6

3.6 Reasoning

The reasoning controller selects geometric tools to compute spatial relations or plan routes, and can use map-derived hints when geometry alone cannot resolve a query.Route planning uses occupancy-grid construction and A* to connect successive landmarks; a reliability gate shifts reliance toward visual evidence when geometric evidence is unreliable.Source: §3.6 · p. 7

4 Experiments

GeoLift is evaluated on full VSI-Bench and MindCube splits, with results showing task-dependent gains and improvements on both benchmarks. Ablations examine the contributions of metric geometry and superpoint association.

4.1 Experimental Setup

Evaluation uses the official full splits of VSI-Bench and MindCube, with VSI-Bench's overall score computed as the unweighted mean of eight category scores.VSI-Bench contains 5,130 questions from 288 egocentric videos; MindCube contains 21,154 sparse-multiview questions, scored with a pooled micro-average.Source: §4.1 · p. 7

4.2 Main Results on VSI-Bench

VSI-Bench category macro-average rises from 55.4 to 64.1, with gains concentrated in relative direction (+28.3 percentage points) and route planning (+22.7 percentage points).GeoLift also exceeds several listed training-based spatial baselines, the direct Qwen3-VL-8B-Instruct baseline, and reported training-free baselines.Source: §4.2 · p. 8

4.3 Results on MindCube

59.8% micro-averaged accuracy on MindCube improves over direct Qwen3-VL-4B inference (23.4%) by 36.4 percentage points.GeoLift exceeds GPT-5.6-Sol in overall accuracy (54.1%) and Around accuracy (61.5%), while GPT-5 retains the highest Rotation score (93.3%).Source: §4.3 · pp. 8–9

4.4 Analysis and Ablations

Metric geometry raises VSI-Bench's category macro-average from 55.4 to 62.8, and superpoint association further raises it to 64.1.Geometry gains include +26.0 points in relative direction and +16.0 in route planning; superpoints add a further +6.7 in route planning and +2.3 in relative direction.Source: §4.4 · p. 9, Table 4 · p. 9

5 Limitations

GeoLift's training-free design depends on frozen visual and geometric modules, so errors in grounding or reconstruction can affect later spatial reasoning.

Errors in 2D grounding or the reconstructed spatial reference can propagate to spatial relations, metric estimates, and route planning.Reliability checks and fallback to direct Video MLLM answers mitigate uncertain geometry but cannot recover missing or ambiguous evidence.Source: §5 · p. 9

6 Conclusion

GeoLift pairs a frozen Video MLLM with a persistent metric 3D representation, using geometric superpoints to associate video observations across views and spatial operators to compute relations and routes. With the same frozen Qwen3-VL-4B backbone, it raises VSI-Bench's category macro-average from 55.4% to 64.1% and MindCube accuracy from 23.4% to 59.8%.

Current Video MLLMs recognize and localize objects in individual frames but are less reliable at integrating spatial evidence across viewpoints and multiple steps.Source: §6 · p. 9
GeoLift uses a persistent metric 3D representation, geometric superpoints for cross-view association, and explicit spatial operators for computing relations and routes.Source: §6 · p. 9
VSI-Bench category macro-average rises from 55.4% to 64.1%, and MindCube overall accuracy rises from 23.4% to 59.8% with the same frozen Qwen3-VL-4B backbone.Source: §6 · p. 9
The results indicate that explicit 3D spatial state broadens the tasks current Video MLLMs can address reliably without benchmark-specific parameter updates.Source: §6 · p. 9