PaperlayerView exact revision on arXiv

Source-linked AI summary

Loc3R-VLM: Language-based Localization and 3D Reasoning with Vision-Language Models

Kevin Qu, Haozhe Qi, Mihai Dusmanu, Mahdi Rad, Rui Wang, Marc Pollefeys

arXiv:2603.18002v1cs.CVcs.AIcs.CL

TL;DR

MLLMs still lack coherent 3D and viewpoint-aware spatial understanding, while many recent methods mainly augment inputs with geometric cues. Loc3R-VLM addresses this gap by jointly learning global layout and situated perspective from monocular video with camera pose priors, achieving strong results across localization and 3D QA benchmarks.

  • Problem

    MLLMs have progressed in vision-language alignment but still lack coherent 3D spatial and viewpoint-aware understanding, while recent methods often provide geometric cues without explicit 3D reasoning supervision.

  • Method

    Loc3R-VLM jointly learns global layout reconstruction and explicit situation modeling from monocular video, using lightweight camera pose priors for geometric and metric-scale alignment.

  • Results

    Loc3R-VLM achieves state-of-the-art language-based localization and outperforms existing approaches on situated and general 3D question-answering benchmarks.

  • Takeaways & Limitations

    Explicit spatial supervision combined with geometric priors yields gains across language-driven localization, situated reasoning, and general 3D question answering.

  • Takeaways & Limitations

    Bird’s-eye-view projection discards vertical detail, limiting reasoning in multi-floor environments and tasks requiring precise height distinctions.

Abstract

from arXiv · show

Multimodal Large Language Models (MLLMs) have made impressive progress in connecting vision and language, but they still struggle with spatial understanding and viewpoint-aware reasoning. Recent efforts aim to augment the input representations with geometric cues rather than explicitly teaching models to reason in 3D space. We introduce Loc3R-VLM, a framework that equips 2D Vision-Language Models with advanced 3D understanding capabilities from monocular video input. Inspired by human spatial cognition, Loc3R-VLM relies on two joint objectives: global layout reconstruction to build a holistic representation of the scene structure, and explicit situation modeling to anchor egocentric perspective. These objectives provide direct spatial supervision that grounds both perception and language in a 3D context. To ensure geometric consistency and metric-scale alignment, we leverage lightweight camera pose priors extracted from a pre-trained 3D foundation model. Loc3R-VLM achieves state-of-the-art performance in language-based localization and outperforms existing 2D- and video-based approaches on situated and general 3D question-answering benchmarks, demonstrating that our spatial supervision framework enables strong 3D understanding. Project page: https://kevinqu7.github.io/loc3r-vlm

1 Introduction

Loc3R-VLM equips 2D vision-language models with 3D reasoning from monocular video by explicitly learning global scene layout and viewpoint-grounded situation modeling. Lightweight camera pose priors support geometric consistency, while the framework achieves strong localization and 3D question-answering results.

  • Existing MLLMs connect language with 2D imagery but lack coherent 3D spatial understanding and viewpoint-aware reasoning.
  • Prior spatial-awareness methods often require precise 3D data at inference and treat global scene understanding and situational awareness as byproducts.
  • Loc3R-VLM jointly reconstructs global layout and models the agent’s situation to support holistic scene representations and perspective-grounded reasoning.
  • The framework integrates lightweight camera pose priors from a pre-trained 3D foundation model to reinforce pose and metric-scale alignment.
  • Loc3R-VLM achieves state-of-the-art language-based localization and surpasses video-based approaches on situated and general 3D question-answering benchmarks.
  • The contributions include a monocular-video 3D understanding framework, combined spatial objectives, and a lightweight camera-pose mechanism for stable geometric grounding.

2 Related Work

Prior 3D vision-language methods use point clouds, geometric input augmentation, or implicit foundation-model features, but face data and scalability constraints. Loc3R-VLM instead learns explicit spatial understanding from monocular video without dense point-cloud input.

  • Point-cloud MLLMs extract geometric and semantic features before aligning them with language, but scarce paired 3D-text data limits generalization.
  • Multi-view and video methods augment 2D vision-language features with 3D coordinates or reconstruction tasks to introduce geometric supervision.
  • Methods using 3D foundation-model representations typically add implicit geometric cues rather than explicitly teaching 3D awareness.
  • Outdoor language-based localization methods generally target LiDAR scenes and provide coarse grounding with limited language generalization and no orientation estimation.
  • Indoor language-based localization requires position and orientation inference under viewpoint ambiguity, occlusion, and fine-grained object relationships.
  • Dense point-cloud representations restrict scalability and generalization, whereas Loc3R-VLM operates directly on monocular video for practical visual inference.

3 Method

Loc3R-VLM equips a vision-language model with 3D spatial understanding from monocular video by combining camera pose priors, global layout reconstruction, and explicit situation modeling. These components jointly ground visual tokens and language in scene structure, agent position, and orientation for viewpoint-aware reasoning.

  • Camera Pose Priors: Loc3R-VLM augments monocular-video vision tokens with latent camera pose priors from CUT3R to provide geometric and metric-scale scene cues.The camera token is projected into the language embedding space and prepended to each frame’s vision-token sequence.
  • Overall Training Objective: The framework jointly trains language modeling, layout reconstruction, and situation objectives so one multimodal representation supports language, reconstruction, and localization.The position head predicts global BEV location and uncertainty, while the orientation head predicts discretized angle logits.
  • Global Layout Reconstruction: Global Layout Reconstruction maps vision patch tokens to coordinates in a shared bird’s-eye-view representation, preserving cross-frame spatial relationships and global context.The BEV frame is gravity-aligned, shared across camera views, and anchored to the first video frame.
  • Global Layout Reconstruction: The layout head predicts each token’s BEV position and axis-wise uncertainty, trained with a Gaussian negative log-likelihood objective.Predicted positions are represented as [x̂_i, ŷ_i], while uncertainty is estimated separately along the two BEV axes.
  • Situation Modeling: Situation modeling inserts dedicated <Pos> and <Ori> tokens between the situation description and question to represent the agent’s position and orientation.These tokens causally attend to camera tokens and spatially enriched vision tokens, supporting viewpoint-aware answer generation.

4 Experiments

Loc3R-VLM is evaluated on language-based localization and situated and general 3D question answering across multiple benchmarks. It achieves strong localization, viewpoint-aware QA, and overall benchmark performance, with ablations supporting complementary roles for situation modeling, layout reconstruction, and camera pose priors.

  • Evaluation Setup: The experiments cover language-based localization, situated 3D question answering, and general 3D question answering across several benchmarks.Evaluation includes SQA3D, ScanQA, MSQA, Beacon3D, and VSI-Bench.
  • Language-based Localization: +25.2% Acc@0.5m and +39.0% Acc@1.0m over View2Cap were achieved for position estimation, while orientation gains were +14.3% Acc@15° and +34.5% Acc@30°.Loc3R-VLM uses no explicit 3D input, unlike the compared dense point-cloud methods.
  • 3D Question Answering: +36.1% Relative Direction, +10.8% Relative Distance, and +8.8% Route Planning were achieved over the second-best generalist baseline on VSI-Bench.The strongest improvements occur in viewpoint-dependent tasks; Loc3R-VLM also performs best on Absolute Distance and Object Size.
  • 3D Question Answering: Loc3R-VLM achieves 62.8 EM on SQA3D and performs best among methods capable of situation localization across both reported benchmarks.It surpasses other 2D methods across most metrics.
  • Ablation Studies: Situation modeling, global layout reconstruction, and camera pose priors each improve performance, while their combination yields the strongest localization and QA results.Ablations describe situation modeling as grounding a local viewpoint and layout reconstruction as organizing the global scene; pose priors especially benefit metric-scale position estimation.

5 Conclusion

Loc3R-VLM equips a 2D Vision-Language Model with 3D understanding from video without explicit 3D input. Its complementary spatial supervision and lightweight pose priors improve localization, situated reasoning, and general 3D question answering.

  • Conclusion: Loc3R-VLM equips a 2D Vision-Language Model with advanced 3D understanding capabilities directly from video data without explicit 3D input.The framework targets language-based localization and spatial reasoning tasks.
  • Conclusion: Its two complementary modules enhance global scene understanding and situational awareness, while lightweight pose priors provide metric-scale consistency.The design is inspired by principles of human cognition.
  • Conclusion: Combining explicit spatial supervision with geometric priors yields substantial gains across language-driven localization, situated reasoning, and general 3D question answering.The conclusion attributes the gains to organizing visual information into global and viewpoint-aware representations.

A Training Data and Evaluation Details

Training uses data from several 3D question-answering and spatial-understanding datasets, with additional details on balancing and evaluation provided in the supplementary material.

  • Training Data: Training data statistics are reported in Table A.1, covering the datasets used to train Loc3R-VLM.The appendix provides detailed training data statistics.
  • Training Data: MSQA training uses only the ScanNet-derived subset, with interleaved images replaced by text in situations and questions.The procedure follows prior work.
  • Training Data: VSI-Bench training combines its official split with custom data created by VLM-3R, while approximately half of available samples are subsampled to balance datasets.The subsampling is intended to maintain balanced contributions across datasets.
  • Evaluation Details: MSQA and Beacon3D are evaluated with GPT-based scoring using gpt-4o-2024-08-06 and the evaluation prompt specified by the respective protocol.The reported metrics follow the protocols of the respective works.

B Robustness to 3D Foundation Model Choice

Replacing the default CUT3R 3D foundation-model encoder with VGGT produces comparable performance. This indicates that Loc3R-VLM is not tightly coupled to a specific 3D representation backbone.

  • Robustness Experiment: A VGGT-based Loc3R-VLM variant achieves performance comparable to the primary CUT3R-based model.The robustness experiment uses a representative subset of the full training data for efficiency.
  • Robustness Conclusion: The framework can integrate alternative 3D foundation models without depending specifically on CUT3R representations.This conclusion follows from replacing CUT3R with VGGT while retaining comparable performance.

C Inference Efficiency

Loc3R-VLM adds a lightweight 3D encoding stage with modest one-time inference costs, while its situation representation links localization quality to situated QA performance.

  • Inference Efficiency: Total latency is 2.6s for the full model on 32 input frames, remaining within practical bounds for VLM applications.Latency and peak memory are measured in fp16 on a single NVIDIA RTX 4090.
  • Inference Efficiency: The CUT3R encoder adds 6.8% peak VRAM overhead and a 1.2s one-time per-video encoding cost on an RTX 4090.CUT3R tokens can be cached and reused across multiple queries for the same video without additional encoding latency.
  • Situation Modeling: The position head outputs both an estimated location and positional uncertainty through a Gaussian negative log-likelihood objective.The uncertainty vector quantifies confidence in the predicted agent location.
  • Situation Modeling: Accurate localization uses position error ≤1.0m and orientation error ≤45°, the criterion for partitioning SQA3D test samples.The analysis evaluates how localization success relates to downstream QA accuracy and predicted uncertainty.
  • Situation Modeling: QA accuracy is substantially higher when Loc3R-VLM localizes the agent within the threshold, while QA performance drops markedly when localization fails.The result indicates that the internal situation representation grounds questions in the correct viewpoint.
  • BEV Representation: BEV ground-truth coordinates are produced by lifting image patches into 3D, aligning gravity, projecting onto the ground plane, and transforming points into a canonical frame.The first video frame anchors the canonical reference; high-depth-variance patches are masked during loss computation.

F Layout Reconstruction

Global layout reconstruction is evaluated as an auxiliary objective for learning cross-view spatial relationships and metrically consistent scene representations. A 2D BEV representation performs better for QA than direct 3D coordinate prediction while preserving comparable localization accuracy.

  • Evaluation: Layout reconstruction is evaluated on 67 unseen ScanNet test scenes using BEV position error, RMSE, Mean Error, and thresholded accuracy metrics.The model is not explicitly optimized for best possible layout-reconstruction performance.
  • Results: Most predictions fall within a few tens of centimeters of ground truth, with high Acc@0.5m and Acc@1.0m indicating metrically consistent global representations.The evaluation treats layout reconstruction as an auxiliary training objective for global scene structure.
  • Ablation: The 2D BEV representation produces stronger QA performance than direct 3D coordinate prediction while maintaining comparable localization accuracy.Both variants are trained on a representative subset of the original training data.

G Frame Count Analysis

Increasing the video input from 16 to 32 frames improves QA and localization, while further scaling to 40 frames produces a performance plateau. Overall performance remains stable across 16–40 frames.

  • Frame Count: Increasing the frame count from 16 to 32 improves both QA performance and localization accuracy.The larger input provides a more comprehensive view of the 3D scene.
  • Frame Count: Performance plateaus when the input increases further to 40 frames, while the model remains stable across the 16–40-frame range.The evaluation is reported in Table G.1.

H Qualitative Results

Qualitative examples show strong situational awareness and spatial understanding, while failure cases reveal distinct errors in localization and downstream reasoning. Localization mistakes can propagate into incorrect answers, especially in complex or ambiguous situations.

  • Qualitative Examples: Qualitative examples demonstrate strong situational awareness and spatial understanding across diverse scenes and situation-question pairs.The examples cover language-based localization and situated question answering on SQA3D.
  • Failure Cases: Failure cases include correct localization with wrong QA, incorrect localization with correct QA, and incorrect localization with incorrect QA.Correct answers despite inaccurate localization can occur when questions are easy or referenced objects appear only once.
  • Failure Cases: Localization errors can propagate to reasoning, producing incorrect answers despite otherwise plausible reasoning.Complex scenes and ambiguous situation descriptions commonly produce joint localization and QA failures.

J Limitations and Future Work

Loc3R-VLM’s limitations concern vertical detail, scene coverage, and domain scope. The paper proposes architectural, sampling, and adaptation directions for addressing these boundaries.

  • Vertical Granularity: BEV projection discards vertical detail, limiting reasoning in multi-floor environments and tasks requiring precise height-based distinctions.The paper suggests layered BEV architectures or object-centric tokens to reintroduce vertical information while maintaining computational efficiency.
  • Scene Coverage: Fixed-length sampling of 32 frames can create blind spots in expansive scenes with low viewpoint overlap.Future work could use spatially adaptive or coverage-aware frame selection without exceeding the model’s context window.
  • Domain Scope: The approach focuses exclusively on static indoor scenes, leaving dynamic scenes and outdoor environments outside its demonstrated scope.Adapting the framework to those settings is identified as future work.
Loading 2603.18002v1…