PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoReader: Learning Latent Geometry for Spatial Reasoning in Vision-Language Models

Rong Li, Shijie Li, Jiangyong Huang, Lingdong Kong, Jiaming Zhou, Dongyue Lu, Zeying Gong, Baoxiong Jia, Junwei Liang, Siyuan Huang

arXiv:2610.18694v1cs.CVcs.AIcs.LGcs.MM

TL;DR

VLMs struggle to reason about metric and multi-view geometry implicit in RGB, while existing geometry-integration approaches do not ensure that the VLM uses the representations. GeoReader trains geometric tokens to be readable by the VLM before adapting it for spatial QA, reaching 94.3% accuracy on MindCube-Tiny with Qwen3-VL-4B and a six-benchmark average of 72.5% with Qwen3.5-9B.

  • Problem

    VLMs struggle to infer and reason about 3D structure from 2D observations, and existing geometry fusion or latent alignment does not ensure that geometry representations affect answers.

    Source: §1 · p. 1

  • Method

    GeoReader trains lightweight Geometric Readers with geometry-description prediction by a frozen VLM, then freezes them while fine-tuning the VLM for question-conditioned spatial QA.

    Source: §1 · p. 2

  • Results

    GeoReader reaches 94.3% accuracy on MindCube-Tiny with Qwen3-VL-4B and averages 72.5% across six additional benchmarks with Qwen3.5-9B.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The representation that helps most is one the receiving model can read; capturing geometry alone does not ensure its use.

    Source: §5 · p. 9

    Benchmark performance does not establish reliability for safety-critical navigation or robotic control, which requires application-specific validation and safeguards.

    Source: Ethics Statement · p. 10

Abstract

from arXiv · show

Vision-language models (VLMs) struggle with metric and multi-view relationships that remain implicit in RGB observations. Geometry foundation models recover this structure from RGB, but existing methods either fuse the estimated features with visual tokens and learn to use them from question answering alone, or align VLM latents with geometry features. Capturing geometry, however, does not ensure its use: zeroing the aligned latents of a released feature-alignment model leaves its accuracy unchanged. We introduce GeoReader, a framework that learns geometric representations for the VLM that reads them. First, we train lightweight Geometric Readers to produce continuous input tokens, supervising them with geometry descriptions decoded by a frozen VLM, so that the Readers express geometry in a form the target model can already read. We then freeze the Readers to preserve this interface while fine-tuning the VLM on spatial QA, with geometry injection conditioned on the question. Geometry descriptions serve only as training targets: at inference, the VLM answers directly from the continuous geometric tokens without generating a geometry chain of thought. With geometry predicted from RGB images, our approach reaches 94.3 on MindCube-Tiny with Qwen3-VL-4B and, with a multi-source QA mixture, an average of 72.5 across six additional spatial benchmarks with Qwen3.5-9B. These results suggest that a useful geometric representation is one the receiving model can use, not merely one from which geometry can be reconstructed.

1 Introduction

Existing geometry-augmented VLMs either rely on answer supervision to use geometry features or align latents without ensuring those representations affect answers. GeoReader first trains geometry to be readable by the VLM, then trains the VLM to use it for spatial QA.

1 Introduction

  • Prior methods fuse geometry features with visual features or align VLM latents to geometry, but neither approach explicitly bridges geometry representations to language-grounded spatial relations.The authors report that zeroing released 3DThinker latents leaves MindCube-Tiny accuracy essentially unchanged, with answer tokens barely attending to them.Source: §1 · pp. 1–2
  • GeoReader trains lightweight Geometric Readers with geometry-description prediction by a frozen VLM, then freezes them while fine-tuning the VLM for question-conditioned spatial QA.Geometry descriptions are Stage 1 targets; at inference, the VLM answers directly from geometric tokens.Source: §1 · p. 2
  • 94.3% accuracy on MindCube-Tiny with Qwen3-VL-4B and 72.5% average accuracy across six additional benchmarks with Qwen3.5-9B; evaluation spans seven benchmarks and three VLM families.The six-benchmark result uses multi-source QA training.Source: §1 · pp. 2–3
  • With annotation-derived camera geometry, removing geometry reduces accuracy from 89.2% to 77.2%, while shuffling poses across views reduces it to 63.1%.These controlled MindCube-Tiny interventions test the supplied geometry’s role.Source: §1 · p. 3

2 Related Work

Related work covers spatial QA supervision, geometry-augmented VLMs, and learned geometry representations for reasoning. GeoReader focuses on training the interface between estimated geometry and the VLM that answers spatial questions.

2 Related Work

  • Spatial QA work builds synthetic or 3D-grounded data, structures training from perception toward reasoning, and targets depth, viewpoint, or object-localization relations.Examples include SpatialVLM [10], SpatialLadder [31], SpatialQA [7], and N3D-VLM [62].Source: §2 · p. 3
  • Geometry-augmented VLMs vary in geometric source and fusion, including point maps, depth, spatial positions, relative depth, and camera-guided fusion.Examples include VLM-3R [18], Spatial-MLLM [63], VG-LLM [75], and SpaceMind [74].Source: §2 · p. 3
  • Geometry-representation methods use reconstruction, geometry prediction, masking, or latent alignment; GeoReader instead supervises geometric inputs through description prediction by a frozen VLM.At inference, GeoReader uses estimated pose and depth rather than descriptions.Source: §2 · p. 3

3 Method

GeoReader separates making geometry readable to a pretrained VLM from using it for spatial QA, then conditions geometry injection on the question.

3.1 Preliminaries

  • The setup feeds visual and question tokens alongside per-view camera-pose and depth signals as additional inputs.The formulation allows any per-view geometric signal.Source: §3.1 · pp. 3–4

3.2 Geometric Token Encoding

  • Each Geometric Reader turns patch-grid geometry descriptors into one continuous token per view, leaving the VLM to combine geometry with RGB.Each signal has its own Reader, preserving its structure.Source: §3.2 · p. 4

3.3 Frozen-VLM Readability Training

  • Stage 1 teacher-forces geometry descriptions through a frozen VLM, updating only Readers on geometry-bearing tokens.Pair supervision trains on each reference–target view pair so only those views’ tokens can explain the target description.Source: §3.3 · pp. 4–5

3.4 Question-Conditioned Geometry Injection

  • A read pass uses image-question context hq to set each signal’s gate; an adapter maps frozen Reader tokens into placeholder slots, with absent signals gated to zero.The slot embeddings alone mark a missing signal, preserving the input layout.Source: §3.4 · p. 5
  • Geometry tokens follow the question in the answer pass; training updates the VLM and injector while Readers and vision encoder stay frozen, and inference reuses the read-pass cache.At inference, the answer pass computes only the geometric tokens and answer after reusing that cache.Source: §3.4 · p. 5

4 Experiments

GeoReader is evaluated across seven spatial benchmarks using geometry predicted from RGB, alongside matched comparisons and ablations. Results favor GeoReader over existing models and show that readable geometry, appropriate injection, and Reader freezing all matter.

4.1 Experimental Setup

  • Main models use DA3-predicted RGB geometry across seven benchmarks, with about 65K geometry descriptions for Stage 1 and about 135K multi-source QA examples for Stage 2.Ablations use 10K MindCube examples and annotated geometry to exclude estimation errors.Source: §4.1 · pp. 5–6

4.2 Main Results

  • 63.6 versus 60.4 average accuracy on six benchmarks for GeoReader/Qwen2.5-VL-3B and 3DThinker-3B; GeoReader reaches 72.5 with Qwen3.5-9B and leads four of five MindCube-Tiny backbone blocks.Table 2 reports GeoReader’s highest average in every backbone-size block.Source: §4.2 · p. 6, Table 1 · p. 6, Table 2 · p. 7

4.3 Learning VLM-Readable Geometry

  • 89.2% accuracy with both stages exceeds Stage 1 alone (41.4%) and QA training alone (54.7%) on the 1,050-question MindCube-Tiny test set.The ablation uses Qwen2.5-VL-3B.Source: §4.3 · p. 6, Table 6 · p. 8
  • Readability pretraining scores 76.4% versus 54.7% for QA-only random Readers and 62.7% for feature alignment; pair supervision and freezing improve accuracy to 89.2% from 77.8% and 76.4%, respectively.Pair supervision removes the view ambiguity of group supervision; the authors freeze Readers because QA answers do not require them to retain Stage 1 relationships.Source: §4.3 · p. 7
  • GeoReader’s 89.2% exceeds adapted dense-feature fusion (86.1%, 81.1%) and annotated pose descriptors (73.8%) on the same geometry.The alternatives are complete direct-injection interfaces.Source: §4.3 · pp. 7–8

4.4 Using Geometry for the Question

  • The full geometry injector scores 89.2%, versus 78.6% for direct injection and 68.5% for a learned fixed scale.Table 7 compares accuracy by injection design.Source: §4.4 · p. 8, Table 7 · p. 8
  • Zeroing GeoReader’s geometric residual drops accuracy from 89.2% to 77.2%, and shuffling poses drops it to 63.1%; feature-alignment latent interventions leave accuracy near 62.7%.Answer tokens also attend more to GeoReader’s geometric tokens than 3DThinker’s answer tokens attend to its latents on a question both models answer correctly.Source: §4.4 · pp. 8–9, Figure 3 · p. 8, Table 8 · p. 8

4.5 Interface Trade-offs and Qualitative Analysis

  • GeoReader uses descriptions only as Reader targets, reaching 89.2% versus 72.9% when generating them and 81.0% when supplying them in prompts; inference uses 16.2 tokens in 0.5 s.Figure 4 shows two correctly answered cross-view object questions using DA3 geometry.Source: Table 5 · p. 8, §4.5 · p. 9, Figure 4 · p. 9

5 Conclusion

GeoReader converts geometry into a form a frozen VLM can read, then keeps the Readers fixed while the VLM learns spatial QA. With geometry predicted from RGB, it reaches 94.3 on MindCube-Tiny and a six-benchmark average of 72.5.

5 Conclusion

  • 94.3 on MindCube-Tiny with Qwen3-VL-4B and a six-benchmark average of 72.5 with Qwen3.5-9B.Both results use geometry predicted from RGB images.Source: §5 · p. 9
  • Geometric Readers learn to express geometry in a form a frozen VLM can read, then remain frozen while the VLM and a question-conditioned injector learn spatial QA.Geometry descriptions are training targets only; inference uses continuous geometric tokens without generating a geometry chain of thought.Source: §5 · p. 9
  • Zeroing a released feature-alignment model’s latents leaves its accuracy unchanged, whereas removing GeoReader’s geometry lowers accuracy.The conclusion emphasizes that capturing geometry alone does not ensure the answering model uses it.Source: §5 · p. 9
  • GeoReader’s useful representation is the one the receiving model can read, rather than merely one from which geometry can be reconstructed.Source: §5 · p. 9

AI Use Statement

The authors used generative AI to assist with implementation, data and figure generation, and manuscript work, while retaining review and responsibility for the final content.

AI Use Statement

  • Generative AI assisted with training-code implementation, training-data and figure generation, and manuscript organization and editing.Source: AI Use Statement · p. 10
  • The authors manually reviewed the AI-assisted training code and generated training data.Source: AI Use Statement · p. 10
  • The authors take responsibility for the final work, including its text, claims, code, data, and figures.Source: AI Use Statement · p. 10

Reproducibility Statement

The reproducibility statement directs readers to appendices for data composition, geometry sources, model and training details, and evaluation procedures.

Reproducibility Statement

  • Training-data composition and split boundaries are described in Sec. A.1.2, and geometry sources in Sec. A.1.3.Source: Reproducibility Statement · p. 10
  • Sec. 3 describes the model and training objectives, while Sec. A.2.1 details the Geometric Reader architecture and Sec. A.2.3 gives training settings.Injection designs and text-geometry recipes are in Secs. A.2.2 and A.4.7.Source: Reproducibility Statement · p. 10
  • Evaluation settings and geometry-content interventions are documented in Sec. A.3 and Sec. A.4.6, respectively.Source: Reproducibility Statement · p. 10

Ethics Statement

The ethics statement identifies the dataset basis and cautions that benchmark results do not establish reliability for safety-critical deployment.

Ethics Statement

  • The work uses publicly available datasets under their respective licenses or terms.Source: Ethics Statement · p. 10
  • Predicted geometry and answers can be wrong, including under reference-frame confusion.Source: Ethics Statement · p. 10
  • Benchmark performance does not establish reliability for safety-critical navigation or robotic control, which requires application-specific validation and safeguards.Source: Ethics Statement · p. 10

A.1 Training Data and Geometry Sources

This section maps the Stage 1 description data, Stage 2 QA data and test-set exclusion, and the predicted and annotated geometry. Figure 5 illustrates Stage 1 examples from ScanNet.

A.1 Training Data and Geometry Sources

  • The section covers Stage 1 description data, Stage 2 QA data and test-set exclusion, and predicted and annotated geometry.Source: §A.1 · p. 18
  • Figure 5 shows three Stage 1 examples with paired RGB images, camera poses in the first camera’s reference frame, and view 2 depth with nine-grid regions.Source: Figure 5 · p. 19
  • Depth colors are clipped at 5 m, and gray marks invalid depth.Source: Figure 5 · p. 19

A.1.1 Stage 1: Geometry-Description Data

The appendix documents GeoReader’s geometry and QA data, geometry sources, model and evaluation procedures, and additional results. It also examines the effects of geometry sources and input interfaces, how GeoReader uses geometry, predicted-geometry successes and failures, and the public resources supporting the work.

A.1.1 Stage 1: Geometry-Description Data

  • Stage 1 uses approximately 29.5K Camera and 35.4K Depth descriptions from 1,550 ScanNet and ScanNet++ scenes.Camera descriptions encode translation and heading relative to the first view; Depth descriptions label positions on a 3 × 3 image grid and include distances. Separate MindCube pair supervision uses 3,375 reference–target pairs from 2,072 training groups.Source: §A.1.1 · pp. 19–20

A.1.2 Stage 2: QA Data and Test-Set Exclusion

  • Stage 2 combines approximately 135K QA examples from seven task sources and 14 RGB sources, excluding official benchmark test questions and answers.Stage 1 and Stage 2 training scenes do not overlap with evaluation-benchmark scenes; the multi-source collection is separate from the 10K MindCube QA set used in ablations.Source: §A.1.2 · p. 20

A.1.3 Geometry Sources

  • Main models use Depth Anything 3 to predict camera poses and depth maps from RGB for both training and evaluation; ablations also use annotated MindCube geometry.Annotated inputs contain grid position and facing direction relative to the first view, without object annotations, reasoning chains, or QA answers; benchmark comparisons use predicted geometry.Source: §A.1.3 · p. 21

A.2.1 Geometric Reader Architecture

  • Each Geometric Reader produces one token per view using a learned query that reads patch-descriptor memory, with Reader weights shared across views and no cross-view attention.Stage 1 trains Readers with a frozen VLM; Stage 2 freezes Readers and the visual encoder while fine-tuning the language model and geometry injector.Source: Figure 7 · p. 22, §A.2.1 · pp. 22–23, §A.2.3 · p. 23

A.2.2 Injection Design and Computation

  • The question-conditioned injector routes geometric tokens using the question representation and initially gates available geometry at 0.5.In the two-pass computation, geometry is inserted after question tokens: question positions cannot attend to subsequent geometry, while answer tokens can attend to all preceding inputs; inference reuses the read-pass KV cache.Source: §A.2.2 · p. 23

A.3.1 Benchmark Inputs and Metrics

  • Evaluations use MindCube-Tiny’s 1,050 questions and the full test sets of six additional benchmarks, with each benchmark’s official scoring.The six-set average is the mean of unrounded benchmark scores; literature results retain their source evaluation settings. Main models train for two epochs on the approximately 135K-example QA collection.Source: §A.3.1 · p. 24, §A.2.3 · p. 24, §A.3.4 · p. 25

A.4.1 Geometry versus Vanilla SFT on Six Benchmarks

  • 59.6 accuracy on metric-geometry subtasks with GeoReader exceeds vanilla SFT’s 56.7; on other subtasks, accuracy is 67.3 versus 67.5.All three metric-geometry question types improve: object and room size (+4.8), camera motion (+4.4), and distance and depth (+2.2); neither non-metric question type decreases.Source: §A.4.1 · p. 25, Table 12 · p. 26

A.4.2 MindCube-Tiny Results with Qwen3.5

  • 82.2 Overall accuracy for GeoReader with Qwen3.5-4B exceeds its base model’s 47.8 and Video-ORA-4B’s 48.6 on MindCube-Tiny.Among rises from 43.2 to 93.5 and Around from 59.2 to 92.4, while Rotation falls from 47.5 to 35.5.Source: Table 13 · p. 26, §A.4.2 · p. 26

A.4.3 Predicted versus Annotated Geometry

  • With Qwen2.5-VL-3B, DA3 geometry yields 81.1% versus 89.2% with annotated geometry and 68.6% with vanilla SFT; pair supervision leads recipes at 81.1%.Across six backbones, DA3 exceeds annotated geometry on four and trails it on two, averaging 1.8 points higher; recipe rankings with DA3 match those with annotated geometry.Source: §A.4.3 · p. 27

A.4.4 Reader Pretraining

  • GeoReader trains Readers through geometry-description supervision decoded by a frozen target VLM, then freezes them for QA, unlike 3DThinker’s VGGT-feature targets [12] [57].On the same 1,050 questions, the released first-stage 3DThinker checkpoint’s feature-alignment result is 62.7% [12].Source: §A.4.4 · pp. 27–28

A.4.5 Alternative Interfaces

  • Alternative interfaces adapt VLM-3R-style per-frame fusion [18], SpaceMind-style camera-guided fusion [74], and per-token additive pose fusion, without adding Reader geometric tokens.The adapted fusion rows use frozen VGGT and Qwen visual encoders and train the language model and fusion module with answer loss; the passage gives no outcome values.Source: §A.4.5 · p. 28

A.4.6 Geometry Interventions

  • GeoReader falls from 937/1,050 correct answers with correct geometry to 811/1,050 with zeroed tokens and 663/1,050 with shuffled poses.For feature alignment, the released 3DThinker checkpoint scores 62.7% correct, versus 62.9% with zeroed latents and 61.5% with latents shuffled across questions [12].Source: §A.4.6 · p. 28

A.4.7 Text-Geometry Recipes and Inference Cost

  • The recipe comparison measures generation and latency on an NVIDIA H200 across 1,050 questions, including both GeoReader passes and KV-cache reuse.Direct answers are limited to 128 generated tokens, while Generated Text-CoT is limited to 512; the supplied passages do not report the measured cost values.Source: §A.4.7 · p. 29

A.5.1 Answer-Token Attention

  • Across Among, Around, and Rotation examples, GeoReader’s answer-position geometry attention is 33.8%, 27.7%, and 32.3%, versus 0.2%, 0.4%, and 0.4% for 3DThinker latents [12].Both models answer all three examples correctly; the attention statistics average over layers and answer positions.Source: §A.5.1 · p. 29, Figure 9 · p. 31

A.5.2 Additional Qualitative Results

  • Predicted-geometry examples include correct view-direction and depth judgments, while failures arise from pose-estimation errors, reversed depth order, and reference-frame confusion.In one correct example, predicted turns of 98° and 177° support selecting the window; in a failure, predicted turns of 50° and 124° lead to the wrong view. Other failures follow car/trailer depths of 21.1 m and 25.1 m or answer from the wrong view.Source: §A.5.2 · p. 30, §A.5.3 · p. 30

A.6 Public Resources

  • The public-resource appendix lists 14 RGB training sources plus MindCube and documents dataset, code, checkpoint, and model licenses, including implementations of 3DThinker [12], VLM-3R [18], and SpaceMind [74].It notes that the collection contains approximately 135K training examples.Source: §A.6 · p. 34, §A.6.1 · p. 34, §A.6.2 · pp. 34–35

Related papers

Loading 2610.18694v1…