Learning and Reasoning with Geometry: How Generation Helps Spatial Understanding

1 Introduction

• Geometric generation offers supervision over richer scene structure and visual consequences than VQA’s supervision of specific queried relations.At inference, generated geometry can provide explicit depth and distance measurements for spatial judgments.Source: §1 · pp. 1–2
• GeoUG achieves circular accuracies of 97.9% on CV-Bench-3D, 90.8% on 3DCueBench, and 94.2% on DA-2K while using fewer generation calls.Across the three benchmarks, accuracy is close to or above always generating.Source: §1 · pp. 2–3
• GeoUG integrates spatial understanding with metric-depth and view generation, using a multi-scale codec for metric depth and joint training objectives.The introduction reports task-dependent benefits: depth generation primarily improves depth-related abilities, while view generation benefits understanding across viewpoints.Source: §1 · p. 2
• 3DCueBench balances selected 2D cues across geometric difficulty levels, and generated depth yields larger gains when depth or distance gaps are smaller.The benchmark is designed to examine when geometric evidence improves an answer.Source: §1 · pp. 2–3

2 Related Work

• Prior unified multimodal work studies specialized understanding and generation experts, asymmetric data contributions, and cross-modal complementarity; this work focuses on geometric generation and spatial understanding.Examples include BAGEL, MetaMorph, and studies of pretraining synergy.Source: §2 · p. 3
• Generative supervision and generated intermediates have been studied for visual understanding and reasoning, while this work emphasizes geometric difficulty and generation errors.Related directions include reconstruction, semantic proxy tasks, visual reasoning traces, relative-depth generation, and synthesized viewpoints.Source: §2 · p. 3
• Spatial instruction datasets provide selected attributes, objects, and relations, whereas this work adds metric-depth and multi-view generation objectives as dense geometric supervision.The paper also uses generated representations to support spatial reasoning.Source: §2 · p. 3

3 Method

• GeoUG jointly trains spatial understanding with geometric generation, using metric-depth maps or RGB views as targets in a shared flow-matching generation mechanism.The generated target is VAE-encoded, and predicted flow transforms noise into target latents that the VAE decodes.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1 Geometric Generation Tasks

• MMDC maps metric depth to a bounded, three-channel representation compatible with a pretrained VAE and decodes generated outputs back to metric depth.Different channel scales provide complementary precision across depth ranges, addressing the compression of depth differences far from a single pivot.Source: §3.1 · p. 4, Figure 2 · p. 4

3.2 Interleaved Geometric Reasoning

• Depth generation yields larger reasoning gains on difficult questions with smaller relative gaps, while gains on easier questions are limited and may sometimes be negative.The authors suggest generated-depth or geometric-measurement errors may account for the degradation on easier questions.Source: §3.2 · p. 5, Figure 3 · p. 5
• A before-answer policy performs poorly at selecting questions that benefit from generation: at a 50% budget, it retains 55% of always-generation gains, near random selection’s 50%.Reinforcement learning in this setup tended toward generating for almost no questions or almost all questions.Source: §3.2 · pp. 5–6
• At a 50% generation budget, the draft-conditioned policy retains 86% of the gain from always generating, versus 55% for the before-answer policy.Each policy’s retained gain is normalized by its own direct and always-generate accuracies.Source: §3.2 · p. 6
• The draft-conditioned policy first drafts an answer, then decides whether to retain it or generate depth to verify or revise it; reinforcement learning optimizes this decision for the accuracy–cost trade-off.Training cold-starts with supervised fine-tuning before GSPO-based reinforcement learning; the generation decision uses a draft-conditioned probability and an estimate of post-generation correctness.Source: §3.2 · p. 6

4.1 3DCueBench: Controlled Evaluation of Geometric Reasoning

• On cue-conflicting Distance questions, Seed-2.0-Lite reaches 37.0% accuracy and GPT-5.5 reaches 35.7%, illustrating the challenge of 2D shortcuts for 3D reasoning.In CV-Bench-3D, the lower-box-bottom rule correctly answers 85.3% of Depth questions, and no question has a relative gap below 0.16.Source: §4.1 · p. 7
• 3DCueBench contains 800 Depth and Distance questions, balancing cue-consistent and cue-conflicting examples within relative-gap intervals to control geometric difficulty.Each 2D rule achieves 50% accuracy across the benchmark, so strong performance requires going beyond those rules.Source: §4.1 · p. 7

4.2 Evaluation Protocol

• Evaluation covers single-image depth and distance on CV-Bench-3D, DA-2K, and 3DCueBench; multi-view reasoning on MindCube and MMSI-Bench; and dense depth estimation on five datasets.The dense-depth datasets are NYU-v2, KITTI, ETH3D, Sintel, and DIODE, with AbsRel and δ1 reported.Source: §4.2 · p. 7

5.1 Generative Co-training

• Depth, multi-view, and exo-to-ego co-training improve CV-Bench-3D-Visual by 1.50–1.67 points, with task-dependent gains on DA-2K-Visual, MindCube, and MMSI camera motion.Depth gives the largest DA-2K-Visual gain (+2.08 points); multi-view and exo-to-ego improve MindCube by 2.21 and 2.31 points, while depth lowers MMSI camera-motion accuracy.Source: §5.1 · pp. 7–8, Figure 6 · p. 8

5.2 Interleaved Geometric Reasoning

• GeoUG exceeds DA3 by 2.7 points on CV-Bench-3D and 2.1 points on DA-2K, and reaches 90.8% versus 90.9% on 3DCueBench while answering 32.6% of its questions without depth generation.The comparison uses circular accuracy; both option orders must be correct.Source: §5.2 · pp. 8–9, Table 1 · p. 8
• Always-generating depth improves circular accuracy over direct answering by 6.0, 15.9, and 23.2 points across the three benchmarks, with larger gains on harder 3DCueBench intervals.The 3DCueBench gain is 23.0 points in the two hardest gap intervals, versus 12.5 and 5.0 in easier intervals; generation also reduces option-position bias from 9.7% to 0.1%.Source: §5.2 · p. 9

5.3 Adaptive Interleaved Reasoning

• With λ = 0.1, adaptive reasoning generates depth for 50.3–72.4% of questions and stays within 0.7 points of always generating across all three benchmarks.At matched generation rates, GeoUG exceeds random selection by 3.5, 4.7, and 5.7 points on CV-Bench-3D, 3DCueBench, and DA-2K, respectively.Source: §5.3 · pp. 9, 11, Table 2 · p. 11

5.4 Depth Estimation

• GeoUG-Depth has lower AbsRel and higher δ1 than Marigold-v1.1 and Lotus on every dataset with reported results, and beats Depth Anything V2 on both metrics across five benchmarks.MoGe-V2 retains lower AbsRel; MMDC reduces mean AbsRel from 7.47 to 7.24, and more training data lowers it to 6.23 while raising mean δ1 from 92.6% to 93.9%.Source: §5.4 · p. 11, Table 4 · p. 11, Table 5 · p. 12

6 Conclusion

• Geometric generation serves two complementary roles: supervision for learning spatial relations and selectively acquired evidence during reasoning.The paper studies both roles through GeoUG, a unified framework.Source: §6 · p. 12
• Joint training with depth and multi-view generation yields task-dependent gains in spatial understanding.Source: §6 · p. 12
• Adaptive interleaved reasoning uses reinforcement learning to balance accuracy gains over retaining a draft against generation cost.Across three benchmarks, it maintains accuracy close to or above always generating with fewer generation calls.Source: §6 · p. 12

• Predicted depth maps preserve recognizable scene layout and many object-contour boundaries, while 3D reconstructions expose some gaps and boundary distortions.The examples illustrate spatial structure, not quantitative metric calibration.Source: A Qualitative Results · p. 18, Figure 9 · p. 19
• Compared with BAGEL, the co-trained models follow camera-motion instructions and render first-person exo-to-ego views, though fine details and precise viewing directions can differ from ground truth.Examples include moving the camera to reveal new content and rendering a person's viewpoint rather than depicting that person.Source: A Qualitative Results · pp. 18, 21, Figure 10 · p. 20, Figure 11 · p. 21
• In the illustrated adaptive-reasoning cases, generated depth corrects wrong drafts, direct answering handles correct drafts, and a tool-following failure produces incorrect answers.The distance example shows the policy comparing 3D object distances rather than camera depths; another example shows incorrect tool measurements overriding a correct draft.Source: Figure 12 · p. 22, Figure 13 · p. 22, Figure 14 · p. 23

B.1 Depth sensitivity

• Sensitivity peaks at a channel's pivot depth and falls toward smaller and larger depths; geometrically spaced pivots (2, 8, 32) m provide overlapping precision ranges.The pivots are fixed across datasets, and shifting a pivot translates its sensitivity profile along the log-depth axis.Source: §B.1 · p. 24, Figure 15 · p. 24

B.2 Fusion weights

• Under small, independent, zero-mean encoding errors with equal variance, the fusion weights are inverse-variance weights, so higher-sensitivity channels receive greater weight.The practical decoder substitutes clipped predictions for unknown noiseless values; this is a first-order approximation under the stated assumptions, not an exact optimum for arbitrary errors.Source: §B.2 · p. 24

B.3 Implementation details

• The implementation clips predicted channels before inversion, floors fusion weights, and applies a configurable upper bound to decoded depth to keep computation well-defined near saturation.Invalid depth entries are filled with zero while their validity mask is retained separately.Source: §B.3 · p. 24

C Details of Adaptive Interleaved Reasoning

• The before-answer and draft-conditioned comparisons do not isolate decision position because they differ in model, training questions, and credit assignment.Direct and generated answers within each curve do come from the same checkpoint.Source: C Details of Adaptive Interleaved… · p. 25
• The before-answer policy’s generation probability took only two values, and its gate was just 0.6 points above random selection at the same rate.Under reinforcement learning, changing the cost mainly changed the overall generation rate without producing an effective question-selection policy.Source: C Details of Adaptive Interleaved… · p. 25
• Dividing advantages by a constant 0.5 keeps every advantage linear in the generation cost λ, unlike within-group standard-deviation normalization.Groups whose rewards differ only by generation cost then receive proportionally small updates.Source: C Details of Adaptive Interleaved… · p. 25
• With trajectory-level advantages, the generation cost stops pushing the decision when every rollout generates; in initial runs, the rate rose from 0.64 to 0.99 within about ten updates.Drafts also drifted: 18 of 63 lacked an answer letter after eight updates, versus 7 of 64 initially.Source: C Details of Adaptive Interleaved… · pp. 25–26
• Decision-token advantages compare estimated generation value minus cost with direct-answer correctness, while forced rollouts estimate generation value by draft correctness.The pooled forced rollouts approximate conditional generation value but may not give an unbiased estimate for every reasoning prefix.Source: C Details of Adaptive Interleaved… · p. 26
• Draft-conditioned supervised sequences use drafts from the initializing checkpoint; draft tokens receive no loss, and decision supervision labels stopping for correct direct drafts and generating for wrong drafts.Drafts were wrong on about 22% of questions; the resulting set contained 262K questions and 312K sequences after near-ties were removed.Source: C Details of Adaptive Interleaved… · p. 26
• The RL training pool contains 800 questions, balanced across indoor and outdoor sources, four question types, nearer versus farther targets, and answer letters.Questions were selected to include misleading 2D cues for object-to-object comparisons.Source: C Details of Adaptive Interleaved… · p. 27

D.1 Metric Depth Data

• A relative-depth estimator’s dense, sharp predictions are affine-aligned to valid real measurements, producing metric-scale pseudo-labels for subsequent training.The fitted aligned depth combines the metric scale of observed pixels with the teacher’s dense geometry.Source: §D.1 · p. 29

D.2 Multi-view Data

• Pose-derived camera instructions encode rotation and translation after scale unification; motion data are balanced, with single motions introduced before combined motions.The poses from RealEstate10K and DL3DV lack a consistent metric scale, so the authors re-estimate them with DA3 before mapping translations to instructions.Source: §D.2 · pp. 29–32

D.3 Exo-to-Ego Generation Data

• Exo-to-ego pairs link synchronized Ego-Exo4D third-person frames to first-person targets, producing 171,558 instruction pairs across six activity domains.Instructions describe the camera wearer, and pairs with unparsed output, leaked templates, or references to absent markers are removed.Source: §D.3 · p. 33

E.1 Multi-view and Exo-to-Ego Generation

• Exo-to-ego generation is evaluated on 469 synchronized Ego-Exo4D pairs, and multi-view generation on 1,500 pairs from RealEstate10K, DL3DV, and ScanNet++.Generated views are compared with targets using PSNR, SSIM, and LPIPS.Source: §E.1 · pp. 33–34
• On exo-to-ego generation, our model raises BAGEL’s PSNR from 7.75 to 12.41 dB and lowers LPIPS from 0.839 to 0.670; multi-view metrics improve on every source.The largest multi-view gain is on RealEstate10K: +1.25 dB PSNR and −0.071 LPIPS. Absolute metrics remain low because a source image and instruction do not fully determine the target.Source: Table 8 · p. 34, Table 7 · p. 34, §E.1 · p. 34

E.2 3DCueBench

• 3DCueBench contains 400 Depth and 400 Distance questions from ten datasets, with two answer options and unmarked images.The two tasks are sampled, balanced, and audited independently; object points are given as normalized coordinates.Source: §E.2.1 · p. 34, Table 9 · p. 34, Figure 23 · p. 35

E.2.2 Sampling and cue selection

• Each relative-gap interval contains 100 questions per task with indoor and outdoor balance, while paired cue-consistent and cue-conflicting examples share a source and interval.The smallest-gap CA-1M group shows sensitivity to depth convention: all three definitions agree on 44.7% of pairs, versus 75.7% agreement between surface-based definitions.Source: §E.2.2 · p. 35

E.2.3 CV-Bench-3D audit

• The CV-Bench-3D difficulty audit finds no questions with relative gap below 0.16; median gaps are 0.49 for Depth and 0.76 for Distance.The audit matches question boxes to Omni3D annotations from Hypersim, SUN RGB-D, and nuScenes.Source: §E.2.3 · p. 36

E.2.4 Blind-text audit

• Text-only probes reach 48.9%–50.4%, while gradient-boosted trees reach 56.1% on Depth and 53.5% on Distance under scene-level cross-validation.Splitting by image instead of scene raises probe accuracy by 5.5 percentage points; these results measure residual text-only predictability, not an upper bound on shortcuts.Source: §E.2.4 · p. 36

E.2.5 Results and failure analysis

• Depth accuracy decreases as the relative gap shrinks, reaching 75.0% and 62.6% in the hardest interval, without a passage-supported assignment of either value to a specific model.The supplementary results for Seed-2.0-Lite and GPT-5.5 use ordinary rather than question-level circular accuracy.Source: §E.2.5 · p. 36
• On Distance cue-conflicting questions, Seed-2.0-Lite and GPT-5.5 score 37.0% and 35.7%, despite at least 92% on cue-consistent questions.Their predictions agree with the selected rule on 77.5% and 78.8% of questions, respectively; aggregate accuracy obscures this asymmetry.Source: §E.2.5 · p. 36, Figure 24 · p. 37
• When the higher Depth point is nearer rather than farther, accuracy is 82.4% versus 93.5% for Seed-2.0-Lite and 69.0% versus 86.9% for GPT-5.5.The counterexamples show the higher point can be nearer even when both models choose the farther option.Source: §E.2.5 · p. 36, Figure 25 · p. 37

E.2.6 Difficulty analysis

• GeoUG’s margin over VST-7B-SFT grows from 9.0 points in the easiest interval to 23.0 points in the hardest, as VST-7B-SFT falls from 88.0% to 61.0%.GeoUG drops from 97.0% to 84.0%; DA3 answering from its own depth follows a similar trend, with varying margins across intervals.Source: §E.2.6 · p. 37, Table 11 · p. 38
• The λ = 0.1 policy stays within 1.0 point of always generating in every interval, while the same-rate random gate trails it by 2.0–4.6 points.Always generating improves over direct answering by 23.0 points in the two hardest intervals, and by 12.5 and 5.0 points in the easier ones.Source: §E.2.6 · p. 37, Table 12 · p. 38

Generation pretraining alone does not reliably provide a better initialization for downstream

• At the final SFT checkpoint, generation-initialized models trail the original by 0.34 points on CV-Bench-3D, 2.61/3.38 on DA-2K, and 2.70/6.76 on MMSI camera motion.MindCube differs: depth and multi-view initialization retain advantages of 0.87 and 2.02 points, respectively; the sequential and co-training experiments are not fully matched.Source: Generation pretraining alone does not… · p. 39, Figure 26 · p. 39

F.2 Visual and Textual Referring Formats

• Textual referring changes accuracy inconsistently across models; DA-2K comparisons also combine referring-format changes with circular transformations.The paper cautions that cross-run comparisons are descriptive because preprocessing and evaluator-version equivalence has not been established.Source: §F.2 · pp. 39–40, Table 13 · p. 40

F.3 External Depth Tools

• External-depth-tool systems score higher than direct-answer baselines on CV-Bench-3D and 3DCueBench, but not consistently on DA-2K, and remain below GeoUG and DA3-only answers.The comparisons do not isolate tool effects from prompting, resolution, and training differences.Source: Table 14 · p. 40, §F.3 · p. 40