Source-linked AI summary
Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models
Hongxing Li, Yixin Li, Dingming Li, Zixuan Wang, Yuchen Yan, Wenqi Zhang, Weiming Lu, Yongliang Shen
TL;DR
Spatial reasoning remains difficult for VLMs because RGB inputs do not directly provide geometric evidence, while existing approaches either add inference-time 3D costs or train without geometric supervision. GPD routes question-relevant 3D cues to a teacher during on-policy self-distillation and applies privileged KL only to incorrect trajectories, retaining RGB-only deployment. Across spatial benchmarks, GPD outperforms GRPO and answer-privileged OPSD, including a 37.6 average on four benchmarks with the 4B backbone.
Problem
RGB inputs do not directly provide geometric evidence, so VLMs must infer depth and layout from appearance while spatial judgments remain difficult.
Source: §1 · p. 1Method
GPD routes textual depth, semantic, and BEV cues with the reference answer to an OPSD teacher, applying privileged token-level KL only to incorrect trajectories while keeping deployment RGB-only.
Source: §1 · pp. 1–2Results
37.6 average across MindCube, SPAR, MMSI, and ViewSpatial; GPD outperforms GRPO and answer-privileged OPSD on the 4B backbone.
Source: §4.2 · p. 6Takeaways & Limitations
Question-routed geometric privilege is effective supervision for spatial reasoning across model scales and benchmarks while the deployed model remains RGB-only.
Source: §5 · p. 7
Abstract
from arXiv · showhide
Spatial reasoning remains a persistent weakness of vision-language models (VLMs), because RGB inputs do not directly provide geometric evidence. Existing remedies either inject 3D into the model at inference, paying architecture and latency costs, or train with outcome rewards that supervise only the final answer. Spatial errors originate in perception: a misjudged depth or direction can be corrected only by the scene's true geometry, which the 3D-scanned sources of spatial training corpora already provide. We propose GPD (Geometry-Privileged Distillation), which makes geometric evidence the privilege in on-policy self-distillation (OPSD). For each question, depth, semantic, and bird's-eye-view (BEV) cues are rendered as compact text and routed to the teacher alongside the reference answer; a privileged KL, applied only to incorrect trajectories, augments GRPO, and the deployed model remains RGB-only. On the 4B backbone, GPD achieves 57.1 on VSI-Bench and 37.6 average across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, outperforming both GRPO and answer-privileged OPSD across spatial reasoning benchmarks. Ablations confirm the complementarity of 3D and answer privilege, the advantage of question-conditioned routing over full-context injection, and the benefit of restricting distillation to incorrect trajectories.
1 Introduction
RGB-only inputs leave VLMs to infer depth and layout from appearance, while existing approaches either add inference-time 3D costs or train without geometric supervision. GPD routes question-relevant 3D cues to a teacher during training and improves spatial-benchmark performance while retaining RGB-only inference.
1 Introduction
- RGB frames provide no direct geometric evidence, leaving VLMs to infer depth and layout from appearance while spatial judgments remain difficult.Training-based methods preserve RGB-only inference but use final-answer supervision that does not localize or correct errors such as wrong depth order or flipped left–right relations.Source: §1 · p. 1
- Spatial training corpora derived from 3D scans provide depth, semantics, and layout as training-time privilege, though these signals cannot be deployment inputs.The motivation is to turn available geometry into supervision that an RGB-only model can internalize.Source: §1 · p. 1
- GPD routes question-relevant depth, semantic, and BEV cues to an OPSD teacher, which re-scores RGB-and-question student trajectories token by token.The privileged KL is applied only to incorrect trajectories, and the deployed model remains RGB-only.Source: §1 · pp. 1–2
- 57.1 on VSI-Bench and 37.6 average across MindCube, SPAR, MMSI, and ViewSpatial; on the 4B backbone, GPD outperforms GRPO and answer-privileged OPSD.The evaluation uses Qwen3-VL-Instruct backbones at 2B and 4B scales across five spatial reasoning benchmarks.Source: §1 · p. 2
2 Related Work
Prior work addresses spatial reasoning through explicit 3D augmentation or training-based adaptation. GPD connects these directions by using question-routed 3D cues as teacher-side training privilege, without changing the student architecture or inference inputs.
Spatial Reasoning in VLMs
- Explicit 3D methods add geometric encoders, representations, or inference-time tools, while training-based methods use supervised fine-tuning or outcome-based reinforcement learning.GPD uses teacher-side 3D cues to provide geometry-aware token-level supervision while leaving student architecture and inference inputs unchanged.Source: §2 · p. 2
On-policy Self-distillation
- OPSD re-scores student rollouts token by token using a teacher conditioned on privileged information; GPD instead routes textual depth, semantic, and BEV cues and distills only incorrect trajectories.Other multimodal OPSD variants use privileged crops, zoomed views, or visual-thought traces.Source: §2 · p. 2
3 Method
GPD uses question-relevant 3D cues as offline training-time privilege while keeping the deployed model’s inference interface unchanged. It combines GRPO with token-level teacher supervision selectively applied to incorrect student trajectories.
3.1 Overview of GPD
- GPD trains the same VLM as an RGB-and-question student and a privileged teacher, jointly optimizing GRPO with token-level KL supervision on incorrect trajectories.The teacher re-scores the student’s own sampled trajectory; the student’s inference interface remains unchanged.Source: §3.1 · p. 2, Figure 2 · p. 3, §3.3 · p. 4
3.3 Geometry-Privileged Self-Distillation
- The student samples on-policy responses from RGB observations and the question, while the privileged teacher scores tokens along those responses for distillation.The distillation loss is gated to incorrect trajectories, and its aggregate is clipped at 1.5 before scaling.Source: §3.1 · p. 2, §3.3 · p. 4
3.2 Offline 3D Privilege Construction
- Offline, GPD constructs textual depth, semantic, and BEV summaries, then routes a question-relevant subset alongside the reference answer to form the teacher’s privilege.Depth records distances and near–far order, semantic cues list objects and image regions, and BEV cues describe top-down relations; BEV is included when reliable poses and layout information are available.Source: §3.2 · p. 3
4 Experiments
Across five spatial benchmarks, GPD improves over controlled GRPO and answer-privileged OPSD; ablations test its privilege design, and further analyses examine training dynamics and cue internalization.
4.1 Experimental Setup
- Evaluation spans five benchmarks, with training on VSI, SPAR, and MindCube samples and shared data and precomputed privilege across methods and ablations.The benchmarks cover video spatial subtasks, mental spatial modeling, 3D object relations, multi-image spatial intelligence, and perspective-dependent understanding.Source: §4.1 · p. 5
4.2 Main Results
- 57.1 on VSI-Bench and 37.6 average across four additional benchmarks make GPD the strongest controlled 4B variant, ahead of GRPO and OPSD.At 2B, GPD also leads the controlled averages on VSI-Bench (51.7) and the four additional benchmarks (33.6).Source: §4.2 · pp. 5–6
4.3 Ablation Study
- 41.5 Avg. with answer plus routed 3D text exceeds answer-only 40.2, 3D-only 40.5, and full-text injection 39.8.Routed images score 40.4, 1.1 points below routed text; the authors attribute the difference to structured text aligning cues with symbolic reasoning.Source: Figure 4 · p. 6, §4.3 · p. 6
- 41.5 Avg. occurs at λ = 0.003 with incorrect-only KL; all-trajectory KL gives 40.6, while smaller and larger weights underperform.Incorrect-only variants score 39.6 at λ = 0.001, 40.2 at λ = 0.004, and 39.6 at λ = 0.005.Source: Figure 4 · p. 6, §4.3 · p. 6
4.4 Further Analysis
- Routed text alone beat RGB-only zero-shot accuracy (52.2 vs. 49.1), while routing maps assign cues according to VSI task category.Counting routes to semantics, absolute distance predominantly to depth, room size to BEV, and relative direction to both semantics and BEV.Source: §4.4 · p. 6, Figure 5 · p. 7
- GPD gradually pulls ahead in validation accuracy as GRPO plateaus, with GPD reaching a higher policy-entropy peak earlier in training.The authors interpret the early entropy peak as broader exploration and report that GPD maintains its accuracy advantage when entropy later aligns.Source: §4.4 · p. 6, Figure 6 · p. 7
- Under RGB-only inference, GPD scores 56.3 versus GRPO’s 54.9 and gains less from test-time routed text (+0.6 vs. +2.1).The authors take the smaller privilege gap as evidence that GPD internalizes geometric cues during training.Source: Figure 7 · p. 7, §4.4 · p. 7
- In a representative multiview case, GPD localizes the camera and fountain correctly, while GRPO misjudges the fountain’s position relative to the jar.The authors connect the GRPO error to outcome-only supervision leaving a geometrically incorrect intermediate judgment uncorrected.Source: Figure 8 · p. 7, §4.4 · p. 7
5 Conclusion
GPD uses question-routed geometric cues and reference answers to guide token-level correction of incorrect trajectories, while retaining RGB-only inference. Across model scales and benchmarks, it improves overall accuracy over GRPO and outperforms answer-privileged OPSD.
5 Conclusion
- GPD augments GRPO with an OPSD teacher conditioned on question-routed depth, semantic, and BEV text cues plus the reference answer, applying token-level KL only to incorrect trajectories.The student samples from RGB observations and retains RGB-only inference.Source: §5 · p. 7