PaperlayerView exact revision on arXiv

Source-linked AI summary

GeoScaffold: Learning Compact Geometric Latents via Reconstruction for Efficient Vision-Language Navigation

Yixuan Jiang, Wentong Li, An Liu, Zihao Xin, Fulin Tang, Cong Leng, Yang Gao, Jian Cheng

arXiv:2610.02697v1cs.ROcs.CV

TL;DR

Streaming VLN policies inherit weak geometric reasoning from 2D pretraining, while geometry-aware additions can require inference-time sensors or modules. GeoScaffold trains query states to reconstruct navigation-relevant geometry, then discards the auxiliary modules. The paper reports gains across benchmarks, backbones, and real-world deployment, with an 81 ms per-frame 4B policy that also runs on Jetson.

  • Problem

    Streaming VLN policies inherit weak metric and topological reasoning from 2D pretraining, while geometry-aware extensions can require persistent sensors, encoders, or perception tools at inference.

    Source: §1 · pp. 1–2

  • Method

    GeoScaffold fine-tunes a VLN policy with dedicated query tokens supervised to reconstruct depth, connectivity, and traversability, then discards the auxiliary modules.

    Source: §1 · p. 2, §3.3 · p. 5

  • Results

    Experiments report navigation gains on R2R-CE and RxR-CE, across multiple backbones, and in real-world deployment; the 4B policy averages 81 ms per frame on an RTX 4090 and runs on Jetson.

    Source: §5 · p. 9

  • Takeaways & Limitations

    Geometric reconstruction is supported as a practical training signal for efficient, spatially aware navigation without a resident geometry encoder.

    Source: §5 · p. 9

Abstract

from arXiv · show

Recent vision-and-language navigation (VLN) systems increasingly adopt streaming Video-LLM policies that map egocentric RGB observations and instructions directly to low-level actions. Yet these policies inherit weak 3D geometric priors from 2D pretraining. Existing geometry-aware extensions charge a persistent inference-time price: depth sensors, 3D encoders, or per-step perception tool calls. We propose GeoScaffold, a geometric supervision framework that pays this price once, at training time, by internalizing geometry into the policy itself. It first learns a compact depth tokenizer on depth maps from the training trajectories and freezes it. It then fine-tunes the policy with a handful of learnable geometry query tokens, training their hidden states to reconstruct navigation-critical geometry such as depth, connectivity, and traversability. This supervision turns the query states into compact geometric latents for action decoding, and through the shared weights also internalizes geometry into the backbone's own representations. Like a scaffold, the tokenizer, target generators, and reconstruction heads are discarded after training, leaving the backbone and action interface unchanged. Extensive experiments show that GeoScaffold consistently outperforms leading vision-only navigators on continuous VLN benchmarks, offering a practical paradigm for lightweight edge deployment of spatially aware embodied navigation models.

1 INTRODUCTION

Streaming VLN policies inherit weak metric and topological reasoning from 2D pretraining, while many geometry-aware alternatives retain inference-time machinery. GeoScaffold instead uses training-only geometric reconstruction to shape query states into latents for an RGB-and-text policy.

1 INTRODUCTION

  • 2D-pretrained VLN backbones are strong at semantic recognition but weak at metric and topological reasoning needed for depth-sensitive navigation decisions.Existing geometry-aware methods can add sensors, 3D encoders, or modular pipelines that retain inference-time hardware, latency, memory, or integration costs.Source: §1 · pp. 1–2
  • GeoScaffold supervises dedicated query-token groups to reconstruct depth, connectivity, and traversability, forming compact geometric latents for action decoding.It learns a depth tokenizer offline, then fine-tunes the policy with reconstruction losses alongside the unchanged navigation objective.Source: §1 · p. 2
  • After training, auxiliary modules are discarded, leaving an RGB-and-text deployment path with learned geometry tokens in the prompt.The framework is presented as requiring no depth sensors, 3D encoders, or reconstruction modules at inference.Source: §1 · p. 2
  • The paper reports consistent success-rate gains across backbones and real-world deployment for its training-only geometric supervision approach.The stated contributions also characterize the framework as lightweight and suitable for edge hardware.Source: §1 · p. 2

2 RELATED WORK

Related work develops streaming vision-language-action policies, adds geometry to address their spatial weaknesses, and uses auxiliary perception supervision or query tokens. GeoScaffold applies dedicated query states to reconstruct navigation-relevant geometry during training for RGB-only streaming navigation.

2.1 VLA MODEL FOR NAVIGATION

2.2 GEOMETRY-ENHANCED VLN

  • Geometry-enhanced VLN supplies deployment-time geometry through sensors, feature fields, maps, 3D models, or perception tools, adding latency, memory, or hardware requirements.GeoScaffold instead uses geometry during training and deploys compact latents without an inference-time geometry module.Source: §2.2 · p. 3

2.3 AUXILIARY PERCEPTION SUPERVISION AND QUERY TOKENS

  • Auxiliary perception work uses prediction, reconstruction, feature alignment, or query-based readouts; GeoScaffold reconstructs current-frame depth, connectivity, and traversability from dedicated query states.Related examples include ROSS (Wang et al., 2025a), 3DRS (Huang et al., 2025), and Spatial Forcing (Li et al., 2026a).Source: §2.3 · p. 3

3 METHOD

GeoScaffold adds training-only geometric reconstruction to streaming VLN, using query-token hidden states as compact geometry latents for action decoding. Its two-stage training freezes a depth tokenizer before geometry-scaffolded policy fine-tuning.

3.1 PRELIMINARIES

  • Streaming VLN-CE uses egocentric RGB observations and instructions to predict navigation actions turn by turn.The action space is forward 25 cm, left or right 15°, or stop.Source: §3.1 · p. 3

3.2 MODEL ARCHITECTURE

  • GeoScaffold reverses geometry input fusion: query-token hidden states are trained to predict depth codes and connectivity and traversability masks as compact latents for action decoding.Depth represents metric scale; the masks mark route-relevant passages and free space.Source: §3.2 · p. 4
  • The policy inserts separate learnable query-token groups for depth, connectivity, and traversability before each action block, and action tokens can attend to their hidden states.The query tokens are excluded from language-modeling loss and are never generated as text.Source: §3.2 · p. 4
  • Within each turn, geometry query tokens attend bidirectionally to one another, while all other positions retain causal attention.This avoids imposing an arbitrary order on query slots summarizing the same observation.Source: §3.2 · p. 4

3.3 TWO-STAGE TRAINING STRATEGY

  • A frozen depth tokenizer is trained offline on normalized depth maps from navigation training trajectories before the policy is fine-tuned with geometric supervision.The tokenizer is not exposed to evaluation environments and is frozen after training.Source: §3.3 · p. 5
  • Policy training retains the imitation objective and adds depth-token, decoded-depth reconstruction, connectivity-mask, and traversability-mask losses.The depth reconstruction loss renders predicted code distributions through the frozen decoder; mask losses combine classification and region-overlap terms.Source: §3.3 · pp. 5–6

4 EXPERIMENTS

Experiments use VLN-CE trajectories in Matterport3D, pairing frames with geometry targets used only for training and reporting standard navigation metrics, including SR and SPL. The section compares benchmark performance, inference latency, backbone generality, qualitative behavior, and ablations.

4.2 MAIN RESULTS

  • 81 ms/frame lets GeoScaffold-4B sustain control rates above 12 Hz on one consumer GPU, while geometry-enhanced baselines are 9× to 14× slower.The comparison measures steady-state latency on a single NVIDIA RTX 4090.Source: Figure 4 · p. 6, §4.2 · pp. 6–7
  • GeoScaffold-4B achieves higher SR and SPL than listed 7B VLA systems on both benchmarks; with added data, it reaches 64.8% SR and 60.5% SPL on R2R-CE.Using VLN-CE data alone, GeoScaffold-4B* reaches 55.3% SR and 51.0% SPL on R2R-CE, ahead of JanusVLN-7B* trained on the same data.Source: §4.2 · p. 6, Table 1 · p. 7
  • Across three backbones from 3B to 7B, GeoScaffold improves all four metrics, with SR gains of 5.5–7.9 points and SPL gains of 6.4–8.7 points.The identical training recipe is used without and with GeoScaffold on R2R-CE and RxR-CE data only.Source: Table 2 · p. 7, §4.2 · p. 7

4.3 QUALITATIVE RESULTS

  • In simulation, GeoScaffold follows the hallway to the instructed dining room; in two zero-shot Go2 episodes, it completes geometry-demanding real-world instructions using RGB and language.The real-world instructions test depth-aware reasoning, traversability, and connectivity.Source: Figure 5 · p. 8, §4.3 · p. 8, Figure 6 · p. 8

4.4 ABLATION STUDY

  • Dropping Ltok costs 3.8 SR versus 1.3 for Lrec; replacing VQ codes with continuous latents costs 1.4 SR, and pixel-space regression costs 3.0 SR.The two losses together perform best, and the authors conclude compact, discrete, decodable targets suit a token-based policy.Source: Table 4 · p. 9, §4.4 · p. 9
  • Removing depth supervision causes the largest SR drop, from 55.3 to 51.1; removing traversability or connectivity costs 2.2 or 2.6 SR, showing complementary cues.The masks add passage and free-space cues that depth alone does not capture.Source: §4.4 · pp. 8–9, Table 3 · p. 9
  • Reconstructing from image hidden states causes the largest SR drop (12.1 points), while query tokens without reconstruction reach 47.7 SR.Action-state reconstruction costs 3.5 SR; the authors suggest image-state supervision may degrade visual features used for navigation.Source: Table 5 · p. 9, §4.4 · p. 9
  • Hybrid attention performs best; causal masking costs 1.4 SR, while blocking latents at test time lowers SR from 55.3 to 52.8 and SPL from 51.0 to 47.7.The blocked-latent variant still exceeds the 48.8-SR policy without GeoScaffold, indicating geometry is also internalized in the backbone's representations.Source: Table 6 · p. 9, §4.4 · p. 9

5 CONCLUSION

GeoScaffold internalizes geometry in streaming VLN policies by reconstructing depth, connectivity, and traversability from dedicated query states. Experiments report navigation gains across benchmarks, backbones, and real-world deployment, alongside efficient inference and on-device execution.

5 CONCLUSION

  • GeoScaffold reconstructs depth, connectivity, and traversability from dedicated query states, learning compact geometric latents for action decoding.The auxiliary modules are discarded after training, leaving a policy that navigates from RGB and language without a resident geometry encoder.Source: §5 · p. 9
  • Experiments demonstrate navigation gains on R2R-CE and RxR-CE, across multiple backbones, and in real-world deployment.Source: §5 · p. 9
  • The 4B policy averages 81 ms per frame on an RTX 4090 and also runs on board a Jetson device.The paper concludes that geometric reconstruction is a practical training signal for efficient, spatially aware navigation.Source: §5 · p. 9
Loading 2610.02697v1…