Source-linked AI summary
VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments
Haolin Yang, Yuxing Long, Zihan Yang, Hao Dong
TL;DR
Continuous-environment instruction generation from dense ego-centric RGB streams lacks explicit trajectory structure, unlike graph-based settings. VTInstructor uses visual trajectory prompts to make path geometry explicit and achieves state-of-the-art results across R2R-CE and RxR-CE, with gains of +0.357 and +0.109 CIDEr over the strongest baselines.
Problem
Continuous-environment instruction generation remains largely unexplored because dense ego-centric RGB streams make trajectory cues difficult to recover.
Method
VTInstructor converts trajectory geometry into explicit visual prompts by selecting navigation-critical keyframes and overlaying path, turn, and goal cues.
Results
+0.357 CIDEr on R2R-CE and +0.109 CIDEr on RxR-CE over the strongest baselines accompany state-of-the-art performance across all standard NLG metrics.
Takeaways & Limitations
Frozen followers guided by VTInstructor-generated instructions improve success by 14.7 percentage points, while downstream augmentation yields +3 SR points on both benchmarks.
Takeaways & Limitations
VT-GRPO optimizes automatic NLG metrics rather than directly optimizing navigational success rate.
Abstract
from arXiv · showhide
Navigation instruction generation from ego-centric RGB video in continuous environments is an important yet challenging task for human-robot interaction and scalable dataset construction. Prior instruction generators assume discrete viewpoint graphs with panoramic observations, where trajectory structure is explicit; in continuous environments, however, the agent receives only a dense RGB stream, making trajectory cues difficult to recover. We propose VTInstructor, the first VLN instruction generation framework for continuous environments. Our key idea is to convert implicit trajectory geometry into explicit visual trajectory prompts: EDTC condenses long RGB trajectories into navigation-critical keyframes, VTP overlays path, turn, and goal cues onto these anchors, VTMod injects the resulting trajectory signals into the visual encoder, and VT-GRPO further calibrates this spatial injection during training, all without requiring a navigation graph, pre-built map, or scene reconstruction. On the challenging R2R-CE and RxR-CE Val Unseen benchmarks, VTInstructor sets a new state of the art across all standard NLG metrics, surpassing the strongest baseline by +0.357 CIDEr and +0.109 CIDEr, respectively. Beyond automatic metrics, VTInstructor-generated instructions raise a frozen follower's success rate to 63.3%, a +14.7 percentage-point gain over the best competing instruction source, and provide consistent data augmentation gains of +3 SR points on downstream navigation tasks.
1 Introduction
VTInstructor addresses the difficulty of generating navigation instructions from dense ego-centric RGB streams in continuous environments by converting implicit trajectory geometry into explicit visual trajectory prompts. It achieves state-of-the-art instruction-generation results and improves downstream follower success and data augmentation on R2R-CE and RxR-CE.
- Motivation: Navigation instruction generation supports human-robot interaction and scalable training-data construction, but prior speaker models primarily target discrete graph-based environments.In those settings, trajectories use topological graphs with panoramic observations that make viewpoint relations and trajectory structure explicit.
- Contribution: VTInstructor is the first VLN instruction-generation framework for continuous environments, using ego-centric RGB trajectories and action sequences without a navigation graph or pre-built map.The model receives only RGB frames as visual input and processes the pipeline within a single vision-language backbone.
- Method: VTInstructor converts trajectory geometry into explicit visual prompts by condensing trajectories into keyframes and overlaying path, turn, and goal cues.Quality filtering retains spatially reliable instruction-keyframe pairs, while VTMod injects trajectory signals into the ViT encoder and VT-GRPO calibrates that injection through reinforcement learning.
- Results: +0.357 CIDEr and +0.109 CIDEr are the respective gains over the strongest baseline on R2R-CE and RxR-CE Val Unseen.VTInstructor achieves state-of-the-art performance across BLEU, METEOR, ROUGE-L, CIDEr, and SPICE.
2 Related Work
Prior VLN instruction-generation methods support data augmentation and communication but are largely developed for discrete navigation graphs with privileged panoramic observations. Visual prompting has provided task-relevant spatial annotations across vision-language and robotic manipulation tasks, while continuous-environment speaker models remain unexplored.
- VLN Instruction Generation: Instruction generation complements VLN instruction following by supporting data augmentation and human–robot communication.Speaker-Follower first trained an LSTM speaker for data augmentation, with later work improving generation through speaker–follower cycle consistency.
- Continuous-Environment Gap: Existing instruction-generation methods are developed mainly in discrete navigation graphs with privileged panoramic observations rather than raw first-person continuous perception.R2R-style VLN viewpoints commonly provide 36 discretized views, whereas R2R-CE and RxR-CE extend benchmarks to the continuous Habitat simulator.
- Visual Prompting: Visual prompting steers model attention by augmenting images with task-relevant annotations, including circles, numbered markers, arrows, keypoints, and trajectory traces.These techniques have been applied to referring expression comprehension, visual grounding, action prediction, and robotic manipulation.
3 Method
VTInstructor converts dense ego-centric RGB trajectories into explicit spatial cues through event-driven compression, VTP overlays, spatial visual-token modulation, and calibrated training. The method preserves navigation semantics while aligning trajectory signals with visual patches and language generation.
- Method overview: VTInstructor uses four components: event-driven trajectory compression, VTP rendering and data curation, VTMod injection, and VT-GRPO calibration.These components convert implicit trajectory geometry in dense RGB streams into explicit spatial cues.
- Problem formulation: The agent acts with four primitives—forward +0.25 m, turn left −30◦, turn right +30◦, and stop—while receiving ego-centric RGB observations at each timestep.The generated instruction must describe the trajectory faithfully enough for a human or autonomous follower to reproduce the route.
- Event-driven trajectory compression: RLE compresses consecutive identical actions into physical displacements or rotations, preserving navigational semantics exactly while producing K≪T segments.Forward segments exceeding 4.0 m are split into shorter segments of approximately equal length.
- Event-driven trajectory compression: When more than Fmax = 32 frames remain, DPC retains visually or geometrically significant event frames while preserving the initial and current observations.The final set includes f1, fT, and selected event-boundary frames.
- VTMod injection: VTMod injects VTP features into ViT layer 7 through one-to-one patch-aligned spatial modulation, with a near-zero-initialized gate preserving pretrained visual priors at training onset.The VTP Encoder produces spatial features aligned to ViT patch tokens, while the lazy gate prevents catastrophic forgetting during early SFT.
- VTP rendering: Each retained keyframe receives a three-channel binary VTP mask encoding a path ribbon, rotation arrow, and endpoint marker.Independent binary channels avoid colour ambiguity and let the encoder learn channel-specific spatial patterns.
4 Experiments
Experiments show that VTInstructor substantially improves instruction generation on continuous-environment R2R-CE and RxR-CE, with gains supported by ablations, frozen-follower evaluation, and downstream data augmentation. Its generated instructions achieve navigation utility comparable to human references and improve follower training.
- Comparison with existing models: VTInstructor achieves CIDEr 0.560 on R2R-CE, surpassing Gemini-3.1-Pro-Preview’s 0.203 by +0.357.BLEU-4 is 0.320 versus Gemini’s 0.166.
- Comparison with existing models: On RxR-CE, VTInstructor reaches BLEU-4 0.308 versus MapInstructor’s 0.159 and CIDEr 0.142 versus 0.057.The passage attributes the stronger advantage to VTP-based spatial grounding for longer, more spatially detailed instructions.
- Ablation studies: Adding VTMod raises BLEU-4 from 0.288 to 0.308 (+0.020) and SPICE from 0.223 to 0.238 (+0.015).The component-wise ablation evaluates VTMod, EDTC, and quality filtering on R2R-CE Val Unseen.
- Ablation studies: Gated patchwise projection outperforms cross-attention injection at BLEU-4 0.300 versus 0.295, while VT-GRPO improves BLEU-4 from 0.314 to 0.320.The results indicate that preserving spatial locality and using the gate contrastive reward improve spatial grounding.
- Downstream navigation utility: VTInstructor instructions achieve SR 63.3 and NE 4.47 m with a frozen CorrectNav follower, compared with human-written SR 61.6 and NE 4.53 m.The generated instructions closely match human SPL: 52.7 versus 53.3.
- Downstream data augmentation: Augmenting human training data with 20K VTInstructor instructions improves SR by +3 percent on both benchmarks.NE also decreases by 0.35 m on R2R-CE and 0.60 m on RxR-CE Val Unseen.
5 Limitations and Future Work
VT-GRPO is optimized entirely with automatic NLG metrics against reference instructions, so it is not directly optimized for navigation success. Future work could add sparse feedback from a lightweight frozen follower to better connect instruction quality with downstream navigation.
- Limitations: VT-GRPO uses only automatic NLG metrics against reference instructions as its reward, rather than directly optimizing navigation success rate.This avoids the prohibitive cost of running a navigation follower in the loop.
- Future Work: Future work could add sparse success/failure feedback from a lightweight frozen follower on a small trajectory buffer as an additional reward term.This would more directly bridge instruction quality and downstream navigation.