Source-linked AI summary
PanoVLN: Towards Effective Panoramic Vision-and-Language Navigation
Zhen Wang, Changpeng Wang, Zhe Liu, Zhangyang Qi, Yuxiang Lu, Zimo Zeng, Donglian Qi, Xi Chen
TL;DR
Panoramic observations provide more complete visual context for navigation, but simply replacing perspective images with panoramas yields limited gains. PanoVLN adapts action prediction, training supervision, and visual representation; with a 4B backbone, it surpasses prior state of the art on two unseen benchmarks and navigates real-world routes faster with fewer pauses.
Problem
Whether panoramic observations can improve navigation by making more of the surrounding environment available remains an open question, and simply replacing perspective images yields limited gains.
Source: §1 · p. 2Method
PanoVLN combines longer-horizon action prediction with confidence-guided execution, branching-focused training data, and semantic-geometric features from RGB panoramas.
Source: §3.2 · p. 4, §4.4 · pp. 8–10Results
11.9% and 8.7% higher success rates than prior state of the art on R2R-CE and RxR-CE Val-Unseen, respectively; quadruped trials also report faster navigation with fewer pauses.
Source: §1 · p. 2Takeaways & Limitations
The findings suggest that broader visibility should shape how navigation policies learn and act.
Source: §5 · p. 10
Abstract
from arXiv · showhide
Recent vision-language models (VLMs) have advanced vision-and-language navigation (VLN), enabling models to predict navigation actions from visual observations and language instructions. In this work, we explore VLN with panoramic observations and introduce PanoVLN. The motivation is straightforward: more complete visual context should enable better-informed navigation decisions. For example, a panorama can reveal a passage outside a perspective camera's field of view, allowing the model to identify the intended route without additional exploration. However, we find that simply replacing perspective images with panoramas yields only limited gains. Our diagnosis suggests that fully exploiting wider visibility requires modifications to action prediction, training supervision, and visual representation. First, wider visibility supports longer-horizon action planning. We make the model predict longer action sequences, enabling larger turns and subsequent movement from a single panorama. Specifically, we introduce a confidence-guided execution (CGE) strategy that dynamically determines how many predicted actions to execute before replanning. Second, wider visibility also brings more complex route choices. We therefore construct training routes with frequent branching points and clear instructions to provide targeted supervision for route selection. Third, panoramic navigation requires understanding spatial relationships across viewing directions, beyond recognizing individual landmarks. We combine semantic and geometric features from RGB panoramas to capture both scene content and spatial layout without adding visual tokens. With a 4B backbone and RGB-only input, PanoVLN surpasses the previous SOTA by 11.9% and 8.7% in success rate on R2R-CE and RxR-CE Val-Unseen. Real-world experiments on a quadruped further demonstrate faster navigation with fewer pauses than prior VLN methods.
1 INTRODUCTION
PanoVLN adapts action prediction, training supervision, and visual representation because panoramic inputs alone yield limited gains. With these adaptations, it reports higher success rates on two unseen benchmarks and faster real-world navigation with fewer pauses.
1 INTRODUCTION
- Replacing perspective images with panoramas under the same training and inference setup does not improve performance, motivating targeted adaptations in action prediction, supervision, and visual representation.Source: §1 · p. 2
- Longer action predictions exploit routes visible beyond a perspective image’s edges, while confidence-guided execution uses action uncertainty to choose when to reobserve and replan.Source: §1 · p. 2
- The training dataset contains 98K trajectories across 800 HM3D scenes, with frequent branching points, visually grounded instructions, and more sampling around turns and stops.These choices provide targeted supervision for selecting routes and completing them.Source: §1 · p. 2
- PanoVLN fuses semantic features with geometric features from the same RGB panorama, capturing scene content and spatial layout without additional visual tokens or depth input.Source: §1 · p. 2
- 77.3% success rate on R2R-CE Val-Unseen and 78.0% on RxR-CE Val-Unseen exceed prior state of the art by 11.9% and 8.7%, respectively.On a quadruped, PanoVLN also navigates indoor and outdoor routes in less time, with fewer policy calls and pauses than prior VLN baselines.Source: §1 · p. 2
2 RELATED WORK
Prior VLN work spans waypoint and map-based planning, VLM-based action prediction, and improvements to supervision and data. PanoVLN studies full-surround observations and brings panoramic geometry into this navigation setting.
Vision-and-language navigation
- VLN-CE methods predict reachable locations, maintain spatial representations, or look ahead along candidate routes; VLM policies use visual histories or video to predict commands.Source: §2 · p. 3
- Other work improves waypoint supervision, navigation-data scale, and instruction generation, while PanoVLN builds on these advances to study full-surround visual observations.Source: §2 · p. 3
Panoramic geometry
- Panoramic geometry methods estimate depth or 3D structure, and navigation methods represent spatial layout with maps, memories, or lookahead features; PanoVLN brings panoramic geometry into VLN.Source: §2 · p. 3
3 METHOD
PanoVLN adapts panoramic navigation through longer action-sequence supervision and uncertainty-guided execution, decision-centric training data, and geometry-aware visual features. These changes target limitations of simply replacing perspective inputs with panoramas.
3.1 PRELIMINARIES
- Replacing perspective images with current and historical RGB panoramas while retaining the same setup yields limited gains and can reduce performance.Source: §3.1 · p. 3
3.2 LONGER ACTION-SEQUENCE SUPERVISION AND EXECUTION
- Longer action targets use panoramic route context, while CGE selects an execution prefix using cumulative uncertainty and a minimum-action constraint.Mean uncertainty rises after an initial dip and varies across policy calls.Source: §3.2 · pp. 3–4
3.3 DECISION-CENTRIC DATA CONSTRUCTION
- 98K trajectories across 800 HM3D scenes are constructed to include frequent branching points, with routes retained when the chosen path and alternatives are visible.Source: §3.3 · pp. 4–5
- Instructions are generated from route-event segments and verified for movement, choice, and stop grounding; sampled states emphasize sustained turns and termination.Source: §3.3 · p. 5
3.4 GEOMETRY-AWARE VISUAL REPRESENTATION
- Aligned geometric features extracted from the current RGB panorama are projected into visual-token space and fused with semantic features without adding tokens.Source: §3.4 · pp. 5–6
- PanoVLN achieves the highest SR and SPL on both R2R-CE and RxR-CE Val-Unseen benchmarks.Source: Table 1 · p. 6
4 EXPERIMENTS
Experiments evaluate RGB-only PanoVLN on unseen R2R-CE and RxR-CE splits, in quadruped trials, and through ablations of its training and execution choices. The reported results include higher benchmark success rates, real-world route transfer and efficiency, and benefits from decision-focused data, longer ERP horizons, confidence-guided execution, and panoramic geometry.
4.1 EXPERIMENTAL SETUP
- The evaluation uses RGB-only observations, with R2R-CE and RxR-CE Val-Unseen metrics and quadruped trials without scene-specific fine-tuning.The default model uses a 4B backbone and a frozen PanoVGGT encoder.Source: §4.1 · p. 6, Figure 5 · p. 7
4.2 SIMULATION EXPERIMENTS
- PanoVLN reaches 77.3% SR on R2R-CE and 78.0% SR on RxR-CE, exceeding previous best results by 11.9 and 8.7 percentage points, respectively.The evaluation uses the Val-Unseen splits.Source: §4.1 · p. 6, §4.2 · p. 7
4.3 REAL-WORLD EXPERIMENTS
- PanoVLN maintains instruction-to-route grounding across indoor and outdoor settings and achieves the best overall navigation efficiency among compared methods.Under synchronous execution, CGE selects a confident action prefix before requesting a new observation, reducing interruptions and total navigation time.Source: Table 2 · p. 8, §4.3 · p. 8
4.4 ABLATION STUDIES
- Perspective performance peaks at H = 4, while ERP favors H = 18; longer ERP targets supervise route choices visible in the panorama.ERP performance improves from H = 8 to H = 18 with execution fixed at six actions.Source: Figure 7 · p. 8, §4.4 · p. 8
- CGE outperforms random prefixes and all fixed execution strategies, including one-action execution, by adapting when to reobserve to each state.Long targets teach route selection, while CGE determines when to reobserve.Source: Table 4 · p. 9, §4.4 · p. 9
- Among the compared geometric encoders, PanoVGGT yields the highest success rates on both benchmarks, complementing semantic appearance with spatial cues for route selection.Alternative encoders have mixed effects.Source: Table 5 · p. 9, §4.4 · p. 9
- Turn- and termination-aware sampling gives more supervision to transitions and stopping; comparable OS and higher SR indicate more reliable goal-region termination.The sampling emphasizes sustained-turn onsets and states near termination.Source: §4.4 · pp. 8–9, Table 3 · p. 8
- Our decision-rich routes and clear instructions yield further gains across training scales beyond the gains from regenerating instructions on existing routes.The data comparison uses matched trajectory counts.Source: Table 3 · p. 8, §4.4 · p. 10
5 CONCLUSION
PanoVLN combines panoramic-specific action supervision, confidence-guided execution, decision-centric training, and geometric features, and reports strong benchmark and real-world navigation results. The authors also disclose language and layout editing and describe verification of generated training instructions.
5 CONCLUSION
- PanoVLN combines longer action supervision, confidence-guided execution, decision-centric training, and geometric features for RGB-panorama navigation.The conclusion argues that broader visibility should shape how navigation policies learn and act.Source: §5 · p. 10
- With a 4B backbone, PanoVLN achieves state-of-the-art success rates on R2R-CE and RxR-CE and faster real-world navigation with fewer pauses.The conclusion reports the outcomes without giving benchmark values.Source: §5 · p. 10
A THE USE OF LARGE LANGUAGE MODELS
- GPT-5.6 and GPT-6 edited manuscript language and layout, while the authors reviewed edits and retained responsibility for technical content, results, and conclusions.Qwen3.8-27B generated dataset instructions from trajectory videos and compass images; the authors verified, revised, or discarded samples based on route and stopping descriptions.Source: A THE USE OF LARGE LANGUAGE MODELS · p. 15
B.1 MODEL AND TRAINING SETTINGS
This appendix section points to the default model and training settings in Table 6 and the training and inference prompt in Figure 9. The prompt uses H = 18, and its italic fields are inputs.
B.1 MODEL AND TRAINING SETTINGS
- Table 6 lists the default PanoVLN model and training settings.Its caption specifies that image dimensions are reported as width × height.Source: §B.1 · p. 15, Table 6 · p. 15
- Figure 9 presents the training and inference prompt with H = 18.Italic fields are inputs to the prompt.Source: §B.1 · p. 15, Figure 9 · p. 15
- Figure 9 omits history when it is unavailable.Source: Figure 9 · p. 15
B.2 REAL-WORLD EXECUTION METRICS
The appendix defines trial-level measures of navigation duration, speed, waiting, pauses, calls, and inference latency, and explains how their means are aggregated. It also identifies the evaluation protocol used by the additional training-source and backbone ablations.
B.2 REAL-WORLD EXECUTION METRICS
- Time includes action execution and policy-response waiting, while Speed measures traveled distance per elapsed time with waiting included.Together, these measures reflect robot motion and interruptions during a trial.Source: §B.2 · p. 16
- Wait measures the share of trial time spent in response intervals, whereas Pauses counts stationary intervals longer than one second.A response interval contributes to Wait even if it does not create a pause exceeding the threshold.Source: §B.2 · p. 16
- Calls counts requests, while Latency measures server-side inference time; Wait intervals also include network communication.The measured mean network overhead is 272 ms per call.Source: §B.2 · p. 16
- The real-world analysis averages trial-level metrics across trials, while request-level latency is first averaged within each trial.Speed and Wait are computed per trial before averaging; integer counts for Pauses and Calls can therefore have fractional reported means.Source: §B.2 · p. 16
C MORE ABLATION STUDIES
- The additional training-source and backbone studies use the R2R-CE and RxR-CE Val-Unseen evaluation protocol.They complement the study in Section 4.4.Source: C MORE ABLATION STUDIES · p. 17
C.1 TRAINING DATA COMPOSITION
The training-source ablation finds gains from DAgger and further gains from the constructed dataset, while the backbone comparison favors Qwen3.5-4B modestly across reported metrics. The experiments evaluate both choices on R2R-CE and RxR-CE Val-Unseen.
C.1 TRAINING DATA COMPOSITION
- Adding DAgger to R2R-CE and RxR-CE training raises SR by 5.3 and 7.0 points and SPL by 4.2 and 6.7 points, respectively.This configuration corresponds to PanoVLN† in Table 1.Source: §C.1 · p. 17, Table 7 · p. 17
- Adding the constructed dataset after DAgger further improves SR by 3.4/3.9 points and SPL by 2.7/2.0 points on R2R-CE/RxR-CE, respectively.RxR-CE nDTW also increases from 71.4 to 73.3, indicating closer agreement with the instructed route alongside better completion.Source: §C.1 · p. 17, Table 7 · p. 17
- The cumulative comparison finds complementary gains from the two added training sources; Figure 8 compares data-construction strategies at matched trajectory counts.Source: §C.1 · p. 17
C.2 VISION-LANGUAGE BACKBONE
- Replacing Qwen3-VL-4B with Qwen3.5-4B improves SR by 1.1 and 2.3 points on R2R-CE and RxR-CE, respectively, and improves the remaining reported metrics.With Qwen3-VL-4B, SR is 76.2% and 75.7%, and SPL is 70.4% and 64.9%, on R2R-CE and RxR-CE, respectively; its strong results show PanoVLN is not confined to one backbone.Source: §C.2 · p. 17, Table 8 · p. 17
D TRAINING DATA ANALYSIS
Reference routes often contain substantial turns, and panoramic views retain visual context for directions beyond a forward-facing camera’s field of view.
D TRAINING DATA ANALYSIS
- 53.3% of reference-path turns of at least 30° fall in the 45°–180° categories across 54,457 turns.The pooled R2R-CE and RxR-CE distribution uses 15° angle categories.Source: Figure 10 · p. 18, D TRAINING DATA ANALYSIS · p. 18
- With a forward 90° view aligned to the incoming path, outgoing directions beyond 45° lie outside the view, while ERP retains them before rotation.Source: D TRAINING DATA ANALYSIS · p. 18
E QUALITATIVE RESULTS
The qualitative results illustrate panoramic attention, real-world trials, and example routes from both benchmark environments.
E.1 PAIRED PANORAMIC ATTENTION CASES
- In paired attention cases, PanoVLN highlights the left hallway and floor beside a stair railing, while the baseline emphasizes the floor ahead or corridor.Source: §E.1 · p. 19
E.2 REAL-WORLD NAVIGATION CASES
- Real-world cases cover Hallway, Office, and Campus trials, with an office instruction to turn around and approach a potted tree.Rows show instructions, successive observations, and route annotations; the display includes third-person and first-person ERP views.Source: Figure 12 · p. 20, §E.2 · p. 20
E.3 R2R-CE NAVIGATION CASES
- R2R-CE Val-Unseen cases pair four panoramic observations with trajectory locations and distinguish executed routes in blue from reference routes in green.Source: Figure 13 · p. 21
E.4 RXR-CE NAVIGATION CASES
- RxR-CE Val-Unseen cases link four panoramic observations per case to trajectory locations, with blue and green marking executed and reference routes.Instruction excerpts retain their original wording, with ellipses marking omissions.Source: Figure 14 · p. 22