Source-linked AI summary
BEVBert: Multimodal Map Pre-training for Language-guided Navigation
Dong An, Yuankai Qi, Yangguang Li, Yan Huang, Liang Wang, Tieniu Tan, Jing Shao
TL;DR
Existing VLN pre-training relies on discrete panoramas whose incomplete and duplicated observations can impair spatial understanding. BEVBert instead pre-trains on hybrid maps combining local metric detail with global topological structure, and it achieves state-of-the-art results across four VLN benchmarks. The method is described for discrete environments but is also generalized to continuous VLN.
Problem
Discrete panoramas require implicit correlation of incomplete and duplicate observations, which may hamper an agent’s cross-modal spatial reasoning.
Method
BEVBert uses a local metric map for short-term reasoning and a global topological map for long-term planning, then learns multimodal map representations through map-based pre-training.
Results
BEVBert achieves state-of-the-art performance on four VLN benchmarks, including 73 SR on R2R, 59 SR on R2R-CE, and 54.2 SDTW on RxR test-unseen splits.
Takeaways & Limitations
The learned multimodal map representations enhance spatial-aware cross-modal reasoning and facilitate language-guided navigation.
Takeaways & Limitations
The technical description focuses on VLN in discrete environments, although the method is also reported to generalize to continuous environments.
Abstract
from arXiv · showhide
Large-scale pre-training has shown promising results on the vision-and-language navigation (VLN) task. However, most existing pre-training methods employ discrete panoramas to learn visual-textual associations. This requires the model to implicitly correlate incomplete, duplicate observations within the panoramas, which may impair an agent's spatial understanding. Thus, we propose a new map-based pre-training paradigm that is spatial-aware for use in VLN. Concretely, we build a local metric map to explicitly aggregate incomplete observations and remove duplicates, while modeling navigation dependency in a global topological map. This hybrid design can balance the demand of VLN for both short-term reasoning and long-term planning. Then, based on the hybrid map, we devise a pre-training framework to learn a multimodal map representation, which enhances spatial-aware cross-modal reasoning thereby facilitating the language-guided navigation goal. Extensive experiments demonstrate the effectiveness of the map-based pre-training route for VLN, and the proposed method achieves state-of-the-art on four VLN benchmarks.
1. Introduction
BEVBert addresses spatial limitations of discrete-panorama VLN pre-training with a hybrid map that combines local metric reasoning and global topological planning. The resulting map-based pre-training learns spatially aware multimodal representations and achieves state-of-the-art results on four VLN benchmarks.
- Motivation: Discrete panoramas contain incomplete and duplicate observations, forcing implicit spatial correlation that may hamper cross-modal reasoning.Duplicate bedroom and bookcase images can obscure whether observations refer to the same object or multiple instances.
- Motivation: Metric maps provide precise local scene features but scale inefficiently, whereas topological maps support long-term planning while losing fine-grained spatial information.Large metric maps can make long-horizon modeling computationally prohibitive, while condensed topological nodes lack local detail.
- Approach: BEVBert combines a local metric map for short-term reasoning with a global topological map for long-term action planning.The hybrid design explicitly balances the complementary demands of local spatial understanding and global navigation dependency.
- Approach: The method introduces a map-based pre-training paradigm that learns spatial-aware cross-modal representations from hybrid maps.The contribution is framed as the first exploration of topo-metric maps for VLN pre-training.
- Results: BEVBert achieves state-of-the-art performance on four VLN benchmarks, including 73 SR on R2R, 59 SR on R2R-CE, and 54.2 SDTW on RxR test-unseen splits.These figures are representative results reported in the introduction.
2. Related Work
VLN research has progressed from recurrent action models to transformer-based multimodal pre-training and memory-based methods. Related navigation work exposes a complementary trade-off: metric maps preserve local detail but scale poorly, while topological maps improve long-term planning at the cost of fine-grained reasoning.
- Vision-and-Language Navigation: Early VLN methods used sequence-to-sequence LSTMs, attention mechanisms, reinforcement learning, and data augmentation to improve action prediction and generalization.Later transformer-based models learned generic multimodal representations and incorporated recurrent state, episodic memory, or topology memory.
- Vision-and-Language Navigation: Discrete panoramas require implicit spatial modeling and may hamper learning of generic language-environment correspondence.This limitation motivates multimodal topo-metric maps in the paper.
- Maps for Navigation: Metric maps precisely represent scene layouts but are scale-inefficient and may inadequately support long-term modeling when map size is restricted.Restricting map size reduces computation but can limit long-term planning capacity.
- Maps for Navigation: Topological maps reduce computational demands and support long-term planning, but condensed graph nodes lack fine-grained information for short-term spatial reasoning.Topo-metric maps are presented in robotics as a way to trade off these limitations.
3. Method
BEVBert constructs a hybrid topo-metric map and learns multimodal representations for VLN, combining global planning with local spatial reasoning. Its encoders fuse instructions with graph nodes and metric-map cells before navigation-oriented pre-training and fine-tuning.
- Method Overview: The framework constructs an offline hybrid map from sampled expert paths, learns multimodal map representations with three pre-training tasks, then fine-tunes with online maps.The mapping module builds the hybrid map, while the learning module performs map-instruction interaction before sequential action prediction fine-tuning.
- Topo-Metric Mapping: The global topological map tracks observed nodes and supports long-term actions, while the local metric map provides fine-grained spatial features for short-term reasoning.The hybrid design balances long-term planning on a graph with local reasoning over a grid centered on the current node.
- Topo-Metric Mapping: The topological map represents visited and navigable nodes as graph elements, assigning visual features to nodes and maintaining an action space over observed nodes.The graph is initialized from the expert-path subgraph, and its edges record distances among adjacent nodes.
- Topo-Metric Mapping: The metric map projects nearby nodes’ grid features onto an egocentric 2D grid, averaging collisions and exposing the current and adjacent nodes for local actions.Nearby visited nodes are selected within κ graph order, then projected using depths and poses onto the ground plane.
- Multimodal Map Learning: The topo-map encoder fuses instructions with graph nodes for long-term reasoning, while the metric-map encoder fuses instructions with cells for fine-grained spatial reasoning.Node embeddings include location and navigation-step information; cell embeddings include polar position and navigability information.
- Pre-training Tasks: Hybrid Single Action Prediction combines local metric-map results with the global action space to predict navigation actions.Cells in the local action space are converted into global actions through a cell→node operation.
4. Experiments
BEVBert is evaluated across four VLN benchmarks and analyses show that its hybrid topo-metric maps and map-based pre-training improve spatial reasoning, navigation performance, and decision-making trade-offs.
- Evaluation Setup: BEVBert is evaluated on R2R, R2R-CE, RxR, and REVERIE, covering fine-grained instruction-following and coarse-grained goal-oriented navigation.R2R-CE uses continuous environments, RxR provides detailed path descriptions, and REVERIE uses coarse-grained instructions.
- Comparison with State-of-the-Art: BEVBert improves over prior methods on R2R, including +4 SR and +3 SPL over DUET on the test unseen split.It also reports +40 SR and +32 SPL over Chasing on that split.
- Comparison with State-of-the-Art: BEVBert sets new state of the art on R2R-CE, improving SR by 4 and SPL by 2 over the topo-map-only ETPNav.The authors attribute this result to the hybrid map design.
- Comparison with State-of-the-Art: On RxR test unseen, BEVBert surpasses EnvEdit by 4 SR, 0.8 NDTW, and 2.4 SDTW.RxR’s longer paths and detailed descriptions make fine-grained metric mapping especially relevant to the reported comparison.
- Quantitative and Qualitative Analysis: Performance declines as spatial and numerical instruction complexity increases, but BEVBert consistently outperforms counterparts, especially on RxR.The analysis identifies spatial reasoning as a bottleneck of existing methods.
- Ablation Study: Ablations show that hybrid topo-metric maps balance local spatial reasoning and long-term planning better than topology-only or metric-only alternatives.Topology-only maps lack metric information, while metric-only maps yield poorer navigation performance because of weaker long-term planning.
5. Conclusion
The paper introduces a hybrid map and BEVBert pre-training paradigm for VLN, combining short-term spatial reasoning with long-term planning through explicit spatial representations.
- BEVBert learns visual-textual associations in an explicit spatial representation based on a hybrid map.
- The hybrid map balances short-term reasoning and long-term planning in VLN.
- Experiments validate that multimodal map representations enhance spatial-aware cross-modal reasoning and facilitate language-guided navigation.
- BEVBert achieves state-of-the-art performance on four VLN benchmarks.
Appendices
The appendices provide additional dataset, metric, modeling, experimental, comparison, and failure-case details.
- Appendix A presents additional details about evaluation datasets and metrics.
- Appendix B describes model variants and training objectives.
- Appendix C provides details about experimental setups.
- Appendix D reports more comparisons against state-of-the-art methods.
- Appendix E presents visualizations of failure cases.
A. Evaluation Datasets and Metrics
The evaluation covers four VLN datasets spanning instruction-following and goal-oriented tasks with fine- or coarse-grained instructions, using task-specific navigation metrics.
- Datasets: R2R provides fine-grained, step-by-step instructions for reaching target locations.
- Datasets: R2R, R2R-CE, RxR, and REVERIE differ in task type and instruction granularity.
- Datasets: RxR uses longer paths and more detailed descriptions involving objects and their relations.
- Datasets: REVERIE uses coarse-grained instructions and requires identifying a target object after reaching its location.
- Metrics: R2R and R2R-CE primarily use SR and SPL, with success defined as stopping within 3 meters of the target.
B.1. Adaptation to the REVERIE Dataset
For REVERIE, BEVBert incorporates candidate object annotations into the short-term metric-map branch to support object grounding through multimodal reasoning.
- REVERIE candidate object annotations are fed into the short-term metric-map encoder for object grounding.
- Object features receive orientation position embeddings before contextual encoding with view features.
- Cell and object embeddings form the visual modality, while encoded instructions provide the linguistic modality.
- The short-term transformer performs cross-modal reasoning over visual and linguistic inputs.
- Multimodal object representations are learned through masked region classification and object grounding tasks.
B.2. Pre-training Objectives
BEVBert combines map-based pre-training tasks with fine-tuning objectives tailored to navigation and object grounding. Its training balances expert-path fidelity with on-policy exploration, using dataset-specific task configurations.
- Pre-training task sampling: R2R/R2R-CE and RxR use MLM, HSAP, and MSI pre-training tasks sampled at a 5:5:1 ratio.REVERIE instead uses MLM, HSAP, MRC, and OG at a 1:1:1:1 ratio.
- REVERIE objectives: REVERIE applies MRC to masked objects and OG to predict endpoint target-object scores from multimodal object representations.MRC masks objects with 15% probability and optimizes semantic-label prediction using KL divergence; OG uses negative log-likelihood.
- Fine-tuning objectives: Fine-tuning combines teacher-forcing and student-forcing losses, with an additional OG loss for REVERIE.Teacher-forcing follows ground-truth actions, while student-forcing samples actions and uses pseudo labels.
- Pseudo-label construction: Goal-oriented pseudo labels are used for R2R/R2R-CE and REVERIE, while RxR uses fidelity-oriented labels based on expert-path similarity.The fidelity criterion is NDTW, whereas goal-oriented labels select a ghost node shortest in path length to the final target.
- Loss balancing: The loss balance λ is 0.2 for R2R/R2R-CE and REVERIE, but 0.8 for RxR because longer annotated paths make excessive exploration harmful.This weighting balances expert-path following against exploration during student-forcing.
C. Experimental Setups
The experimental setup includes a learned depth-estimation pipeline and a controlled procedure for evaluating spatial and numerical instruction reasoning. Instruction groups with insufficient examples are excluded for reliability.
- Depth estimation: RedNet estimates depths from RGB images using a U-Net-like architecture trained on Matterport3D train-split houses.Sigmoid outputs and downsized depth images supervise depth prediction and intermediate layers, after which depths are estimated for all viewpoints.
- Instruction selection: Spatial and numerical instructions are extracted using token templates and grouped by the number of special tokens.Groups containing fewer than 40 instructions are omitted to ensure reliable performance estimation.
D. More Comparisons with State-of-the-Art
Additional comparisons evaluate BEVBert against state-of-the-art methods across R2R, RxR, and REVERIE. The reported results show strong performance but a substantial remaining gap to human performance on RxR.
- Cross-dataset comparisons: Tables 12–14 provide additional state-of-the-art comparisons on R2R, RxR, and REVERIE, with each dataset’s main metrics highlighted.These tables cover the three datasets separately rather than reporting a single aggregate benchmark result.
- Performance comparison: BEVBert achieves state-of-the-art performance on all metrics in seen splits, while remaining below human performance.On RxR test unseen, humans achieve 93.9 SR and 76.9 SDTW, compared with BEVBert’s 64.4 SR and 54.2 SDTW.
E. More Qualitative Examples
Qualitative examples identify early state-tracking failures and ambiguity-related errors in navigation. The cases illustrate incorrect endpoints, backtracking problems, and premature or directionally wrong actions.
- Early lost: Early mistakes can disrupt state tracking, causing excessive backtracking or preventing recovery to the correct path.One case instead skips the required turn and stops at a visually plausible but incorrect kitchen-counter location.
- Ambiguity: Ambiguous instructions can lead to wrong-direction travel, premature turns, or stopping before the instruction is complete.Examples include confusion over another bedroom, the second room on the left, hallway direction, and when “down the hallway” ends.