PaperlayerView exact revision on arXiv

Source-linked AI summary

MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigation

Lingfeng Zhang, Xiaoshuai Hao, Qinwen Xu, Qiang Zhang, Xinyao Zhang, Pengwei Wang, Jing Zhang, Zhongyuan Wang, Shanghang Zhang, Renjing Xu

arXiv:2502.13451v5cs.RO

TL;DR

VLN agents must follow language instructions while navigating unseen environments, but historical-frame memory can impose storage and computational costs. MapNav replaces those frames with an updated, text-annotated semantic map processed by an end-to-end VLM, achieving state-of-the-art performance across simulated and real-world environments while maintaining a constant 0.17MB memory footprint.

  • Problem

    Existing VLN approaches rely heavily on historical observations as spatio-temporal context, creating storage and computational overhead during navigation.

  • Method

    MapNav builds an Annotated Semantic Map from RGB-D, pose, and semantic information, updates it each timestep, and uses it with current observations and instructions in an end-to-end VLM.

  • Results

    MapNav achieves state-of-the-art navigation performance in simulated and real-world environments while maintaining a constant memory footprint of 0.17MB regardless of trajectory length.

  • Takeaways & Limitations

    Annotated Semantic Maps provide a memory representation for VLN that replaces traditional historical frames and supports structured spatial understanding.

  • Takeaways & Limitations

    Semantic segmentation may produce imprecise object labels under challenging conditions such as occlusions or varying lighting.

Abstract

from arXiv · show

Vision-and-language navigation (VLN) is a key task in Embodied AI, requiring agents to navigate diverse and unseen environments while following natural language instructions. Traditional approaches rely heavily on historical observations as spatio-temporal contexts for decision making, leading to significant storage and computational overhead. In this paper, we introduce MapNav, a novel end-to-end VLN model that leverages Annotated Semantic Map (ASM) to replace historical frames. Specifically, our approach constructs a top-down semantic map at the start of each episode and update it at each timestep, allowing for precise object mapping and structured navigation information. Then, we enhance this map with explicit textual labels for key regions, transforming abstract semantics into clear navigation cues and generate our ASM. MapNav agent using the constructed ASM as input, and use the powerful end-to-end capabilities of VLM to empower VLN. Extensive experiments demonstrate that MapNav achieves state-of-the-art (SOTA) performance in both simulated and real-world environments, validating the effectiveness of our method. Moreover, we will release our ASM generation source code and dataset to ensure reproducibility, contributing valuable resources to the field. We believe that our proposed MapNav can be used as a new memory representation method in VLN, paving the way for future research in this field.

1 Introduction

VLN requires agents to interpret language while navigating complex, unseen environments, and existing approaches face limitations in continuous navigation. MapNav replaces historical frames with an updated Annotated Semantic Map that supports structured navigation and achieves strong performance across simulated and real-world settings.

  • VLN combines natural-language understanding, visual perception, and real-time decisions in complex, unseen environments.
  • Existing discrete VLN methods use waypoint graphs but do not reflect the continuity of real-world navigation.
  • MapNav replaces traditional historical frames with an end-to-end VLM model using an Annotated Semantic Map as its memory representation.
  • The ASM is initialized per episode and updated at every timestep with obstacles, explored areas, agent position, trajectories, semantic objects, and textual region labels.
  • MapNav outperforms state-of-the-art methods in both simulated and real-world environments, while released code and data support reproducibility.

2 Related Work

VLN research spans graph-based discrete navigation, continuous-environment control, and structured map representations. MapNav contributes an annotated top-down semantic map that replaces historical frames and supplies textual navigation cues to an end-to-end VLM.

  • Discrete VLN methods navigate predefined graph nodes, whereas VLN-CE supports unrestricted movement through low-level control or waypoint-based methods.
  • The framework feeds the current RGB observation, ASM, and instruction into an end-to-end VLM that generates navigation actions in natural language.
  • MapNav introduces an Annotated Semantic Map that is updated each timestep and enhanced with textual labels for key regions.
  • Prior map-based methods use topological, value, or semantic maps to represent viewpoints, waypoint utility, or object-level information.

3 Methodology

MapNav replaces historical-frame memory with an incrementally updated Annotated Semantic Map (ASM) that combines top-down spatial structure, object semantics, and textual labels. The agent encodes the ASM with observations and instructions, then directly generates executable navigation actions through a VLM.

  • Annotated Semantic Map Generation: MapNav constructs a top-down ASM from RGB-D, pose, and semantic segmentation data, updating it at each timestep for object mapping and navigation structure.Point clouds are projected onto a 2D plane, while segmentation masks are aligned with point-cloud data for object-wise mapping.
  • MapNav Agent: The MapNav agent processes current RGB observations and ASM features through separate encoders, then aligns and concatenates them with instruction tokens in a shared multimodal representation.The architecture uses spatial-aware feature transformations and modality-specific MLP projectors before multimodal fusion.
  • Annotated Semantic Map Generation: ASM augments semantic maps with explicit natural-language labels for salient regions, grounding abstract spatial representations in concepts such as chairs, plants, and beds.Connected components and geometric centroids determine annotation placement for regions exceeding the minimum area threshold.
  • MapNav Agent: MapNav directly converts VLM language outputs into discrete navigation actions instead of using a separate action decoder.A pattern-matching ruleset maps synonymous natural-language expressions to executable actions, including stopping and turning commands.
  • Evaluation Setup: Table 1 compares MapNav with state-of-the-art methods on the Val-Unseen splits of R2R-CE and RxR-CE, while Table 2 evaluates simple and semantic instruction following across five real-world scenes.The supplied table descriptions identify baselines, RGB-history conventions, waypoint predictors, panoramic views, and the two real-world instruction-following settings.

4 Experiments

Experiments evaluate MapNav on simulated and real-world VLN benchmarks, comparing ASM-based memory with historical-frame and other map representations. MapNav improves navigation results while reducing memory growth and retaining the main benefit of ASM over additional temporal accumulation.

  • Experimental setup: 1 Million training pairs combine R2R and RxR trajectories, DAgger data, collision-recovery samples, and general vision-language co-training data.Evaluation covers VLN-CE val-unseen splits for R2R and RxR, plus 50 real-world instructions across five environments.
  • Simulated results: 23.5% SR and 26.5% SPL improvements on R2R result from adding ASM to single-frame RGB compared with Navid using current RGB.Without ASM, MapNav improves SR by 14.1% and SPL by 15.7% on R2R, and by 6.9% and 5.4% on RxR.
  • Simulated results: 2.5% SR and 1.3% SPL gains on R2R, plus 8.8% SR and 6.5% SPL gains on RxR, come from MapNav using ASM with two historical RGB frames versus Navid using all RGB frames.The result indicates that ASM can replace extensive historical-frame sequences while remaining competitive with all-frame approaches.
  • Real-world results: 30% SR improvements over Navid occur under semantic instructions in both the lecture hall and living room, while MapNav outperforms WS-MGMAP and Navid across the reported real-world settings.The comparison uses SR and NE for simple and semantic instructions.
  • Efficiency analysis: 0.17MB constant memory is used by MapNav regardless of trajectory length, whereas Navid reaches 276MB at 300 steps because it accumulates historical RGB observations.The table compares memory consumption and average processing time across navigation-step counts.
  • Ablation studies: SR: 41% and SPL: 38% show only slight gains with four historical RGB frames, supporting ASM as the core advantage rather than temporal accumulation.This ablation supports choosing ASM as a more efficient alternative to extensive historical RGB sequences.

5 Conclusion

MapNav uses Annotated Semantic Maps for VLN and reports state-of-the-art performance with lower memory use than traditional approaches. Its enriched top-down maps integrate RGB-D and pose data with annotations, maintaining a constant memory footprint across trajectory lengths.

  • Conclusion: MapNav uses Annotated Semantic Maps to replace traditional historical frames in vision-and-language navigation.The method converts RGB-D and pose data into enriched top-down maps and integrates them with Vision-Language Models.
  • Conclusion: 0.17MB constant memory is maintained regardless of trajectory length while MapNav delivers superior navigation performance.The paper contrasts this footprint with traditional approaches that use historical observations.

Limitations

MapNav’s semantic segmentation can produce imprecise object labels under occlusions or varying lighting. The paper identifies stronger semantic understanding and multi-view scene understanding as directions for addressing this limitation.

  • Limitation: Semantic segmentation may produce imprecise object labels under challenging conditions such as occlusions or varying lighting.The authors propose more advanced semantic understanding for dynamic scenes and partial observations.
  • Future directions: Real-world generalization remains an intended extension, with the authors proposing use of real-world image data from VLM pre-training.The paper also plans to extend MapNav toward interactive navigation and manipulation using object affordances and physical interaction capabilities.

Appendix

The appendix supplies supplementary material covering dataset construction, training, robot implementation, qualitative and additional experiments, and VLM attention visualization. These sections extend the main paper’s experimental and implementation details.

  • Appendix overview: The supplementary material provides details omitted from the main manuscript because of page limitations.It is organized into sections covering dataset construction, training procedures, robot setup, and experiments.
  • Implementation details: Dataset construction and model training procedures are documented in Sections A and B.Section A covers dataset construction, while Section B outlines model training.
  • Implementation details: Section C describes the real-world MapNav robot setup and implementation specifics.This section complements the reported real-world evaluation.
  • Additional results: Sections D through G provide qualitative and additional results from simulated and real-world environments.The appendix separately includes simulator and real-world qualitative results and further experimental validation.
  • Analysis: Section H analyzes VLM attention visualization across different map representations.This analysis provides an additional view of how the model processes alternative map inputs.

A Dataset Construction

MapNav constructs a large, diverse VLN dataset by combining expert demonstrations, DAgger interaction data, collision recovery samples, and general vision-language data. This hybrid strategy targets both navigation coverage and robustness to unforeseen obstacles.

  • Dataset Construction: The dataset combines expert trajectories, DAgger-collected data, and collision recovery samples to address limited diversity and scale in VLN training.These sources provide complementary coverage of optimal navigation, interactive learning, and recovery from collisions.
  • Dataset Construction: Approximately 200k new training pairs were collected through DAgger from R2R and RxR environments.The interactive phase supplements expert demonstrations with trajectories generated by the pretrained model.
  • Dataset Construction: Approximately 25k collision recovery pairs were collected for each of R2R and RxR.These samples target unforeseen obstacles and challenging navigation situations.
  • Dataset Construction: The enhanced dataset totals approximately 1 Million training pairs and represents both optimal pathfinding and practical recovery strategies.The authors report improved resilience and adaptability across diverse indoor environments and navigation challenges.
  • Dataset Construction: Co-training adds approximately 600k general vision-language samples to 500k R2R navigation samples for visual reasoning and generalization.The auxiliary data complements navigation-specific training with broader visual understanding tasks.

B Details of Training

MapNav uses a multimodal architecture built on LLaVA-OneVision, with a SigLIP visual encoder, projector, and language model. Its real-world system streams RGB-D observations to a server, generates ASMs, and returns executable actions to a quadrupedal robot.

  • Training Details: The model comprises a visual encoder, projector, and large language model within the LLaVA-OneVision framework.The visual encoder uses SigLIP-so400m-patch14-384 with 384x384 inputs and 1152-dimensional visual embeddings.
  • Training Details: Training used 8 NVIDIA A100 GPUs for approximately 30 hours, totaling 240 GPU hours.The reported run processed approximately 500k step-wise samples.
  • Real-World System: The real-world setup uses a Unitree Go2 Edu robot equipped with an Intel RealSense D435i depth camera mounted 40cm above the robot.RGB-D observations are sent to an NVIDIA A100 server for ASM generation and action inference.
  • Real-World System: The system transmits actions inferred from generated ASMs back to the Go2 for execution.This connects camera sensing, server-side MapNav processing, and robot control in the real-world pipeline.

D Qualitative Results.

Qualitative evaluations show MapNav using ASMs to follow complex multi-waypoint instructions in simulation and semantic-landmark instructions in the real world. Without ASMs, the simulated agent struggles to find the correct path.

  • Simulated Results: In simulation, the ASM-equipped agent successfully identifies the shortest path while following instructions with multiple waypoints.The visualization compares navigation with and without annotated semantic maps.
  • Visualization: The qualitative figures cover both simulated and real-world navigation views, including third-person, egocentric, and ASM visualizations.The displayed examples pair instructions and timestep-specific observations with map representations.
  • Simulated Results: Without annotated semantic maps, the simulated agent struggles to find the correct path.The comparison highlights the role of ASM-based semantic information in path planning.
  • Real-World Results: In real-world tests, MapNav executes simple instructions and handles complex tasks involving semantic landmarks.The ASM visualization shows real-time path adaptation during these tasks.

E More Experiments

Additional experiments examine input modalities, semantic segmentation choices, and qualitative generalization across simulated and real-world environments. The reported comparisons favor ASM-based inputs and Mask2Former, while visualizations span diverse layouts and instructions.

  • Input Modality Ablation: RGB-only input provides the baseline, RGB+Depth degrades performance, and the complete RGB-plus-ASM model is evaluated as the proposed configuration.The authors attribute the RGB+Depth degradation to limitations in processing depth information.
  • Segmentation Ablation: Mask2Former achieves the best performance across all reported metrics on the R2R Val-Unseen split.The study compares Mask2Former with YOLOv8 and MobileSAM for ASM generation.
  • Simulated Qualitative Experiments: Visualization experiments present 24 navigation trajectories across six pages from the R2R and RxR datasets.The cases cover varied room layouts, navigation objectives, and complex multistep instructions.
  • Simulated Qualitative Experiments: The simulated cases show successful interpretation and execution across diverse scenarios, supporting the reported spatial-understanding and memory-representation capabilities of ASMs.The visualizations pair generated ASMs with navigation actions.
  • Real-World Qualitative Experiments: Six real-world episodes span five environments: an office, meeting room, lecture hall, tea room, and living room.The episodes include both simple and semantic instructions.

H VLM Attention Visualization Analysis

The attention analysis compares map representations and finds that Annotated Semantic Maps concentrate VLM attention on labeled semantic objects and navigation-relevant paths. Additional visualizations and demonstrations illustrate the map-based inputs and navigation scenarios used in the analysis.

  • Map representation comparison: Semantic Maps show sparse attention without convergence on semantic objects, whereas ASM guides attention toward those objects through textual labels.The comparison is presented in the visualization of different map representations.
  • Attention alignment: ASM produces sharp attention peaks above 0.8 aligned with labeled objects and navigation-relevant areas.The analysis uses Vicuna-13B and VLM Visualizer attention heatmaps across map representations.
  • Attention alignment: ASM’s red trajectory lines receive substantial attention, complementing attention to semantic objects with structured navigation cues.The reported visualizations attribute this grounding to explicit textual annotations and structured trajectory representation.
  • Map representation comparison: The analyzed map descriptions portray routes using colored lines or arrows, while ASM descriptions include labels for areas and objects such as plants, chairs, sofas, and sinks.These descriptions provide qualitative examples of the map representations examined.
Loading 2502.13451v5…