PaperlayerView exact revision on arXiv

Source-linked AI summary

GridMM: Grid Memory Map for Vision-and-Language Navigation

Zihan Wang, Xiangyang Li, Jiahao Yang, Yeqi Liu, Shuqiang Jiang

arXiv:2307.12907v4cs.CVcs.AI

TL;DR

VLN memory representations may fail to preserve the spatial and fine-grained visual information needed for instruction grounding. GridMM builds a dynamically growing egocentric grid memory with instruction-relevance aggregation, and experiments across discrete and continuous benchmarks show superiority over existing methods.

  • Problem

    Existing VLN approaches use recurrent states, topological maps, or top-down semantic maps, with limitations in preserving trajectory information, unrestricted visual content, or object attributes.

  • Method

    GridMM projects historical observations into a dynamically growing top-down egocentric grid and uses instruction relevance aggregation to capture fine-grained visual clues.

  • Results

    The method outperforms existing methods on many benchmark datasets spanning discrete and continuous environments.

  • Takeaways & Limitations

    GridMM provides global space-time perception and local detailed clues for navigation.

Abstract

from arXiv · show

Vision-and-language navigation (VLN) enables the agent to navigate to a remote location following the natural language instruction in 3D environments. To represent the previously visited environment, most approaches for VLN implement memory using recurrent states, topological maps, or top-down semantic maps. In contrast to these approaches, we build the top-down egocentric and dynamically growing Grid Memory Map (i.e., GridMM) to structure the visited environment. From a global perspective, historical observations are projected into a unified grid map in a top-down view, which can better represent the spatial relations of the environment. From a local perspective, we further propose an instruction relevance aggregation method to capture fine-grained visual clues in each grid region. Extensive experiments are conducted on both the REVERIE, R2R, SOON datasets in the discrete environments, and the R2R-CE dataset in the continuous environments, showing the superiority of our proposed method.

1. Introduction

VLN agents must ground language in time-dependent observations while representing visited environments for navigation. GridMM addresses limitations of recurrent, topological, and semantic-map memories with a dynamically growing visual grid map and instruction-relevant feature aggregation.

  • VLN requires agents to understand natural-language instructions and act in discrete or continuous 3D environments.
  • Recurrent states compress trajectory history into fixed-size vectors, which may omit essential information for environment understanding and instruction grounding.
  • Top-down semantic maps represent spatial relations more precisely but cannot represent objects outside predefined labels or fully preserve attributes such as “wood” and “blue.”
  • GridMM structures the globally visited environment with a dynamically growing grid map whose regions store fine-grained visual features.
  • Instruction relevance aggregation captures visual clues relevant to navigation instructions within GridMM.
  • The method is evaluated across discrete and continuous environments and outperforms existing methods on many benchmark datasets.

2. Related work

Prior VLN work uses recurrent history encodings and map-based representations across discrete and continuous settings. GridMM extends map representations with a dynamically expanding egocentric grid that combines spatially enhanced visual features with instruction-conditioned representations.

  • VLN research covers step-by-step instruction following, dialog navigation, and remote object grounding, all requiring time-dependent visual observations.
  • Continuous-environment VLN requires low-level action selection and is designed to be closer to real-world navigation than topological tasks.
  • Navigation-map research includes topological structures for backtracking or global planning and top-down semantic maps for spatial representation.
  • GridMM projects stored features into a unified N×N egocentric square map and aggregates cell features with instruction relevance.
  • Unlike BEVBert’s visual-only metric-map cells, GridMM cell representations adapt to instructions and contain visual and linguistic information.

3. Method

GridMM constructs a dynamically growing, egocentric grid memory from historical visual observations, then encodes instruction-relevant cell features, spatial information, and navigation trajectories for action prediction.

  • 3.1. Navigation Setups: The navigation setup provides panoramic RGB and depth observations, navigable views or waypoints, and agent-position information in discrete and continuous environments.
  • 3.2. Grid Memory Mapping: Grid memory stores fine-grained visual features with their absolute coordinates, accumulating observations across navigation steps.Features are extracted from panoramic RGB images and indexed over spatial positions before map construction.
  • 3.2. Grid Memory Mapping: The map transforms historical features into dynamically relative coordinates centered on the current agent and aligned with its current direction.Its side length expands with the visited environment, and the resulting map is divided into N×N cells.
  • 3.3.2. Grid Memory Encoding: Each grid cell aggregates multiple visual features into one embedding vector, producing map features that preserve spatial relations across the N×N grid.
  • 3.3.1. Instruction and Observation Encoding: Action reasoning combines instruction, panoramic observation, grid-map, and trajectory representations, using candidate views for local prediction and candidate waypoints for global prediction.Observation embeddings include visual and geometric information, while the trajectory represents historical and candidate waypoint features.
  • 3.3.2. Grid Memory Encoding: Instruction relevance scores select and aggregate cell features according to their relevance to the navigation instruction.The method computes feature-token relevance, applies row-wise max-pooling, and then forms the cell embedding.

4. Experiment

GridMM is evaluated across discrete and continuous VLN benchmarks, where it generally outperforms prior methods. Ablations indicate that instruction-aware grid features, trajectory information, egocentric coordinates, and dynamically growing maps contribute to performance.

  • Datasets and metrics: The model is evaluated on REVERIE, R2R, SOON, and R2R-CE, covering both discrete and continuous environments.
  • Comparison to prior methods: GridMM achieves state-of-the-art performance on most metrics and significantly outperforms previous methods on R2R-CE.On REVERIE val unseen, it exceeds DUET by 4.39% SR and 2.74% SPL.
  • Map comparison: All map-based variants outperform the no-map baseline, while GridMM surpasses topological and semantic maps.The semantic-map variant remains inferior to the topological-map and other compared variants.
  • Feature comparison: Grid features outperform object features because they represent broader visual information, including house structure and background, with greater semantic generalization.
  • Ablation study: Instruction relevance aggregation outperforms average pooling by filtering irrelevant grid features and capturing instruction-relevant visual clues.
  • Map design: Navigation performance increases with map scale, but larger maps impose heavy computational costs for slight gains; the chosen scale is 14×14.The dynamically growing map expands from about 10 meters to about 20 meters during navigation, adapting to visited environments.

5. Conclusion

The paper concludes that GridMM combines global space-time perception with local instruction-relevant visual clues for VLN. It reports improved navigation accuracy but leaves multi-floor environment handling open.

  • GridMM structures visited environments with a top-down, egocentric, dynamically growing map and aggregates instruction-relevant visual clues.
  • The method provides global space-time perception and local detailed clues, enabling more accurate navigation results.
  • Handling multi-floor environments remains an open limitation of the approach.

A. Datasets

The evaluation covers discrete Matterport3D-based VLN benchmarks and continuous trajectories produced in Habitat. These datasets differ in instruction style, path structure, and object-grounding requirements.

  • Dataset scope: The study evaluates discrete environments using R2R, REVERIE, and SOON, and continuous environments using R2R-CE and related VLN-CE settings.
  • Discrete environments: Discrete benchmarks use Matterport3D houses with navigable locations represented by panorama images and GPS coordinates.
  • Continuous environments: R2R-CE and RxR-CE transfer discrete paths from R2R and RxR into continuous trajectories on the Habitat simulator.

B. Performance in RxR-CE

GridMM achieves competitive results on longer-trajectory navigation in the RxR-CE continuous-environment benchmark.

  • GridMM achieves competitive results on longer trajectory navigation such as RxR-CE.

C.1. Training Details

Training uses dataset-specific pre-training and fine-tuning procedures, with model selection based on SPL on the validation-unseen split.

  • REVERIE training combines original and DUET-augmented data for pre-training, followed by fine-tuning on REVERIE.Pre-training runs for 100k iterations with batch size 32 and learning rate 5e-5; fine-tuning runs for 50k iterations with batch size 4 and learning rate 1e-5.
  • SOON training uses cleaned original data, while R2R pre-training follows DUET settings with additional augmented data.
  • R2R-CE transfers the R2R-pre-trained model to continuous environments and fine-tunes it for 30 epochs.Fine-tuning uses batch size 8 and learning rate 1e-5 on three RTX3090 GPUs.
  • The best model is selected using SPL on the validation-unseen split for every dataset.

C.2. Ablation Details

The ablation details describe two alternative map representations: a top-down semantic map and object features used as substitutes for grid features.

  • The top-down semantic-map ablation projects segmented observation pixels into a unified 448×448 map and encodes 32×32 patches.A pre-trained UNet produces segmentation, and a convolution layer maps patches to embeddings with hidden size 768.
  • The object-feature ablation detects objects with VinVL and uses the 10 highest-confidence object features as substitutes for grid features.Object coordinates are taken from bounding-box center points.

D. Analysis of Computational Cost

The computational-cost analysis measures GFLOPs across trajectory and instruction lengths and compares cached versus uncached GridMM computation with BEVBert.

  • GFLOPs are computed using operation counts for matrix multiplication, two-layer MLPs, self-attention, and cross-attention blocks.
  • Figure 7 varies trajectory length with instruction length fixed at 32, while Figure 8 varies instruction length with trajectory length fixed at 15.Visual- and text-encoder costs are omitted from both comparisons.
  • GridMM without cache updates every grid-map cell at every navigation step, whereas caching reuses previous results and significantly reduces computational cost.With fixed grid features, cached computation recomputes feature positions but not their instruction relevance values.
  • GridMM with cache has lower GFLOPs than BEVBert because it uses fewer than 200 map features as cross-attention keys and values, versus more than 400 metric-map features in BEVBert.
Loading 2307.12907v4…