PaperlayerView exact revision on arXiv

Source-linked AI summary

Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation

Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, Ivan Laptev

arXiv:2202.11742v1cs.CV

TL;DR

Vision-and-language navigation requires agents to follow instructions in unseen environments while grounding language in visual scenes and exploring toward targets. DUET addresses this with dual-scale graph transformers and online topological maps, achieving state-of-the-art results across REVERIE, SOON, and R2R.

  • Problem

    Vision-and-language navigation requires agents to follow language instructions and reach target locations in unseen environments, while balancing fine-grained grounding with efficient exploration.

  • Method

    DUET constructs online topological maps and dynamically combines coarse-scale global-map encoding with fine-scale local cross-modal encoding for action planning and language grounding.

  • Results

    DUET achieves state-of-the-art performance on REVERIE, SOON, and R2R, including more than 20% higher success rate on REVERIE and SOON and a 4% success-rate increase on R2R.

  • Takeaways & Limitations

    Dynamic fusion of complementary global exploration and local grounding supports both long-term action planning and fine-grained navigation.

  • Takeaways & Limitations

    The approach is limited by instability and compute costs when local-action models must be rerun repeatedly to backtrack multiple steps.

Abstract

from arXiv · show

Following language instructions to navigate in unseen environments is a challenging problem for autonomous embodied agents. The agent not only needs to ground languages in visual scenes, but also should explore the environment to reach its target. In this work, we propose a dual-scale graph transformer (DUET) for joint long-term action planning and fine-grained cross-modal understanding. We build a topological map on-the-fly to enable efficient exploration in global action space. To balance the complexity of large action space reasoning and fine-grained language grounding, we dynamically combine a fine-scale encoding over local observations and a coarse-scale encoding on a global map via graph transformers. The proposed approach, DUET, significantly outperforms state-of-the-art methods on goal-oriented vision-and-language navigation (VLN) benchmarks REVERIE and SOON. It also improves the success rate on the fine-grained VLN benchmark R2R.

1. Introduction

VLN requires agents to follow language instructions while navigating unseen environments, combining fine-grained visual-language grounding with efficient exploration. DUET addresses limitations in action scope, memory, and representation through topological maps and dual-scale graph-transformer encoding.

  • Goal-oriented VLN requires grounding rooms and objects while efficiently exploring unseen environments to reach a target.
  • Recurrent navigation memories condense rich history into fixed-size vectors, while local-action transformers require N model runs to backtrack N steps.
  • Topological maps support long-term planning by tracking visited and navigable locations and selecting goals across the global action space.
  • DUET dynamically fuses fine-scale current-location encoding with coarse-scale map encoding to balance language grounding and large-graph reasoning.
  • More than 20% SR improvement is reported on REVERIE and SOON, while R2R SR increases by 4%.

2. Related work

Related VLN work progressed from recurrent cross-modal models to transformers and topological maps. DUET targets the remaining combination of recurrent state tracking and insufficient fine-scale language-grounding representations.

  • Early VLN methods mainly used recurrent neural networks with cross-modal attention, while later approaches adopted transformer-based architectures.
  • Metric maps are difficult to construct accurately in real time, motivating topological representations for pre-exploration and backtracking.
  • Recent topological-map VLN methods support global action planning but retain recurrent state tracking and lack fine-scale representations for language grounding.
  • DUET addresses these limitations with a dual-scale graph transformer over topological maps.

3. Method

DUET builds a topological map from observations and combines coarse global-map reasoning with fine-grained local visual-language reasoning to plan actions over global and local spaces.

  • Problem formulation: VLN models an agent navigating an initially unseen graph from language instructions, panoramic images, object features, and neighboring-node observations.The agent traverses navigable nodes toward a target and identifies the specified object after stopping.
  • Topological Mapping: DUET gradually constructs a topological map by adding the current node and neighboring unvisited nodes, updating edges and node visual representations over time.The map contains visited, navigable, and current nodes, with new observations updating representations.
  • Topological Mapping: A multi-layer transformer models spatial relations among current-node image and object features, while navigable-node representations accumulate partial views and average repeated observations.The resulting pooled node features support coarse-scale graph reasoning; current-node image and object features remain available for fine-scale reasoning.
  • Global Action Planning: The coarse-scale encoder predicts over map nodes, the fine-scale encoder predicts local actions from current observations, and dynamic fusion combines both action predictions.The planning module uses encoded instructions with global map features and fine-grained current-location features.
  • Coarse-scale Cross-modal Encoder: Graph-aware cross-modal transformers encode instruction-node relations and graph layout, using location and navigation-step embeddings plus a stop node for global action prediction.Graph-aware self-attention incorporates pairwise graph distances, and scores for visited nodes can be masked when revisiting is unnecessary.
  • Fine-scale Cross-modal Encoder: The fine-scale encoder adds absolute and relative location embeddings, processes image and object tokens with a cross-modal transformer, and supports local action prediction and object grounding.A special visual stop token represents the stop action.
  • Training and Inference: Policy learning combines expert demonstrations with pseudo supervision from an interactive demonstrator that selects shortest-distance navigable targets during sampled trajectories.The pseudo demonstrator uses the environment graph during training, while the current policy supplies sampled trajectories.

4. Experiments

DUET is evaluated on goal-oriented REVERIE and SOON and fine-grained R2R, using navigation, object-grounding, and ablation analyses. It consistently outperforms prior methods, while ablations show complementary roles for coarse and fine scales, dynamic fusion, and graph-aware encoding.

  • Datasets and Metrics: REVERIE and SOON require object grounding and exploration, whereas R2R provides step-by-step instructions without object localization.REVERIE supplies predefined object boxes; SOON requires predicting object centers using detector candidates.
  • Datasets and Metrics: Navigation is assessed with TL, NE, SR, OSR, and SPL, while REVERIE and SOON additionally use RGS and RGSPL.Higher is better for all metrics except TL and NE.
  • Ablation Study: Fine-scale encoding supports target-object selection, coarse-scale encoding supports exploration, and dynamic fusion improves SPL by 1.79% over average fusion.Graph-aware self-attention is more beneficial for SPL, emphasizing shorter paths to targets.
  • Ablation Study: Synthetic speaker data does not improve performance, and the authors hypothesize that policy learning requires cleaner data.The passage contrasts noisy synthetic data with the usefulness of auxiliary proxy tasks during pretraining.
  • Comparison with State of the Art: 14.03% SR, 3.53% SPL, and 5.75% RGSPL gains over HAMT on REVERIE val unseen increase to 22.11%, 9.39%, and 8.98% on test unseen.DUET beats prior state-of-the-art methods across all evaluation metrics on the three REVERIE splits.
  • Comparison with State of the Art: 20.54% SR and 12.19% SPL gains over GBE on SOON test unseen demonstrate stronger performance than the previous graph-based approach.Overall SOON performance remains lower than REVERIE because SOON has fewer and more challenging training data.
  • Qualitative Analysis: Figure 5 depicts start and target locations with flags and global actions with dashed lines, contrasting DUET’s exploration with HAMT’s local-action limitation.The caption states that DUET can correct previous decisions and explore more efficiently.
  • Comparison with State of the Art: 6% and 4% SR gains on R2R val unseen and test unseen, respectively, coexist with comparable SPL performance.The authors associate longer map-based trajectories with encouraged backtracking, while coarse-scale DUET still outperforms prior graph-based approaches.

5. Conclusion

DUET combines online topological mapping with dual-scale graph-transformer reasoning for long-term planning and fine-grained language grounding. It reports state-of-the-art performance across REVERIE, SOON, and R2R, while remaining limited by seen–unseen gaps and discrete environments.

  • Conclusion: DUET uses online topological maps, coarse-scale global reasoning, and fine-scale local representations within a dynamically combined navigation policy.The model constructs maps over time, updates node visual representations, and predicts the next map location or a stop action.
  • Conclusion: DUET achieves state-of-the-art performance on REVERIE, SOON, and R2R.
  • Conclusion: The model addresses long-term action planning and fine-grained language grounding through complementary coarse- and fine-scale encoders.The coarse scale represents the global map, while the fine scale represents local observations.
  • Conclusion: DUET is not always successful, shows a gap between seen and unseen environments, and is restricted to discrete environments.The paper identifies these issues as directions for future work.
  • Conclusion: Masked language modeling and masked region classification are used as auxiliary pretraining tasks alongside single-step action prediction and object grounding.MLM targets grounded language representations and cross-modal alignment, while MRC predicts semantic labels for masked image regions.

A.2. Speaker Model for Data Augmentation

The speaker model synthesizes REVERIE instructions from visual observations and semantic labels of target objects and rooms. Using these synthetic instructions expands the training set substantially.

  • A.2. Speaker Model for Data Augmentation: The speaker model generates REVERIE instructions from panorama image features, target-object labels, and target-room labels.Its architecture contains a panorama encoder and a sentence decoder.
  • A.2. Speaker Model for Data Augmentation: The model is trained only on the REVERIE training split for 50 epochs with pretrained GloVe embeddings.
  • A.2. Speaker Model for Data Augmentation: 19,636 synthetic instructions expand REVERIE training from 10,466 to 30,102 instruction-path pairs.

B.1. Dataset

The evaluation uses Matterport3D-based VLN benchmarks with standard seen and unseen house splits. REVERIE and SOON emphasize object-oriented navigation, while SOON requires detector-assisted object candidates because it lacks per-panorama object boxes.

  • B.1. Dataset: REVERIE and SOON are the primary benchmarks because they require both fine-grained object grounding and exploration, while R2R provides step-by-step instructions without object localization.
  • B.1. Dataset: All benchmarks use 90 Matterport3D houses, with training, val seen, val unseen, and test subsets.Val seen houses overlap with training, whereas val unseen and test houses differ from training.
  • B.1. Dataset: SOON annotates target-object locations and center-point orientations but does not provide annotated object bounding boxes for each panorama.Its four-corner annotations are described as noisy, motivating object-box preprocessing.
  • B.1. Dataset: A pretrained BUTD detector supplies candidate objects for SOON, which are selected using semantic similarity and Euclidean distance to the annotated target.This converts SOON object grounding toward the REVERIE setting.

B.3. Evaluation Metrics

The evaluation distinguishes object-grounding success across REVERIE and SOON and uses standard dataset and trajectory analyses. Ablations show a trade-off between exploration and object grounding when balancing behavior cloning.

  • B.3. Evaluation Metrics: REVERIE success requires reaching a location where the target object is visible and selecting it among annotated panorama candidates.
  • B.3. Evaluation Metrics: SOON success requires reaching within 3 meters of a target location and placing the predicted object-center orientation inside the annotated object polygon.
  • B.3. Evaluation Metrics: About 50% of SOON polygons formed from annotated corner points do not contain the objects’ center points.This annotation issue motivates caution when interpreting polygon-based object-grounding evaluation.
  • B.3. Evaluation Metrics: Larger behavior-cloning weight λ harms exploration, with λ = 1 producing the worst OSR and SR.
  • B.3. Evaluation Metrics: Removing behavior cloning yields strong OSR, SR, and SPL but is less competitive for object grounding.Behavior cloning guarantees arrival at target locations in sampled trajectories, supporting object-grounding training.

C.2. Backtrack ratio in inference

DUET uses global-map backtracking to explore and correct decisions, with coarse- and fine-scale modules receiving different emphasis across navigation stages.

  • Backtrack behavior: 48.6% of DUET’s predicted trajectories backtrack on REVERIE val unseen, compared with 13.7% on val seen.On R2R val unseen, the backtrack ratio decreases to 23.2% when step-by-step instructions are provided.
  • Backtrack behavior: Backtracking lets DUET jump to previously partially observed nodes through the global action space rather than selecting neighboring nodes locally.This supports exploration and correction of previous decisions.
  • Scale weighting: The coarse-scale module receives average weights of 0.36 at the beginning, 0.45 in the middle, and 0.42 at the end of navigation.The fine-scale module is emphasized at the beginning and end, while the coarse-scale module receives more attention during exploration.

C.4. Failure analysis

REVERIE evaluation shows that navigation often fails before object grounding: 46.98% reaches the correct location, while object localization succeeds 68.43% of the time after arrival.

  • Navigation outcomes: 46.98% of REVERIE cases reach the correct location, while 29.82% stop at an incorrect room type.Another 23.20% reaches the correct room type but an incorrect location.
  • Object grounding: 68.43% of objects are correctly localized once the agent reaches the correct location.The result separates location-finding errors from object-grounding errors.
  • Interpretation: Fine-grained scene understanding remains challenging in REVERIE.The reported navigation and grounding breakdown identifies both room/location selection and object localization as evaluation outcomes.

D. Qualitative Examples

Qualitative examples show DUET exploring alternate directions and correcting decisions, while failures expose sensitivity to fine-grained objects and language variation.

  • REVERIE examples: DUET efficiently explores another direction after initially moving incorrectly on REVERIE examples.HAMT is shown alongside DUET for comparison in the REVERIE trajectories.
  • R2R examples: On R2R, DUET corrects previous decisions when follow-up instructions do not match visual observations.The examples also include ambiguity at the starting point despite step-by-step instructions.
  • Failure cases: A REVERIE failure occurs when DUET reaches a bathroom that lacks the fine-grained objects specified by the instruction.The example indicates that selecting the correct room does not guarantee selecting the correct object-containing location.
  • Failure cases: R2R predictions are not robust across different language instructions describing the same trajectory.The agent succeeds on one instruction but fails on two others for that trajectory.
Loading 2202.11742v1…