PaperlayerView exact revision on arXiv

Source-linked AI summary

History Aware Multimodal Transformer for Vision-and-Language Navigation

Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, Ivan Laptev

arXiv:2110.13309v2cs.CVcs.AI

TL;DR

VLN systems commonly encode navigation history in recurrent states, which can limit long-horizon memory and multimodal decision making. HAMT hierarchically encodes panoramic and temporal history with transformers, combines it with text and current observations, and trains with proxy tasks followed by reinforcement learning. It achieves state-of-the-art performance across diverse VLN benchmarks, with particularly large gains on longer trajectories.

  • Problem

    Recurrent states may lose essential information from extended navigation histories, while limited history and task-specific visual training can hinder generalization.

  • Method

    HAMT hierarchically encodes view images, panorama-level spatial relations, and temporal history, then fuses text, history, and current observations for action prediction.

  • Results

    HAMT achieves state-of-the-art navigation performance across diverse VLN tasks, with larger gains for long-horizon navigation.

  • Takeaways & Limitations

    The results demonstrate improved accuracy and generalization compared with dominant recurrent methods, particularly on longer trajectories.

Abstract

from arXiv · show

Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches to VLN implement memory using recurrent states. Instead, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy. HAMT achieves new state of the art on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN (R4R, R2R-Back). We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories.

1 Introduction

VLN agents must align language with continually changing visual observations while retaining useful information over long trajectories. HAMT addresses these challenges with a fully transformer-based architecture that jointly models instructions, history, and current observations, and reports state-of-the-art results across diverse tasks.

  • Motivation: VLN requires agents to understand instructions, perceive visual scenes, and navigate to target locations across varied task settings.The paper cites fine-grained indoor and outdoor navigation, remote object finding, and dialog-based navigation.
  • Challenges: Recurrent history representations compress observations and actions into fixed-size states, risking loss of essential information on extended trajectories.The paper illustrates this with instructions requiring an agent to remember its starting location after visiting an object.
  • Challenges: Generalization remains challenging because prior methods use limited visual-task optimization and may lack history during training, increasing overfitting risk.The cited discussion contrasts pretrained text-image representations with the need for navigation-specific visual and historical learning.
  • Approach: HAMT uses unimodal transformers for text, history, and observations plus a cross-modal transformer for long-range multimodal dependencies.Its history encoding is designed to handle the computational cost of processing sequences of previous observations.
  • Contributions: HAMT is trained with auxiliary proxy tasks and reinforcement learning, and outperforms state of the art across diverse VLN tasks with larger gains on long-horizon navigation.The contribution statement identifies end-to-end proxy-task training followed by RL policy improvement.

2 Related work

Prior VLN work combines recurrent memory, graphs, transformers, and reinforcement learning, but existing transformer approaches do not provide a fully transformer-based end-to-end architecture. HAMT extends this direction with hierarchical history encoding for long-term spatio-temporal dependencies.

  • Vision-and-language navigation: Early VLN systems used LSTM-based sequence-to-sequence navigation, panoramic action spaces, cross-modal alignment, progress estimation, and graph-based scene relationships.Reinforcement learning is also commonly used to improve navigation policies.
  • Vision-and-language navigation: Recent transformer VLN models improve text or multimodal fusion, but several retain recurrent components or rely on CNN-based visual features.The cited examples include PRESS, SIA, and PTA, while HAMT is presented as the first full transformer architecture for VLN.
  • Memory-based policy for navigation: Memory-based navigation methods include recurrent states and topological or graph memories, yet graph-based approaches still use LSTMs for state tracking.The related work frames HAMT as an alternative for modeling long-term spatio-temporal dependencies.
  • Multimodal pretraining with transformers: Multimodal pretraining with transformer architectures has achieved strong results in vision-language tasks, including visual question answering, retrieval, and captioning.ViLT demonstrates end-to-end learning of image representations with associated text using a Vision Transformer.

3 Method

HAMT is a transformer-based VLN policy that jointly encodes instructions, long-horizon history, and current observations. Its hierarchical history encoder models panoramic spatial relationships and temporal relationships across panoramas while reducing computation for long trajectories.

  • Architecture: HAMT jointly encodes text, history, and the current observation for sequential action prediction using unimodal and cross-modal transformers.The cross-modal transformer captures multimodal relationships among the instruction, historical observations, and current observation.
  • History representation: The history consists of previous panoramic observations and actions, which recurrent approaches instead condense into a fixed-size state vector.The paper identifies information loss in condensed recurrent states as a challenge for extended trajectories.
  • History representation: HAMT hierarchically encodes view images within each panorama and temporal relationships across panoramas to balance computational efficiency and information integrity.View images are embedded with ViT, processed by a panoramic transformer, and then represented at the panorama level for temporal encoding.
  • History representation: The hierarchical history encoder has computational cost O(tK^2+t^2), reduced from O(t^2K^2) for flattened history encoding.The flattened approach treats every view image as a token, causing quadratic growth with the full history sequence length.
  • Training: HAMT is trained end-to-end with proxy tasks covering language, multimodal alignment, action prediction, and spatial reasoning before reinforcement-learning policy fine-tuning.The proxy-task framework includes MLM, MRM, ITM, SAP/SAR, and SPREL objectives.

4 Experiments

Experiments evaluate HAMT across diverse VLN datasets and ablate its history encoding, proxy-task training, and fine-tuning objectives. HAMT consistently improves navigation, with especially strong gains on longer-horizon and path-fidelity tasks.

  • Evaluation settings: HAMT is evaluated on seven datasets spanning fine-grained, high-level, dialog-based, and long-horizon VLN.The datasets are R2R, RxR, REVERIE, R2R-Last, CVDN, R4R, and R2R-Back.
  • Ablation studies: Hierarchical panoramic history encoding improves val unseen R2R performance by 4.2% in SR and 3.6% in SPL over recurrent history encoding.The comparison uses ResNet152 visual features and contrasts recurrent, temporal-only, and hierarchical history representations.
  • Ablation studies: Proxy-task pretraining improves val unseen R2R performance by 16.7% in SR and 18.0% in SPL over training from scratch with RL+IL.Replacing ResNet152 with ViT further improves both seen and unseen performance, while end-to-end ViT training adds 2.1% SPL on val unseen.
  • Ablation studies: Mixing RL and IL achieves the best fine-tuning performance, because proxy-task training stabilizes RL while IL offsets noisy action prediction.Pure RL improves SR more than IL but fluctuates more, and its shortest-path-focused reward yields relatively small SPL gains.
  • State-of-the-art comparisons: HAMT improves R2R SPL over RecBERT by 5.9% on val seen and 7.0% on val unseen splits.It also achieves state-of-the-art performance under the single-run setting on the unseen testing split.
  • State-of-the-art comparisons: On long-horizon R4R, HAMT improves CLS by 8.2% and nDTW by 9.5% relative to RecBERT, with larger relative nDTW gains for longer instructions.On R2R-Back, improvements are more significant because successful return navigation requires remembering the route to the target.

5 Conclusion

HAMT is presented as an end-to-end transformer for VLN that efficiently incorporates long-horizon history into action prediction. It achieves state-of-the-art performance across diverse VLN tasks, with improved accuracy and generalization compared with recurrent methods.

  • HAMT combines instructions, observations, and long-horizon history to derive multimodal action predictions.The model is trained with proxy tasks and then fine-tuned with reinforcement learning.
  • HAMT achieves state-of-the-art navigation performance across a diverse range of challenging VLN tasks.
  • The reported results demonstrate improved accuracy and generalization compared with dominant recurrent methods.
  • The paper identifies continuous-action VLN and larger navigation-dataset pretraining as directions for future work.

Supplementary Material for HAMT

The supplementary material provides additional model details, experimental setup, datasets, metrics, implementation details, full results, ablations, and qualitative analyses.

  • Section A provides additional details about the model.
  • Section B describes the experimental setup, including datasets, metrics, and implementation details.
  • Sections C and D report additional results and ablations, while Section E presents qualitative results.

A.1 Proxy tasks in training

HAMT uses multiple proxy tasks and hierarchical transformer components to learn multimodal alignment and history-sensitive representations. The supplementary material also specifies model variants, dataset constructions, and evaluation rules.

  • Proxy tasks: HAMT is trained with five proxy tasks, including masked language modeling, masked region modeling, and instruction trajectory matching.The full task set also includes SAP, SAR, and SPREL.
  • Proxy tasks: Masked language modeling predicts randomly masked instruction tokens conditioned on the instruction and full trajectory.Tokens are masked with probability 15%, and the objective minimizes negative log-likelihood.
  • Proxy tasks: Masked region modeling predicts semantic labels for masked observations using the instruction and neighboring observations.Its objective minimizes the divergence between target and predicted class distributions.
  • Proxy tasks: Instruction trajectory matching evaluates whether an instruction and trajectory are aligned while encouraging sensitivity to temporal order.The task uses negative trajectories, including randomly selected and temporally shuffled examples.
  • Model variants: An encoder-decoder HAMT variant removes text-to-vision cross-modal attention and reuses encoded text embeddings for greater efficiency with long instructions.
  • Dataset and evaluation setup: R2R-Back appends a return command and requires following a reverse path back to the starting location.Its evaluation requires reaching the original destination and then returning near the start, with SPL based on total traversed distance.

B.3 Implementation Details

The implementation details describe proxy-task sampling, inference-time comparisons on R2R, and supplementary navigation results on RxR and REVERIE.

  • Training: Proxy-task training samples MLM, MRM, ITM, SAP, SAR, and SPREL in a 5:2:2:1:1:1 ratio.The optimizer is AdamW, with image augmentation and stochastic depth used during end-to-end training.
  • Navigation evaluation: The supplementary material reports navigation performance on RxR validation and test splits and on REVERIE.
  • Inference evaluation: Inference time is compared between HAMT and RecBERT on 2,349 R2R unseen-validation instructions using a single Tesla P100 GPU.The comparison uses models with the same numbers of language-transformer and cross-modal-transformer layers.
  • Inference evaluation: Table 11 presents computation-time results for inference on the R2R unseen-validation split.

C.3 REVERIE dataset

HAMT supports REVERIE’s joint navigation and remote object-grounding tasks by augmenting viewpoint features with object information and adding an object-grounding head. Its R2R-optimized visual features may limit object-feature generalization.

  • HAMT concatenates object features with original view features and predicts target objects from object-token embeddings.The model is fine-tuned from end-to-end pretraining on R2R and uses ground-truth object boxes to extract REVERIE object features.
  • The REVERIE evaluation covers both navigation and object grounding on validation-unseen and test splits.
  • HAMT’s object representations may be less generalizable because its ViT was optimized on R2R rather than large-scale object-detection data.

D.1 History in training with proxy tasks

History improves HAMT’s proxy-task training and unseen-environment generalization compared with a matched PREVALENT model without history. The reported navigation gap is especially clear on R2R val unseen.

  • PREVALENT’s lack of trajectory input prevents instruction-trajectory matching with the ITM proxy task.
  • 57.5 SPL versus 52.7 shows HAMT outperforming PREVALENT without history on R2R val unseen after fine-tuning.The comparison uses the same proxy tasks except ITM, attributing the gain to history encoding.
  • HAMT and PREVALENT perform similarly on val seen SAP accuracy, but PREVALENT performs much worse on val unseen.The authors associate history input with better alignment between history and instruction for action prediction and generalization.

D.2 Visual features in training with proxy tasks

HAMT’s visual-feature and training-strategy experiments compare ViT with ResNet features, proxy-task choices, end-to-end optimization, and history encoding across navigation settings. The results favor optimized ViT features, selected proxy tasks, and two-stage training, particularly for long-horizon navigation.

  • Visual features: ViT features outperform ResNet152 features with and without proxy-task training.
  • Visual features: End-to-end feature optimization improves SPL by 2.1% on val unseen but decreases SPL by 0.8% on val seen.The same model is selected using val unseen, and longer training may improve val seen performance.
  • Proxy tasks: SAP/R and SPREL are beneficial proxy tasks, and end-to-end ViT features are superior to fixed ViT features.
  • Training strategy: 53.5 SPL for single-stage e2e training is below 56.5 SPL without e2e training on val unseen after 25k iterations.The authors hypothesize that limited VLN data and higher task complexity make single-stage training less effective.
  • History encoding: Recurrent-state history encoding performs worst on R2R-Back because returning to the start requires remembering the route to the target.
  • Action prediction: The cross-modal transformer is relatively insensitive to which observation, history, and text tokens provide action-prediction inputs on val unseen.
  • Qualitative comparison: HAMT’s instruction understanding improves over RecBERT in the illustrated stair-navigation example.

E Qualitative results

Qualitative trajectories show HAMT handling instruction interpretation, scene recognition, direction following, and state–instruction alignment better than RecBERT in several R2R examples. Failures remain in unusual scenes and fine-grained scene or object recognition.

  • Across Figures 6–9, HAMT better interprets instructions, recognizes scenes, follows directions, and aligns observations with instructions than RecBERT.Figures 10–11 document remaining scene- and object-recognition failures.
  • HAMT correctly recognizes the kitchen and follows the instruction where RecBERT navigates back and forth in wrong locations.
  • HAMT follows the instruction and stops successfully when RecBERT chooses the wrong initial direction.
  • HAMT tracks the third right turn and stops correctly where RecBERT loses alignment after two correct turns.
  • HAMT still fails to recognize a liquor shelf and explores until reaching the maximum navigation steps.
  • HAMT fails to locate the correct direction at the first step in an unusual scene.
Loading 2110.13309v2…