PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards Learning a Generalist Model for Embodied Navigation

Duo Zheng, Shijia Huang, Lin Zhao, Yiwu Zhong, Liwei Wang

arXiv:2312.02010v3cs.CVcs.AI

TL;DR

Existing embodied-navigation agents are mostly task-specific and generalize poorly to unseen scenarios. NaviLLM adapts LLMs with schema-based instruction to unify diverse tasks as generation problems and train on varied data. It achieves state-of-the-art results on CVDN, SOON, and ScanQA, with strong generalizability to unseen tasks.

  • Problem

    Previous embodied-navigation methods primarily focus on task-specific agents and lack generalizability across diverse or unseen scenarios.

  • Method

    NaviLLM adapts an LLM and uses schema-based instruction to cast diverse embodied-navigation tasks into unified generation problems.

  • Results

    NaviLLM achieves state-of-the-art results on CVDN, SOON, and ScanQA and demonstrates strong generalizability on unseen tasks.

  • Takeaways & Limitations

    A single model can support a broad range of embodied-navigation capabilities while maintaining strong performance across multiple benchmarks and unseen tasks.

Abstract

from arXiv · show

Building a generalist agent that can interact with the world is the intriguing target of AI systems, thus spurring the research for embodied navigation, where an agent is required to navigate according to instructions or respond to queries. Despite the major progress attained, previous works primarily focus on task-specific agents and lack generalizability to unseen scenarios. Recently, LLMs have presented remarkable capabilities across various fields, and provided a promising opportunity for embodied navigation. Drawing on this, we propose the first generalist model for embodied navigation, NaviLLM. It adapts LLMs to embodied navigation by introducing schema-based instruction. The schema-based instruction flexibly casts various tasks into generation problems, thereby unifying a wide range of tasks. This approach allows us to integrate diverse data sources from various datasets into the training, equipping NaviLLM with a wide range of capabilities required by embodied navigation. We conduct extensive experiments to evaluate the performance and generalizability of our model. The experimental results demonstrate that our unified model achieves state-of-the-art performance on CVDN, SOON, and ScanQA. Specifically, it surpasses the previous stats-of-the-art method by a significant margin of 29% in goal progress on CVDN. Moreover, our model also demonstrates strong generalizability and presents impressive results on unseen tasks, e.g., embodied question answering and 3D captioning.

1. Introduction

Embodied navigation spans diverse instruction-following, interaction, and question-answering tasks, but prior agents are largely task-specific and generalize poorly across scenarios. NaviLLM adapts LLMs with schema-based instruction to unify these tasks and achieves strong benchmark and unseen-task performance.

  • Motivation: Prior embodied-navigation methods address diverse tasks but generally lack generalization across scenarios.These tasks include fine- and coarse-grained navigation, human-agent interaction, and embodied question answering.
  • Motivation: Adapting LLMs to embodied navigation is promising, but their potential in this setting remains largely unexplored.LLMs have shown generalizability in text, vision, video, 3D understanding, and robotic manipulation.
  • Approach: NaviLLM casts diverse embodied-navigation tasks into generation problems through schema-based instruction.Schemas describe task information, visual observations, and navigation history, enabling one generative formulation for multiple tasks.
  • Approach: NaviLLM integrates data from diverse datasets to support navigation, localization, 3D question answering, trajectory summarization, and embodied question answering.The unified training setup is designed to mitigate data scarcity while broadening task coverage.
  • Results: 29% relative improvement over the previous state-of-the-art is reported on CVDN, while NaviLLM also achieves state-of-the-art results on CVDN, SOON, and ScanQA.The authors additionally report strong generalizability on unseen tasks such as embodied question answering and 3D captioning.

2. Related Work

Embodied-navigation research has expanded from instruction following to dialog, object localization, and question answering, while related work explores multitask learning, foundation models, and multimodal LLMs. NaviLLM is positioned as a unified embodied-AI model spanning navigation and 3D understanding.

  • Embodied Navigation: R2R, CVDN, SOON, REVERIE, and EQA cover instruction following, dialog-based navigation, object localization, and question answering.These benchmarks require different combinations of navigation, interaction, localization, and answering abilities.
  • Generalization: MT-RCM addresses overfitting to specific datasets through multitask modeling, whereas NaviLLM emphasizes LLM-based generalizability across broader tasks and datasets.The related-work comparison distinguishes NaviLLM by its broader task and dataset scope.
  • Multimodal LLMs: Multimodal instruction-tuning methods extend LLMs to visual inputs, but NaviLLM is designed specifically for embodied AI, including navigation and 3D understanding.The paper contrasts its embodied focus with prior multimodal work on images and videos.
  • LLMs in Embodied Tasks: Existing embodied-task approaches either translate visual information into text for planning or fine-tune LLMs on robotic action sequences.The paper presents these as two lines of work for integrating LLMs into embodied tasks.

3. Method

NaviLLM combines a scene encoder with an LLM and uses schemas for tasks, observations, history, and output hints. This representation lets one generative model handle navigation, localization, summarization, 3D question answering, and embodied question answering.

  • Architecture: NaviLLM uses a scene encoder to transform current visual observations into scene representations that are supplied to an LLM.The model comprises a scene encoder and an LLM, with schemas constructed from the resulting representations.
  • Scene Encoding: A pretrained ViT extracts visual features from each viewpoint image, whose features are then fused with a transformer encoder.The fused representations model spatial relationships among viewpoints, with angle and GPS information also incorporated.
  • Schema-Based Instruction: Schema-based instruction represents the task, current visual observation, trajectory history, and expected output in a unified multimodal format.The task may be an instruction, target object, or user question; the output hint specifies an action, answer, or summary.
  • Multi-task Learning: The schemas convert multiple embodied tasks into generation problems optimized with a unified cross-entropy objective.This supports joint learning across diverse task datasets.
  • Vision-Language Navigation: For VLN, the model predicts the ID of a viewpoint to move toward from navigation instructions, reachable-viewpoint observations, and updated history.History is updated with the scene representation of the selected viewpoint as the agent moves.
  • Additional Tasks: The framework also covers object localization, trajectory summarization, 3D question answering, and two-stage embodied question answering.These tasks reuse or omit history and combine navigation and question-answering schemas according to their requirements.

4. Experimental Setup

The experiments combine multiple embodied-navigation and 3D-question-answering datasets, task-specific metrics, and established baselines to evaluate NaviLLM across seen and unseen environments. The setup also specifies the model, training procedure, task constructions, and benchmark comparisons.

  • Model Details: The model fine-tunes a multi-view fusion module and Vicuna-7B-v0 while keeping the EVA-CLIP-02-Large scene-encoder ViT frozen.The fusion module is a 2-layer transformer encoder with hidden size 1024, and object features are additionally extracted from ViT-B16.
  • Training Data: NaviLLM is trained with a two-stage procedure on combined data from navigation, object-grounding, 3D-QA, and augmented datasets.Pretraining uses teacher forcing; multi-task fine-tuning alternates teacher forcing and student forcing, with LLaVA-23k added during fine-tuning.
  • Task Coverage: The setup evaluates navigation, object localization, trajectory summarization, 3D-QA, and held-out embodied question answering using task-specific data constructions.Trajectory summarization reverses instruction-trajectory pairs, while 3D-QA includes questions constructed from fine-grained R2R annotations; EQA is held out for generalization testing.
  • Metrics: Navigation evaluation uses SR, SPL, OSR, TL, and GP, with GP adopted for CVDN and SPL used for the other listed navigation datasets.SR measures reaching the target within a distance threshold, while SPL weights success by the ratio of ground-truth to actual path length.
  • 3D-QA Evaluation: ScanQA evaluation uses val and test-with-objects splits, reporting EM, METEOR, ROUGE-L, CIDER, and BLEU-4 against representative 3D-QA baselines.The listed baselines include VoteNet+MCAN, ScanRefer+MCAN, and 3D-LLM, identified as the current ScanQA state-of-the-art method.

5. Experimental Results

Experiments show that NaviLLM achieves strong performance with one model across established benchmarks and generalizes to held-out and unseen tasks. Ablations and visualizations further examine the roles of multi-task learning, pretrained LLMs, and schema-based instruction.

  • NaviLLM achieves state-of-the-art performance on CVDN, SOON, and ScanQA while remaining comparable to leading methods on R2R and REVERIE.
  • 7.90 GP on CVDN places NaviLLM first on the leaderboard, attributed to LLM-based dialog comprehension and unified training that mitigates data scarcity.
  • 26.3% EM on ScanQA improves 7.2% over 3D-LLM, indicating strong performance on 3D question answering.
  • Generalization Ability on Unseen Tasks: On held-out VLN tasks, NaviLLM improves GP by 32% on CVDN and SPL by 136% on SOON over task-specific DUET models, while achieving 28.10% SR on REVERIE.
  • Generalization Ability on Unseen Tasks: Zero-shot MP3D-EQA evaluation achieves 47.78% SR, 35.60% SPL, and 44.5% question-answering accuracy, combining navigation and question answering.
  • Ablation and Visualization: Multi-task learning improves all tasks, pretrained LLM initialization is important, and Figure 3 illustrates trajectory summarization, object navigation, EQA, and 3D captioning on unseen settings.

6. Conclusion

NaviLLM is presented as the first generalist model for embodied navigation, using schema-based instruction to unify diverse tasks and datasets. It achieves state-of-the-art results on several benchmarks and shows strong generalizability to unseen tasks.

  • NaviLLM adapts LLMs to diverse embodied-navigation tasks through schema-based instruction.The design unifies the tasks into a generation problem and allows data from various datasets to be used.
  • The single model achieves SoTA results on CVDN, SOON, and ScanQA, while matching the latest models on R2R and REVERIE.
  • NaviLLM demonstrates strong generalizability and promising results on unseen tasks.

A. Dataset Statistics

The paper reports dataset sizes and environment or scene coverage for CVDN, SOON, R2R, REVERIE, ScanQA, and EQA across their training, validation, and test splits.

  • CVDN: CVDN contains 4,742 training questions, 907 unseen-validation questions, and 1,384 test questions across 57, 10, and 16 environments, respectively.
  • SOON: SOON contains 2,780 training instructions, 339 unseen-validation instructions, and 1,411 test instructions across 34, 5, and 14 houses, respectively.
  • R2R: R2R contains 14,039 training instructions, 2,349 unseen-validation instructions, and 4,173 test instructions across 61, 11, and 18 houses, respectively.
  • REVERIE: REVERIE contains 10,466 training instructions, 3,521 unseen-validation instructions, and 6,292 test instructions across 60, 10, and 16 houses, respectively.
  • ScanQA: ScanQA contains 25,563 training questions, 4,675 validation questions, and 6,179 test questions across 562, 71, and 97 scenes, respectively.
  • EQA: EQA testing retains 849 entries after filtering validation data with inaccurate endpoints.

B. Hyperparameter Details

The experiments use a fixed view count across tasks and task-specific history lengths for the navigation settings.

  • The view number n is fixed at 36 for all tasks, while history round t varies by task.The history rounds are 15 for R2R, 15 for REVERIE, 30 for CVDN, and 20 for SOON.

C. Ablation for Schema Elements

Removing schema elements causes a significant performance drop, indicating that the proposed schemas help distinguish different tasks.

  • Removing schema elements significantly reduces performance, supporting their effectiveness in distinguishing different tasks.

D. Detailed Comparison on Benchmarks

The section compares NaviLLM with existing methods across CVDN, SOON, R2R, REVERIE, and ScanQA, while also documenting schema ablations and training data. On ScanQA, NaviLLM reaches second place among the top five leaderboard teams as of November 2023.

  • NaviLLM ranks second among the top five teams on the ScanQA leaderboard as of November 2023.
  • The study includes an ablation of schemas using models trained for 2,500 steps and evaluated on validation-unseen sets.
  • NaviLLM is compared with state-of-the-art methods on CVDN, SOON, R2R, and REVERIE through detailed benchmark tables.
  • Training-data statistics report the numbers of 3D scenes and instances, including augmentation datasets supplied by prior work.
  • The method uses instruction templates covering trajectory summarization and object localization tasks derived from R2R, SOON, and REVERIE.
Loading 2312.02010v3…