PaperlayerView exact revision on arXiv

Source-linked AI summary

An Embodied Generalist Agent in 3D World

Jiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu, Puhao Li, Yan Wang, Qing Li, Song-Chun Zhu, Baoxiong Jia, Siyuan Huang

arXiv:2311.12871v3cs.CVcs.AIcs.CLcs.LG

TL;DR

Existing generalist models have limited 3D input capacity and rarely address 3D grounding, embodied reasoning, and acting. LEO unifies multimodal 3D perception and action through two-stage training and LLM-assisted data generation, achieving strong performance across diverse tasks while retaining limitations in novel-scene generalization and action control.

  • Problem

    Generalist models have limited 3D input capacity and rarely explore 3D grounding, embodied reasoning, and acting tasks.

  • Method

    LEO uses a unified multimodal architecture and objective with 3D vision-language alignment, 3D vision-language-action instruction tuning, and LLM-assisted 3D-language data generation.

  • Results

    LEO demonstrates proficiency across 3D captioning, question answering, embodied reasoning, navigation, and manipulation, achieving state-of-the-art performance on most evaluated tasks.

  • Takeaways & Limitations

    LEO is an initial step toward embodied generalist agents that extend LLM abilities from text to the 3D world and embodied tasks.

  • Takeaways & Limitations

    LEO has limited generalization to novel scenes and a notable gap between vision-language learning and embodied action control.

Abstract

from arXiv · show

Leveraging massive knowledge from large language models (LLMs), recent machine learning models show notable successes in general-purpose task solving in diverse domains such as computer vision and robotics. However, several significant challenges remain: (i) most of these models rely on 2D images yet exhibit a limited capacity for 3D input; (ii) these models rarely explore the tasks inherently defined in 3D world, e.g., 3D grounding, embodied reasoning and acting. We argue these limitations significantly hinder current models from performing real-world tasks and approaching general intelligence. To this end, we introduce LEO, an embodied multi-modal generalist agent that excels in perceiving, grounding, reasoning, planning, and acting in the 3D world. LEO is trained with a unified task interface, model architecture, and objective in two stages: (i) 3D vision-language (VL) alignment and (ii) 3D vision-language-action (VLA) instruction tuning. We collect large-scale datasets comprising diverse object-level and scene-level tasks, which require considerable understanding of and interaction with the 3D world. Moreover, we meticulously design an LLM-assisted pipeline to produce high-quality 3D VL data. Through extensive experiments, we demonstrate LEO's remarkable proficiency across a wide spectrum of tasks, including 3D captioning, question answering, embodied reasoning, navigation and manipulation. Our ablative studies and scaling analyses further provide valuable insights for developing future embodied generalist agents. Code and data are available on project page.

1. Introduction

LEO addresses limited 3D understanding and underexplored embodied tasks by unifying multimodal perception, reasoning, planning, and action in one agent. It combines two-stage training, LLM-assisted data generation, and broad evaluation across 3D tasks.

  • Motivation: Existing generalist models have limited 3D scene-level understanding because 3D datasets are smaller and more costly to annotate than 2D datasets.Previous 3D vision-language models also under-explore large-scale unified pretraining and efficient finetuning.
  • LEO: LEO accepts egocentric 2D images, 3D point clouds, and text, then uses a unified interface, architecture, and objective for 3D tasks.Its visual tokens and text tokens form sequences processed by a decoder-only LLM with both text and action tokens.
  • Training: LEO is trained through 3D vision-language alignment and 3D vision-language-action instruction tuning using comprehensive object-level and scene-level datasets.An LLM-assisted pipeline generates 3D vision-language data with scene graphs, Object-centric Chain-of-Thought, and refinement procedures.
  • Evaluation: Across captioning, question answering, embodied reasoning, navigation, and manipulation, LEO achieves strong performance and generalization, including state-of-the-art results on most tasks.Ablations attribute performance to alignment, data diversity, instruction tuning, and object-centric representation, while scaling analyses find a scaling law.
  • Contributions: The work releases LEO’s data, code, and model weights alongside extensive experiments and analyses for embodied generalist agents.

2. Model

LEO converts multimodal inputs and responses into token sequences processed by a pretrained LLM with a prefix-language-modeling objective. Its architecture combines 2D image tokens, object-centric 3D tokens, text, and embodied actions.

  • Multimodal representation: LEO represents egocentric 2D images, object-centric 3D inputs, textual instructions, and responses as one ordered multimodal token sequence.This design supports both textual responses and embodied action commands within the same architecture.
  • Learning objective: The model trains with GPT-style prefix language modeling, predicting response tokens conditioned on system messages, multimodal inputs, and instructions.The prefix comprises tokens from the system message through the instruction.
  • Generation: A pretrained LLM aligns multimodal tokens and generates text or reserved action tokens, which are mapped back to embodied commands when needed.Inference uses beam search for textual responses and converts action-token outputs into action commands.
  • 3D representation: Object-centric 3D tokens encode individual point-cloud objects and use spatial attention biases based on relative position and size to capture 3D relations.
  • Parameter tuning: LEO contains approximately 7B parameters, with approximately 142M tuned while the pretrained 3D encoder and LLM remain frozen.The trainable components include the 2D image encoder, Spatial Transformer, and LoRA parameters.

3. Datasets

LEO uses two datasets for 3D vision-language alignment and 3D vision-language-action instruction tuning, covering understanding, dialogue, planning, navigation, and manipulation. An LLM-assisted pipeline uses scene graphs, O-CoT prompting, and refinement to expand and improve 3D-language data.

  • 3. Datasets: LEO-align targets 3D vision-language alignment, while LEO-instruct targets instruction following across diverse 3D vision-language-action tasks.The two-stage split addresses data scarcity and separates alignment from broader capability training.
  • LEO-align: LEO-align includes object-level captions, object-in-the-scene referring expressions, and scene-level captioning tasks.
  • LEO-instruct: LEO-instruct spans grounded scene understanding, question answering, dialogue, planning, navigation, and manipulation in 3D scenes.Tasks require either natural-language responses to scenes and questions, coherent responses to complex instructions, or embodied acting.
  • Data generation: Scene-graph prompting supplies object attributes and spatial relations, enabling LLMs to generate detailed 3D-language data beyond object-box-based context.
  • Data generation: Object-centric Chain-of-Thought asks LLMs to expose candidate object labels and IDs during generation, while subgraph sampling increases scene-graph diversity.
  • Quality control: Human-defined refinement removes inadequate responses, rewrites unnatural narratives, and corrects reasoning or hallucination errors against scene-graph ground truth.Quantitative checks report consistent quality improvements from scene-graph prompting, O-CoT prompting, and refinement, with the complete pipeline outperforming 3D-LLM.

4. Capabilities and Analyses

LEO is evaluated as a unified embodied generalist across 3D vision-language, dialogue and planning, navigation, and manipulation tasks. Results show strong task performance, while ablations and scaling analyses identify benefits and limitations of alignment, broad instruction tuning, data balance, and model scaling.

  • Evaluation scope: LEO handles 3D captioning, question answering, embodied reasoning, dialogue, planning, navigation, and manipulation within a comprehensive evaluation.The evaluation spans perception, grounding, reasoning, planning, and acting, including Scan2Cap, ScanQA, SQA3D, ObjNav, and CLIPort.
  • 3D vision-language understanding and reasoning: LEO surpasses state-of-the-art single-task and task-specific fine-tuned models on 3D dense captioning and 3D question answering.Its LLM-based approach supports open-ended responses while retaining strong quantitative results.
  • Dialogue and planning: LEO generates scene-grounded plans and dialogue with concrete objects, plausible actions, and detailed spatial relations.Quantitative comparison is unavailable for dialogue and planning because comparable benchmarks do not exist.
  • Embodied action in 3D world: LEO reaches comparable robotic-manipulation performance, performs better on some unseen tasks, and achieves comparable navigation success with better SPL on MP3D-val.On HM3D-val, results confirm zero-shot generalization to novel scenes; truncated past actions may contribute to lower navigation success than recurrent baselines.
  • Ablation studies: Alignment improves performance, especially on Scan2Cap, while broad generalist instruction tuning improves cross-scene and cross-task generalization over a specialist.Including embodied acting data can reduce 3D vision-language performance, likely because of a language-action gap and imbalanced data scale.
  • Data balance and scaling: Balanced tuning data mitigates hallucination, and scaling data and model size generally lowers loss, although gains from larger LLMs may begin to saturate.The analysis also finds that alignment consistently improves losses and that data scale should match model capacity.
  • Limitations and future directions: The authors identify limited generalization to novel scenes and a notable gap between vision-language learning and embodied action control as important limitations.They propose richer 3D VL data, continued VL-action bridging, and safety and alignment research as future directions.

B. Data

The data covers object-, scene-, language-, reasoning-, navigation-, and manipulation-oriented tasks for understanding and acting in 3D environments. It combines existing datasets with LLM-generated examples and demonstrations.

  • Object- and scene-level tasks: Object-level captioning uses single 3D objects to predict detailed language descriptions from Cap3D.
  • Object- and scene-level tasks: Object-in-the-scene captioning trains LEO to predict referring expressions that describe an object’s relations to surrounding scene elements.
  • Object- and scene-level tasks: Scene-level captions describe key objects, attributes, functionalities, inter-object relations, room types, and styles, with approximately 20K captions generated from scene graphs.
  • Language and reasoning tasks: 3D question answering combines ScanQA and SQA3D with approximately 35K LLM-generated 3RScanQA pairs refined for quality.
  • Language and reasoning tasks: The dataset includes approximately 11K multi-round 3D dialogues and approximately 14K scene-aware task-plan pairs generated on 3RScan scenes.
  • Embodied interaction tasks: Embodied tasks cover ObjNav using multimodal inputs and discrete motor commands, plus CLIPort manipulation with discretized actions and 100K demonstrations per selected task.

B.3. Design of Seed Tasks for LLM-assisted 3D Data Generation

The seed-task design uses scene-graph information to prompt LLMs for captions, question answering, dialogue, and planning data. Few-shot demonstrations and structured response requirements guide generation.

  • Caption generation: Object Scene Caption describes an object together with adjacent-object spatial relationships, while Scene Caption describes key objects and relations across the whole scene.
  • QA and dialogue generation: QA prompts query scene graphs about attributes, counting, existence, spatial relations, object types, affordances, room types, and other 3D properties.
  • QA and dialogue generation: Dialogue prompts support multi-round conversations, including higher-level tasks such as searching for specific objects.
  • Planning generation: Planning prompts ask ChatGPT to generate a high-level task followed by 5–10 object-interaction action steps.
  • Prompt construction: Few-shot prompts pair scene-graph content with human-labeled responses before ChatGPT generates responses for new scene graphs.
  • Prompt construction: Dialogue prompts require object attributes, spatial relations, commonsense topics, conversational context, and an intermediate thought identifying involved objects.

B.5. Analysis of the Object-Centric Chain-of-Thought

The analysis examines Object-centric Chain-of-Thought (O-CoT) for improving LLM-generated 3D data, alongside rule-based refinement procedures that detect and correct response errors.

  • O-CoT analysis: The experiment varies two demonstration seeds to generate dialogues across all 3DSSG scenes and evaluates Object Counting accuracy.
  • O-CoT analysis: O-CoT consistently improves answer accuracy for Object Counting questions, although errors remain afterward.
  • Response refinement: Refinement filters raw responses using human-defined procedures grounded in the 3D scene graph.
  • Response refinement: The refinement handles object counting, existence, non-existence, negative responses, and unexpected object IDs.
  • Response refinement: Regular-expression matching detects errors and supports correction, while responses retaining IDs after failed rewrites are discarded.

B.7. Subgraph Sampling

Subgraph sampling is evaluated as a way to diversify scene-caption prompts while preserving consistency. Scene-graph prompting is also compared with box-based prompting for spatial relations.

  • Subgraph sampling: Sampling rates vary according to the number of nodes in each scene.
  • Subgraph sampling: The comparison generates captions for the same scene from both the full graph and a sampled subgraph, then uses GPT-4 to assess similarities and differences.
  • Subgraph sampling: Subgraph sampling maintains both consistency and diversity in scene captions according to the comparative evaluation.
  • Prompting comparison: Scene-graph and box-based prompting are compared using similar demonstrations and identical new-scene queries for fairness.
  • Prompting comparison: The comparison focuses mainly on spatial relations because 3D-LLM does not elaborate on attribute-related prompts.
  • Prompting comparison: Scene-graph prompting produces more diverse and reliable spatial relations than box-based prompting in the illustrated comparison.

B.9. Additional Comparision Regarding Dataset Quality

The dataset-quality comparison contrasts LEO’s generated object captions with ScanScribe, focusing on grammatical correctness in 3RScan examples.

  • LEO’s object-caption dataset did not manifest the grammar errors observed in ScanScribe.The comparison uses object captions from 3RScan scenes, including the scene represented in Table A.7.

C. Data Examples

The appendix presents examples of LEO’s object-, scene-, and planning-data instructions alongside multimodal prompts, spatial features, embodiment encoding, and action representations.

  • Multimodal representation: LEO combines egocentric image tokens with object-centric 3D tokens in an interleaved multimodal sequence.The sequence includes text prefixes, an ego-view image field, and scene-object tokens.
  • Data examples: The examples cover object-level and scene-level caption instructions, planning instructions, and navigation or manipulation prompts.Navigation instructions specify a goal and past actions; manipulation follows the same pattern with a task description.
  • 3D feature encoding: Object-centric 3D features use a frozen ScanNet-pretrained PointNet++ encoder with 1024 sampled points per object.The resulting object features are processed for downstream 3D representation.
  • Spatial feature encoding: Spatial Transformer processing explicitly fuses pairwise object relations into attention and outputs object tokens for the LLM.The spatial relations use object-center distance and horizontal and vertical angles, with a three-layer, eight-head transformer.
  • Embodiment encoding: An embodiment token is inserted into the 3D object list to support embodiment-aware reasoning in navigation, embodied reasoning, and object-in-the-scene captioning.Embodiment information includes the agent’s position and orientation, while object-in-the-scene captioning assumes the agent is located at the target object.
  • Action representation: LEO represents navigation with four action tokens and discretizes CLIPort manipulation poses into 516 tokens.The action tokens are mapped back to commands after the LLM produces its response.

G.2. Model Ablation

The ablations examine language-model scale, point-cloud backbones, and dialogue or planning data, showing distinct effects across model components and capabilities.

  • LLM ablation: Scaling from OPT-1.3B to Vicuna-7B yields notable improvements, while Vicuna-7B and Vicuna-13B perform comparably.The result suggests potential performance saturation beyond the 7B scale in the tested settings.
  • Point-cloud backbone: Replacing PointNet++ with Point-BERT produces no notable performance difference on the reported tasks.The Point-BERT checkpoint is adapted for 6-channel XYZRGB input and language-aligned 3D object representations.
  • Dialogue and planning data: Removing dialogue and planning data causes more hallucinations on unanswerable questions and worse NLP performance than the default model.The evaluation compares models using human preferences summarized with TrueSkill.
  • Evaluation setup: The ablation evaluates answerable, unanswerable, and NLP question subsets, with 30 representative questions collected for each subset.Unanswerable questions lack sufficient information in the provided 3D scenes, whereas NLP questions test language functionality.

H. Evaluation Details

The evaluation details refine open-ended QA scoring, describe navigation data and policy constraints, and report that data refinement consistently improves results, especially on QA reasoning.

  • 3D question answering: Exact-match scoring can wrongly penalize valid open-ended answers, motivating a refined protocol that accepts subset matches between prediction and ground truth.The authors still speculate that this simple refinement is insufficient for fully sound evaluation of flexible human language.
  • 3D question answering: The appendix provides representative ScanQA validation examples to justify the refined exact-match protocol.Table A.16 is explicitly described as documenting the protocol’s rationale.
  • Embodied navigation: The Object Navigation training set uses all 57 MP3D training scenes and approximately 60K shortest-path episodes, evaluated on MP3D and HM3D validation tasks.The evaluation includes the original MP3D validation split and a newly introduced HM3D task.
  • Embodied navigation: LEO’s navigation policy is feed-forward, using the instruction, current 2D and 3D state, and past four actions to predict the next action.The absence of recurrence is described as a trade-off between performance and training efficiency, with more sophisticated recurrent policies reserved for future work.
  • Data refinement: Data refinement produces consistent improvements, primarily benefiting reasoning and QA tasks.The authors attribute this pattern partly to refinement focusing mainly on QA and dialogue data.

I.2. Data Comparison

Comparisons examine how training data and model design affect LEO across 3D understanding and embodied tasks. LEO’s collected data and object-centric representation show advantages, while qualitative results reveal both grounding strengths and response limitations.

  • Training Data: LEO’s collected training data consistently outperforms LL3DA data on Scan2Cap and ScanQA.The comparison reports a consistent performance gap favoring the original collected data.
  • Model Comparison: With the same training data, LEO outperforms LL3DA on most metrics, with a significant advantage on ScanQA.The ScanQA gap indicates a major advantage for object-centric 3D representation in reasoning tasks.
  • Model Comparison: LEO consistently outperforms 3D-LLM when both use the same 3D-LLM data, despite using a much smaller training-data subset.This comparison is made against 3D-LLM’s task-specific fine-tuning results, whereas LEO is evaluated after instruction tuning without task-specific fine-tuning.
  • Embodied Acting: LEO generalizes to novel objects in object navigation, whereas human demonstrations produce weaker performance for agents without recurrence.The authors attribute the demonstration-data difficulty to greater exploration and identify the absence of recurrence as a constraint.
  • 3D Understanding: LEO correctly refers to queried objects and describes spatial relationships, but its responses remain confined to simple, insufficiently diverse formats.The qualitative Scan2Cap results identify more flexible responses while maintaining accuracy as future work.
  • Embodied Reasoning: SQA3D examples show embodied reasoning about current location and orientation, including identifying objects at the agent’s location and counting beds in front of it.These examples require understanding both the scene and the situation of embodiment.
Loading 2311.12871v3…