PaperlayerView exact revision on arXiv

Source-linked AI summary

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang

arXiv:2502.09560v3cs.AIcs.CLcs.CV

TL;DR

MLLM-based embodied agents lack comprehensive evaluation despite their potential for real-world tasks. EmbodiedBench addresses this gap with a standardized benchmark and unified agent framework spanning action levels and capabilities, finding strong high-level performance but persistent low-level manipulation difficulty.

  • Problem

    MLLM-based embodied agents remain underexplored because comprehensive evaluation frameworks are lacking.

  • Method

    EmbodiedBench evaluates agents on 1,128 tasks across four environments and six capability-oriented subsets using a unified framework combining visual perception, examples, history, and feedback.

  • Results

    MLLMs perform strongly on high-level tasks but struggle with low-level tasks, with GPT-4o scoring 28.9% on EB-Manipulation.

  • Takeaways & Limitations

    EmbodiedBench highlights low-level manipulation and long-horizon planning as key challenges for developing more capable vision-driven embodied agents.

  • Takeaways & Limitations

    The evaluation is conducted solely in simulated environments without real-world experiments.

Abstract

from arXiv · show

Leveraging Multi-modal Large Language Models (MLLMs) to create embodied agents offers a promising avenue for tackling real-world tasks. While language-centric embodied agents have garnered substantial attention, MLLM-based embodied agents remain underexplored due to the lack of comprehensive evaluation frameworks. To bridge this gap, we introduce EmbodiedBench, an extensive benchmark designed to evaluate vision-driven embodied agents. EmbodiedBench features: (1) a diverse set of 1,128 testing tasks across four environments, ranging from high-level semantic tasks (e.g., household) to low-level tasks involving atomic actions (e.g., navigation and manipulation); and (2) six meticulously curated subsets evaluating essential agent capabilities like commonsense reasoning, complex instruction understanding, spatial awareness, visual perception, and long-term planning. Through extensive experiments, we evaluated 24 leading proprietary and open-source MLLMs within EmbodiedBench. Our findings reveal that: MLLMs excel at high-level tasks but struggle with low-level manipulation, with the best model, GPT-4o, scoring only 28.9\% on average. EmbodiedBench provides a multifaceted standardized evaluation platform that not only highlights existing challenges but also offers valuable insights to advance MLLM-based embodied agents. Our code and dataset are available at https://embodiedbench.github.io.

1. Introduction

EmbodiedBench addresses the limited evaluation of MLLM-based embodied agents with a comprehensive benchmark spanning action levels and agent capabilities. It combines diverse environments, fine-grained evaluation, a unified agent framework, and extensive model studies.

  • Existing MLLM embodied-agent evaluation remains underexplored, especially regarding vision and low-level navigation and manipulation.
  • EmbodiedBench contains 1,128 testing instances across four environments with hierarchical high-level and low-level action demands.EB-ALFRED and EB-Habitat emphasize task decomposition and planning, while EB-Navigation and EB-Manipulation require precise perception and spatial reasoning.
  • Its capability-oriented framework evaluates basic solving, commonsense reasoning, complex instruction understanding, spatial awareness, visual perception, and long-horizon planning.
  • The unified agent framework integrates egocentric visual perception, few-shot examples, interaction history, and environment feedback for decision-making.The study evaluates leading closed-source and open-source MLLMs using this framework.
  • 40%–70% performance reductions occur when visual input is removed on some tasks, while high-level-task impact is minimal.Ablations also examine image resolution, multi-step image input, and visual in-context learning.
  • The work contributes a benchmark suite, an efficient MLLM agent framework, and evaluations and ablations yielding insights for vision-driven agent design.

2. Related Work

Prior work includes embodied-agent systems, simulators, and evaluation benchmarks, but EmbodiedBench distinguishes itself through multi-domain coverage and fine-grained comparison.

  • Embodied-agent research uses LLMs mainly for high-level planning and MLLMs for perception or direct and code-based decision-making.
  • The field has developed a variety of simulators and evaluation benchmarks for embodied agents.
  • EmbodiedBench covers household, manipulation, and navigation tasks as a multi-domain benchmark.
  • Its fine-grained evaluation is multidimensional rather than limited to overall accuracy.

3. Problem Formulation

The paper formulates embodied agents through hierarchical action levels and a vision-and-language POMDP. High-level actions decompose into executable low-level primitives, while agents act from visual histories and language goals.

  • Low-level actions are atomic translational or rotational commands directly executable by robots.A robotic arm action may include translation, rotation, and gripper state; moving forward 0.1 m is another example.
  • High-level actions are sequences of low-level executable primitives, such as rotating, scanning, and moving toward a target.
  • Vision-driven agents make sequential decisions from visual perception and language instructions in a partially observable environment.
  • The objective is to maximize task-success probability until completion or the maximum horizon is reached.
  • The agent maintains an interaction history of images and actions and selects actions through a policy conditioned on the instruction and history.

4. EmbodiedBench

EmbodiedBench evaluates MLLM-based embodied agents across four environments, spanning high-level planning and low-level navigation and manipulation. Its benchmark combines diverse task settings, capability-oriented subsets, and a unified pipeline for multimodal reasoning and executable planning.

  • Benchmark environments: EmbodiedBench comprises four environments covering high-level task decomposition and planning alongside low-level actions requiring precise perception and spatial reasoning.The environments are EB-ALFRED, EB-Habitat, EB-Navigation, and EB-Manipulation.
  • High-level tasks: EB-ALFRED supports eight high-level skill types and a dynamic action space ranging from 171 to 298 actions.The simulator provides egocentric observations and textual feedback on action validity and failure reasons.
  • High-level tasks: EB-Habitat uses 282 language instruction templates and 70 high-level skills, with navigation restricted to receptacle-type objects.Its constraints require agents to visit multiple locations to find desired items.
  • Low-level tasks: EB-Navigation evaluates locating target objects from visual observations and textual feedback using low-level robot-executable actions.Success requires reaching within a specified distance of the target without direct positioning data.
  • Low-level tasks: EB-Manipulation controls a robotic arm with a 7-dimensional action vector, enhanced by action discretization, detection boxes, and object pose estimation.Position components are divided into 100 bins and orientation components into 120 bins.
  • Capability-oriented evaluation: Six capability subsets assess base task solving, commonsense reasoning, complex instruction understanding, spatial awareness, visual appearance, and long-horizon planning.The Long Horizon subset typically requires more than 15 steps in EB-ALFRED.
  • Agent pipeline: The unified agent pipeline combines multimodal inputs, reflection, reasoning, and structured executable plans, supporting dynamically sized multi-step plans and replanning after failures.Inputs include language instructions, visual perceptions, demonstrations, interaction history, and task-specific information.

5. Experiments

Experiments benchmark 24 MLLMs across hierarchical embodied tasks and capability-focused subsets, revealing strong high-level performance but persistent low-level and long-horizon weaknesses. Ablations show that visual design choices substantially affect manipulation performance, while planning errors remain prevalent.

  • Benchmark Results: 24 MLLMs are evaluated across high-level and low-level embodied tasks in EMBODIEDBENCH.The benchmark includes 8 proprietary and 16 open-source models.
  • Benchmark Results: GPT-4o leads proprietary low-level results with 57.7% on EB-Navigation and 28.9% on EB-Manipulation, while Claude-3.5-Sonnet leads high-level results.Claude-3.5-Sonnet scores 64.0% on EB-ALFRED and 68.0% on EB-Habitat.
  • Benchmark Results: Low-level tasks depend more strongly on vision: removing vision reduces GPT-4o’s EB-Navigation performance from 57.7% to 17.4%, with long-horizon planning falling to 0%.High-level tasks can remain comparable or improve without visual input.
  • Benchmark Results: Models perform unevenly across capability subsets, and long-horizon planning is consistently the most difficult subset.On EB-Habitat, Claude-3.5-Sonnet drops from 96% on the base subset to 58%, while GPT-4o drops from 86% to 64%.
  • Language-centric Ablation: Language-centric ablations show that removing environment feedback lowers success by 10% for GPT-4o and 8% for Claude-3.5-Sonnet, while zero-shot prompting reduces success to around 40%.The comparisons use the EB-ALFRED base subset.
  • Visual-centric Ablation: Visual-centric ablations identify 500 × 500 as the strongest tested camera resolution, show large gains from detection boxes, and find visual ICL superior to language-only ICL.Removing detection boxes lowers GPT-4o from 39.6% to 27.1%; visual ICL improves Claude-3.5-Sonnet by 16.7%. Multi-step images often confuse current MLLMs.

6. Conclusion

EMBODIEDBENCH evaluates MLLM-based embodied agents across varying action levels and capability-oriented subsets, identifying low-level manipulation and long-horizon planning as key challenges. It also examines how vision input matters across tasks.

  • EMBODIEDBENCH evaluates MLLM-based embodied agents across tasks with varying action levels and capability-oriented subsets.
  • The experiments identify difficulties in low-level manipulation and long-horizon planning.
  • The findings show that the significance of vision input varies across tasks.

Limitations

The evaluation is limited to simulated environments and does not include real-world experiments. Simulation improves reproducibility while reducing time, financial costs, and safety risks, but leaves practical deployment untested.

  • The evaluation is conducted solely in simulated environments, without real-world experiments.
  • Simulation offers reproducibility while reducing time, financial costs, and safety risks compared with real-world testing.
  • Future work could develop more realistic simulations or standardized, cost-effective real-world test suites.

Impact Statement

The paper situates MLLMs within the broader development of embodied agents, where foundation models support perception, decision-making, and action in physical and simulated environments. Existing work spans planning, control, perception, policy generation, simulators, and evaluation benchmarks.

  • Foundation models, including LLMs and MLLMs, transform how embodied agents perceive, decide, and act in physical and simulated environments.
  • LLMs have been used for high-level planning and low-level control, while MLLMs support perception tasks such as object-attribute identification and action recognition.
  • MLLMs have also been applied to end-to-end action generation, visual markers, masks, constraints, guidance, and code generation for policies or value functions.
  • Vision Language Action Models use MLLMs or language-conditioned diffusion models and are trained on low-level robotics action data.
  • The field has developed diverse simulators and evaluation benchmarks alongside these embodied-agent approaches.

B. Future Research Directions

Future research directions target broader task diversity, stronger low-level spatial reasoning, long-horizon planning, multi-step visual understanding, visual in-context learning, multimodal-agent training, and robustness. These directions aim to address current capability and deployment gaps.

  • Expanding Task Diversity: Future benchmarks should expand task diversity to realistic settings such as autonomous driving, multi-agent collaboration, and human-agent interaction.
  • Low-Level Tasks and Spatial Reasoning: Current agents struggle with spatial reasoning and low-level control, motivating stronger integration of spatial reasoning with low-level action planning.
  • Long-Horizon Planning: Long-horizon planning remains challenging, motivating hierarchical planning, memory-augmented methods, and world models.
  • Multi-step/Multi-view Image Understanding: Current MLLMs struggle with multi-step and multi-view image inputs, motivating improved temporal reasoning and spatial awareness.
  • Visual In-context Learning (ICL): Visual in-context learning is effective for embodied decision-making and can improve adaptability without fine-tuning, but better methods remain open.
  • Training Multimodal Embodied Agents: Training MLLMs for embodied tasks through embodied pretraining, imitation learning, or reinforcement learning could enhance performance.
  • Robustness and Generalization of MLLM Agents: Real-world applicability requires further study of MLLM-agent robustness and generalization.
  • These research directions could move the field closer to real-world applications.

C. Details about EMBODIEDBENCH Environments and Datasets

EMBODIEDBENCH combines high-level household and rearrangement tasks with low-level navigation and manipulation environments, using varied action hierarchies and capability-focused evaluation. Its datasets adapt existing simulators and benchmarks while adding task, action-space, and instruction improvements.

  • EB-ALFRED: EB-ALFRED covers seven household task types using a dynamic 171–298-action space and improved support for multiple object instances.The simulator also corrects execution errors and refines instructions to improve task solvability.
  • EB-Habitat: EB-Habitat samples 50 instances from each of six capability-oriented subsets, producing 300 testing instances.The subsets cover commonsense reasoning, complex instructions, visual appearance, spatial relationships, and long-horizon tasks.
  • EB-Navigation: EB-Navigation requires agents to use first-person visual observations and validity feedback to select low-level movement, rotation, and camera-tilting actions.The environment evaluates action validity after execution, including collisions or obstacles, and returns only valid or invalid feedback.
  • Agent behavior and evaluation: Multi-step planning reduces average planner steps by around 50%–80% relative to environment steps, with GPT-4o using 2.6 planner steps versus 12.9 environment steps in EB-Manipulation.More capable and larger models generally require fewer planner and environment steps, although the reported averages have inherent inaccuracies.
  • Agent behavior and evaluation: Resolution affects environments differently: low-level EB-Manipulation and EB-Navigation perform best at 500×500, while moderate changes yield 2%–4% gains elsewhere.The results suggest that insufficient detail and excessive visual complexity can both hinder task execution.

F.4. Detection Boxes

The ablations show that visual augmentation is task-dependent: detection boxes and visual in-context learning can help, whereas temporal or multi-view image inputs may hurt performance. The resulting design enables detection boxes only for EB-Manipulation.

  • Detection boxes: Detection boxes improve EB-ALFRED and EB-Manipulation, including a nearly 10% boost for EB-Manipulation.The gains are attributed to improved object recognition and interaction for models such as GPT-4o and Claude-3.5-Sonnet.
  • Detection boxes: Detection boxes hinder EB-Navigation because they can obscure visual cues needed for path planning.A single target-object box was tested to reduce the clutter associated with multiple boxes.
  • Detection boxes: Detection boxes are enabled by default only for EB-Manipulation, reflecting task-specific differences in visual requirements.They are disabled for the other task groups because visual strategies do not transfer uniformly across environments.
  • Multi-step images: Adding the latest three sequential images decreases performance, particularly in EB-Manipulation, rather than improving decision-making.The authors suggest models struggle to relate multiple sequential images to the current state.
  • Multi-view images: Multi-view inputs combine front and top-down images for EB-Navigation or front and wrist views for EB-Manipulation.The experiment tests whether simultaneous viewpoints improve performance in the two low-level environments.
  • Visual in-context learning: Visual in-context learning supplies image observations alongside examples of successful low-level actions, improving Claude-3.5-Sonnet’s manipulation performance by 16.7%.The images help connect action choices with object positions in the scene.

G. Further Discussion on Chat History as Input for EB-Navigation

Chat history affects EB-Navigation differently across model families. Proprietary models generally benefit, while open-source models show mixed preferences, so chat history is used only for navigation rather than globally.

  • Observed effects: EB-Navigation is the most sensitive environment to historical input among the four benchmark environments.The analysis concerns prior conversations paired with images, distinct from the multi-step image setting.
  • Observed effects: Proprietary MLLMs tend to benefit from chat history, particularly on long-horizon tasks, whereas open-source models show mixed preferences.Llama-3.2 and InternVL improve with chat history, while Qwen2-VL performs best with single-step input.
  • Design choice: Chat history is adopted only for EB-Navigation because the authors lack a definitive explanation for the model-family discrepancy.They hypothesize that training on multimodal multi-turn dialogue may influence performance on this input format.

H.2. Error Analysis for EB-Navigation

EB-Navigation errors span perception, reasoning, planning, and output formatting. The analysis links failures to object recognition, 3D spatial reasoning, movement precision, and valid structured action generation.

  • Error taxonomy: Navigation failures are categorized as perception, reasoning, and planning errors, each associated with a distinct stage of the agent pipeline.Perception errors occur during visual state description, reasoning errors during reflection and reasoning, and planning errors during plan generation.
  • Perception errors: Perception errors include missing a visible target and hallucinating a target that is absent.Hallucinated detections can prompt premature conclusions and incorrect decisions instead of further exploration.
  • Reasoning errors: Reasoning errors arise when models infer paths incorrectly or fail to reflect on spatial structure after failed actions and feedback.These failures indicate difficulty constructing coherent 3D representations from sequential 2D observations.
  • Planning errors: Planning errors persist even when the general direction is correct because distance estimation and movement execution can be misaligned.One example is overshooting the target by taking three steps instead of two.
  • Output-format errors: Larger models tend to produce fewer JSON format errors, while smaller models more often fail to generate valid JSON files.Format errors occur across the six subsets and affect proprietary models as well as smaller open-source models.
  • Prompt and action execution: The navigation prompt requires visual-state description, reasoning, planning, and a final action ID in JSON format.The action space provides eight movement, rotation, and camera-tilting actions, while invalid-action feedback guides subsequent plans.

J. Supplementary Case Studies of Successful Planning

The case studies show successful planning examples across four environments, alongside failure cases exposing planning, perception, and spatial-reasoning challenges.

  • Successful planning examples: Successful planning examples span EB-ALFRED, EB-Habitat, EB-Navigation, and EB-Manipulation.Examples feature Claude-3.5-Sonnet, InternVL2.5-78B, GPT-4o, and Gemini-1.5-pro, respectively.
  • EB-ALFRED: In EB-ALFRED, one task asks the agent to slice a tomato, microwave it, and place it in the sink.
  • EB-Habitat: In EB-Habitat, the instruction requires moving a spoon and sponge to the brown table and cleanser to the black table.
  • EB-Navigation and EB-Manipulation: EB-Navigation and EB-Manipulation examples involve approaching a pot closely and stacking a star on a cylinder.These instructions respectively require spatial navigation and object placement.
  • Failure cases: Failure cases reveal errors in executable planning, object-color perception, and spatial reasoning across EB-ALFRED, EB-Manipulation, and EB-Navigation.The examples include continuing to interact with the first book instead of locating “Book 2,” misidentifying an object’s color, and failing to navigate around a countertop after recognizing blockage.
Loading 2502.09560v3…