EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

• Existing MLLM embodied-agent evaluation remains underexplored because prior benchmarks such as VisualAgentBench focus mainly on high-level planning and do not resolve the roles of vision, navigation, or manipulation.The benchmark is motivated by the need for standardized and automated evaluation frameworks enabling comprehensive assessment and comparison.Source: §1 · p. 1
• 1,128 testing instances across four environments support evaluation from high-level task decomposition and planning to low-level navigation and manipulation requiring precise perception and spatial reasoning.EB-ALFRED and EB-Habitat focus on high-level tasks, whereas EB-Navigation and EB-Manipulation require translational or rotational control.Source: §1 · pp. 1–2
• Six capability-oriented subsets assess basic task solving, commonsense reasoning, complex instruction understanding, spatial awareness, visual perception, and long-horizon planning.This fine-grained framework complements prior evaluations centered mainly on overall accuracy or module-specific performance.Source: §1 · p. 2
• MLLMs excel at high-level tasks but struggle with low-level manipulation, while long-horizon planning is the most challenging evaluated capability.The benchmark evaluates 24 leading closed-source MLLMs and open-source models ranging from 7B to 90B parameters.Source: §1 · p. 2
• The unified agent framework integrates egocentric visual perception, few-shot in-context examples, interaction history, and environment feedback for decision-making across high- and low-level tasks.The contributions comprise the benchmark suite, an efficient MLLM agent framework, and extensive evaluations and ablation studies for vision-driven agent design.Source: §1 · p. 2
• 40%–70% performance degradation after removing vision demonstrates that visual input is crucial for low-level tasks but has minimal impact on high-level tasks.Ablations also examine image resolution, multi-step image input, and visual in-context learning.Source: §1 · p. 2

Embodied Agents

• LLMs primarily support high-level planning, while MLLMs handle perception, directly generate actions, or produce code for policy and value functions.The cited approaches span planning, perception, end-to-end action generation, and code-based policy or value-function development.Source: §2 · p. 2

Simulators and Benchmarks

• The field has developed a variety of simulators and evaluation benchmarks for embodied agents.Examples include simulators and benchmarks cited across prior work, with additional related works listed in Appendix A.Source: §2 · pp. 2–3

Benchmark Comparison

• Table 1 compares EmbodiedBench with existing benchmarks as a multi-domain, fine-grained evaluation spanning household, manipulation, and navigation tasks.The comparison distinguishes multi-dimensional evaluation from overall accuracy and notes that some related benchmarks cover household, games, or Web domains.Source: Table 1 · p. 3, §2 · p. 3

Definition of Action Levels

• Actions are classified as low-level executable commands or high-level sequences of low-level primitives, following Ma et al. (2024b) and Belkhale et al. (2024).Low-level actions specify translational or rotational displacements and may include a gripper state; high-level actions decompose goals such as finding an object into sequential behaviors.Source: §3 · p. 3

Vision-driven Agents

• Vision-driven agents are modeled as language-conditioned POMDPs over hidden states, visual observations, action spaces, transition dynamics, instruction goals, and task-completion rewards.The reward is 1 when the underlying state satisfies the instruction and 0 otherwise.Source: §3 · p. 3
• The agent maintains an image-action history, follows a policy π(a_t|L, h_t), and optimizes expected terminal reward until success or the maximum horizon is reached.The terminal timestep τ occurs when the instruction is achieved or the allowed interaction horizon ends.Source: §3 · p. 3

4. EmbodiedBench

• EmbodiedBench comprises EB-ALFRED, EB-Habitat, EB-Navigation, and EB-Manipulation, covering varied action levels and six core embodied-agent capabilities.The benchmark introduces new datasets and enhances existing simulators for comprehensive evaluation.Source: §4 · p. 3

4.1. High-level and Low-level Tasks

• The four environments span high-level household skills and low-level navigation or manipulation, with dynamic action spaces, visual feedback, navigation constraints, and discretized robotic-arm control.EB-ALFRED supports eight skill types and 171–298 actions; EB-Habitat evaluates 70 skills from 282 instruction templates; EB-Navigation uses eight low-level actions; EB-Manipulation uses discretized position and orientation controls with object localization aids.Source: §4.1 · pp. 4–5

4.2. Capability-oriented Data Collection

• The capability-oriented data covers base solving, commonsense reasoning, complex instructions, spatial awareness, visual appearance, and long-horizon planning across the four environments.EB-ALFRED and EB-Habitat each contain 300 test instances, EB-Navigation contains 300 across five subsets, and EB-Manipulation contains 228; Navigation omits spatial awareness and Manipulation omits long horizon.Source: §4.2 · p. 5

4.3. Vision-driven Agent Design

• The unified agent pipeline processes multimodal inputs, reflects on feedback, reasons about goals, and produces structured executable plans with dynamically chosen multiple actions.Agents primarily use current-step images, while manipulation receives detection boxes and object positions; failed or invalid plans trigger replanning from the latest state.Source: §4.3 · pp. 5–6

5.1. Experimental Setups

• 24 MLLMs are evaluated with temperature 0, 2048 maximum completion tokens, 500 × 500 images, task-specific step limits, and task success rate as the primary metric.The benchmark includes 8 proprietary and 16 open-source models.Source: §5.1 · p. 6

5.2. Benchmark Results

• 57.7% to 17.4%: removing vision sharply reduces GPT-4o’s EB-Navigation success and collapses long-horizon planning to 0%, whereas high-level tasks remain largely text-dependent.GPT-4o (Lang) and GPT-4o-mini (Lang) can match or outperform vision-enabled counterparts on EB-ALFRED and EB-Habitat.Source: §5.2 · p. 6
• 96% to 58%: long-horizon performance drops substantially from base scores, and Claude-3.5-Sonnet’s EB-Habitat decline exceeds GPT-4o’s 86% to 64% drop.Long-horizon subsets are consistently the most difficult across high- and low-level tasks; models also diverge across capabilities, with GPT-4o outperforming Claude-3.5-Sonnet on EB-Habitat long-horizon tasks.Source: §5.2 · p. 6
• 28.9%: GPT-4o leads proprietary low-level results on EB-Manipulation, while Claude-3.5-Sonnet leads high-level tasks and InternVL3-78B is the strongest open-source model.Claude-3.5-Sonnet scores 64.0% on EB-ALFRED and 68.0% on EB-Habitat; GPT-4o scores 57.7% on EB-Navigation and 28.9% on EB-Manipulation, while InternVL3-78B reaches 53.7% and 26.3%.Source: §5.2 · p. 6, Table 2 · p. 7, Table 3 · p. 8

5.3. Language-centric Ablation

• 10% and 8%: removing environment feedback lowers GPT-4o and Claude-3.5-Sonnet success rates, while 0-shot prompting reduces success to around 40%.These language-centric findings, together with vision-removal results, indicate that high-level tasks depend more heavily on textual information.Source: §5.3 · p. 7

5.4. Visual-centric Ablation

• 39.6% to 27.1%: removing detection boxes reduces GPT-4o’s manipulation success, while 500 × 500 images outperform 300 × 300 and 700 × 700 resolutions.Multi-step images confuse current MLLMs, whereas visual ICL significantly outperforms language-only ICL, boosting Claude-3.5-Sonnet by 16.7%.Source: §5.4 · pp. 8–9

5.5. Error Analysis

• 55%: planning errors dominate EB-ALFRED failures, while EB-Manipulation has 44% planning and 33% perception errors.EB-ALFRED commonly exhibits missing steps and invalid actions; manipulation failures include inaccurate gripper poses and wrong object recognition despite detection boxes.Source: §5.5 · p. 9

Conclusion

• EMBODIEDBENCH evaluates MLLM-based embodied agents across tasks with varying action levels and capability-oriented subsets.The benchmark is intended to support research toward more capable and versatile vision-driven embodied agents.Source: §6 · p. 9
• Experiments identify low-level manipulation, long-horizon planning, and task-dependent vision-input significance as key challenges.These findings highlight areas requiring improvement in future vision-driven embodied agents.Source: §6 · p. 9

Limitations

• Evaluation is limited to simulated environments, which improve reproducibility, cost, and safety but leave real-world applicability untested.Future work could pursue more realistic and complex simulations or standardized, cost-effective real-world test suites to bridge this gap.Source: Limitations · p. 9

Impact Statement

• The work aims to advance vision-driven embodied agents, while leaving societal consequences unhighlighted and directing future-work discussion to Appendix B.This statement appears in the impact discussion rather than the related-work survey.Source: Impact Statement · p. 9

A. Additional Related Works

• Embodied-agent integration has progressed from LLM-based planning and control to MLLM perception and policy-making.MLLMs support object attribute identification, visual relation extraction, action recognition, direct action generation, visual guidance, and code-generated policies or value functions.Source: A. Additional Related Works · p. 17
• Vision Language Action Models use MLLMs or language-conditioned diffusion models as foundations and train on low-level robotics action data.World-model approaches instead use language-conditioned diffusion models to predict future states from actions or task descriptions.Source: A. Additional Related Works · p. 17
• Existing benchmarks often lack multimodal input evaluation, limiting assessment of vision-driven embodied agents.This limitation applies to ALFWorld, AgentBench, Lota-bench, and Embodied Agent Interface.Source: A. Additional Related Works · p. 17
• Most benchmarks focus narrowly on high-level household tasks or isolated low-level manipulation and navigation control.The passage contrasts household-focused benchmarks with VLMbench for manipulation and GOAT-bench for navigation.Source: A. Additional Related Works · p. 17
• VisualAgentBench evaluates MLLMs across multiple domains but remains limited to high-level household and Minecraft tasks without fine-grained capability assessment.Embodied Agent Interface and VLABench add fine-grained metrics but primarily target LLMs and VLAs rather than MLLMs.Source: A. Additional Related Works · p. 17
• EmbodiedEval covers multiple MLLM evaluation domains but excludes low-level manipulation and capability-oriented evaluation, with only 328 test instances.Its domains include navigation, object interaction, social interaction, attribute question answering, and spatial question answering.Source: A. Additional Related Works · p. 17

Expanding Task Diversity

• Future benchmarks should expand task diversity across realistic settings such as autonomous driving, multi-agent collaboration, and human-agent interaction to assess adaptability and generalization.Source: B. Future Research Directions · p. 17

Low-Level Tasks and Spatial Reasoning

• Improving spatial reasoning and low-level control requires integrating spatial reasoning with action planning through 3D visual grounding and alignment.Source: B. Future Research Directions · p. 18

Long-Horizon Planning

• Long-horizon planning could advance through hierarchical planning, memory-augmented methods, and world models for complex, multi-step tasks.Source: B. Future Research Directions · p. 18

Multi-step/Multi-view Image Understanding

• Future work should improve multi-frame and multi-view comprehension, temporal reasoning, and spatial awareness, potentially using video pretraining.Source: B. Future Research Directions · p. 18

Visual In-context Learning (ICL)

• Visual in-context learning enables adaptable embodied decision-making without fine-tuning, but more effective methods for embodied tasks remain an open problem.Source: B. Future Research Directions · p. 18

Training Multimodal Embodied Agents

• Training multimodal embodied agents could explore embodied pretraining, imitation learning, offline and online reinforcement learning, and end-to-end perception-reasoning-action optimization.Source: B. Future Research Directions · p. 18

Robustness and Generalization of MLLM Agents

• Robustness and generalization research should evaluate embodied agents under adversarial settings, dynamically generated environments, and domain shifts.Source: B. Future Research Directions · p. 18

C. Details about EMBODIEDBENCH Environments and Datasets; D. Model Versions; E. Definitions and Examples of Capability-oriented Subsets; F. Additional Experiment Results

• The benchmark spans four environments with task-dependent step limits and early stopping for excessive invalid actions or empty plans, while experiments use proprietary APIs and locally deployed open-source MLLMs.Table 5 defines six capability-oriented subsets, and the additional experiments analyze subgoal rates, step counts, camera resolution, detection boxes, multi-step images, multi-view images, and visual in-context learning.Source: C. Details about EMBODIEDBENCH… · p. 18, D. Model Versions · p. 21, E. Definitions and Examples of… · p. 21

C.1. EB-ALFRED

• EB-ALFRED supports all seven ALFRED task types through multi-instance objects, dynamic 171–298-action spaces, simulator fixes, and corrected instructions.Its egocentric simulator provides eight high-level action types plus textual validity feedback, while the dataset contains 300 instances evenly divided across six capability subsets.Source: §C.1 · p. 19

C.2. EB-Habitat

• EB-Habitat uses 282 language templates and 70 high-level skills, constraining navigation to receptacles and providing 300 instances across reorganized capability subsets.Built with Habitat 2.0, YCB, and ReplicaCAD, it complements EB-ALFRED by requiring visits to multiple locations to find target objects.Source: §C.2 · p. 19

C.3. EB-Navigation

• EB-Navigation evaluates seven robot-executable motion actions using first-person observations and validity-only feedback, with 90 scene tasks expanded to 300 capability-oriented instances.Tasks specify poses, targets, and language instructions, enforce adjustable initial distances and accessible objects, and support automated dataset generation.Source: §C.3 · p. 20

C.4. EB-Manipulation

• EB-Manipulation controls a 7-DoF Franka Panda through four task categories, discretized actions, indexed detection boxes, 3D poses, and validity-only feedback.Its dataset contains 228 testing instances; the visual-appearance subset has 36 tasks, while each other subset has 48 tasks across four categories.Source: §C.4 · pp. 20–21

F.1. Subgoal Success Rate

• Subgoal success rates generally exceed final task success rates, while Gemini-1.5-Pro surpasses Claude-3.5-Sonnet on EB-ALFRED subgoals despite lower final task success.GPT-4o’s subgoal performance is within 0.2 of Claude-3.5 in both high-level environments, showing that subgoal completion does not ensure full-task completion.Source: §F.1 · p. 24, Table 6 · p. 24

F.2. Average Planner and Environment Steps

• GPT-4o uses 2.6 planner steps versus 12.9 environment steps on EB-Manipulation, reducing model inferences by nearly 80% through multi-step planning.Across most cases, multi-step planning reduces planner steps by approximately 50%–80% relative to environment steps.Source: §F.2 · p. 25, Table 8 · p. 25

F.3. Camera Resolution

• 500×500 resolution yields the best performance for EB-Manipulation and EB-Navigation, whereas EB-ALFRED gains only 2%–4% as resolution increases.The results indicate that overly low resolution omits task-relevant detail, while overly high resolution can add distracting complexity.Source: §F.3 · pp. 25–26, Figure 7 · p. 26

F.4. Detection Boxes

• Detection boxes improve EB-ALFRED and EB-Manipulation, with EB-Manipulation gaining nearly 10%, but they hinder EB-Navigation by obscuring path-planning cues.A single target-object box consistently improves navigation accuracy, yet boxes remain disabled there by default; they are enabled by default only for EB-Manipulation.Source: §F.4 · p. 26, Figure 8 · p. 26, Table 9 · p. 26

F.5. Multi-step Images

• Adding previous or latest-three environment frames does not improve decisions and particularly reduces EB-Manipulation performance.Models may struggle to relate sequential images to their current state, highlighting challenges in using temporal continuity.Source: §F.5 · p. 27, Figure 10 · p. 27, Figure 11 · p. 27

F.6. Multi-view Images

• Multi-view inputs from front/top-down or front/wrist cameras reduce performance, particularly for GPT-4o, despite theoretically richer spatial context.GPT-4o and Claude-3.5-Sonnet may struggle with multi-view feature fusion or increased input complexity.Source: §F.6 · p. 28, Figure 13 · p. 28, Figure 14 · p. 28

F.7. Visual In-context Learning (ICL)

• Claude-3.5-Sonnet achieves a 16.7% improvement with visual ICL over language-only ICL in manipulation tasks.Visual demonstrations connect successful low-level actions with object positions; the study limits examples to two to avoid excessive visual input.Source: §F.7 · pp. 29–30, Figure 15 · p. 29, Figure 16 · p. 30
• Visual ICL significantly outperforms language-only ICL, indicating that image-conditioned demonstrations can strengthen embodied agents’ understanding of low-level action-object relationships.The two-example limit may slightly reduce performance compared with main results without visual ICL.Source: §F.7 · pp. 29–30

F.8. Additional Ablation Study Conclusion

• Task-specific visual integration is essential: moderate resolution increases, bounding boxes, and visual ICL can help, while extreme resolution, multi-step or multi-view images, and navigation boxes may hurt.The findings motivate more advanced fusion techniques for combining diverse visual inputs.Source: §F.8 · p. 30

G. Further Discussion on Chat History as Input for EB-Navigation

• EB-Navigation is the most sensitive of the four environments to historical input, with chat history affecting models differently.Chat history consists of prior conversations paired with images, unlike the multi-step image setting in Appendix F.5.Source: G. Further Discussion on Chat History… · p. 30
• Proprietary MLLMs generally benefit from chat history, particularly on long-horizon EB-Navigation tasks.Source: G. Further Discussion on Chat History… · p. 30
• Open-source models have mixed preferences: Llama-3.2 and InternVL improve with chat history, whereas Qwen2-VL performs best with single-step input.Qwen2-VL uses only the summarized “interaction history” adopted in the other environments.Source: G. Further Discussion on Chat History… · p. 30
• Because no definitive explanation exists for the discrepancy, the paper adopts chat history only for EB-Navigation rather than as a global design choice.The authors hypothesize that training on large amounts of multi-turn multimodal dialogue may influence compatibility with this input format.Source: G. Further Discussion on Chat History… · p. 30

H. Error Definitions and Additional Analysis

• The error analysis defines three main categories—perception, reasoning, and planning—corresponding to distinct stages of the agent pipeline.Perception errors occur during visual state description, reasoning errors during reflection and reasoning, and planning errors during language and executable plan generation.Source: §H.1 · p. 30

H.1. Error Type Definition

• Perception errors arise in visual state description, reasoning errors in reflection and reasoning, and planning errors in language or executable plan generation.Table 11 provides a detailed breakdown of sub-errors for each category.Source: §H.1 · p. 30

H.2. Error Analysis for EB-Navigation

• Navigation failures fall into perception, reasoning, and planning errors that significantly hinder MLLM agents from reaching target objects.Perception errors include wrong recognition and hallucinated targets; reasoning errors involve flawed path logic and weak spatial reflection; planning errors involve inaccurate movement and distance estimation.Source: §H.2 · pp. 30–31, Error Type Definition · p. 32

Error Type Definition

• The error taxonomy highlights needed improvements in object recognition, hallucination reduction, 3D spatial reasoning, and alignment between model outputs and executable actions.Models struggle to form coherent 3D representations from sequential 2D observations, refine actions from feedback, and align distance estimates with movement execution.Source: Error Type Definition · p. 32

H.3. Format Errors

• Larger models tend to produce fewer invalid JSON outputs, whereas smaller models show more format errors, and EB-ALFRED has more errors because its trajectories are more complex and longer.Even proprietary models are not immune to format errors across the six subsets.Source: §H.3 · p. 32

I.1. Prompts

• The prompts define environment-specific action spaces, validity constraints, planning limits, visual-state descriptions, history and feedback reflection, and JSON action outputs.Navigation prompts emphasize forward and lateral movement, sparse rotation, and moving as close as possible to the target; manipulation prompts encode discrete 3D gripper positions, orientations, and gripper state.Source: §I.1 · pp. 34, 36

I.2. Skill Sets

• EB-ALFRED uses scene-dependent navigation and object-manipulation skills, whereas EB-Habitat uses a static skill set; navigation and manipulation action spaces are embedded in their prompts.The appendix lists representative receptacles and objects for EB-ALFRED and refers to the prompts for EB-Navigation and EB-Manipulation action spaces.Source: §I.2 · p. 39, Table 13 · p. 39, Table 14 · p. 40

I.3. In-context examples

• In-context demonstrations are environment- and task-specific: EB-ALFRED uses 10 examples generally and 7 for long-horizon tasks, while EB-Habitat uses 10, EB-Navigation 3, and EB-Manipulation 2–8 examples by task complexity.The examples are selected to differ from test instances and, for EB-ALFRED, cover task types while avoiding multiple object instances and variable action indices.Source: §I.3 · p. 41
• In-context examples show long-horizon household instructions decomposed into ordered discovery, manipulation, and state-changing actions.Examples include microwaving an apple slice, placing multiple objects on a specified table, closing a refrigerator, and relocating a sponge.Source: §I.3 · pp. 43–44
• Navigation examples use iterative visual reasoning and environmental feedback to revise movement plans around obstacles until the agent can complete the task.After an invalid forward move reveals an obstacle, the plan shifts leftward before moving forward and then rightward toward the floor lamp.Source: §I.3 · p. 45
• Manipulation examples ground plans in detected object properties and coordinates, then sequence gripper movements for placement, stacking, insertion, and wiping tasks.The examples identify colors, shapes, and 3D coordinates before selecting target objects and specifying grasp, transport, release, or wiping operations.Source: §I.3 · p. 46

I.4. Output JSON Schema

• The JSON schemas require visual state descriptions, reasoning and reflection, language plans, and nonempty executable plans, with task-specific action representations.Schemas for ALFRED, Habitat, and Navigation require action IDs and names, while Manipulation additionally specifies object descriptions with coordinates and 7-dimensional discrete actions.Source: §I.4 · p. 48
• The schema requires visual_state_description, reasoning_and_reflection, language_plan, and executable_plan fields in the output object.The executable_plan contains an action field described as a 7-dimensional discrete action in the prompt-specified list format.Source: §I.4 · p. 49

J. Supplementary Case Studies of Successful Planning

• Successful planning examples span Claude-3.5-Sonnet, InternVL2.5-78B, GPT-4o, and Gemini-1.5-pro across EB-ALFRED, EB-Habitat, EB-Navigation, and EB-Manipulation.The examples provide detailed reasoning and planning traces in Figures 18–21.Source: J. Supplementary Case Studies of… · p. 50
• Claude-3.5-Sonnet successfully plans slicing a tomato, heating it in a microwave, and placing it in a sink in EB-ALFRED.This case is presented as the EB-ALFRED planning example in Figure 18.Source: Figure 18 · p. 50
• InternVL2.5-78B successfully plans moving the spoon and sponge to the brown table and the cleanser to the black table in EB-Habitat.This case is presented as the EB-Habitat planning example in Figure 19.Source: Figure 19 · p. 51
• GPT-4o successfully plans navigating to the pot and getting as close as possible to it in EB-Navigation.This case is presented as the EB-Navigation planning example in Figure 20.Source: Figure 20 · p. 52
• Figure 21 presents a Gemini-1.5-pro planning example in EB-Manipulation for stacking the front star on top of the right cylinder.Source: Figure 21 · p. 53

K. Supplementary Case Studies of Unsuccessful Planning

• Unsuccessful cases reveal challenges in executable planning, object perception, and spatial reasoning across EB-ALFRED, EB-Manipulation, and EB-Navigation.The cases highlight limitations in object recognition, planning, and spatial reasoning, with details in Figures 22–24.Source: K. Supplementary Case Studies of… · p. 54
• In EB-ALFRED, the agent continues interacting with the first book instead of locating Book 2 by the seventh action, producing a planning error.In EB-Manipulation, it misidentifies an object's color; in EB-Navigation, it recognizes a countertop blockage but fails to navigate around it.Source: Figure 22 · p. 54, Figure 23 · p. 55, Figure 24 · p. 56