Source-linked AI summary
EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, Lei Shi, Maosong Sun
TL;DR
Existing benchmarks inadequately assess MLLMs as embodied agents because they lack diverse interactive tasks and scenes, impose restrictive interfaces, or rely heavily on textual states. EmbodiedEval introduces a unified interactive 3D benchmark with diverse scenes, tasks, and action spaces, and finds that current MLLMs remain substantially below human-level embodied performance. The benchmark identifies grounding, spatial reasoning, planning, and exploration as areas for improvement.
Problem
Existing embodied benchmarks lack task and scene diversity, may impose rigid interfaces, and can underrepresent visual grounding and spatial reasoning needed for real-world interaction.
Method
EmbodiedEval is an interactive benchmark that evaluates MLLMs in 3D simulation using task descriptions, egocentric visual histories, and unified movement, interaction, and answering actions.
Results
Current MLLMs largely fall short of human-level performance on embodied tasks, with performance varying across categories and notable declines on spatial and long-horizon tasks.
Takeaways & Limitations
EmbodiedEval provides insights for improving MLLMs’ grounding, spatial reasoning, planning, and exploration capabilities.
Takeaways & Limitations
The evaluation set contains 327 test cases because verifying every scene, task, and correctness individually is time-consuming; more cases are planned.
Abstract
from arXiv · showhide
Multimodal Large Language Models (MLLMs) have shown significant advancements, providing a promising future for embodied agents. Existing benchmarks for evaluating MLLMs primarily utilize static images or videos, limiting assessments to non-interactive scenarios. Meanwhile, existing embodied AI benchmarks are task-specific and not diverse enough, which do not adequately evaluate the embodied capabilities of MLLMs. To address this, we propose EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied 3D scenes, each of which is rigorously selected and annotated. It covers a broad spectrum of existing embodied AI tasks with significantly enhanced diversity, all within a unified simulation and evaluation framework tailored for MLLMs. The tasks are organized into five categories: navigation, object interaction, social interaction, attribute question answering, and spatial question answering to assess different capabilities of the agents. We evaluated the state-of-the-art MLLMs on EmbodiedEval and found that they have a significant shortfall compared to human level on embodied tasks. Our analysis demonstrates the limitations of existing MLLMs in embodied capabilities, providing insights for their future development. We open-source all evaluation data and simulation framework at https://github.com/thunlp/EmbodiedEval.
1 Introduction
Existing benchmarks do not comprehensively evaluate MLLMs as embodied agents because they lack interactive diversity, use rigid formats, or rely heavily on textual environment states. EmbodiedEval addresses these gaps with diverse 3D interactions, tasks, and scenes, while experiments reveal substantial shortcomings relative to humans.
- Research gap: Existing embodied benchmarks lack diversity in tasks and scenes, with some offering only a few predefined tasks within limited room types.ALFRED, for example, includes seven predefined tasks across four room types.
- Research gap: Rigid input-output formats and text-based environment representations hinder evaluation of mainstream MLLMs and underrepresent visual grounding and spatial reasoning.These capabilities are described as essential for real-world interaction.
- EmbodiedEval: EmbodiedEval introduces a comprehensive interactive benchmark with diverse object and human interactions, systematically generated tasks, and varied realistic 3D scenes.Its scenes include house rooms, large residences, gyms, stores, and offices.
- Findings: Mainstream MLLMs largely fall short of human-level performance, with notable declines on spatial and long-horizon tasks.The benchmark is intended to provide insights for improving grounding, spatial reasoning, planning, and exploration.
2 Related Works
Prior MLLM and embodied-agent benchmarks cover perception, egocentric understanding, question answering, navigation, and interaction, but remain fragmented. EmbodiedEval targets their limited task variety, template dependence, and task-specific interfaces with a unified evaluation setting.
- MLLM evaluation: MLLM benchmarks mainly assess perception, cognition, challenging visual tasks, or egocentric images and videos rather than comprehensive embodied capabilities.The related work surveys benchmarks across these evaluation directions.
- Embodied-agent benchmarks: Existing embodied-agent benchmarks cover question answering, navigation, and object interaction, but provide limited task variety and incomplete capability coverage.They rely substantially on predefined task templates.
- EmbodiedEval: EmbodiedEval evaluates embodied agents through interactive 3D simulation rather than only static or text-based representations.Its evaluation process feeds task descriptions and egocentric observation histories to the model and returns new observations after actions.
- Benchmark limitations: Task-specific observation and continuous action spaces in many embodied benchmarks are inadequate for effectively evaluating MLLMs.This motivates a benchmark designed around more accessible model interfaces.
3 EmbodiedEval
EmbodiedEval uses a unified simulation-based decision process in which MLLMs receive task descriptions, action options, and egocentric visual histories, then act through movement, interaction, or answering. Its benchmark is constructed from diverse scenes and generated tasks with manual verification, producing broad task coverage and detailed statistics.
- Evaluation process: Each evaluation episode initializes a 3D scene and visual observation, then repeatedly applies model actions, records feedback, and updates the observation history.Episodes end when predicates indicate success, an answer is incorrect, or the maximum steps are exceeded.
- Evaluation formulation: The unified input space contains task descriptions, action options, and egocentric visual observations, while outputs support movement, interaction, and answering.Movement uses directional choices on scene-specific navigation graphs rather than 3D position selection.
- Task design: The benchmark covers navigation, object interaction, social interaction, and question-answering tasks to assess varied embodied capabilities.The task categories include both environmental modification and human-agent interaction.
- Benchmark construction: Dataset construction combines scene collection, task generation, and task annotation, with candidate tasks generated from seeds across more than 30 existing datasets.Annotators align tasks to scenes, configure actions and success criteria, and verify executability in simulation.
- Dataset statistics: The benchmark contains 328 tasks in five categories across 125 unique scenes, with 575 predicate instances and 1533 options.Options include 1213 textual answers and 320 interactions; expert demonstrations average 10.72 steps per episode.
- Evaluation metrics: Performance is evaluated with Success Rate, Goalcondition Success, and Success weighted by Path Length.These metrics measure complete success, partial goal achievement, and execution efficiency, respectively.
4 Experiments
Experiments evaluate 19 MLLMs on EmbodiedEval using success, partial-goal, and efficiency metrics, revealing large gaps from human performance and weaknesses in interaction, spatial, and long-horizon tasks. Error analysis attributes failures to grounding hallucinations, insufficient exploration, spatial reasoning, and planning.
- Experimental Setup: 19 MLLMs are evaluated against random and non-expert human agents using multiple visual observation formats and a 24-step task limit.Image MLLMs receive multiple egocentric images, while video MLLMs receive videos of the interaction process.
- Evaluation Metrics: Success Rate measures complete task completion, Goalcondition Success measures achieved goal conditions, and SPL measures success weighted by path efficiency.SPL compares execution efficiency with expert demonstrations.
- Main Results: 25.00% overall success is achieved by GPT-4o versus 97.26% for non-expert humans, while GPT-4o reaches 32.42% GcS.LLaVA-OneVision-72B is the strongest open-source MLLM at 12.80% overall success.
- Task-Type Performance: Models perform relatively better on attribute question answering and navigation, but spatial question answering and interaction tasks expose substantial weaknesses.Spatial QA success often halves relative to attribute QA, while interaction tasks require affordance and social-cue understanding.
- Long-Horizon Tasks: Longer tasks reduce success rates, and additional allowed steps provide diminishing gains as models struggle with longer histories.For Gemini-Flash, adding images generally decreases performance except in spatial question answering, which benefits from additional spatial context.
- Error Analysis: Four recurring error categories are grounding hallucination, insufficient exploration, lack of spatial reasoning, and wrong planning.These errors include perceiving nonexistent objects, incomplete environment coverage, directional misunderstandings, and repetitive or random actions.
5 Conclusion
EMBODIEDEVAL is presented as an interactive benchmark for comprehensively evaluating MLLMs on embodied tasks. Experiments find that current MLLMs perform poorly, motivating further development of their embodied capabilities.
- EMBODIEDEVAL provides an interactive framework for evaluating MLLMs on comprehensive embodied tasks.The dataset is extensively annotated to support evaluation accuracy, diversity, and quality.
- Current MLLMs perform poorly on embodied tasks, indicating substantial room for improving their embodied capabilities.
6 Limitations
The evaluation set is limited by the time-consuming process required to verify every scene, task, and correctness condition. Consequently, it contains 327 test cases, with more cases planned for future research.
- 327 test cases comprise the evaluation set because each scene, task, and correctness condition requires individual verification.The authors plan to incorporate more cases in future research.
7 Potential Risks
The paper acknowledges risks associated with deploying MLLMs as interactive embodied agents. These include biased decisions in critical scenarios and privacy or data-security concerns in real-world environments.
- Reliance on MLLMs for critical decisions may produce biased outcomes because of limitations in understanding contextual nuances.
- Real-world deployment raises privacy and data-security concerns because these systems often require substantial personal and environmental data.
A Task Samples
EMBODIEDEVAL broadens attribute question answering beyond traditional EQA by covering diverse object and scene attributes, including multi-attribute and multi-object reasoning.
- Attribute questions cover category, shape, material, color, function, state, location, existence, quantity, and comparative analysis.
- The benchmark also includes complex reasoning across multiple attributes and multiple objects.
B Details of Evaluation Framework
The evaluation framework discretizes navigation and interaction while preserving task-relevant behavior, and combines annotated answer choices with predicate-based success evaluation.
- Navigation: Navigation uses graph-based movement, allowing view rotation at points and movement between adjacent points instead of continuous 3D control.The graph is intended to be more natural and adaptable than grid-world movement while limiting high-level task restrictions.
- Navigation: The navigation action space contains directional decisions such as moving forward and turning left or right between navigation points.Agents choose directions among navigation points rather than selecting arbitrary 3D positions.
- Object interaction: Interaction uses discrete actions with explicit spatial and state prerequisites, including visibility, proximity, nearby sinks, or holding an object.Tasks may include both necessary and distracting interaction actions.
- Question answering: Question answering provides eight carefully annotated options designed to be meaningful, challenging, and strongly distractive within each scene.The answer space is illustrated through examples in the benchmark materials.
- Evaluation: Predicate functions map simulator states to boolean success values and can evaluate either final conditions or events occurring during the episode.Examples include reaching a location and passing a specified navigation point.
- Evaluation: A task succeeds only when all predicates are true, allowing multi-step requirements such as passing through a kitchen, returning, and selecting an answer.The example combines agent_pass, agent_at, and choose predicates.
C Details of Task Annotation
Task annotation combines procedurally generated and manually edited scenes with feasibility checks, expert review, and simulator-based verification. The framework also standardizes evaluation conditions and documents diverse task, action, scene, and answer examples.
- Task annotation: Each task sample specifies a scene, task description, output space, and success criteria, with predicates added for multiple sub-goals.Annotators adapt task wording to fit the selected scene while preserving the task’s core content.
- Task annotation: Annotation requires unambiguous tasks, observation-based questions with eight distractive options, and simulator-executable navigation and action choices.Annotators verify feasibility under the same observational constraints used by agents.
- Annotation system: A Unity-based annotation system supports scene and task exchange, content viewing, visualized action spaces, and guided workflows governed by predefined rules.The system is designed to improve efficiency and precision during complex annotation.
- Quality control: Eight expert annotators receive systematic training, and each task is independently checked for correctness and quality by at least three reviewers.The annotation process uses two rounds, beginning with task-completion checks.
- Scene construction: Scenes are built from a curated Objaverse subset of about 15,000 indoor objects spanning more than 500 categories, then procedurally generated and interactively refined.GPT-3.5 filters unsuitable assets and annotates object occurrences, positions, and functions; Gemini-1.5-Flash annotates orientations.
- Evaluation setting: Evaluation uses temperature 0 as the standard because deterministic outputs remove randomness, although models perform slightly better at temperature 1.The authors associate the higher temperature result with greater exploration and fewer repetitive errors.
G Error Cases
The supplied passages identify Figures 23, 24, 25, 27, and 28 in the “G Error Cases” section, but provide no figure content or findings.
- Figure 23 is referenced in the section.
- Figure 24 is referenced in the section.
- Figure 25 is referenced in the section.
- Figure 27 is referenced in the section.
- Figure 28 is referenced in the section.