PaperlayerView exact revision on arXiv

Source-linked AI summary

Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making

Manling Li, Shiyu Zhao, Qineng Wang, Kangrui Wang, Yu Zhou, Sanjana Srivastava, Cem Gokmen, Tony Lee, Li Erran Li, Ruohan Zhang, Weiyu Liu, Percy Liang, Li Fei-Fei, Jiayuan Mao, Jiajun Wu

arXiv:2410.07166v3cs.CLcs.AIcs.LGcs.RO

TL;DR

LLM performance in embodied decision making is difficult to compare because tasks, module interfaces, and evaluation metrics vary across studies. EMBODIED AGENT INTERFACE addresses this by standardizing goals, four decision-making modules, and fine-grained error metrics. The benchmark identifies substantial reasoning and grounding errors while showing differentiated strengths across models and tasks.

  • Problem

    Existing embodied decision-making evaluations vary in tasks, LLM module interfaces, and metrics, often reducing assessment to a single success rate.

  • Method

    The paper introduces an object-centric interface using LTL goal specifications, standardized interfaces for four LLM modules, and fine-grained trajectory and goal metrics.

  • Results

    The benchmark finds grounding and reasoning errors across models, while o1-preview leads overall and reaches 74.9% versus 64.2% on BEHAVIOR.

  • Takeaways & Limitations

    The interface provides a common basis for comparing embodied decision-making abilities and locating specific error types across tasks and modules.

  • Takeaways & Limitations

    The evaluation is limited to states, actions, and goals expressible in abstract language over relational object graphs.

Abstract

from arXiv · show

We aim to evaluate Large Language Models (LLMs) for embodied decision making. While a significant body of work has been leveraging LLMs for decision making in embodied environments, we still lack a systematic understanding of their performance because they are usually applied in different domains, for different purposes, and built based on different inputs and outputs. Furthermore, existing evaluations tend to rely solely on a final success rate, making it difficult to pinpoint what ability is missing in LLMs and where the problem lies, which in turn blocks embodied agents from leveraging LLMs effectively and selectively. To address these limitations, we propose a generalized interface (Embodied Agent Interface) that supports the formalization of various types of tasks and input-output specifications of LLM-based modules. Specifically, it allows us to unify 1) a broad set of embodied decision-making tasks involving both state and temporally extended goals, 2) four commonly-used LLM-based modules for decision making: goal interpretation, subgoal decomposition, action sequencing, and transition modeling, and 3) a collection of fine-grained metrics which break down evaluation into various types of errors, such as hallucination errors, affordance errors, various types of planning errors, etc. Overall, our benchmark offers a comprehensive assessment of LLMs' performance for different subtasks, pinpointing the strengths and weaknesses in LLM-powered embodied AI systems, and providing insights for effective and selective use of LLMs in embodied decision making.

1 Introduction

The paper introduces EMBODIED AGENT INTERFACE to standardize embodied decision-making tasks, LLM module interfaces, and fine-grained evaluation. It represents both state and temporally extended goals and supports systematic error analysis across multiple benchmarks and modules.

  • Motivation: Existing evaluations lack standardization for embodied tasks, LLM-interfaced modules, and metrics beyond a single success rate.These gaps make comparisons across domains and systems difficult.
  • Goal standardization: LTL formulas represent both state-based and temporally extended goals over trajectories, enabling alternative goal interpretations.The interface uses object-centric states and actions expressed in abstract language terms.
  • Module standardization: The interface standardizes four LLM-based modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling.It defines common input-output specifications so LLM-based and external modules can be integrated and compared.
  • Evaluation: Fine-grained metrics automatically identify hallucination, affordance, and planning errors through trajectory and goal evaluation.Trajectory evaluation checks executability, while goal evaluation checks whether plans achieve correct outcomes.
  • Findings: 45.2% of evaluated errors are trajectory feasibility errors, including 19.5% missing-step and 14.2% additional-step errors.The reported failures often involve overlooked preconditions, such as standing before acting or opening a closed container.
  • Findings: o1-preview significantly outperforms other models on BEHAVIOR, achieving 74.9% versus 64.2%.It also leads in several module-level comparisons across BEHAVIOR and VirtualHome.

2 Embodied Agent Interface Based on LTL

The Embodied Agent Interface uses object-centric representations and LTL to unify embodied goals, subgoals, action sequences, and trajectories across four standardized LLM abilities. It evaluates these abilities through module-specific inputs, outputs, and metrics grounded in simulator execution and formal matching.

  • Interface and representations: The interface represents environments with objects, relational states, actions, goals, subgoals, and trajectories, while standardizing four modules: goal interpretation, subgoal decomposition, action sequencing, and transition modeling.States and actions use abstract language forms, and each module has a defined input-output specification.
  • Interface and representations: LTL expresses both state-based and temporally extended goals, supports alternative interpretations, and provides a unified language across module inputs and outputs.The framework uses temporal ordering, Boolean connectives, quantifiers, and state or action propositions.
  • Ability modules and metrics: Subgoal decomposition is refined by breadth-first search into executable action sequences, then evaluated using trajectory feasibility and goal-satisfaction metrics.The search accommodates multiple action sequences that can realize a subgoal while restricting actions per subgoal to keep the candidate set finite.
  • Ability modules and metrics: Goal interpretation maps an initial state and natural-language instruction to an LTL goal, evaluated with an F1 set-matching score against compatible simple ground-truth goals.The evaluation expands complex ground-truth goals into simple alternatives and selects the best matching score.
  • Ability modules and metrics: Action sequencing generates actions from the initial state, goal, and transition model, with execution failures categorized as missing steps, additional steps, wrong temporal order, or affordance errors.Evaluation also checks full and partial goal satisfaction after simulator execution.
  • Ability modules and metrics: Transition modeling generates PDDL operators and is evaluated through logic matching against human annotations and the feasibility of plans supported by the generated models.Logic matching compares generated preconditions and effects with annotated operator definitions using F1-based formula matching.

3 Dataset Annotations and Benchmark Implementations

The benchmark annotates complex tasks in BEHAVIOR and VirtualHome, adds goal-action and transition-model information, and adapts simulators for formal evaluation. BEHAVIOR requires grounded handling of quantified goals with multiple valid goal options.

  • Dataset annotations: The annotation effort targets long-horizon, scene-complex tasks in BEHAVIOR and VirtualHome, adding goals, trajectories, goal actions, and related simulator support.Goal-action annotations capture necessary actions without post effects, such as touching an object in a petting task.
  • BEHAVIOR representation: BEHAVIOR goals may contain quantifiers that must be translated into grounded atomic propositions, with multiple grounded assignments satisfying one quantified goal.The benchmark calls these alternative grounded assignments goal options.
  • BEHAVIOR representation: 4,164.4 goal options are available per BEHAVIOR task on average, alongside an average of 6.7 grounded goals.These alternatives reflect the multiple assignments that can satisfy a single quantified goal.
  • Dataset annotations: Table 2 reports simulator dataset statistics and highlights annotations newly collected in this paper.The table summarizes the benchmark resources used for evaluation.

4 Results

Across BEHAVIOR and VirtualHome, the evaluation reveals model, simulator, goal-type, and error-pattern differences across four embodied decision-making abilities. Results show strong overall models alongside persistent grounding, feasibility, relational reasoning, and transition-modeling weaknesses.

  • Model and ability comparison: o1-preview leads overall, reaching 74.9% on BEHAVIOR versus 64.2% for the comparison baseline.It also leads several abilities across both simulators, while commercial models generally outperform open-weight models.
  • Simulator differences: Models achieve lower trajectory feasibility but higher goal-interpretation scores on BEHAVIOR than VirtualHome.The paper associates this contrast with BEHAVIOR’s longer average horizon of 14.6 steps and VirtualHome’s larger state space.
  • Goal-type differences: Relational goals are harder than object-state goals, with lower spatial-relation recall and lower goal-satisfaction rates.The difficulty also extends to non-spatial relations such as holding(toothbrush), which can be a precondition for brushing teeth.
  • Goal interpretation: LLMs often translate language word-by-word, confusing intermediate subgoals with final goals and missing precise physical relations.Examples include predicting open(freezer) for drinking water and omitting onfloor(gym_shoe, floor) when cleaning sneakers.
  • Planning and transition errors: Trajectory errors are mainly runtime errors, especially missing and additional steps caused by overlooked preconditions or continuing after goal achievement.Transition models also misrepresent object states and spatial relations across preconditions and effects, potentially producing infeasible plans.
  • System design implications: The benchmark examines robustness through sensitivity analysis, modularized versus pipeline-based comparisons, and replanning, finding similar trajectory feasibilities across module compositions.These experiments investigate how LLM-based and external modules can be integrated.

5 Related Work

Related work uses LLMs for varied embodied decision-making roles and often combines multiple modules through prompting or pipelines. The paper situates its benchmark within this diverse landscape of task and interface assumptions.

  • Existing embodied-agent uses: Prior embodied-agent studies use LLMs for combinations of goal interpretation, subgoal decomposition, action sequencing, and related planning functions.These combinations include goal interpretation with action sequencing, goal interpretation with subgoal decomposition, and action sequencing with subgoal decomposition.
  • Literature overview: Table 8 summarizes existing work leveraging LLMs for embodied agents.The related-work survey covers the broader literature accompanying the benchmark.

6 Conclusions and Future Work

The paper concludes that EMBODIED AGENT INTERFACE standardizes embodied decision-making evaluation while exposing fine-grained LLM weaknesses and their task cofactors. Its current scope is limited to abstract language descriptions and relational-graph environments.

  • Conclusions: EMBODIED AGENT INTERFACE standardizes LTL goal specifications, four decision-making modules, and fine-grained metrics with automatic error identification.The framework supports systematic benchmarking across embodied decision-making tasks.
  • Conclusions: The evaluation highlights limitations in complex-goal interpretation and reasoning, associated with trajectory length, goal complexity, and spatial-relation goals.These cofactors are used to attribute observed errors.
  • Future work: The current evaluation covers states, actions, and goals described in abstract language over relational graphs of objects.Future extensions include sensory inputs, actuation outputs, vision-language models, memory, geometric reasoning, and navigation.

Checklist

The checklist reports that the paper addresses scope, ethics, assets, reproducibility, and limitations through affirmative responses or referenced supplementary sections. It also notes that error bars were not reported because inference is resource-intensive and proprietary APIs are costly.

  • Claims and limitations: The authors report that the paper’s main claims accurately reflect its scope and contributions and that its limitations are described.The checklist points readers to Appendix Section S for the limitations discussion.
  • Ethics: The authors report discussing potential negative societal impacts and conforming to ethics review guidelines.These discussions are referenced in Appendix Section S.
  • Reproducibility and compute: Code, data, and reproduction instructions are included in the supplementary materials, while training details are reported.The checklist says error bars were not applicable because LLM inference is resource-intensive and proprietary APIs are too costly.
  • Assets: Existing assets were cited and their licenses were discussed in the website and supplemental material.The checklist also states that new assets were included in the supplemental material or provided by URL.
  • Data and participants: The resources are reported as open-access public data without personally identifiable information or offensive content.No human-subject participation or compensation was involved.

A Summary of Empirical Findings

The benchmark evaluates embodied decision making across standardized goals, modules, and fine-grained error categories. Experiments reveal persistent grounding and reasoning weaknesses, while performance varies across models, simulators, and ability modules.

  • Empirical findings: Most LLMs struggle to ground natural-language instructions in objects, states, and relations, sometimes predicting intermediate goals instead of final goals or omitting spatial relations.Examples include predicting open(freezer) for “drinking water” and missing required object relationships in serving tasks.
  • Empirical findings: 41.2% of trajectory executions contain runtime errors, including 15.5% missing-step and 16.2% additional-step errors caused often by overlooked preconditions.Models may omit standing before acting from a seated or lying state, fail to open closed containers, or repeat actions for already achieved goals.
  • Empirical findings: Subgoal decomposition is not strictly easier than action sequencing, although o1-preview reaches 89.4% success in VirtualHome and 57.0% in BEHAVIOR.Other strong results include 89.1% for Gemini 1.5 Flash and Claude-3.5 Sonnet in VirtualHome.
  • Benchmark design: EAI standardizes embodied tasks with object-centric representations and LTL goals that cover both state-based and temporally extended requirements.Its interface represents objects, states, relations, actions, and trajectories while supporting alternative goal interpretations.
  • Benchmark design: Fine-grained metrics identify parsing, hallucination, affordance, missing-step, additional-step, and temporal-order errors beyond final task success.The metrics are designed to automatically locate distinct failures in translating goals and producing executable behavior.
  • Benchmark design: The framework unifies four LLM ability modules—goal interpretation, subgoal decomposition, action sequencing, and transition modeling—with standardized interfaces.These modules specify how LLM-based components connect to embodied environments and external modules.

D.2 Action Sequencing: Trajectory Error Detection for Missing Step, Additional Step, Wrong Temporal Order, Affordance Error

Action sequencing evaluates whether generated actions are syntactically valid, executable, and sufficient to achieve object, relation, and action goals. Its trajectory analysis separates runtime failures into actionable error categories, while results show strong formatting performance but persistent grounding and planning mistakes.

  • Inputs and evaluation: Action sequencing takes an initial world state, goal, and environment vocabulary, then requires an executable sequence of parameterized actions.Each action contains an action name and an ordered list of object arguments.
  • Inputs and evaluation: Trajectory evaluation checks parsing, hallucinated names, runtime executability, and goal achievement against node, edge, and action goals.The framework separately evaluates whether the trajectory runs and whether the final environment state satisfies the specified outcomes.
  • Runtime error categories: Affordance errors occur when object properties disallow an action, while additional-step errors occur when an action’s intended effect is already satisfied.For example, opening a shelf is invalid when the shelf cannot be opened, and toggling an already-on light is redundant.
  • Runtime error categories: Missing-step detection identifies omitted necessary actions, while historical and precondition checks distinguish failures involving action prerequisites and previously achieved effects.The automated procedure checks affordability, redundancy, current preconditions, and whether effects or prerequisites were satisfied earlier.
  • Observed model behavior: Grammar errors are uncommon for leading proprietary models, whereas open-source models make more format and parsing mistakes; goal interpretation still shows hallucinations and omitted spatial relations.The study reports no parsing errors for some leading models but recurring intermediate-goal substitutions and incorrect relationships such as chicken directly on table.

E.2.1 Result Analysis

Across VirtualHome and BEHAVIOR, o1-preview is the strongest overall model, while error patterns reveal that task complexity, precondition checking, and grounded reasoning remain major challenges.

  • VirtualHome results: 89.4% task success makes o1-preview the top VirtualHome model, with Gemini 1.5 Flash and Claude-3.5 Sonnet at 89.1%.
  • Error analysis: Grammar errors are generally rare, but models still hallucinate invalid predicates or objects, including GPT-4o’s undefined POUR action and Llama 3 8B’s nonexistent scene objects.
  • Error analysis: Runtime failures differ by simulator: VirtualHome commonly produces additional steps, whereas BEHAVIOR is dominated by missing steps caused by stricter preconditions and more complex tasks.
  • Goal satisfaction: State goals exceed 91% success for several VirtualHome models, but BEHAVIOR state and relation goals are harder because quantifiers fail when any contained requirement is missed.
  • Cross-simulator comparison: BEHAVIOR reaches only 57.0% task success for o1-preview versus 89.4% in VirtualHome, reflecting more complex quantified representations and stricter precondition checks.

F.3 Result Analysis

Sensitivity analysis shows strong performance on straightforward actions but pronounced weaknesses on actions requiring spatial reasoning, tool use, and multi-condition object manipulation.

  • VirtualHome sensitivity: 0.09 success for plug_in in object affordance and object states identifies it as a consistently difficult action.
  • VirtualHome sensitivity: Spatially demanding actions remain difficult: walk_towards reaches 0.63 and walk_into 0.17 in spatial-relations tasks.
  • VirtualHome sensitivity: Plug_in and walk_towards are especially challenging because they require alternative device preconditions or complex spatial constraints.
  • VirtualHome sensitivity: 0.00 success for slice appears in both object states and spatial relations, indicating a recurring weakness in precise tool-based actions.
  • BEHAVIOR sensitivity: In BEHAVIOR, straightforward actions outperform nuanced actions, while place_inside is difficult because it combines inside, holding, and handsfull constraints.
  • Cross-task implications: Across task categories, plug_in, walk_towards, slice_carvingknife, and place_inside expose deficiencies in multi-faceted constraints, precise movements, and tool use.

G.3 Result Analysis

Pipeline-based and modularized methods achieve similar trajectory executable rates, but composing modules reduces goal success through error accumulation. Most advanced models avoid grammar errors, while runtime errors—especially missing and additional steps—remain widespread.

  • Modularized and pipeline-based methods have similar trajectory executable rates across G + Q and G + Φ.
  • Pipeline-based methods suffer error accumulation because composing modules can reduce downstream goal success.Goal-interpretation errors can mislead downstream action and subgoal generation.
  • SOTA LLMs generally avoid grammar errors, unlike less advanced models including Gemini 1.0 Pro, GPT-3.5-turbo, and some open-source models.
  • Runtime errors persist across models, with missing necessary steps recurring during generation.
  • Replanning experiments append feedback and previous plans after execution or goal-satisfaction failures.

H.3 Result Analysis

Replanning improves action-sequencing performance by more than 10% on success and executable-success rates, including under stochastic failures. However, feedback can also prompt unnecessary extra actions.

  • More than 10% improvement occurs in success rate and executable success rate with replanning.Other metrics also improve except additional-step error rate.
  • Replanning can over-correct by generating unnecessary actions, increasing additional-step errors.
  • Replanning improves performance when actions fail stochastically, with the replanning gap generally widening at higher failure probabilities.

I Prompt and Analysis

The paper uses structured prompts to connect natural-language goals and robot tasks to symbolic, temporally ordered outputs. Prompt design addresses formatting failures, while experiments examine prompt transfer, human comparison, and visual-input limitations.

  • I.1 Prompt of Goal Interpretation: The goal-interpretation prompt constrains outputs to parsable JSON and designated object-state and relationship vocabularies.
  • I.1 Prompt of Goal Interpretation: Goal interpretation prompts convert household instructions into symbolic unary object states and binary object relationships using scene and initial-state information.
  • I.2 Prompt of Subgoal Decomposition: Subgoal decomposition prompts require a temporally ordered Boolean-expression list representing a complete transition from initial states to final goals.
  • I.2 Prompt of Subgoal Decomposition: These prompts encode robot constraints, including one-object-per-hand, free-hand requirements, intermediate states, valid connectives, and avoidance of redundant or invalid states.
  • I Prompt and Analysis: Many models struggle with strictly formatted outputs, so the paper replaces long strings with JSON and Python-list structures to leverage code-generation abilities.
  • I Prompt and Analysis: Model-specific prompts do not produce significant differences, suggesting that prompt improvements can generalize across large models.
  • I Prompt and Analysis: GPT-4o outperforms humans in subgoal decomposition and transition modeling but trails humans in goal interpretation and action-sequencing execution success.
  • K.1 Integration of Visual Inputs in Long-Horizon Decision Making: Visual-input evaluation finds lower end-to-end LLaVA performance than Llama 3, while scene graphs improve LLaVA without matching Llama 3 and entangle perception with planning errors.

K.1.3 Why LLMs Benchmarking is Useful

The benchmark supports standardized evaluation of LLM and VLM embodied decision making across modules, tasks, and perception-to-planning pipelines. Its results show that structured inputs can improve planning, while perception errors remain important constraints.

  • The benchmark evaluates LLMs and VLMs across standardized ability modules, including goal interpretation, subgoal decomposition, action sequencing, and transition modeling.
  • Adding scene graphs and broader context improved Llama’s goal-interpretation F1 from 31.5% to 31.9% and action-sequencing success from 11.1% to 13.9%.
  • The same input strategy improved LLaVA’s goal-interpretation F1 from 25.8% to 25.9% and action-sequencing success from 11.0% to 12.3%.
  • Scene-graph perception errors include unnecessary state, missing state, spatial hallucination, and object hallucination, with predicate errors reducing subgoal-decomposition success.
  • The dataset combines natural-language descriptions, symbolic goals, action trajectories, transition models, and human demonstration videos for systematic module evaluation.

L.2 Data Statistics and Distribution

The datasets target long-horizon embodied decision making with differing task lengths, goal structures, and simulator characteristics. VirtualHome has shorter average trajectories, whereas BEHAVIOR has longer actions and more complex goals.

  • VirtualHome contains 801 goals: 340 state goals, 299 relation goals, and 162 action goals, with action trajectories averaging 8.76 steps.
  • BEHAVIOR averages 14.6 actions and 3.7 goals per task, reflecting longer trajectories and more complex goal structures.
  • BEHAVIOR BDDL goals can yield multiple grounded-goal combinations, called goal options, increasing the variety of valid task specifications.
  • Task selection emphasizes long-horizon complexity: BEHAVIOR uses tasks with human demonstrations, while VirtualHome includes executable tasks in a typical scene.
  • Transition-model tasks are categorized into object affordance, object states, object orientation, spatial relations, and non-spatial relations using predicate-based IDF scores.
  • VirtualHome, BEHAVIOR, and AI2-THOR provide complementary capabilities involving household tasks, human activities, physics, task complexity, and state-space variation.

M.4.1 Annotation Quality Evaluation

The annotation process combines human expertise, iterative refinement, and automated checks, with high reported quality and annotator consistency. Evaluation uses accuracy, coverage, preference, and weighted variability measures.

  • Annotation quality is assessed with Action Accuracy, Action Coverage, Human Preference, and Weighted Mean Squared Error.
  • Weighted MSE averages squared deviations from task-level mean scores and weights task-level MSEs by evaluation frequency.
  • All annotation attributes receive mean scores above 3.7 out of 5, with relatively low Weighted MSE values indicating annotator consistency.
  • The evaluation environment includes visual-physical and symbolic simulators sharing the same actions and object states.

N.1.1 Building the Symbolic Simulator and Transition Model Implementation

The symbolic simulator defines action preconditions and post-effects while supporting automatic navigation and runtime error analysis. The framework standardizes modules and exposes both capabilities and important scope limitations.

  • Object interactability requires that an object is not enclosed and is within reach, although automatic navigation removes the reachability requirement for LLM task completion.
  • The VirtualHome evaluation environment records environment graphs and performs runtime error analysis for all 42 possible actions.
  • The framework standardizes goal specifications, module interfaces, and fine-grained error metrics across embodied decision-making evaluations.
  • Reported errors include grounding failures, missing or additional planning steps, and overlooked action preconditions.
  • Potential deployment requires attention to safety, ethics, privacy, security, and harmful biases, especially in safety-critical applications.
  • The approach abstracts environments as relational object graphs and therefore does not directly capture rich multimodal perception, low-level dynamics, sensory inputs, or precise actuation.
Loading 2410.07166v3…