Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making

Task standardization

• The interface represents object-centric states and actions and specifies both state-based and temporally extended goals as linear temporal logic formulas.This supports alternative goal interpretations and task-success criteria defined over trajectories.Source: §1 · p. 2, Figure 1 · p. 2

Module standardization

• It formalizes goal interpretation, subgoal decomposition, action sequencing, and transition modeling with standardized input-output specifications that integrate LLM-based and external modules.The interface enables modular comparisons across differing assumptions about knowledge, planners, and primitive skills.Source: §1 · pp. 2–3, Figure 2 · p. 2

Evaluation metrics

• Fine-grained trajectory and goal evaluation automatically identifies hallucination, affordance, action-order, missing-step, and additional-step errors across applicable modules.Trajectory evaluation tests simulator executability, while goal evaluation checks whether plans achieve correct outcomes; the two apply to overlapping but distinct module sets.Source: §1 · p. 3, Figure 3 · p. 3

Empirical findings

• Trajectory feasibility errors occur in 45.2% of cases, including missing-step errors at 19.5% and additional-step errors at 14.2%, often because models overlook preconditions.Models may omit standing before acting, opening closed containers before retrieval, or commonsense intermediate steps such as placing turkey on a plate before putting it on a table.Source: §1 · p. 3
• LLMs struggle to ground natural-language instructions, and evaluation performance worsens with longer trajectories or more complex environments.Subgoal decomposition remains as difficult as action sequencing in abstract spaces, while errors include hallucinated entities, reporting bias, and intermediate subgoals incorrectly treated as final goals.Source: §1 · pp. 1, 3
• o1-preview significantly outperforms other evaluated LLMs, especially on BEHAVIOR (74.9% vs. 64.2%), while showing broad strength across the four modules.It excels in goal interpretation on VirtualHome and in action sequencing, transition modeling, and subgoal decomposition on both benchmarks; Claude-3.5 Sonnet and Mistral Large lead selected module–benchmark combinations.Source: §1 · p. 4

2 Embodied Agent Interface Based on LTL

• The interface represents embodied tasks with objects, relational states, actions, goals, subgoals, and trajectories, while standardizing four LLM modules and their input-output specifications.The four modules are goal interpretation G, subgoal decomposition Φ, action sequencing Q, and transition modeling T.Source: §2 · p. 4

2.1 Representation for Objects, States and Actions

• States are finite object universes with relational Boolean features, and actions are primitive functions over objects expressed in abstract language forms.Examples include object properties, relations, and actions such as is-open(cabinet1), is-on(rag0, window3), and soak(rag0).Source: §2.1 · p. 4

2.2 Representation for Goals, Subgoals, Action Sequences, and State-Action Trajectories

• LTL provides a unified language for state constraints, action constraints, temporal orders, alternative trajectories, and evaluation across goals, subgoals, and action sequences.The implemented fragment supports Boolean and quantified formulas with the temporal connective then; formulas classify finite state-action trajectories, while additional temporal connectives remain extensible.Source: §2.2 · pp. 4–5

2.3 Ability Module 1: Goal Interpretation G : ⟨s0, lg⟩→g

• Goal interpretation maps an initial state and natural-language instruction to a simple LTL goal, evaluated by maximum F1 set matching against equivalent ground-truth goals.Generated goals contain an ordered action sequence and final-state propositions, while the ground-truth goal may be translated into multiple simple LTL goals.Source: §2.3 · p. 5

2.4 Ability Module 2: Subgoal Decomposition Φ : ⟨s0, g⟩→¯ϕ

• Subgoal decomposition maps a task to an LTL subgoal sequence, which breadth-first search refines into possible action trajectories evaluated for feasibility and goal satisfaction.Because each subgoal can admit multiple action sequences, the reported scores take the maximum over bounded candidate sequences.Source: §2.4 · pp. 5–6

2.5 Ability Module 3: Action Sequencing Q : ⟨s0, g⟩, M →¯a

• Action sequencing maps a task and transition model to actions, assessing executable trajectories through failure types and evaluating complete and partial LTL goal satisfaction.Feasibility errors include missing steps, additional steps, wrong temporal order, and affordance errors.Source: §2.5 · p. 6

2.6 Ability Module 4: Transition Modeling T : ⟨s0, g⟩, o →⟨pre, eff⟩

• Transition modeling generates PDDL preconditions and effects for operators, evaluated by logic-matching F1 and whether an external planner finds a plan achieving the goal.Logic matching recursively handles complex formulas, while planning success is binary according to whether the generated operators support a plan from the initial state.Source: §2.6 · p. 6

Annotations

• BEHAVIOR and VirtualHome are selected as evaluation simulators because of their task length and scene complexity.In the VirtualHome subset, 80.7% of task categories include instructions with more than 10 action steps, while 33% of instructions exceed 10 steps.Source: §3 · p. 6
• The dataset adds Goal Action annotations for necessary actions without post effects, alongside goal and trajectory annotations.The goal action touch in “pet the cat” exemplifies an action required to complete the task despite producing no post-effect state change.Source: §3 · p. 6
• BEHAVIOR BDDL goals are grounded into atomic propositions, with multiple grounded goal options potentially satisfying one quantified goal.Each task averages 6.7 grounded goals and 4,164.4 goal options.Source: §3 · p. 7

Implementation on simulators

• The authors implement EvalGibson to add BEHAVIOR an action-transition-model layer, providing 30 state-changing actions, and revise VirtualHome for accurate evaluation.Large-model evaluation settings are described separately, while implementation details appear in the appendices.Source: §3 · p. 7

4 Results

• o1-preview, Claude-3.5 Sonnet, and Gemini 1.5 Pro are the top overall models, with o1-preview leading except in object-state reasoning, where Gemini 1.5 Pro leads.Llama-3-70B and Mistral-Large-2402 are the strongest open-weight models, but remain behind commercial models.Source: §4 · p. 7
• 74.9% versus 64.2%: o1-preview outperforms other models particularly on BEHAVIOR, leading several module–simulator combinations and subgoal decomposition across both simulators.Claude-3.5 Sonnet leads BEHAVIOR goal interpretation and VirtualHome transition modeling, while Mistral Large leads VirtualHome action sequencing; Mixtral-8x22B and Llama-3-70B Instruct lead open-weight transition modeling and goal interpretation, respectively.Source: §4 · p. 7
• BEHAVIOR has lower trajectory-feasibility scores but higher goal-interpretation scores than VirtualHome, reflecting its longer horizons and VirtualHome’s larger state spaces.BEHAVIOR tasks average 14.6 steps, whereas VirtualHome goals can require searching larger state spaces; trajectory performance inversely correlates with sequence length, while goal performance inversely correlates with environment complexity.Source: §4 · p. 7
• Relational goals are harder than object-state goals: spatial relations have lower goal-interpretation recall and satisfaction, while some non-spatial relations are even harder in transition modeling.The passage gives holding(toothbrush) as a required precondition for brushing teeth.Source: §4 · p. 7

4.1 Ability Module Analysis

• LLMs often confuse intermediate subgoals with final goals and translate natural-language goals word-by-word instead of grounding them in the environment state.In the VirtualHome Drink task, GPT-4o incorrectly includes open(freezer) and inside(water, glass) as final goals.Source: §4.1 · p. 9
• Runtime errors dominate trajectory-feasibility failures, with missing-step and additional-step errors more common than wrong-order or affordance errors.Additional-step errors are most frequent even for powerful models, such as opening a box twice after the goal is already achieved.Source: §4.1 · p. 9
• Object goals are generally easier to achieve than relational goals in subgoal decomposition and action sequencing.The analysis contrasts object goals such as toggled_on with relational goals such as ontop(agent, chair).Source: §4.1 · p. 9
• Transition modeling varies substantially across models, with relational preconditions and effects, object orientation, and directionally biased state predictions remaining difficult.Claude-3 Opus reaches 63% on VirtualHome object states but performs weakly on spatial relations; models overstate object states in effects and spatial relations in preconditions, often producing infeasible plans.Source: §4.1 · pp. 9–10

• Prior embodied decision-making research uses LLMs for diverse tasks, with a comprehensive summary provided in Appendix P and a quick overview in Table 8.Source: §5 · p. 10, Table 8
• LLMs have been combined across modules through chain-of-thought prompting or pipelined queries, including goal interpretation, subgoal decomposition, action sequencing, and transition modeling.Reported combinations include goal interpretation with action sequencing [13] [14] [16] [17] [18] [19] [22] [23] [24] [25] [26] [28] [30] [31] [32], goal interpretation with subgoal decomposition [2], action sequencing with subgoal decomposition [34] [18], and action sequencing with transition modeling [8] [28] [32] [13].Source: §5 · p. 10
• This work standardizes the interface between LLMs and decision-making modules to support seamless integration, modular evaluation, and fine-grained metrics, while also reviewing agent interfaces [18] and simulation benchmarks.The additional related work on agent interfaces and simulation benchmarks appears in Appendix P.Source: §5 · p. 10

• EMBODIED AGENT INTERFACE standardizes LTL goal specifications, unifies decision-making tasks through a common interface and four ability modules, and provides fine-grained metrics with automatic error identification.The evaluation highlights limitations in complex-goal interpretation and reasoning, associating errors with trajectory length, goal complexity, and spatial-relation goals.Source: §6 · p. 10
• The current evaluation is limited to abstract-language states, actions, and goals represented through relational graphs; future work targets sensory inputs, actuation outputs, VLMs, memory, geometric reasoning, and navigation.The proposed extensions include episodic and state memory systems and are discussed further in Appendix K.Source: §6 · p. 10

1. Goal Interpretation:

• Gemini 1.5 Pro achieves the highest goal-interpretation F1-score, while Claude-3 Opus achieves the highest successful goal-retrieval Recall across both simulators.In VirtualHome, Gemini 1.5 Pro reaches 82.0% F1 and Claude-3 Opus reaches 89.1% Recall; LLMs commonly hallucinate states, omit relations, or generate intermediate goals.Source: §1 · p. 20

2. Action Sequencing:

• In BEHAVIOR, o1-preview achieves the highest task and execution success rates, 81.0% and 91.0%, whereas VirtualHome allows Mistral Large and Gemini 1.5 Pro to outperform it.Runtime errors are common, especially missing and additional steps caused by overlooked preconditions; state goals are easier than relation goals, and increasing task complexity reduces success.Source: §2 · p. 20

3. Subgoal Decomposition:

• o1-preview leads subgoal decomposition with 89.4% success in VirtualHome and 57.0% in BEHAVIOR, showing that decomposition is not necessarily easier than action sequencing.Gemini 1.5 Flash and Claude-3.5 Sonnet also reach 89.1% in VirtualHome.Source: §3 · p. 20

4. Transition Modeling:

• Transition modeling remains difficult for non-spatial relations, while targeted model strengths emerge for object states and orientations; inconsistent predicted action spaces can make planning infeasible.In BEHAVIOR, Claude-3.5 Sonnet reaches 78.8% F1 for object states and o1-preview reaches 83.5% F1 for non-spatial relations, but VirtualHome’s best non-spatial-relation F1 is only 11.9%.Source: §3 · p. 21

6. Pipeline-Based vs. Modularized; 7. Replanning and Feedback

• Pipeline-based composition causes error accumulation without improving execution rates, whereas feedback-based replanning raises GPT-4o’s BEHAVIOR task success from 47.0% to 59.0%.The pipeline reduces GPT-4o’s Goal Interpretation–Subgoal Decomposition task success from 48.0% to 38.0%, while replanning raises execution success from 53.0% to 63.0% but increases additional-step errors from 0.0% to 3.0%.Source: §3 · pp. 21–22

B.1 Motivation

• EAI addresses limitations of high-level planning benchmarks by grounding natural-language goals in object-centric states, relations, and temporally dependent LTL transitions while providing fine-grained error metrics.It unifies goal interpretation, subgoal decomposition, action sequencing, and transition modeling for embodied environments.Source: §B.1 · p. 22

B.2 Input and Output Details

• The interface accepts task instructions, agent instructions with in-context examples, and environment representations, while modules produce grounded goals, state sequences, action sequences, or transition effects.Subgoal states can require multiple actions, and transition modeling maps operator preconditions to post-effects.Source: §B.2 · pp. 23, 24, 26

B.3 Grounding to Markov Decision Process; B.4 The Relationship between Ability Modules

• EAI grounds embodied decision making in an MDP tuple and maps its four modules to goal input, trajectory output, transition learning, and fine-grained goal evaluation.The protocol isolates individual modules using existing data or tools for the remaining components, enabling detailed diagnosis of strengths, weaknesses, and integration choices.Source: §B.3 · pp. 26–27, §B.4 · pp. 27–28

C.1 Why LTL

• LTL unifies state goals, action sequences, action preconditions and postconditions, subgoal sequences, and candidate subgoal sequences in a compact representation supporting unified evaluation.Its expressiveness captures state constraints, action constraints, temporal dependencies, and alternatives, while formula similarity provides a common accuracy metric.Source: §C.1 · p. 28

C.2 Comparision with Traditional LTL Representation

• The interface extends standard LTL with relational states, quantifiers including ∃=n, and a finite-trajectory then operator for more expressive task planning.The recursive logical structure and connectives remain consistent with standard LTL.Source: §C.2 · p. 29

C.3 Syntax and Semantics of LTL Formulas

• Its finite-trajectory formulas use state and action propositions, logical connectives, quantifiers, and then to specify what must occur before what.The then operator supports temporal ordering without requiring globally or until, and formulas are evaluated over state-action trajectories.Source: §C.3 · pp. 29–30

D.1 Goal Interpretation: State Goal, Relation Goal and Action Goal

• Each goal dimension is scored with precision, recall, and F1 to quantify false positives, missed goals, and overall translation capability.Together, these metrics provide a comprehensive evaluation of converting natural-language goals into symbolic representations that agents can understand and execute.Source: Figure 17 · p. 31, §D.1 · p. 31
• Goal interpretation translates natural-language goals, the initial world state, and simulator-recognized actions and states into symbolic goals for embodied agents.The pipeline tracks grammatically incorrect, structurally incoherent, and hallucinated object or state predictions before comparing valid outputs with ground truth.Source: §D.1 · p. 31
• The evaluation separates goal interpretation into single-object state goals, object-object relation goals, and agent-action goals.Action goals occur in some simulators, such as VirtualHome, but not in BEHAVIOR-100.Source: §D.1 · p. 31

D.2 Action Sequencing: Trajectory Error Detection for Missing Step, Additional Step, Wrong Temporal Order, Affordance Error

• Action sequencing evaluates executable action plans through parsing, hallucination, argument-number, affordance, additional-step, missing-step, and wrong-order errors, followed by goal satisfaction.Inputs include the initial state, goal, environment vocabulary, objects, and node, edge, and action goals; success checks final node, edge, and action conditions.Source: §D.2 · pp. 32–34

D.3 Subgoal Decomposition: Converting Subgoal Trajectory to Action Trajectory with BFS Searching

• Subgoal decomposition converts temporally ordered declarative states into executable trajectories by repeatedly using BFS to connect the current state to each subgoal.BFS finds shortest paths when they exist, but can become computationally expensive in larger action and state spaces; the benchmark’s limited spaces avoid that issue.Source: §D.3 · pp. 34–35

D.4 Transition Modeling: Evaluating with PDDL Planners

• Transition modeling evaluates predicted PDDL action preconditions and effects using logical-form matching, precision, recall, F1-score, planner success rate, and task-category sensitivity analysis.The logical metric recursively matches predicted and ground-truth clauses through bipartite matching, while planner success measures whether executable solutions reach desired goals.Source: §D.4 · pp. 37–38

D.5 Average Performance

• Overall performance averages GoalF1, SubgoalTask SR, ActSeqTask SR, and half-weighted Transition Modeling F1 and planner success rate.The aggregate combines the four ability modules using the stated weighting equation.Source: §D.5 · p. 38

E Full Results with 18 models

• The full-results section analyzes each ability module across 18 models and additionally examines potential correlations with factors in Appendix E.5.The supplied overview states that BEHAVIOR involves longer action sequences and complex subgoal decomposition, whereas VirtualHome more strongly challenges environmental-state understanding.Source: Figure 23 · p. 38, E Full Results with 18 models · p. 39

E.1 Goal Interpretation

• Goal Interpretation benchmarks how LLMs translate natural-language task instructions into actionable symbolic goals in VirtualHome and BEHAVIOR, with qualitative error analyses by error and goal type.The reported goal categories include state or unary goals, spatial or binary goals, and VirtualHome-specific action goals.Source: §E.1 · p. 39, §E.1.1 · p. 39

E.1.1 Result Analysis

• The Goal Interpretation results aggregate state, spatial, and VirtualHome action goals into an Overall micro average alongside category-specific performance.The supplied passages identify Table 9 as the quantitative benchmark but do not provide its cell values.Source: §E.1.1 · p. 39
• o1-preview leads overall F1 in VirtualHome, Claude-3.5 Sonnet leads in BEHAVIOR, Gemini 1.5 Pro is close, and Llama-3 70B Instruct leads open-source models in both simulators.These comparisons summarize the overall precision, recall, and F1 evaluation reported in Table 9.Source: §E.1.1 · p. 39, Table 9 · p. 40
• Category leaders vary: o1-preview and Claude-3.5 Sonnet excel on spatial goals, Gemini 1.5 Pro leads VirtualHome action and BEHAVIOR state goals, and Cohere Command R leads VirtualHome state goals.Source: §E.1.1 · p. 39

E.1.2 Case Studies

• Commercial GPT and Claude models rarely make format errors with clear prompts and examples, whereas Llama 3 8B and Mixtral 8x22B make some parsing mistakes.Grammar errors include hallucinations and formatting errors such as JSON parsing failures.Source: §E.1.2 · p. 39
• Both proprietary and open-source models may substitute intermediate goals for final predictions, causing false positives through Chain-of-Thought reasoning.Source: §E.1.2 · pp. 39–40
• Models also omit uncommon but essential spatial relations, such as predicting chicken ONTOP(table) instead of the ground-truth chicken–plate–table relations, causing missing goals.This imprecision occurs across Behavior and VirtualHome and is consequential for embodied planning and robotic control.Source: §E.1.2 · p. 40

E.2.1 Result Analysis

• The dominant decomposition errors differ by simulator: VirtualHome models most often add already-satisfied steps, whereas BEHAVIOR models most often miss precondition-satisfying steps.VirtualHome also shows temporal-order and affordance errors, while BEHAVIOR missing-step errors exceed half of cases for most models and remain 25.0% for o1-preview; stricter preconditions and longer tasks drive this gap.Source: §E.2.1 · p. 42, §E.2.2 · p. 45
• o1-preview achieves the highest task success rates in VirtualHome (89.4%) and BEHAVIOR (57.0%), while BEHAVIOR is more challenging because its tasks involve complex quantifiers and stricter precondition checks.In BEHAVIOR, o1-preview also reaches 62.0% execution success, followed by GPT-4o at 49.0% task success and 55.0% execution success.Source: §E.2.1 · p. 42, §E.2.2 · p. 45
• In BEHAVIOR, LLMs perform less well on state and relation goals because quantified goals fail if even one constituent goal is unmet; o1-preview achieves 56.5% state and 69.4% relation success, highest among models.The evaluation metric treats quantifiers as combined entities for state and relation goals, contributing to lower observed success rates.Source: §E.2.1 · pp. 42–43

E.2.2 Case Studies

• Case studies identify parsing, parameter-length, hallucination, wrong-order, missing-step, affordance, additional-step, and goal-satisfaction failures across both simulators.Examples include nonexistent objects, invalid action affordances, omitted actions such as DRINK, and plans that break previously achieved relations by moving an object.Source: §E.2.2 · pp. 43–44

E.3 Action Sequencing

• For action sequencing, Mixtral Large performs best in VirtualHome, whereas longer BEHAVIOR sequences favor stronger reasoning models, especially o1-preview.VirtualHome tasks average only 3–5 actions, while BEHAVIOR tasks average 14.6 steps and impose stricter preconditions; overall, even the best models complete only about half of BEHAVIOR tasks.Source: §E.3 · pp. 45–46
• Action-sequencing failures are dominated by missing steps in both simulators, reflecting unsatisfied preconditions and limited commonsense reasoning, while grammar performance generally improves with model size, newer versions, and higher-quality training.Wrong-order errors reflect planning deficiencies, affordance errors reflect misunderstanding object properties, and additional-step errors can arise from in-context memory failures or incorrect assumptions about the initial state.Source: §E.3 · pp. 46–47

E.4.1 Logic Form Accuracy

• In VirtualHome, Claude-3 Opus achieves 63.0% F1 in object states, while Claude-3.5 Sonnet achieves 90.9% in object orientation and 78.8% in BEHAVIOR object states.Gemini 1.5 Pro reaches 77.7% F1 in VirtualHome object affordance; BEHAVIOR spatial relations reach 58.6% for Claude-3.5 Sonnet and non-spatial relations 83.5% for o1-preview.Source: §E.4.1 · pp. 49–50
• VirtualHome non-spatial relations remain difficult, with Gemini 1.5 Flash reaching only 7.9% F1 because models miss complex preconditions and effects involving object and agent states.Examples include omitted container and hand-occupancy conditions for grab, plus incorrect effects describing how held objects interact.Source: §E.4.1 · pp. 50–51

E.4.2 Planner Success Rate Analysis

• o1-preview achieves 100% success in VirtualHome object orientation and 90.2% in BEHAVIOR non-spatial relations, demonstrating robust planner performance across environments.The results assess whether predicted transitions enable feasible plans from initial states to goals.Source: §E.4.2 · p. 51
• Non-spatial relations generally achieve higher planner success than spatial relations, but object orientation remains difficult and precise spatial interactions challenge even strong models.Model performance varies substantially by category; GPT-4o struggles on non-spatial relations, while Claude-3 Opus performs poorly on object states and affordances.Source: §E.4.2 · p. 51

E.4.3 Error categorization

• Missing predicates omit necessary action conditions, whereas additional predicates over-constrain plans; property hallucinations can introduce unsupported vocabulary and impossible constraints.Examples include replacing lieable with sittable, assigning a stained state to a cleaning brush, and predicting an undefined wearable property.Source: §E.4.3 · p. 52

Consistency of predicted action space

• Mixed LLM-predicted and ground-truth action spaces reduce planner success because inconsistent action definitions can make otherwise valid transitions incompatible.GPT-4o’s plug_in prediction passes independently, but conflicts with the ground-truth switch_on requirements when combined with ground-truth actions.Source: Consistency of predicted action space · p. 54, Figure 35 · p. 54

E.5 Correlection with Action Length and Goal Complexity

• GPT-4o goal success declines as total, state, and relation goals increase, as ground-truth action sequences lengthen, and as action preconditions become more complex.Tool-dependent actions such as Soak, Slice, and Clean are especially challenging; OPEN reaches 1.00 execution success, while both tested SLICE cases fail.Source: §E.5 · pp. 54–55, Figure 36 · p. 55

F.1 Motivation and Problem Formulation

• Sensitivity analysis replaces each ground-truth action with its LLM-predicted counterpart and uses PDDL planning to identify actions that affect task success by category.The method provides finer-grained, per-action diagnostics beyond whole-action-space planner success and highlights actions for improvement or downstream fine-tuning.Source: §F.1 · p. 57, §F.2 · p. 57

F.3 Result Analysis

• VirtualHome results showed strong performance on straightforward actions, but substantial difficulties on complex interactions and spatial predictions, with “plug_in” and “walk_towards” among the most challenging actions.“plug_in” scored 0.09 in object affordance and object states; “walk_towards” scored 0.63 in spatial relations and 0.65 in non-spatial relations.Source: §F.3 · pp. 57, 59
• BEHAVIOR results likewise showed robust performance on simple actions but severe weaknesses in precise tool use, manipulation, and compound spatial interactions.“release” and “slice_carvingknife” both had 0.00 success in non-spatial relations, “slice” had 0.00 in object states and spatial relations, and “place_inside” required coordinating spatial, holding, and handsfull conditions.Source: §F.3 · pp. 59, 61
• The sensitivity analysis recommends broader scenario coverage, specialized fine-tuning, and targeted error analysis focused on underperforming actions.These interventions are intended to improve robustness and applicability to complex real-world interactions.Source: §F.3 · p. 61
• Across both environments, action-specific failures reflect missing handling of multifaceted preconditions, precise movements, fine motor skills, and advanced spatial awareness.The analysis attributes “plug_in” errors to an overlooked has_switch precondition and “walk_towards” errors to added or missing spatial constraints.Source: §F.3 · pp. 59, 61

G.1 Motivation and Problem Formulation

• The pipeline-based study addresses complete tasks by converting ambiguous natural-language goals into symbolic goals before downstream decision making.For example, “cook food” may require symbolic conditions such as cooked(chicken) and ontop(chicken, plate).Source: §G.1 · p. 62
• Given an initial state s0 and natural-language goal lg, the objective is to generate a feasible action sequence ¯a and decompose lg into a subgoal trajectory ¯ϕ.Source: §G.1 · p. 62

G.2 Implementation Details

• Implementation composes Goal Interpretation G with either Action Sequencing Q or Subgoal Decomposition Φ to produce action sequences or subgoal trajectories in BEHAVIOR.The rules are G: ⟨s0, lg⟩→g, followed by Q: ⟨s0, g⟩→¯a or Φ: ⟨s0, g⟩→¯ϕ.Source: §G.2 · p. 62

G.3 Result Analysis

• Pipeline-based methods have lower goal success because goal-interpretation errors mislead downstream modules into producing less accurate action and subgoal sequences.Despite this degradation, LLMs can still generate feasible and executable action sequences.Source: §G.3 · p. 62
• SOTA LLMs generally avoid grammar errors, unlike Gemini 1.0 Pro, GPT-3.5-turbo, and some open-source models, while all models remain prone to runtime errors.The runtime failures arise when generation omits necessary steps; the authors attribute grammar-error differences to instruction tuning and model scale or training rigor.Source: §G.3 · pp. 62–63
• Table 16 reports similar trajectory executable rates for modularized and pipeline-based methods across G + Q and G + Φ, while pipelines suffer error accumulation.The comparison covers goal interpretation with action sequencing or subgoal decomposition, where M denotes modularized and P pipeline-based methods.Source: §G.3 · pp. 62–63, Table 16 · p. 63

H.1 Motivation and Problem Formulation

• The authors regard feedback-based replanning as necessary because agents should learn from failures and warnings, adjust plans within the simulator, and avoid repeating mistakes.[9][81][82]The current setting gives the agent only one planning or prediction attempt, motivating the replanning extension.Source: §H.1 · p. 63

H.2 Implementation Details

• For each failed task, the system appends error messages, failure details, and the previous plan to the task prompt before requesting a revised plan.Failures include execution errors such as grammar or runtime errors and unsatisfied node, edge, or action goals.Source: §H.2 · p. 64
• The feedback explicitly identifies missing-step runtime errors by stating that an action is not executable and requires another action beforehand.For unsatisfied goals, the message lists unmet node, edge, and action goals and asks the model to check and revise the sequence.Source: §H.2 · p. 64

H Replanning and Feedback

• The implementation permits at most three replanning attempts, accumulating all previous plans and error messages, and evaluates this setting on GPT-4o for action sequencing.Each retry prompt records the retry count and the predicted action sequence from the preceding attempt.Source: §H.2 · p. 64

H.3 Result Analysis

• Replanning improves success rate and executable success rate by more than 10%, while also improving other metrics except additional steps error rate.These results are reported for action sequencing in Table 17.Source: Table 17 · p. 64, §H.3 · p. 64
• Replanning enables models to learn from errors, adjust action sequences toward task outcomes, and reduce repetition of similar errors across iterations.The resulting improvement demonstrates enhanced adaptability through feedback-based strategy revision.Source: §H.3 · p. 64
• Replanning can over-generate actions, increasing additional step errors and creating unnecessary actions that require more precise guidance to avoid.The model’s revisions may overstate actions when it iterates on previous attempts and modifies them.Source: §H.3 · p. 64

H.4 Replanning with Stochastic Actions

• The stochastic-action experiment evaluates GPT-4o on BEHAVIOR action-sequencing tasks with simulated failure probabilities and up to three replanning attempts.This experiment targets action failures that arise during action sequencing.Source: §H.4 · p. 64
• Replanning is helpful when actions fail stochastically, according to the BEHAVIOR results in Table 18.The comparison explicitly evaluates conditions with and without replanning.Source: Table 18 · p. 65, §H.4 · p. 65
• The performance gap between replanning and no replanning generally increases as the action failure probability becomes larger.Thus, replanning’s relative benefit is greater under more frequent stochastic failures.Source: §H.4 · p. 65

Prompt of Goal Interpretation

• The goal-interpretation prompt requires parsable JSON and restricts unary outputs to a fixed vocabulary of household object states.The system prompt requires output beginning with { and ending with }, without explanations; allowed unary states include Cooked, Open, Frozen, Dusty, Stained, Sliced, Soaked, and Toggled_On.Source: Prompt of Goal Interpretation · p. 65
• Goal interpretation maps natural-language household objectives into symbolic unary object states and binary object relationships using relevant objects and initial states.Unary goals use designated object-state predicates, while binary goals encode relationships such as NextTo, Inside, OnFloor, Touching, and Under.Source: Prompt of Goal Interpretation · p. 65

Prompt of Subgoal Decomposition

• Subgoal decomposition asks the model to generate a complete, temporally ordered list of boolean state expressions connecting initial states to final goals.The prompt defines first-order predicate vocabulary and connectives, while requiring logical, detailed, format-compliant state transitions.Source: Prompt of Subgoal Decomposition · p. 66
• Subgoal plans incorporate manipulation constraints, including one object per hand, a free hand for most actions, valid holding states, and nonredundant transitions.The prompt requires intermediate states when effects cannot occur directly and permits conjunctions or alternatives when temporal ordering or equivalence allows them.Source: Prompt of Subgoal Decomposition · p. 66

Prompt of Action Sequencing

• Action sequencing converts initial states, target states, and interactable objects into a sequential list of action commands that changes the environment to the target.States are represented as unary or binary predicates, and each command specifies an action name and one or more target objects.Source: Prompt of Action Sequencing · p. 67
• The action-sequencing vocabulary covers grasping, placing, releasing, opening, closing, cooking, cleaning, freezing, slicing, soaking, drying, toggling, and spatial relations.Commands distinguish left- and right-hand operations and support transfers, next-to placement, next-to-on-top placement, and under placement.Source: Prompt of Action Sequencing · p. 68
• Action commands encode execution preconditions, such as empty hands for grasping, held objects for placement, open containers for insertion, and appropriate tools or states for transformations.Examples require a knife for slicing, a soaked cleaning tool or toggled-on cleaner for cleaning, and a toggled-on sink for soaking.Source: Prompt of Action Sequencing · p. 68
• Action-sequencing prompts enforce affordance and execution constraints, including hand capacity, valid object names, object accessibility, hand availability, and post-slicing object identities.They also specify PLACE, RELEASE, and target-object requirements for executable commands.Source: Prompt of Action Sequencing · p. 69
• Action-sequencing prompts require multiple formatted commands that transform the initial environment state into the target state through sequential execution.The prompt provides initial and target states, interactable objects, and explicit output-format constraints.Source: Prompt of Action Sequencing · p. 69

I.4 Prompt of Transition Modeling

• The transition-modeling prompt casts the LLM as a software engineer generating only parsable JSON containing PDDL action definitions.The system prompt requires output to begin with “{” and end with “}”, without explanatory notes.Source: §I.4 · p. 70, Prompt of Transition Modeling · p. 71

Prompt of Transition Modeling

• Transition modeling supplies typed VirtualHome predicates describing states, spatial relations, hand occupancy, and object affordances as the domain vocabulary for predicted actions.The predicate inventory includes properties such as grabbable, cuttable, can_open, movable, and hangable.Source: Prompt of Transition Modeling · p. 71
• Given PDDL objects, initial conditions, goals, action names, and parameters, the model writes action parameters, preconditions, and effects so actions can reach the goal.The required action structure comprises an action name, parameters, precondition, and effect, with predicate arity and types matching the domain file.Source: Prompt of Transition Modeling · pp. 71–72
• Preconditions must use disjunctive normal form with typed predicates and restricted logical operators, while effects support conjunction, negation, conditional WHEN clauses, and universal FORALL clauses.The prompt formally defines the permitted clause structures and requires predicate arguments to match declared numbers and types.Source: Prompt of Transition Modeling · p. 72
• The hang_up_clothes example demonstrates deriving alternative right- or left-hand preconditions and conditional hand-release effects that establish the clothes-on-hanger goal.The example requires clothing, a hangable target, hand possession, and proximity, then adds ontop while removing either hand-holding predicate.Source: Prompt of Transition Modeling · pp. 72–73
• Transition-modeling prompts require exact domain predicates, declared action parameters, valid preconditions and effects, and PDDL-formatted JSON output that connects the initial state to the goals.The prompt permits actions without preconditions or effects and discourages unnecessary WHEN usage or additional arguments.Source: Prompt of Transition Modeling · p. 74

I.6 Prompt Analysis and Learned Lessons

• Structured JSON and Python-list outputs replace long strings because models often produce unparsable formats or forbidden explanations despite explicit instructions.The prompting lessons also favor system prompts, stop sequences, shorter strings, and start/end tokens to improve format following and reduce repetition.Source: §I.6 · p. 76

I.7 Further Consideration about Prompt Variability

• The model-agnostic prompt design keeps format error rates below 3.8% across evaluated models, supporting fairer comparisons without model-specific prompt bias.The design was iteratively tested across all models to ensure accurate prompt interpretation.Source: §I.7 · p. 76
• Model-specific prompt tuning produces no significant performance differences, supporting generalizable prompt improvements across large models.Across Claude, GPT-4o, and Gemini experiments, no model-specific prompt consistently dominates; all reported benchmark results use the finalized general_improved prompt.Source: §I.7 · p. 78

J Human Performance Comparison

• GPT-4o is outperformed by humans in Goal Interpretation and Action Sequencing Execution Success Rate, but exceeds humans in Subgoal Decomposition and achieves perfect Planner Success Rate in Transition Modeling.The results suggest complementary strengths: GPT-4o handles long-context logical reasoning and scene-graph tracking, whereas humans better handle nuanced understanding and execution.Source: J Human Performance Comparison · p. 78, Table 19 · p. 78

K.1 Integration of Visual Inputs in Long-Horizon Decision Making

• The study extends its primarily text-based evaluation with VLM experiments because visual information is critical for practical embodied AI and text-only assumptions may be limiting.The additional experiments examine whether VLMs can support long-horizon planning.Source: §K.1 · p. 78

K.1.1 Challenges with Existing VLMs

• Existing VLMs are not designed for scene-level reasoning in complex multi-room environments, while prior embodied evaluations mainly emphasize perception rather than long-horizon decision making [87] [91].The discussion specifically concerns environments such as BEHAVIOR and VirtualHome.Source: §K.1.1 · pp. 78–79

K.1.2 Comparative Experiments with LLMs and VLMs

• LLaVA performs significantly worse than Llama 3 end-to-end, improves with scene-graph input, but still trails Llama 3 using scene graphs, while entangling perception and planning errors.The comparison uses BEHAVIOR experiments with scene-graph input to Llama 3, image-only input to LLaVA, and combined image and scene-graph input to LLaVA.Source: §K.1.2 · p. 79, Table 20 · p. 79

K.1.3 Why LLMs Benchmarking is Useful

• Adding scene graphs and broader context improved Llama’s goal-interpretation F1 from 31.5% to 31.9% and action-sequencing success from 11.1% to 13.9%.The same approach improved LLaVA’s goal-interpretation F1 from 25.8% to 25.9% and action-sequencing success from 11.0% to 12.3%.Source: Table 21 · p. 79, §K.1.3 · p. 80

K.1.4 Implications for Future Research

• The findings motivate better VLM integration of visual and textual information, modular evaluations separating perception from decision making, and adaptation of structured-input and broader-context strategies.These directions are presented as priorities for improving VLM long-horizon decision making and embodied reasoning.Source: §K.1.4 · p. 80

K.2 Impact of Perception and State Estimation Errors

• The perception-impact experiment used VLM-generated scene graphs as LLM inputs for BEHAVIOR subgoal decomposition, targeting a module described as highly sensitive to perception performance.Different VLMs generated scene graphs from images before LLM-based planning.Source: §K.2 · p. 80, §K.2.1 · p. 80

K.2.2 Results Analysis

• Claude-3.5 Sonnet achieved the highest tested performance at 75.0%, while higher predicate error rates reduced subgoal-decomposition success and accurate object relationships proved crucial.The evaluated perception errors include unnecessary state, missing state, spatial hallucination, and object hallucination; missing state and spatial hallucination are grouped as predicate errors.Source: Table 22 · p. 80, §K.2.2 · p. 81, Figure 54 · p. 81

K.3 Assumptions on Scene Graphs in Our Benchmark

• The benchmark adopts scene graphs because they represent semantic and relational environment structure needed for reasoning about objects and complex task planning.Generating relational graphs from real-world visual data is identified as an active research area.Source: §K.3 · p. 81

L.1 Dataset Structure

• The dataset stores natural-language task descriptions, symbolic goals including LTL forms, symbolic action trajectories, and transition models for evaluating four ability modules.BEHAVIOR tasks additionally include VR demonstrations of ground-truth action trajectories.Source: §L.1 · p. 81
• The annotation design extends RobotHow [5] with precise symbolic goals and transition-model annotations, reducing noisy-state effects and capturing logical preconditions and post-effects.For example, on(light) is presented as more precise than combining facing(agent, light) with on(light).Source: §L.1 · pp. 82–83
• The general data structure supports systematic module integration, unified evaluation across environments, downstream applications, and language grounding between natural-language commands and symbolic goals or actions.It is designed to extend beyond specific simulators and facilitate comparison of methods across domains.Source: §L.1 · p. 83

L.2 Data Statistics and Distribution

• 801 RobotHow goals comprise 340 state, 299 relation, and 162 action goals, with trajectories averaging 8.76 steps and reaching 54 steps.One-third of instructions exceed 10 steps, and 21 of 26 task categories include such instructions.Source: §L.2 · p. 84

L.3 Goal Complexity Analysis

• BEHAVIOR tasks average 14.6 actions and 3.7 goals, while each task averages 6.7 grounded goals and 4164.4 goal options generated from quantified BDDL goals.Different grounded-goal combinations can satisfy the same BDDL goal, creating multiple goal options.Source: §L.2 · p. 84, §L.3 · p. 85, §M.2.2 · p. 89

L.4 Task List

• The benchmark selects long-horizon, complex tasks: VirtualHome contributes executable household activities from a typical scene, while BEHAVIOR contributes all tasks with human demonstrations across realistic domains.Examples include cooking, washing dishes, bottling fruit, object manipulation, tool use, and goal-directed behavior.Source: §L.4 · pp. 85–86

L.5 Task Categorization

• Tasks are categorized into object affordance, object states, object orientation, spatial relations, and non-spatial relations using IDF-weighted predicate scores, with K = 2 categories assigned per task.Table 25 reports the resulting task counts for VirtualHome and BEHAVIOR.Source: §L.5 · p. 87, Table 25 · p. 88

M.1 Simulator Comparison and Selection

• VirtualHome and BEHAVIOR are selected over AI2-THOR because they better support extended, goal-oriented decision making through complex household activities and diverse human demonstrations.VirtualHome emphasizes long high-level action sequences, whereas BEHAVIOR provides detailed low-level scripted actions and broad human activities.Source: §M.1 · p. 88, Environments and Assets · p. 89

M.2 BEHAVIOR

• BEHAVIOR-100 annotation adds ground-truth action sequences and natural-language descriptions, deriving executable PDDL problems and modified domain files from BDDL tasks and VR human demonstrations.The action-sequence pipeline combines automatic object-state tracking and segmentation with manual mapping of state changes to actions; sampled solvable goal options are bounded by max_num grounded goals.Source: §M.2 · p. 89, §M.2.1 · p. 89, §M.2.3 · p. 89

M.3 VirtualHome

• VirtualHome annotation manually standardizes 338 task instructions across 30 categories by identifying wildcard representations for common final states, then extracting concrete goals and annotating state and relation goals.Source: §M.3 · p. 89, §M.3.1 · p. 90, §M.3.2 · p. 90

M.4 Quality Verification

• Annotation quality is checked automatically in VirtualHome and BEHAVIOR for executable actions, goals, and desired outcomes, and is additionally assessed by humans for action accuracy, coverage, and preference.These checks jointly target simulator consistency and perceived annotation quality.Source: §M.4 · p. 90

M.4.1 Annotation Quality Evaluation

• Annotation quality was evaluated using action accuracy, action coverage, and human preference.These metrics respectively assess correspondence to required actions, coverage of necessary actions, and human judgments of quality and naturalness.Source: §M.4.1 · p. 91
• Weighted MSE quantified score variability by averaging per-attribute squared deviations from task means and weighting task-level MSEs by evaluation frequency.This weighting accounts for differing numbers of evaluations per task while balancing task variability and overall consistency.Source: §M.4.1 · p. 91
• Mean annotation scores exceeded 3.7/5 for all attributes, while relatively low Weighted MSE values indicated consistency among annotators.The score distribution was also visualized in Figure 63.Source: §M.4.1 · p. 91, Figure 63 · p. 91
• The annotation process combined human expertise, iterative refinement, and automated checks to produce a high-quality, reliable dataset.The reported metrics support the dataset’s quality and inter-annotator consistency.Source: §M.4.1 · p. 91

N.1 BEHAVIOR Implementation Details

• BEHAVIOR included an iGibson-based simulator with visual rendering and PyBullet physical simulation, alongside a symbolic simulator recording activities as graph-based states.The two simulators provide distinct visual-physical and symbolic evaluation environments.Source: §N.1 · p. 92

N Simulator Implementation Details

• Both BEHAVIOR simulators shared the same action set and object-state representation.This common interface connects the visual-physical and graph-based implementations for behavioral-task evaluation.Source: §N.1 · p. 92

N.1.1 Building the Symbolic Simulator and Transition Model Implementation

• The symbolic simulator specifies each action through arguments, preconditions, and post-effects, with object interactability based on containment and reachability.Because auto-navigation invokes NAVIGATE_TO automatically, LLMs need not satisfy the reachability requirement.Source: §N.1.1 · p. 92, Table 27 · p. 92
• BEHAVIOR’s transition models cover manipulation, placement, transfer, toggling, opening, drying, soaking, and unfreezing actions under explicit object and hand-state conditions.Examples include placing held objects relative to targets, toggling interactable objects, and requiring appropriate states for OPEN, DRY, SOAK, and UNFREEZE.Source: §N.1.1 · pp. 93–94

N.1.2 BEHAVIOR State Space and Action Space

• BEHAVIOR defines 15 object states and 30 actions, separating unary from binary states and state actions from spatial actions.The complete state and action names are provided in Table 28.Source: §N.1.2 · p. 94, Table 28 · p. 95

N.2.1 VirtualHome State Space and Action Space

• VirtualHome provides the evaluation foundation through an EnvironmentGraph-backed MotionPlanner, runtime error analysis for 42 actions, and modified executors aligned with expected outcomes.Its state and action configuration is summarized in Table 29.Source: §N.2.1 · p. 94, Table 29 · p. 95

O.1 Decoding Parameters

• All evaluated LLMs use temperature zero to standardize decoding and target the arg max under each model’s distribution.The setup limits sampling to temperature scaling because several models do not support other sampling methods.Source: §O.1 · p. 94
• Although completion sampling can introduce instance-level randomness, low-temperature arg-max decoding makes this effect insignificant in the evaluation scenarios.This qualification concerns the specific decoding setup used for the prompts.Source: §O.1 · pp. 94, 96

O.2 Evaluation Cost

• A complete evaluation run requires 180 model-module runs, approximately 192,280,000 tokens and 84,960 queries, costing $1540.70 through Together AI pricing for open-source models.The dataset is released under CC-BY-4.0, with public hosting, a persistent DOI, and structured schema.org metadata.Source: §O.2 · p. 96, §Q.1 · p. 98, §Q.2 · p. 99

P.3 Embodied Agent Benchmarks

• The interface evaluates goal interpretation, subgoal decomposition, action sequencing, and transition modeling through standardized interfaces, fine-grained error metrics, and LTL-based representations of complex temporally extended goals.This addresses limitations of prior benchmarks that focus on narrower modules or lack detailed planning-error analysis.Source: §P.3 · p. 98, §P.2 · p. 98

Q.5 URLs of Code and Re-productivity

• The codebase, evaluator scripts, Dockerized simulator tooling, and batch LLM inference integrated with HELM are publicly available for reproducing module evaluations across VirtualHome and BEHAVIOR.Together AI APIs are provided as an option for open-source model inference, with environment-specific computation resources documented for each evaluator.Source: §Q.5 · p. 99, §Q.5.1 · p. 99, §Q.5.2 · p. 99

How many instances are there in total (of each type, if appropriate)?

• The dataset contains 338 VirtualHome and 100 BEHAVIOR task plans, with 2,960 and 1,460 steps, respectively, plus annotated goals and transition models for both environments.VirtualHome includes 801 goal conditions and 33 transition models; BEHAVIOR includes 673 goal conditions and 30 transition models.Source: How many instances are there in total… · p. 100
• The benchmark targets zero-shot evaluation of long-horizon tasks with complicated goals selected from VirtualHome and BEHAVIOR-100, using task annotations, ground-truth trajectories, transition models, and BEHAVIOR demonstration videos.The accompanying evaluator executes plans in modified simulators and supports visualization, while each instance is associated with a ground-truth action sequence.Source: How many instances are there in total… · pp. 100–101, Any other comments? · p. 100

Any other comments?

• The annotations were authored by expert researchers, supplemented with manual goals, trajectories, transition models, and natural-language instructions, and verified by executing plans and checking satisfied goals.VirtualHome annotations were cross-examined with alternative trajectories, while BEHAVIOR action sequences were checked against multiple demonstration videos.Source: How many instances are there in total… · p. 101, Any other comments? · p. 102
• Human annotators create missing goals, trajectories, and transition models from scratch, while simulator execution validates the goals, trajectories, and transition models.The annotation process is detailed in Appendix M, and code supporting it is available through the project repository.Source: Any other comments? · p. 103
• The dataset supports evaluation of 15 LLMs, with zero-shot evaluation code planned for release and future benchmarking of VLMs left open.The dataset will be publicly available in JSON format with task goals, trajectories, instructions, symbolic definitions, transition models, and an evaluator for fine-grained metrics.Source: Any other comments? · pp. 103–104

When will the dataset be distributed?

• The dataset will be publicly released under VirtualHome and BEHAVIOR licenses for research and commercial use, with distribution planned through the project website.The authors will maintain it and host download and evaluation tools through GitHub.Source: When will the dataset be distributed? · p. 104, Who will be… · p. 104

How can the owner/curator/manager of the dataset be contacted (e.g., email address)?

• The maintainers will announce updates and extensions through the website and GitHub, host public errata, and review dataset contributions submitted through pull requests.No older dataset versions currently exist; standard open-source practices will support community contributions.Source: How can the owner/curator/manager of… · p. 105

S.1 Broader Impact

• The framework standardizes goals, modules, and interfaces while fine-grained evaluation identifies LLM strengths, weaknesses, and failure modes for more reliable embodied agents.These insights can guide targeted training, safeguards, fallback mechanisms, monitoring, and intervention strategies across robotics, autonomous systems, and human-robot interaction.Source: §S.1 · pp. 105–106

S.2 Limitations

• The current framework abstracts environments as relational graphs and omits rich multimodal perception, low-level physics, sensory inputs, actuation outputs, and fine-grained manipulation.It therefore isolates symbolic reasoning but does not fully capture the perception-language-action integration required in physical environments.Source: §S.2 · p. 106

S.3 Potential Negative Social Impact

• Deployment may create privacy, security, bias, employment, and inequality risks, requiring robust safety measures, ethical frameworks, regulatory guidance, and diverse stakeholder engagement.Potential benefits include healthcare, education, manufacturing, and logistics applications.Source: §S.3 · p. 106

S.4 Potential Future Directions

• Future work targets multimodal grounding, memory, physical reasoning, navigation, counterfactual prediction, diverse benchmarks, and fine-tuning from fine-grained evaluation feedback.Proposed extensions include VLM integration, low-level control, scene-graph memory, richer physics simulations, and specialized decision-making models trained on embodied trajectories.Source: §S.4 · p. 107