Source-linked AI summary
AtomTree: A Hierarchical Framework for State-Aware Embodied Instruction Following with LLMs
Haotian Zhai, Tianming Sha, Junnan Li
TL;DR
Long-horizon embodied instruction following remains challenging when planners rely on costly labeled trajectories or one-shot plans that use limited evolving-state information. AtomTree incrementally builds a state-aware subgoal tree, using memory and hierarchical backtracking to update plans. On ALFRED, it outperforms prior zero-shot planners across all four splits.
Problem
Embodied instruction following needs long-horizon planning under partial observability, while large labeled trajectories are costly and many LLM planners rely on one-shot plans with limited use of evolving state and history.
Source: §I · pp. 1–2Method
AtomTree builds a state-conditioned subgoal tree, using node memory and hierarchical backtracking to regenerate a subtree from the latest state when an action stalls.
Source: §I · p. 2Results
ATOMTREE achieves the best performance among zero-shot methods on all four ALFRED splits; on Valid Seen, its SR rises from 18.78 to 40.97 versus SOCRATIC-PLANNER[13].
Source: D. Results · p. 6Takeaways & Limitations
The results suggest state-driven hierarchical reasoning can mitigate error accumulation from one-pass sequential planning in long-horizon settings.
Source: §I · p. 2
Abstract
from arXiv · showhide
Embodied instruction following in interactive environments requires agents to interpret natural language commands and complete long-horizon tasks. Existing planners either rely on large labeled trajectories, or use few-shot and zero-shot LLM prompting that produces a whole plan in a single serial reasoning process. These approaches depend on linguistic common sense, commit to a plan from the initial instruction, and only update after failures, which weakens robustness and state awareness. We present AtomTree, a hierarchical framework for state-aware task planning with LLMs. AtomTree incrementally builds a tree of subgoals and actions, where the LLM decides at each step, from the current symbolic state, whether to further decompose or to output a grounded low-level action. A lightweight memory module summarizes past outcomes for reuse in later decisions. When a low-level action stalls, AtomTree triggers hierarchical backtracking replanning to return to the parent node and regenerate the subtree plan from the latest state, preventing continued drift along an incorrect trajectory. In zero-shot evaluation on the ALFRED benchmark, AtomTree outperforms prior LLM-based planners. Project page: https://zhaihaotian.github.io/atomtree-website/.
I. INTRODUCTION
On ALFRED, AtomTree addresses the difficulty of long-horizon embodied instruction following by making planning state-aware and hierarchical. Its zero-shot evaluation reports stronger performance than prior LLM-based planners, while error analysis identifies planning as a growing failure mode as task length increases.
I. INTRODUCTION
- Embodied instruction following requires perception, subgoal planning, navigation, and object interaction under partial observability, while generalizable long-horizon planning with limited supervision remains open.The motivating task setting includes household instructions such as cleaning a mug and placing it into a coffee machine.Source: §I · p. 1
- Large-scale instruction and expert-action datasets are costly to collect, while manually designed task procedures do not scale across heterogeneous tasks.The passage describes both annotation and demonstration collection as expensive and time-consuming.Source: §I · p. 1
- 68.7% of failures for tasks with ground-truth PDDL length greater than 7 are planning errors, up from 38.9% when length is no greater than 7.As the PDDL-length threshold rises from 5 to 15, planning errors rise from around 50% to nearly 100% of failures (Pearson r = 0.8892, p ≈ 2.1 × 10^-5).Source: §I · pp. 1–2
- Existing LLM planners commonly generate a full plan in one pass and replan only after failures, leaving decisions weakly conditioned on evolving environment state.A “white table” instruction may not match the environment’s available object categories, illustrating the gap between language and executable ontology.Source: §I · p. 2
- AtomTree treats planning as a state-conditioned sequential decision process, choosing at each node whether to decompose the current objective or execute an atomic action.This produces a tree of decomposition and execution nodes intended to mitigate error accumulation from one-pass sequential planning.Source: §I · p. 2
- Each AtomTree node conditions its decision on the current environment state, instruction segment, and internal memory, which summarizes historical observations for downstream decisions.If a low-level action stalls, hierarchical backtracking returns to its parent node and regenerates the subtree from the latest state, rather than only replanning the stalled action.Source: §I · p. 2
- Hierarchical backtracking replanning adjusts the plan from a parent node after local failure, preventing continued deviation along an incorrect sequential trajectory.The mechanism is triggered when a low-level action stalls for an extended period.Source: §I · p. 2
- In zero-shot ALFRED evaluation, ATOMTREE surpasses state-of-the-art few-shot LLM planners on multiple primary metrics and performs better than some few-shot and supervised methods, especially on long-horizon tasks and complex instructions.The introduction also reports stronger overall performance than representative few-shot, zero-shot, and supervised baselines.Source: §I · p. 2
A. Embodied Instruction Following
Embodied instruction following combines language understanding with perception, planning, navigation, and interaction, often over long horizons. Prior approaches include supervised action prediction and LLM-based planning, while AtomTree uses dynamic state-conditioned decomposition and execution rather than relying on fixed or one-shot plans.
A. Embodied Instruction Following
- End-to-end supervised embodied instruction-following methods map observations and instructions to fine-grained actions, but rely heavily on large annotated trajectories or task-specific supervision and generalize poorly to new environments.The task itself typically requires long-horizon navigation and object interaction in 3D environments.Source: A. Embodied Instruction Following · p. 2
- Recent modular or hierarchical LLM planners generate subgoals or action sequences and use feedback or external modules, but mostly retain one-shot sequential planning with limited use of evolving state and execution history.AtomTree instead dynamically decides at each step whether to decompose a subgoal further or emit executable actions based on current state.Source: A. Embodied Instruction Following · p. 2
B. LLM for Embodied Planning
- Decompose-then-solve strategies and structured prompting, including Chain-of-Thought, provide a methodological foundation for applying hierarchical planning to embodied tasks.The passage states that these strategies can improve LLM performance on complex compositional tasks.Source: B. LLM for Embodied Planning · pp. 2–3
- AtomTree’s overview contrasts its evolving tree of state-conditioned decomposition and execution with a fully supervised end-to-end model and an LLM planner using retrieved few-shot examples.The tree progressively refines higher-level goals into executable actions.Source: Figure 2 · p. 3
- Embodied planning commonly uses LLMs as high-level planners; examples combine language models with affordance values, environment feedback, programmatic prompting, or explicit state tracking.The passage names SayCan, Inner Monologue-style methods, ProgPrompt, AutoTAMP, and LLM-State as examples.Source: B. LLM for Embodied Planning · p. 3
- Long-horizon systems with bi-stage or hierarchical structures still often rely on long single-shot plans or predefined hierarchies, limiting support for dynamic states and long action sequences.The passage frames these limitations in EIF scenarios such as ALFRED.Source: B. LLM for Embodied Planning · p. 3
- The related ReAcTree system is described as using explicit control-flow nodes and retrieval-based episodic memory, whereas AtomTree is characterized as zero-shot with lightweight symbolic memory.The passage identifies ReAcTree as concurrent work closely related to AtomTree.Source: B. LLM for Embodied Planning · p. 3
III. PRELIMINARIES
The framework evaluates embodied instruction following from high-level goals alone, using a state-conditioned hierarchy that chooses between decomposing a subgoal and executing an atomic action. The resulting subgoal tree is expanded and executed in a closed loop.
III. PRELIMINARIES
- The evaluation discards ALFRED's step-by-step instructions and conditions planning only on the high-level goal.The authors state that this matches underspecified human instructions that rarely spell out every intermediate step.Source: §III · p. 3
- At each decision step, the planner uses the current state and subgoal to choose between Decompose and an executable high-level atomic action.The action set includes navigation, finding, and object interactions; the low-level controller maps these to primitive actions.Source: §III · p. 4
IV. METHOD
- ATOMTREE maintains an evolving subgoal tree, expanding an active subgoal or executing one atomic action before continuing with updated observations and memory.The low-level controller uses deterministic components, including map-based navigation and rule-based interaction executors.Source: §IV · p. 4, §III · p. 4, A. State-Aware Decision · p. 5
A. State-Aware Decision
- The decision prompt supplies the current subgoal, state, and memory so the LLM can choose whether to decompose the goal or execute it directly.The state includes the held object and objects detected in the current egocentric view.Source: Figure 3 · p. 4, A. State-Aware Decision · p. 4
B. State-Aware Node Memory
ATOMTREE's lightweight symbolic memory provides long-horizon context under partial observability. It records objects seen and action outcomes, then reuses them in later decisions and replanning.
B. State-Aware Node Memory
- The memory combines accumulated seen-object names with a history of high-level actions and their final outcomes.Records can capture outcomes such as a failed Find(Apple) or successful Navigation(Table).Source: B. State-Aware Node Memory · p. 5
- As new objects are discovered, the seen-object memory can ground an initially ambiguous description such as “white table” to a scene category such as DiningTable.The planner can then generate a grounded navigation and placement sequence for the previously ambiguous subgoal.Source: B. State-Aware Node Memory · p. 5
- The same serialized memory supports decisions, decomposition, and backtracking replanning without passing the full interaction trace to the LLM.It lets the policy reuse discovered objects and prior successes or failures.Source: B. State-Aware Node Memory · p. 5
C. Hierarchical Backtracking Replanning
ATOMTREE handles failed or stalled execution by returning to the parent subgoal and replanning from updated state and memory. It replaces the attempted child subtree and resumes execution.
C. Hierarchical Backtracking Replanning
- Hierarchical backtracking addresses execution failures such as interaction errors, occluded targets, or high-level assumptions invalidated by new observations.Replanning occurs at the subgoal-tree level.Source: C. Hierarchical Backtracking Replanning · p. 5
- When an action fails or the controller stalls under its budget, the planner backtracks to the parent subgoal and reconsideres it using updated state and memory.The parent decision can produce a revised ordered subgoal list or an atomic action for immediate execution.Source: C. Hierarchical Backtracking Replanning · p. 5
- After a failed Find(Apple), recorded container candidates can support revising the subtree to open a Cabinet before searching again.Once the target is found, the planner resumes the original navigation objective.Source: C. Hierarchical Backtracking Replanning · p. 5
A. Benchmark and Splits
The experiments use ALFRED validation and test splits spanning seen and held-out scenes. ATOMTREE runs zero-shot with DeepSeek-V3 and no training on ALFRED trajectories.
A. Benchmark and Splits
- ALFRED validation and test splits each include seen and unseen scenes, with 820 Valid Seen, 821 Valid Unseen, 1,533 Test Seen, and 1,529 Test Unseen episodes.Seen scenes appear in ALFRED training; unseen scenes are held out for evaluation.Source: A. Benchmark and Splits · p. 5
B. Models and Inference Setup
- All experiments use DeepSeek-V3 in a zero-shot pipeline without training on ALFRED trajectories or language-annotated learning data.A small number of exemplars constrain output format; the sampling temperature is 1 and the generation cap is 2,024 tokens per LLM call.Source: B. Models and Inference Setup · p. 5
C. Metrics
ATOMTREE reports stronger ALFRED performance than prior zero-shot and few-shot LLM planners, with comparisons limited to goal-only settings. The section defines SR and GC and presents results across four benchmark splits.
C. Metrics
- SR measures the percentage of episodes completing the full task, whereas GC measures the fraction of goal conditions satisfied at episode end.GC captures partial progress when the task is not fully completed.Source: C. Metrics · p. 5
D. Results
- Table I compares SR and GC across four splits, while Table II reports SR ablations on Valid Seen and Valid Unseen.The comparison includes only methods evaluated with goal-only instructions; methods requiring step-by-step instructions are excluded.Source: Table I · p. 6, Table II · p. 6, D. Results · p. 6
- ATOMTREE achieves the best zero-shot performance across all four ALFRED splits, with large SR gains over SOCRATIC-PLANNER [13].SR rises from 18.78 to 40.97 on Valid Seen, 15.35 to 42.39 on Valid Unseen, 18.66 to 39.92 on Test Seen, and 13.47 to 41.20 on Test Unseen.Source: D. Results · p. 6, Table I · p. 6
- In the zero-shot setting, ATOMTREE surpasses FLARE [4] on Test Seen (SR 31.96 to 39.92) and Test Unseen (SR 32.57 to 41.20).It is also competitive with several supervised methods, exceeding FILM [10] and MCR-AGENT [35] on both test splits while remaining close to PROMPTER [11] and CAPEAM [16], especially on unseen scenes, without trajectory training.Source: D. Results · p. 6
- The results indicate stronger and more robust performance under limited supervision, particularly on unseen and long-horizon tasks.The method uses only the goal instruction.Source: D. Results · p. 6
E. Ablation Study
The ablation progressively adds hierarchical planning, state-and-history memory, and backtracking replanning to a reactive baseline. Each component improves SR on both validation splits, with memory providing the largest gain.
E. Ablation Study
- Each added component consistently improves SR on Valid Seen and Valid Unseen.The additions are hierarchical planning, state-and-history memory, and hierarchical backtracking replanning.Source: E. Ablation Study · p. 6
- Memory provides the largest improvement by helping prevent forgotten completed actions and repetitive predictions, especially for composite subgoals such as cleaning.Fig. 4 contrasts an incorrect repeated cleaning-related subgoal without memory with AtomTree proceeding to the fridge after cleaning.Source: E. Ablation Study · p. 6, Figure 4 · p. 7
- Hierarchical planning gives the first clear gain by letting the model choose between further decomposition and direct action execution.The baseline is ReAct-style: it re-queries after each atomic action but has no hierarchy, memory, or backtracking.Source: E. Ablation Study · p. 6
- Backtracking adds a smaller gain by revising the subtree after failure using the updated state and memory.The ablation table reports SR on the two validation splits.Source: E. Ablation Study · p. 6
F. Qualitative Analysis
The qualitative examples contrast memory- and state-aware decisions with repeated or irrelevant predictions. The conclusion reports stronger performance than prior zero-shot and few-shot LLM planners and points to state-aware hierarchy under limited supervision.
F. Qualitative Analysis
- Memory lets the planner move from completed lettuce-cleaning steps to navigating to the fridge instead of repeating cleaning-related subgoals.Without memory, it may fail to recognize that the lettuce has already been washed, leading to redundant subgoals or loops.Source: F. Qualitative Analysis · p. 7, Figure 5 · p. 7
- After placing the first spray bottle, the few-shot planner predicts an unnecessary cabinet action, while AtomTree follows the current state and execution history.Fig. 5 compares the two planners on a multi-instance task.Source: F. Qualitative Analysis · p. 7, Figure 5 · p. 7
VI. CONCLUSION
- ATOMTREE outperforms prior zero-shot and few-shot LLM planners and is competitive with several supervised methods, especially on long-horizon and unseen tasks.The conclusion presents state-aware hierarchical planning as a promising direction for robust embodied agents under limited supervision.Source: §VI · p. 7