PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs

Yining Hong, Huang Huang, Manling Li, Li Fei-Fei, Leonidas Guibas, Jiajun Wu, Yejin Choi

arXiv:2602.21198v3cs.LGcs.AIcs.CLcs.CVcs.RO

TL;DR

Embodied LLMs struggle to learn from failures, causing repeated mistakes during deployment. The paper introduces Reflective Test-Time Planning, which combines pre-action candidate evaluation, post-execution reflection, and hindsight updates to the policy and reflection model. Across embodied benchmarks and deployments, the framework improves performance, generalizes to HM3D and a Franka Panda arm, and supports better credit assignment through retrospective reflection.

  • Problem

    Embodied LLMs can plan tasks but cannot reflect on what went wrong or why, leaving deployment as repeated independent trials rather than accumulated experience.

  • Method

    Reflective Test-Time Planning combines internal reflection for candidate-action selection with external reflection and hindsight-driven test-time updates to the action policy and internal reflection model.

  • Results

    The framework achieves significant gains over baseline models across two embodied benchmarks, with zero-shot HM3D generalization, real-robot experiments, and better retrospective credit assignment than step-wise feedback at lower computational overhead.

  • Takeaways & Limitations

    Reflection-in-action and reflection-on-action are mutually dependent, while retrospective reflection supports long-horizon credit assignment and behavioral correction during deployment.

  • Takeaways & Limitations

    Reflection-in-action operates in imagination and may miss unforeseen physical constraints or environmental dynamics that cause execution failures.

Abstract

from arXiv · show

Embodied LLMs endow robots with high-level task reasoning, but they cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials where mistakes repeat rather than accumulate into experience. Drawing upon human reflective practitioners, we introduce Reflective Test-Time Planning, which integrates two modes of reflection: \textit{reflection-in-action}, where the agent uses test-time scaling to generate and score multiple candidate actions using internal reflections before execution; and \textit{reflection-on-action}, which uses test-time training to update both its internal reflection model and its action policy based on external reflections after execution. We also include retrospective reflection, allowing the agent to re-evaluate earlier decisions and perform model updates with hindsight for proper long-horizon credit assignment. Experiments on our newly-designed Long-Horizon Household benchmark and MuJoCo Cupboard Fitting benchmark show significant gains over baseline models, with zero-shot generalization to photorealistic HM3D environments and real-robot experiments on a Franka Panda arm. Ablations confirm that reflection-in-action and reflection-on-action are mutually dependent, and that retrospective reflection achieves better credit assignment than step-wise external feedback at lower computational overhead. Qualitative analyses further highlight behavioral correction through reflection.

1 Introduction

Embodied LLMs can plan tasks but remain unable to learn from failures, while existing reflection mechanisms largely operate in isolation. Reflective Test-Time Planning unifies internal pre-action evaluation, external post-action reflection, and hindsight-based updates, improving adaptation across embodied benchmarks.

  • Embodied LLMs remain brittle static oracles that cannot learn from failures, so deployment repeats mistakes instead of accumulating experience.
  • Human-inspired reflection alternates between internal simulation before action and outcome-based revision of beliefs and strategies afterward.
  • Existing approaches support verbal reflection-on-action or action-selection reflection-in-action largely in isolation, leaving persistent learning from reflections underexplored.
  • Reflective Test-Time Planning unifies reflection-in-action and reflection-on-action through candidate-action evaluation before execution and language-based assessment afterward.
  • Retrospective reflection re-evaluates earlier decisions with hindsight, enabling model updates that address long-horizon credit assignment and underlying error causes.
  • The framework gains substantially over baselines across newly designed household and cupboard-fitting benchmarks, generalizes zero-shot to HM3D and real-robot experiments, and reduces repetitive failures.

2 Related Works

The related work spans test-time adaptation, embodied multimodal LLMs, verbal reflection, and classical planning or reinforcement learning. The paper distinguishes its adaptive deployment-time parameter updates and reflection-based learning from fixed-parameter or single-loop approaches.

  • Test-Time Adaptation: Test-time adaptation updates models during inference under distribution shifts, with parameter-efficient variants including LoRA; this work uses self-supervised signals from verbal assessments.
  • Multimodal Embodied Large Language Models: Multimodal embodied LLMs pursue zero-shot generalization, spatial grounding, and multisensory interaction using large-scale robotic data and 3D representations.
  • Reflection and Self-Improvement in AI Agents: Verbal self-reflection methods store natural-language critiques to guide future actions, but these reflections are used as text rather than deployment-time model updates.
  • Planning and Reinforcement Learning for Embodied Agents: Classical planning and reinforcement learning typically correct actions without revising the failure-producing assumptions, corresponding to single-loop learning.

3 Reflective Test-Time Planning

Reflective Test-Time Planning adapts embodied agents during deployment by combining pre-action candidate evaluation, post-execution reflection, and hindsight-based model updates. The framework uses execution experience to refine both action selection and reflective reasoning.

  • Framework overview: Three interacting models generate actions, evaluate candidates internally, and assess executed outcomes externally during deployment.The action LLM πθ produces actions, while internal and external reflection LLMs provide pre-action and post-execution assessments.
  • Test-time training: Hindsight-corrected reflections form training pairs that update both the internal reflection LLM and the action LLM during test-time training.The resulting dataset includes retrospectively evaluated actions and supports learning from deployment experience.
  • Reflection-in-action: Reflection-in-action samples diverse candidate actions and selects the highest-scoring option after internal evaluation.Unlike greedy generation, the agent mentally evaluates multiple options before committing to one.
  • Limitations and complementarity: Internal reflection can fail because it reasons in imagination rather than reality, making external reflection necessary for grounding in execution outcomes.An action may receive a favorable internal score yet fail because of unforeseen physical constraints or environmental dynamics.
  • Reflection-on-action: Reflection-on-action stores executed actions and external feedback in a working-memory buffer, triggering consolidation at K steps or key milestones.External feedback assesses immediate outcomes and their causes, providing experience for later training.
  • Retrospective reflection: Retrospective reflection re-evaluates earlier actions with full hindsight to address failures whose consequences emerge beyond the next timestep.The method uses the current working-memory context, historical actions, prior reflections, and current observations for retroactive assessment.

4 Experiments on Long-Horizon Household Tasks

The Long-Horizon Household benchmark evaluates failure recovery across multi-step household tasks, while ablations examine reflection components and test-time training signals. Results show substantial gains over baselines and indicate that reflection-in-action and reflection-on-action work together.

  • Task Construction: The benchmark defines Fitting, Selection, Preparation, and Hybrid categories spanning geometric, preference, sequential-dependency, relational, and occlusion failures.Tasks are based on BEHAVIOR-1K environments and require multi-step planning across rooms.
  • Task Construction: Validated trajectories provide observation, action, reflection, and score tuples for initializing three embodied LLM modes before deployment.The models are trained jointly and instantiated as action, internal-reflection, and external-reflection models at deployment.
  • Results: Retrospective reflection provides better long-horizon credit assignment than immediate external reflection while reducing test-time overhead.Immediate external reflection evaluates only locally visible next-step consequences, whereas retrospective reflection reassesses earlier decisions with hindsight.
  • Results: 44.7% fitting success exceeds 10.6% for 3DLLM-Mem, 2.1% for ReflectVLM, and 0% for PPO.Fitting benefits most because constrained spatial tasks require iterative refinement from execution feedback.
  • Results: Removing RIA reduces Preparation to 3.17% and Hybrid to 3.23%, below 11.1% and 12.9% when both reflection components are removed.The ablations indicate mutual dependence: RIA improves action selection, while ROA supplies hindsight correction and test-time learning.
  • Results: Reflective Test-Time Planning generalizes zero-shot to photorealistic HM3D environments and outperforms baselines despite synthetic BEHAVIOR-1K training.Qualitative results illustrate continual learning and active perception from accumulated scene experience.

5 Experiments on the Cupboard Fitting Task

The controlled MuJoCo Cupboard Fitting benchmark isolates learning from placement failures under geometric and long-horizon constraints. Reflective Test-Time Planning improves fit and correct-placement rates, transfers to real robots, and benefits from both reflection modes and retrospective correction.

  • Task Design: The task requires a Franka Panda arm to place colored geometric objects into compatible cupboard compartments under multi-object spatial and long-horizon constraints.Fit rate counts objects placed in any compartment, while correct rate counts objects placed in designated compartments.
  • Results: 60.2% fit rate and 25.3% correct rate are achieved by the full method, substantially outperforming all baselines.The benchmark is designed to isolate reflective test-time training mechanisms after placement failures.
  • Results: Removing RIA lowers fit rate to 53.5%, removing ROA lowers it to 45.2%, and removing both lowers it further to 44.5%.The ablation pattern shows that candidate selection and test-time adaptation jointly support learning from failures.
  • Results: LoRA-based test-time training reaches 60.2% fit rate versus 57.4% for full base-weight training while using fewer trainable parameters.This result supports parameter-efficient adaptation for reflection-on-action.
  • Results: Retrospective reflection outperforms Receding Horizon Planning while saving 5× test-time compute.The comparison concerns long-horizon planning in the Cupboard Fitting benchmark.
  • Real-Robot Results: The full model generalizes better than the no-RIA-and-ROA ablation on a Franka Panda arm, with the gap widening under real-world uncertainty.The real-robot setting includes grasp imprecision and object slippage; retrospective reflection corrects earlier placement decisions mid-episode.

6 Conclusion

Reflective Test-Time Planning couples pre-action and post-execution reflection to improve embodied agents’ adaptation during deployment. The framework’s added computation is incurred at test time and is presented as practical because it converts execution time into reflective learning.

  • Reflective Test-Time Planning couples reflection-in-action for pre-action evaluation with reflection-on-action for post-execution assessment.
  • The framework adds candidate sampling, internal reflection scoring, and periodic reflection-on-action-based LoRA updates at inference time.
  • ∼3× higher per-step wall-clock time is observed across Long-Horizon Household and Cupboard Fitting tasks compared with the vanilla baseline.
  • The latency occurs during deployment without requiring additional supervised data or environment rollouts beyond normal task execution.
  • RIA filters poor actions before execution, while ROA reduces repeated failures through hindsight-driven updates.
  • Reflective computation produces persistent behavioral improvements, unlike baseline time spent on rollouts without policy or reasoning updates.

C.3 Compute-Matched Experiment

The compute-matched experiment tests whether the method’s gains arise from reflection rather than simply receiving more wall-clock time. A 3× rollout budget does not reproduce the full model’s performance, while engineering optimizations reduce overhead substantially.

  • Compute-Matched Experiment: A time-matched vanilla baseline receives a 3× steps budget to approximate the full model’s inference time.
  • Compute-Matched Experiment: Even with tripled time, the baseline fails to correct early decisions, revisits states without strategic change, and repeats placement and navigation failures.
  • Compute-Matched Experiment: The time-matched baseline plateaus below the full model on all benchmarks and does not improve over the vanilla baseline.
  • Compute-Matched Experiment: Reflective updates change the decision process, whereas rollout expansion only increases trajectory length without improving competence or hindsight reasoning.
  • Compute-Matched Experiment: The authors conclude that ∼3× test-time latency provides adaptation benefits that proportional rollout-step scaling cannot recover.
  • Efficiency Improvements: Combined engineering techniques reduce overhead to ∼0.45× while maintaining a 31.5% success rate.
  • HM3D Generalization: The HM3D evaluation tests deployment-time adaptation under a domain gap from synthetic BEHAVIOR-1K training scenes to photorealistic environments.

F.1 Human Evaluation of Vϕe Quality

The human evaluation assesses whether the external evaluator produces reliable supervision across distribution shifts. Its reflections achieve near-human-oracle performance under both environment-level and task-level shifts.

  • Human Evaluation: Human annotators rate Vϕe outputs for factual correctness, causal quality, and usefulness as a learning signal.
  • Human Oracle Comparison: The human-oracle comparison replaces Vϕe reflections with human-written reflections from the same execution outcomes.
  • Human Oracle Comparison: Evaluation covers HM3D environment-level shift and two held-out BEHAVIOR task types representing task-level shift.
  • Human Oracle Comparison: Near-identical performance confirms that Vϕe provides a reliable teacher signal under both distribution shifts.
  • Human Oracle Comparison: Vϕe is a post-hoc describer grounded in observed execution outcomes rather than a forward predictor of unobserved futures.
  • Initialization: Test-time reflection cannot function without minimal SFT initialization.

G.2 Base Model Scale Analysis

The base-model scale analysis examines how embodied prior and model size affect reflection quality and task fit. SFT repairs formatting, but fit performance still depends on sufficient embodied knowledge and adaptability.

  • Evaluation Dimensions: The analysis measures semantic similarity, human evaluation, format parse rate, and fit rate across model scales.
  • Without SFT: Without SFT, Qwen2-VL 2B has lower semantic similarity and human-evaluation scores, indicating unreliable reflections with insufficient embodied prior.
  • SFT Effects: Qwen2.5-VL 3B and 7B show high semantic quality but near-zero format parse rates without SFT, while SFT fixes format across models.
  • Fit Rate: After SFT, Qwen2.5-VL 3B reaches the optimal fit rate, whereas 2B underperforms from insufficient embodied prior and 7B degrades during test-time adaptation.
  • Generalization: The framework also evaluates ambiguous instructions and two held-out task types never seen during SFT.
  • Commercial Baselines: Compared with zero-shot commercial models, the framework achieves 33.7% average success rate versus 20.7% for GPT-5 and 19.7% for Gemini 2.5 Pro.

L GPT-5 Data Quality Validation

The paper validates generated initialization data against physics simulation and human judgment, then analyzes reflection-related hyperparameters on Cupboard Fitting.

  • GPT-5 Data Quality Validation: 100% simulator consistency and 98% human consistency follow physics-simulation filtering of GPT-5-generated trajectories.Raw outputs achieved 85% simulator consistency and 80% human consistency before filtering.
  • Reflection Hyperparameters: The ablations vary candidate count and sampling temperature to measure how internal reflection affects task performance.The candidate-count study evaluates N from 1 through 10, while the temperature study evaluates T from 0.0 through 2.0.
  • Reflection Hyperparameters: 60.0% performance occurs at N=6 candidate actions, while increasing the budget to N=10 lowers performance to 58.8%.Performance rises from 53.0% at N=1 before plateauing beyond six candidates.
  • Reflection Hyperparameters: 60.0% performance occurs at T=1.25, remaining near-peak at 59.8% for T=1.5 before dropping to 47.0% at T=2.0.The results support a moderate-temperature range for balancing candidate diversity and quality.

N.3 LoRA Configuration Analysis

LoRA test-time training performs best at an intermediate configuration, while larger adapters can cause mode collapse and excessive action budgets can slightly degrade performance.

  • LoRA Configuration: 60.0% performance occurs at (r=8, α=16), the optimal LoRA configuration in the ablation.Smaller configurations provide limited adaptation capacity, whereas larger configurations degrade sharply.
  • LoRA Configuration: 34.8% performance at (32,32) falls below the 53.0% no-adaptation baseline.The paper attributes this collapse to overparameterized adapters, aggressive learning rates, and limited retrospective training data.
  • Action Budget: 60.0% performance occurs at a 50-step action budget, compared with 51.5% at 30 steps and 59.4% at 100 steps.The 30-step setting often cannot complete multi-object placement, while the 100-step setting shows slight degradation.
  • Action Budget: Excessive action budgets can accumulate errors through inefficient exploration and longer downstream consequences.The analysis identifies excessive exploration and compounding errors as mechanisms behind the 100-step degradation.

O Single-Step Action Generation vs. Receding Horizon Planning

The paper favors single-step action generation over receding-horizon sequence planning because it integrates execution feedback and reflective test-time training more efficiently.

  • Single-Step Procedure: The single-step procedure executes one action, observes its outcome, updates the model, and then selects the next action.This sequence aligns action decisions with ground-truth execution feedback rather than imagined futures.
  • Experimental Comparison: 60.0% fit rate for single-step generation exceeds the 57.8% fit rate of receding-horizon planning.Receding-horizon planning also requires approximately 4× more computation per step in the reported comparison.
  • Computational Trade-offs: Receding-horizon planning discards 80% of its sequence-generation computation because only the first action is executed.The paper argues that this computation can instead support more candidates, additional training epochs, or larger retrospective-memory windows.
  • Compatibility with Test-Time Training: Sequence planning conflicts with test-time training because hypothetical future predictions are generated before execution outcomes update the model.The paper describes interference between imagined future states and learning from actual placement feedback.
  • Implicit Long-Horizon Planning: Retrospective reflection distills multi-step lookahead into single-step action evaluation by training internal reflection on hindsight about long-horizon consequences.A placement that initially succeeds can later be downgraded when it blocks a larger object.

P.3 Task Categories

The Long-Horizon Household benchmark organizes tasks around distinct failure modes and generates physically grounded trajectories with structured reflection annotations.

  • Task Categories: Four task categories target spatial reasoning, object selection, sequential dependencies, and combinations of these failure modes.The categories are Fitting, Selection, Preparation, and Hybrid tasks.
  • Fitting Tasks: Fitting tasks test geometric reasoning and capacity constraints through constrained placement and packing.Typical failures include oversized placements, blocking larger storage spaces, and unrecognized occlusion.
  • Selection Tasks: Selection tasks require comparing and retrieving items under preferences or constraints across explored rooms.Failures include choosing inferior items, committing before exploration, and failing to revise decisions after new information.
  • Preparation Tasks: Preparation tasks test ordered actions and prerequisites across multiple rooms and objects.Examples include adding toppings before placing a base plate or cooking before retrieving ingredients.
  • Task Generation: GPT-5 task prompts provide scene context, specify failure modes, and require structured trajectories containing actions, reflections, scores, and execution results.The generation pipeline uses BEHAVIOR-1K scene graphs and validates physical plausibility through simulation.
  • Cupboard Fitting Setup: Cupboard Fitting requires placing 6–10 objects into 6–8 compartments while satisfying object-compartment compatibility and long-horizon dependencies.The benchmark reports correct rate and fit rate, with execution status and updated visual observations after each action.
Loading 2602.21198v3…