Source-linked AI summary
Inner Monologue: Embodied Reasoning through Planning with Language Models
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, Brian Ichter
TL;DR
Embodied planning requires language models to reason over changing world states and action outcomes, but prior approaches provide limited evidence about language-mediated feedback. This paper introduces Inner Monologue, which combines LLM planning with textual feedback and pretrained robotic skills without additional training; across three robotic settings, environment feedback significantly improves instruction completion and supports replanning in unseen scenarios.
Problem
Embodied robots must reason about available skills, their effects, and changing observations, while prior language-based planning work did not study how language can provide embodied feedback for closed-loop planning.
Method
Inner Monologue injects textual feedback from perception models or humans into LLM planning prompts while combining the planner with pretrained robotic skills.
Results
Environment feedback significantly improves high-level instruction completion across three simulated and real robotic manipulation settings and enables replanning in complex unseen scenarios.
Takeaways & Limitations
Closed-loop language feedback supports retrying after stochastic failures, replanning under infeasibility, and requesting human feedback for ambiguous instructions.
Takeaways & Limitations
The approach assumes access to oracle scene descriptions in some settings, and low-level control policies can bottleneck instruction completion and task scope.
Abstract
from arXiv · showhide
Recent works have shown how the reasoning capabilities of Large Language Models (LLMs) can be applied to domains beyond natural language processing, such as planning and interaction for robots. These embodied problems require an agent to understand many semantic aspects of the world: the repertoire of skills available, how these skills influence the world, and how changes to the world map back to the language. LLMs planning in embodied environments need to consider not just what skills to do, but also how and when to do them - answers that change over time in response to the agent's own choices. In this work, we investigate to what extent LLMs used in such embodied contexts can reason over sources of feedback provided through natural language, without any additional training. We propose that by leveraging environment feedback, LLMs are able to form an inner monologue that allows them to more richly process and plan in robotic control scenarios. We investigate a variety of sources of feedback, such as success detection, scene description, and human interaction. We find that closed-loop language feedback significantly improves high-level instruction completion on three domains, including simulated and real table top rearrangement tasks and long-horizon mobile manipulation tasks in a kitchen environment in the real world.
1 Introduction
Embodied robot planning requires semantic understanding, sequencing, feedback interpretation, and adaptation when actions fail. Inner Monologue combines LLM reasoning with textual environment feedback and pretrained skills to support closed-loop replanning without additional training.
- Embodied interaction requires robots to sequence basic behaviors, recognize unsuccessful plans, and integrate high-level planning, perception, and low-level control.
- LLMs may serve as interactive problem solvers by combining multiple feedback sources for robotic manipulation, beyond interpreting natural-language instructions.
- Inner Monologue uses language as an interpretable interface for combining textual feedback with LLM planning through few-shot prompting without additional training.
- The system combines perception models, human feedback, and pretrained manipulation skills in a shared language prompt to execute user instructions.
- Without additional training beyond a frozen language model and pretrained skills, Inner Monologue handles complex, long-horizon, and unseen tasks in simulation and on two real robotic platforms.It can retry after stochastic failures, replan under systematic infeasibility, and request human feedback for ambiguous queries.
2 Related Work
Prior work applies planning and language to robotics, but language-model planners commonly assume that proposed actions execute successfully. Inner Monologue addresses this gap by incorporating grounded environmental feedback into language-based planning.
- Task and motion planning jointly addresses high-level discrete task planning and low-level continuous motion planning.
- Language-model planning methods decompose abstract instructions into executable low-level steps, often in a zero-shot manner.
- Existing approaches effectively assume successful execution of every proposed step, limiting robustness to intermediate failures in dynamic environments or with weak low-level policies.
- Inner Monologue explores grounded environmental feedback to make language-model planning responsive to execution outcomes.
3 Leveraging Embodied Language Feedback with Inner Monologue
Inner Monologue closes the loop between an LLM planner and an embodied robot by injecting textual observations during interaction. It supports several feedback modalities, including success detection, passive scene descriptions, and actively queried scene information.
- Inner Monologue treats LLMs as interactive problem solvers that incorporate embodied environment observations into grounded planning.
- Problem Statement: The planner selects short-horizon skills from a pretrained library to fulfill a natural-language instruction while receiving textual feedback from the environment.
- The framework continually injects information from feedback sources into language prompts as the robot interacts with the environment.
- The analysis studies textual feedback generally rather than assuming one specific fusion algorithm or one method for extracting environmental information into language.
- Sources of Feedback: Feedback includes task-specific success detection and scene-specific passive or active descriptions.
- Sources of Feedback: Success Detection expresses whether a low-level skill succeeded as language feedback.
- Sources of Feedback: Passive Scene Description supplies structured feedback automatically and consistently, without active prompting by the planner.
- Sources of Feedback: Active Scene Description responds to planner queries with unstructured answers from a person or pretrained model, allowing targeted information gathering.
4 Experimental Results
Across simulated and real-world robotic domains, Inner Monologue uses language-model planning with embodied feedback to execute, recover from failures, and adapt during long-horizon tasks. Closed-loop feedback supports replanning, recovery, and several unscripted interaction capabilities, though these capabilities vary in consistency without similar prompted examples.
- Experimental Setup: Inner Monologue is evaluated across simulated tabletop rearrangement, real-world tabletop rearrangement, and real-world kitchen mobile manipulation using domain-specific prompts and feedback models.The shared formulation decomposes human instructions into actionable steps while incorporating embodied feedback.
- Simulated Tabletop Rearrangement: In simulation, Object + Scene feedback performs best because it tracks goal conditions and currently achieved goals while generalizing to unseen tasks without further training.The evaluation includes four seen and four unseen tasks, with test-time disturbances.
- Real-World Tabletop Rearrangement: Closed-loop scene description and success detection enable real tabletop robots to replan after clutter-induced omissions and recover from noisy policy failures.Object feedback addresses occluded objects, while success detection supports retrying failed pick-and-place actions; the components are complementary.
- Real-World Mobile Manipulator in a Kitchen Setting: With adversarial disturbances, Inner Monologue substantially outperforms SayCan because embodied feedback enables appropriate recovery modes, whereas SayCan lacks explicit high-level retry behavior.Success and Object feedback reduce planning failures and overall failure rate, while also introducing new failure modes.
- Emergent Capabilities: Human feedback lets the planner switch goals mid-task, propose alternatives when goals become infeasible, interpret Chinese instructions, and answer temporally grounded scene questions.These capabilities emerge without similar prompted examples, including switching tasks twice and replacing an unsuccessfully attempted heavy block with a lighter one.
- Emergent Capabilities: Emergent capabilities are inconsistently expressed when similar examples are absent from prompts, likely reflecting limitations of current language models.The authors identify further investigation of these behaviors and limitations as a future direction.
5 Limitations
The paper’s results are constrained by assumptions about textual scene feedback and by the capabilities of low-level control policies. Failure modes also arise from inaccurate success detection, planning errors, and control errors.
- Oracle scene descriptors are assumed in some experiments, although learned scene description and object recognition are studied separately.
- False negatives from success detectors cause unnecessary retries, while false positives create adversarial partial observability.
- Additional failure sources include LLM planning errors and control errors, including plans that ignore feedback and reference absent objects.
- Low-level control policies can bottleneck instruction completion and limit the tasks over which the LLM can reason.
- Future work includes fully automated feedback, uncertainty-aware information aggregation, and additional safety and ethics feedback modules.
6 Conclusion
The paper formulates Inner Monologue as closed-loop language feedback for embodied robotic planning and evaluates it across simulated and real manipulation settings. Environment feedback significantly improves high-level instruction completion, particularly under adversarial disturbances, while enabling replanning in complex unseen settings.
- Inner Monologue combines environment feedback with LLM planning and robotic control policies across three simulated and real robotic manipulation settings.
- Environment feedback significantly improves high-level instruction completion, especially in challenging scenarios with adversarial disturbances.
- Closed-loop language feedback enables replanning in complex unseen settings.
A Inner Monologue Implementation Details
The implementation details describe multiple Inner Monologue variants that inject textual environment feedback into LLM planning and connect the planner to pretrained robotic policies. Feedback includes object recognition, success detection, and scene descriptions across simulated and real tabletop systems.
- Three Inner Monologue implementations differ in their internal components while each incorporates textual environment feedback into planning.
- Simulated success detection marks pick-and-place execution successful when the picked object is within 4cm of the place object and satisfies the height condition.
- Scene descriptions track achieved sub-goals and, for sorting, visible objects and objects already moved into a plate.
- The low-level tabletop policy uses object bounding boxes, object identities, depth images, camera intrinsics, and camera pose to output 3D pick and place locations.
- Pick positions are computed by deprojecting depth pixels and transforming points from the camera frame into the robot base frame.
- Test-time planar translation noise uses σ=1.5cm for block stacking and σ=0.7cm for object sorting, with samples capped at 1.5σ.
A.3 Inner Monologue for Real-World Mobile Manipulation in a Kitchen Setting
The kitchen mobile-manipulation implementation combines a large language model, learned and scripted low-level policies, and textual feedback from human or learned perception. It includes learned success detection, object-recognition comparisons, and active human questioning.
- The kitchen system uses PaLM with learned and scripted navigation and manipulation policies, including behavior-cloning policies trained on teleoperated and autonomous data.
- Human-provided object recognition supplies textual lists of objects visible to the robot camera.
- ViLD and MDETR perform worse than humans but provide reasonably effective automated object feedback without domain-specific fine-tuning.
- ViLD fails to detect objects 28.0% of the time in 10 representative kitchen mobile-manipulation episodes.
- The foresight success detector uses initial and final observations plus the attempted low-level skill to predict whether execution succeeded.
- A hindsight predictor outputs probabilities over possible skills and is combined with the foresight model to reduce false-positive success predictions.
- Active scene description lets the LLM choose whether to continue planning or ask a human an unconstrained question after each planning step.
B.1 Simulated Tabletop Rearrangement Environment
The simulated tabletop environment evaluates long-horizon rearrangement and generalization using multiple block-and-bowl tasks, with a multi-task CLIPort policy as an additional baseline.
- The environment contains 8 tasks involving up to 4 blocks and 3 bowls, randomized across colors, locations, and task instructions.Tasks use 10 possible colors, enforce 15cm initial separation, and sample instructions involving available objects or locations.
- A multi-task CLIPort policy is trained on 4 of the 8 tasks to test generalization to new instructions and long-horizon tasks.The baseline uses 20,000 demonstrations across the four training tasks and differs from Inner Monologue’s low-level primitive.
B.2 Real Tabletop Rearrangement
The real tabletop evaluation contains block stacking and object sorting tasks in which occlusions and clutter make scene-description feedback important for completing the instructions.
- The real tabletop domain includes a block stacking task and an object sorting task.
- In block stacking, the robot completes a three-cube tower despite initially perceiving only two blocks because of occlusions.Without scene-description feedback, the planner has a 50% chance of completing the task.
- In object sorting, the robot separates three fruits and three bottles from clutter onto two remaining plates.Occlusions and reduced detection performance can hide objects initially, so scene-description feedback remains necessary.
B.3 Real Kitchen Mobile Manipulation
The real kitchen evaluation uses a mobile manipulator to execute unstructured instructions over named locations containing household objects in a mock office kitchen.
- The kitchen environment places an RGB-observing mobile manipulator among 15 household items distributed across multiple named locations.
- The robot must complete unstructured human instructions by sequencing vision-based navigation and manipulation skills.
- The mock office kitchen setup includes 5 predicted locations and 15 objects, with a 7-degree-of-freedom arm and gripper policies trained from RGB observations.
- Kitchen mobile-manipulation evaluation covers 3 families of instructions on the mock kitchen scene.
C Additional Results
Additional examples illustrate Inner Monologue’s feedback-driven execution traces across simulated and real tabletop tasks and kitchen manipulation, alongside robustness to feedback ordering and instruction typos.
- Additional emergent capabilities: Figure 9 reports that Inner Monologue can handle different feedback injection orders and remain robust to typos in instructions.The reported flexibility extends beyond the fixed feedback order typically used in each domain.
- Simulated tabletop prompts: The simulated tabletop prompt tracks a goal state, completed subgoals, and successive actions until all blocks reach the requested location.The trace shows the robot updating its thought after each scene-feedback message and stopping when the goal state is complete.
- Real tabletop prompts: The real tabletop traces show successful retries after failed pick-and-place actions, changing visibility through occlusion, and explicit STOP actions.
- Kitchen prompts: Kitchen prompt examples show multi-step plans with failed and successful pickups, navigation, placement, and human-directed object rearrangement.