Inner Monologue: Embodied Reasoning through Planning with Language Models

Introduction

• Embodied interaction requires robots to sequence broad repertoires of behaviors over long tasks and switch approaches when plans fail, while integrating high-level planning, perceptual feedback, low-level control, and semantic world knowledge.Prior approaches have emphasized planning such as TAMP or hierarchical learning such as HRL.Source: §1 · p. 1
• LLMs' world knowledge and prompted reasoning motivate investigating whether they can combine feedback sources as interactive problem-solvers for embodied tasks [3] [7] [8] [10] [11] [12] [13].The motivating question extends their use beyond interpreting natural-language instructions to reasoning in robotic manipulation.Source: §1 · p. 1
• Inspired by human inner monologue, the framework combines immediate action choices, observations of outcomes, and corrective actions in response to feedback [18].The example of trying keys illustrates how an observed failure can prompt a different action.Source: §1 · p. 1
• The work investigates the link between language-based planning and language-informed embodied feedback, which prior work had not studied, using few-shot prompting without additional training.Natural language serves as a universal, interpretable interface for grounding communication and incorporating conclusions into an overarching inner monologue.Source: §1 · pp. 1–2
• The system chains perception models, pretrained manipulation skills, and human feedback in a shared language prompt to execute user instructions.Perception includes language-conditioned semantic classification and scene description; skills are invoked by the language model from a set of pretrained actions with textual descriptions.Source: §1 · p. 2
• Without additional training, Inner Monologue handles long-horizon and unseen tasks across simulation and two real-world platforms, retrying failures, replanning infeasible actions, and requesting feedback on ambiguous queries.The formulation also supports continued adaptation to new instructions, self-proposed goals, interactive scene understanding, and multilingual interactions.Source: §1 · p. 2

Task and Motion Planning

• Task and motion planning jointly solves discrete task planning and continuous motion planning, using optimization, symbolic reasoning, learned methods, language, or hierarchical learning.This work combines pre-trained LLM semantic knowledge with trained low-level skills to find feasible plans.Source: §2 · p. 2

Task Planning with Language Models

• Prior LLM planners decompose high-level instructions into executable steps zero-shot [20] [21], but assume successful execution and may not handle intermediate failures or weak low-level policies robustly.Inner Monologue explores incorporating grounded environment feedback into the LLM as each plan step is produced.Source: §2 · pp. 2–3

Fusing Vision, Language, and Control in Robotics

• Robotics work has fused vision, language, and control using visual-language models such as CLIP [65], CLIPort [76], and Socratic Models [19].Socratic Models combine foundation models and language-conditioned policies through language as a common interface, demonstrating simulated vision-based manipulation.Source: §2 · p. 3

3.1 Problem Statement

• A robot executes short-horizon skills from a pretrained policy library, while a pretrained LLM planner [20] [21] selects a skill sequence for the high-level instruction using textual environment feedback.Feedback can include success detection, object detection, scene descriptions, visual question answering, or human input.Source: §3.1 · p. 3

3.2 Inner Monologue

• Inner Monologue continually injects environment feedback into LLM prompts so planners can correct plans and replan in closed loop, even before affordance-based grounding methods are applied.Earlier embodied LLM planning work used one-way skill lists; Inner Monologue lets the planner incorporate grounded feedback during interaction.Source: §3.2 · p. 3
• The study treats feedback incorporation as a case study, not a fixed algorithm: implementations vary across LLMs, control fusion, environments, tasks, and policy acquisition, using pretrained LLMs with few-shot prompting rather than fine-tuning.It does not assume a specific method for fusing planning with low-level control or translating environment feedback into language.Source: §3.2 · p. 3

3.3 Sources of Feedback

• Success feedback is a language-form binary judgment of whether a low-level skill succeeded, produced by engineered simulation detectors or learned real-world detectors [80] [81].The paper groups this task-specific feedback separately from scene descriptions.Source: §3.3 · pp. 3–4, Figure 2 · p. 4
• Passive Scene Description supplies structured scene feedback automatically at each planning step, including object-recognition outputs and task-progress descriptions.The paper calls object-recognizer outputs Object feedback and simulated tabletop task-progress descriptions Scene feedback.Source: §3.3 · p. 4, Figure 2 · p. 4
• Active Scene Description lets the planner query for unstructured scene, task, or user-preference information; this work uses human responses, including responses to structured VQA-style questions and open-ended preference queries.The planner's question and the response are both included in the prompt; answers may also come from another pretrained model, though this work considers human-provided responses.Source: §3.3 · p. 4, Figure 2 · p. 4

4.1 Simulated Tabletop Rearrangement

• Object + Scene performs best by tracking goal conditions and achievements; all Inner Monologue variants transfer to unseen tasks, while CLIPort struggles with disturbed long-horizon generalization and often needs oracle termination.CLIPort’s oracle indicates task completion; its alternative evaluation uses fixed-step termination.Source: Table 1 · p. 6, §4.1 · p. 6

4.2 Real-World Tabletop Rearrangement

• Closed-loop Object + Success feedback enables replanning and recovery from pick-and-place mistakes; the components are complementary, addressing failures from initially occluded objects and unsuccessful actions that otherwise are not retried.The experiments inject policy-action noise and also face noisy object and success detections from real-world perception and clutter.Source: §4.2 · pp. 6–7, Table 2 · p. 6

4.3 Real-World Mobile Manipulator in a Kitchen Setting

• Under adversarial disturbances, SayCan’s success rate is close to 0%, while Inner Monologue significantly outperforms it through feedback-dependent recovery; Success and Object feedback reduce planning failures but introduce new failure modes.The system can retry or replan after natural failures as well as forced policy failures.Source: Table 3 · p. 7, §4.3 · pp. 7–8

4.4 Emergent Capabilities

• Without similar prompted examples, Inner Monologue adapts to changing human instructions, proposes alternative goals when a task becomes infeasible, and interprets new instructions in Chinese.It can switch tasks after a changed instruction, respond to “please stop,” and solve the block task by proposing a lighter object after the purple block proves too heavy.Source: §4.4 · p. 8
• It can answer scene questions using past actions and feedback, handle occasional changes in feedback order and typos, but these emergent behaviors vary in consistency without similar examples in the prompt.Its multilingual understanding can also extend to symbols and emojis.Source: §4.4 · p. 9

Limiting assumptions and failure modes

• The method assumes oracle textual scene descriptions and can fail through detection, planning, or control errors, including ignored feedback that yields skills for absent objects.False negatives from the success detector trigger extra retries, while false positives add adversarial partial observability.Source: §5 · p. 9

Limitations of results and future work

• Low-level control policies limit both overall instruction completion and the tasks the LLM can reason over, regardless of improvements to its reasoning.Source: §5 · p. 9
• Future work could automate Inner Monologue without a human oracle and better aggregate inaccurate feedback, including uncertainty descriptions and safety or ethics modules.The proposed automation could build on advances in image/video captioning and visual-question answering.Source: §5 · p. 9

• Environment feedback significantly improves high-level instruction completion, especially in challenging scenarios with adversarial disturbances.Source: §6 · p. 9
• Inner Monologue combines different sources of environment feedback with methods that fuse LLM planning and robotic control policies.Source: §6 · p. 9
• The methods were studied across three robotic manipulation settings in simulation and the real world, with emergent capabilities showing replanning in complex unseen settings.Source: §6 · p. 9

A Inner Monologue Implementation Details

• The three Inner Monologue implementations incorporate textual environment feedback into planning but differ in their internal components across experimental settings.Table 4 compares versions implemented in three environments.Source: A Inner Monologue Implementation Details · p. 15, Table 4 · p. 15

A.1 Inner Monologue for Simulated Tabletop Rearrangement

• In simulation, InstructGPT plans pick-and-place actions executed by a CLIP-based Transporter Net trained on 20,000 demonstrations covering all objects.Each demonstration pairs a language instruction and RGB-D observation with expert pick and place coordinates obtained from simulator object poses.Source: §A.1 · p. 15
• At each simulated episode’s start, the planner receives the scene’s object list without coordinates, which the system obtains directly from simulator state.Spatial manipulation is handled by the low-level policy.Source: §A.1 · p. 15
• Simulation feedback reports success when the picked object is within 4 cm of and above its target, except that the height condition is waived for location targets.Scene feedback lists satisfied desired subgoals after each action, and the planner can add a chain-of-thought summary.Source: §A.1 · p. 15

A.2 Inner Monologue for Real-World Tabletop Rearrangement

• The real tabletop policy maps object detections, target names, depth, camera intrinsics, and camera pose to 3D pick and place positions in the robot frame.Pick coordinates are computed by deprojecting the detected bounding-box center and transforming it from camera frame using the wrist camera pose.Source: §A.2 · p. 16
• For real-world sorting, the place point is chosen farthest from other objects in the target plate, and the robot lowers from 15 cm above until detecting 5N contact.A height guard helps prevent accidentally picking up the plate. Pick-position translation noise has standard deviation 1.5 cm for stacking and 0.7 cm for sorting, capped at 1.5 standard deviations.Source: §A.2 · p. 16
• Real success detection compares robot-frame 2D object and intended-place centers, using a 3 cm threshold for stacking and 10 cm for sorting.MDETR detections can be inconsistent, especially in clutter; the thresholds account for bounding-box offsets and objects rolling within plates.Source: §A.2 · p. 16
• Real tabletop scene descriptions track visible and newly revealed objects; sorting feedback also lists objects successfully moved into a plate to guide stopping at task completion.For stacking, descriptions additionally track previously visible objects that are no longer visible.Source: §A.2 · p. 16

Large Language Model

• The kitchen system uses PALM, a 540B-parameter language model trained on web documents, books, Wikipedia, conversations, and GitHub code.The implementation identifies PALM [8] as its language model.Source: §A.3 · p. 17

Low-level Policies

• Low-level control combines learned manipulation policies with scripted navigation and countertop placing, while SayCan-style affordance values come from trained RL Q-networks.The behavior-cloning policies use 68,000 teleoperated demonstrations and 12,000 autonomous successes collected over 11 months with 10 robots; navigation uses a ground-truth map and learned perception, and placing follows precomputed motions.Source: §A.3 · p. 17

Environment Feedback: Object Recognition

• 28.0% of the time, ViLD [77] failed to detect objects across 10 representative kitchen episodes; ViLD and MDETR were less accurate than humans but still reasonably effective.The study evaluated pretrained models without domain-specific fine-tuning. Human-provided object recognition reports objects visible to the robot camera.Source: Table 5 · p. 17, §A.3 · p. 17

Environment Feedback: Success Detection

• The foresight success detector estimates whether a specified skill succeeded from initial and final images plus the skill instruction, returning “[success: no]” below a threshold.It combines CLIP image and text embeddings, is fine-tuned with binary cross-entropy, and is paired with a hindsight predictor to reduce false positives by checking whether the inferred achieved skill matches the requested one.Source: §A.3 · pp. 17–18

Environment Feedback: Active Scene Description

• After each planning step, the agent can continue or ask a human an open-ended question, whose answer may confirm scene facts or refine the high-level plan.The questions have no predetermined format or scope.Source: §A.3 · p. 18

B.1 Simulated Tabletop Rearrangement Environment

• Across eight tasks, scenes randomly initialize up to four blocks and three bowls, with varied colors and locations and at least 15 cm between objects.Instructions specifying an object or location are sampled from available scene objects or nine locations; each task has a binary reward based on ground-truth object states.Source: §B.1 · p. 18
• A multi-task CLIPort baseline [76] is trained on 20,000 demonstrations from four tasks to assess long-horizon performance and generalization to new instructions.It shares the Inner Monologue low-level policy’s architecture and training details, but is a different policy; Table 1 gives the train/test split.Source: §B.1 · p. 18

B.2 Real Tabletop Rearrangement

• The domain contains two tasks: stacking a block tower and sorting fruits and bottles onto separate plates.In sorting, the objects begin cluttered on one of three plates; the other two are the destination plates.Source: §B.2 · pp. 18–19
• In block stacking, occlusion hides one of three cubes, leaving the planner a 50% chance of success without scene-description feedback.Two cubes are initially stacked, and the robot can complete the tower either by placing the third cube atop them or placing the pair atop the third.Source: §B.2 · p. 18
• In sorting, clutter and reduced object detection can hide some of the three fruits and three bottles, so scene-description feedback is needed.The objects are realistic-looking, lightweight rigid plastic toys that support robust suction grasps and MDETR detections.Source: §B.2 · p. 19

B.3 Real Kitchen Mobile Manipulation

• The robot must complete unstructured human instructions by executing vision-based, concurrent, continuous closed-loop navigation and manipulation policies.The setup follows SayCan [21].Source: §B.3 · p. 19
• The mock kitchen setup has 5 predicted locations and 15 objects, and uses a mobile manipulator with a 7-degree-of-freedom arm and gripper.The robot's policies are trained from RGB observations.Source: Figure 8 · p. 19, §B.3 · p. 19
• The evaluation covers 3 families of instructions in the mock kitchen scene.Table 6 lists the instructions used in the Kitchen Mobile Manipulation tasks.Source: Table 6 · p. 19

C Additional Results

• Inner Monologue handles feedback inserted at different prompt positions and remains robust to instruction typos, which the authors suggest may be inherited from the backbone LLM.The figure contrasts varying feedback placement with the usual provision of feedback after the prompt and planning steps.Source: Figure 9 · p. 20

D Prompts

• The simulated tabletop prompt interleaves a goal-state thought, block-placement actions, and scene feedback about completed goals to guide subsequent actions.The examples show the robot updating its thoughts as blocks are moved toward the requested location.Source: D Prompts · p. 21
• The real tabletop prompt supplies visible and occluded objects alongside action-success signals, including failed and successful placements before the robot stops.In the example, a failed yellow-block placement is followed by a successful retry, and later observations report occluded blocks.Source: D Prompts · pp. 21–22
• The kitchen prompt examples interleave planned actions with success and scene feedback, including retries and adjustments when observations reveal different available objects.Examples include retrying a pickup after failure and choosing an observed snack instead of the initially requested item.Source: D Prompts · pp. 23–24
• The human-interaction kitchen prompt lets the robot ask clarifying or confirmation questions and incorporate changed preferences or task instructions while acting.Examples include asking which drink the person prefers, confirming an item choice, and responding to a request to keep a drink rather than discard it.Source: D Prompts · pp. 24–25