Robotic Control via Embodied Chain-of-Thought Reasoning

• Reactive policies may struggle to generalize to novel scenes or unfamiliar objects, while prompted CoT and semantic sub-task plans alone do not ensure reasoning grounded in observations and robot state.Source: §1 · pp. 1–2
• ECoT interleaves semantic sub-task plans with grounded object-box predictions and robot-state-dependent movement reasoning before action prediction, using synthetic supervision generated with detectors and language models.Source: §1 · p. 2, Figure 1 · p. 2
• 28%: ECoT increases OpenVLA’s absolute success rate across tasks involving new objects, scenes, viewpoints, and instructions, without additional robot training data [7].Source: §1 · p. 2
• Embodied CoT makes policy failures more interpretable and lets humans correct behavior by modifying faulty reasoning chains with natural-language feedback.Source: §1 · p. 3

Scaling robot learning

• Training generalist robot policies [13] on diverse robot datasets [22] [24] [13] [26] is presented as a path to robustness and generalization.Many such policies accept natural-language prompts for varied manipulation tasks, and some control multiple robot embodiments.Source: §2 · p. 3

Vision-language models for robot generalization

• VLAs [7] offer a scalable alternative for integrating pretrained vision-language models into robot policies, with state-of-the-art generalist performance [7] and generalization to new objects and scenes.The related work notes that existing VLAs do not sufficiently leverage the underlying models’ ability to reason through the steps required for a task.Source: §2 · p. 3

Reasoning for language and control

• The paper trains a VLA to generate high- and low-level chain-of-thought reasoning and actions from instructions and observations, grounding both forms of reasoning in the agent’s environment.The authors argue that reasoning should support low-level control as well as high-level sub-task planning, and report that this grounded formulation is critical to leveraging (V)LM reasoning for control.Source: §2 · p. 3
• Prior robotics approaches use pretrained or fine-tuned LLMs to decompose tasks into high-level sub-tasks, relying on pretrained low-level policies for execution.The authors contrast this with their position that reasoning can also benefit low-level control and should be grounded in visual observations and agent state.Source: §2 · p. 3
• Step-by-step prompting improves LLM performance on complex tasks, and chain-of-thought has become standard in language and vision-language model training.Related work also applies LLM reasoning to high-level robotic task planning.Source: §2 · p. 3

3 Preliminaries: Vision-Language-Action Models

• VLAs autoregressively predict the next robot action from the current image observation and task instruction, converting continuous actions into discrete tokens.Each action dimension is discretized into 256 bins [7].Source: §3 · pp. 3–4
• ECoT interleaves linguistic subtask reasoning with steps grounded in the scene and robot state, which the experiments identify as key to improving policy performance with chain-of-thought reasoning.The grounded steps connect reasoning to visual features and the robot state before action prediction.Source: Figure 3 · p. 4
• OpenVLA combines a fused SigLIP and/or DinoV2 visual encoder with a Llama 2 7B backbone to map image and task tokens to action tokens [7].It builds on the Prismatic VLM and is fully open-source.Source: §3 · p. 4

4 Embodied Chain-of-Thought Reasoning for Visuomotor Policies

• Instead of relying on prompting small VLA backbones to reason, ECoT post-hoc labels robot data with reasoning strings and trains models to predict reasoning tokens before action tokens.The labels use features extracted from pre-trained models, producing observation-reasoning-action tuples for autoregressive training.Source: §4 · p. 4

4.1 Designing Embodied Chain-of-Thought Reasoning Steps

• ECoT progresses from rephrasing the TASK and planning steps to choosing a SUBTASK, predicting a MOVE command, and grounding reasoning in GRIPPER position and OBJECT bounding boxes.The authors note that alternative reasoning tasks and orderings remain unexplored, and automatic selection of sensible chains is future work.Source: §4.1 · pp. 4–5

4.2 Generating Embodied Chain-of-Thought Data at Scale

• The synthetic-data pipeline generated ECoT annotations for Bridge v2’s more than 2.5M transitions over 7 days.It combines Prismatic-7B scene descriptions, Grounding DINO object boxes, proprioception-derived movement primitives, OWLv2/SAM gripper positions, and Gemini-generated plans and sub-tasks. Object detections require box confidence above 0.3 and text confidence above 0.2; movement directions over the next four time steps map to 729 templates, and gripper projections are fitted per trajectory with RANSAC.Source: Figure 4 · p. 5, §4.2 · pp. 5–6

4.3 Efficient Chain-of-Thought Inference for Robot Policies

• Reasoning increases tokens per timestep from 7 for OpenVLA to 350 for ECoT; synchronous and asynchronous strategies reuse high-level reasoning across steps to improve inference speed.Synchronous execution updates high-level reasoning every N steps; asynchronous execution uses one policy instance to update reasoning and another to predict lower-level reasoning and actions.Source: §4.3 · p. 6

5.1 Experimental Setup

• Experiments used WidowX with Bridge V2’s 60k demonstrations and 314 trials per approach across spatial-relation, unseen-object, and unseen-instruction tasks.Policies received a single third-person camera view and natural-language instruction, and predicted end-effector velocity actions.Source: §5.1 · p. 6

5.2 Embodied Chain-of-Thought Reasoning Improves Policy Generalization

• ECoT’s aggregate absolute success rate exceeded Octo, OpenVLA (Bridge), RT-2-X, and naïve CoT by 45%, 22%, 19%, and 18% in-distribution, and 48%, 34%, 16%, and 16% out-of-distribution.ECoT also outperformed OpenVLA (Bridge) on all but one generalization evaluation and surpassed RT-2-X despite its 10 additional robot datasets and 7x larger network.Source: Table 1 · p. 7, §5.2 · p. 7
• ECoT and OpenVLA (Bridge) shared the same VLM base and robot fine-tuning data; ECoT’s reasoning traces ground sub-tasks in object boxes and gripper position before action.The reported distinction between the policies was ECoT’s use of chain-of-thought reasoning.Source: §5.2 · p. 7

5.3 Diagnosing Policy Failures Through Inspecting Reasoning Chains

• Inspecting ECoT’s reasoning exposed a failure in which it mistook a hammer for a screwdriver and reached for the wrong object.Reasoning chains are not a foolproof account of an end-to-end policy’s actions, though they often correlate strongly with execution.Source: §5.3 · p. 7

5.4 Chain-of-Thought Reasoning Enables Interactive Policy Correction

• ECoT’s success rate increased by 48% on the most challenging tasks after a human provided one natural-language intervention during a rollout.ChatGPT adapted the reasoning chain to the feedback, which was held fixed for five steps; without intervention, ECoT averaged 32% success on those tasks. OpenVLA and RT-2-X did not benefit to the same degree from the correction.Source: §5.4 · p. 8

5.5 Efficient Chain-of-Thought Inference

• Both proposed inference accelerations improved speed while at least matching performance; asynchronous execution was fastest but doubled inference compute.The five-step freeze approach was used for Table 1 because it offered the best performance-speed tradeoff.Source: §5.5 · p. 9, Table 2 · p. 9

5.6 Additional Analysis

• Moving bounding-box prediction earlier in the reasoning chain sped up inference by 30–50%, although the frozen-box policy performed below base ECoT while outperforming the baselines.The authors adopted this faster structure for subsequent experiments because it also supported rollout visualization.Source: §5.6 · p. 9
• Co-training retained conversational question-answering ability but yielded no measurable evaluation-task gains; fine-tuning a pretrained VLA with ECoT data nearly matched the original ECoT model in 20k rather than 80k steps.Fine-tuning showed comparable performance after 2,500 steps, a reported 30x compute reduction. The model also generated ECoT reasoning for other robot embodiments without embodiment-specific reasoning annotations, but in SIMPLER’s real-to-sim setting its reasoning was faulty and did not improve over OpenVLA.Source: §5.6 · pp. 9–10

6 Discussion and Limitations

• 69% aggregate success was achieved by the base ECoT policy; all Table 3 policies exceeded OpenVLA (Bridge) (29%), RT-2-X (46%), and Octo (14%) on the same trial subset.The table evaluates a large subset of trials in the harder out-of-distribution view setting; other design choices achieved higher performance on some tasks.Source: Table 3 · p. 10
• ECoT improves VLA performance without additional robot training data by grounding reasoning in scene and robot state, including object boxes, the end-effector, and low-level movements.The discussion contrasts these grounded reasoning steps with simply applying the language-modeling CoT recipe.Source: §6 · p. 10
• ECoT uses the same fixed reasoning-step order for every task; future work could select a subset based on robot and scene state and optimize the model to choose steps.The paper identifies the lack of task-adaptive reasoning-chain structure as a limitation.Source: §6 · p. 10
• Scaling ECoT training to a larger OXE dataset subset could improve transfer to more robots, while its still-limiting runtime constrains higher-frequency control.Runtime optimizations in Section 4.3 improve achievable control frequencies; exploring other approaches for improving LLM throughput may enable higher-frequency CoT control.Source: §6 · pp. 10–11

A Grounding DINO Detections and Prismatic Descriptions

• Figure 8 shows Prismatic VLM captions of Bridge observations alongside associated bounding boxes generated by Grounding DINO.The examples illustrate scene descriptions and detections together.Source: A Grounding DINO Detections and… · p. 18, Figure 8 · p. 18
• Figure 9 provides qualitative examples of successful and failed chain-of-thought reasoning.Source: Figure 9 · p. 19
• Figure 10 shows example starting scenes and associated prompts for all task types.Source: Figure 10 · p. 20

B List of Movement Primitives

• Movement labels are assigned from state differences four steps ahead, using a 0.03 threshold on each axis.Axes below threshold are omitted; when no movement is detected, the label is stop.Source: B List of Movement Primitives · p. 21
• The primitive vocabulary combines directional movement, tilt, clockwise or counterclockwise rotation, and opening or closing the gripper.Movement directions cover forward/backward, left/right, and up/down.Source: B List of Movement Primitives · p. 21

C Prompts

• Scene-description generation uses a prompt about scene contents and spatial relations, optionally prefixed with the task instruction; the Gemini labeling prompt is shown in Figure 11.The task instruction is included only when the demonstration has a corresponding instruction containing at least one space character.Source: C Prompts · p. 22, Figure 11 · p. 23
• OpenVLA policy prompts are inserted into the original authors’ template [7], after which the agent generates reasoning text when trained to do so and an action.The prompts and example starting scenes for the language-conditioned policies are provided in Figure 10.Source: C Prompts · p. 22
• The prompt used for ChatGPT during human-intervention experiments is provided in Figure 12.Source: C Prompts · p. 22, Figure 12 · p. 24
• The annotation prompt asks for a task overview, remaining high-level movements and their justifications, then precise step-by-step reasoning based on progress, relevant objects, and the executed trajectory.It also asks annotators to include relevant object locations, obstacles, movement directions, and why each action is appropriate.Source: C Prompts · p. 23
• Each step’s reasoning is specified with <task>, <plan>, <subtask>, <subtask_reason>, <move>, and <move_reason> tags.The prompt requires one separate reasoning string for each trajectory step, with the same integer keys as the trajectory features.Source: C Prompts · p. 23
• The response checklist calls for a task description, executed high-level movements, a solution plan, one precise reasoning string per trajectory step, and a final FINISHED label.The example annotation context specifies a successful cloth-unfolding task and describes the robot’s environment and primitive movements.Source: C Prompts · p. 23