PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Strategies for Efficient Embodied Reasoning

William Chen, Suneel Belkhale, Suvir Mirchandani, Oier Mees, Danny Driess, Karl Pertsch, Sergey Levine

arXiv:2505.08243v2cs.RO

TL;DR

Embodied chain-of-thought improves robot policy generalization, but requires specialized reasoning data and slows inference. The paper isolates representation learning, curricularization, and expressivity as possible mechanisms, then introduces lightweight recipes that retain performance while reducing latency. Its results show improved VLA representations, state-of-the-art LIBERO performance, and approximately 3x faster inference than standard robot reasoning.

  • Problem

    Embodied reasoning improves VLA generalization but requires specialized robot reasoning data and can slow inference substantially.

  • Method

    The paper isolates representation learning, learning curriculum, and effective expressivity through lightweight embodied reasoning recipes, including reasoning pre- or co-training and test-time reasoning dropout.

  • Results

    The proposed approaches retain improved generalization, achieve state-of-the-art performance on LIBERO, and reach 3.5+ Hz versus 1-1.2 Hz for conventional embodied chain-of-thought.

  • Takeaways & Limitations

    Learning to generate reasonings improves VLA representations, while using reasoning during action prediction helps leverage those features for performance.

  • Takeaways & Limitations

    The proposed approaches still require robot reasoning training data, which can be difficult or expensive to extract.

Abstract

from arXiv · show

Robot chain-of-thought reasoning (CoT) -- wherein a model predicts helpful intermediate representations before choosing actions -- provides an effective method for improving the generalization and performance of robot policies, especially vision-language-action models (VLAs). While such approaches have been shown to improve performance and generalization, they suffer from core limitations, like needing specialized robot reasoning data and slow inference speeds. To design new robot reasoning approaches that address these issues, a more complete characterization of why reasoning helps policy performance is critical. We hypothesize several mechanisms by which robot reasoning improves policies -- (1) better representation learning, (2) improved learning curricularization, and (3) increased expressivity -- then devise simple variants of robot CoT reasoning to isolate and test each one. We find that learning to generate reasonings does lead to better VLA representations, while attending to the reasonings aids in actually leveraging these features for improved action prediction. Our results provide us with a better understanding of why CoT reasoning helps VLAs, which we use to introduce two simple and lightweight alternative recipes for robot reasoning. Our proposed approaches achieve significant performance gains over non-reasoning policies, state-of-the-art results on the LIBERO-90 benchmark, and a 3x inference speedup compared to standard robot reasoning.

1 Introduction

The paper examines embodied chain-of-thought reasoning as an alternative way to improve VLA generalization, while addressing its data and inference costs. It analyzes why reasoning helps and introduces ECoT-Lite recipes that preserve benefits with greater practicality.

  • VLAs combine large transformer architectures with semantic knowledge from vision-language-model pre-training to support generalizable robot policies.
  • Embodied chain-of-thought reasoning splits action prediction into intermediate reasoning steps, offering an alternative to collecting increasingly large robot datasets.
  • Detailed reasoning annotations and extended inference can make a single action prediction take multiple seconds.
  • The paper develops lightweight ECoT-Lite recipes that isolate hypotheses about reasoning while retaining most regular chain-of-thought generalization benefits.
  • ECoT-Lite achieves state-of-the-art LIBERO performance, exceeds conventional VLA performance on BridgeData V2 by 10-19%, and reaches 3.5+ Hz versus 1-1.2 Hz for conventional reasoning.

2 Related Work

Prior work shows that robot reasoning can improve policy generalization, but has not established why. This paper positions its contribution as a dissection of representation-learning and test-time-compute effects in embodied robotics.

  • VLAs adapt vision-language models to output robot actions, using internet-scale vision and language pre-training to improve robustness and generalization.
  • Chain-of-thought methods generate intermediate steps before answers, with prior analyses linking longer reasoning to greater transformer expressivity.
  • Robot reasoning predicts subtasks, grounded motion features, object boxes, or semantic keypoints before final actions.
  • Prior robotics studies report improved generalization from intermediate reasonings but do not dissect whether CoT or its additional training signal is responsible.
  • This work investigates both representation learning and test-time compute to explain why training on and using robot CoT benefits embodied policies.

3 Preliminaries

The paper formulates VLAs as policies that generate tokenized robot actions from language and observations, then extends this formulation with embodied reasoning before action prediction.

  • A VLA models a policy that samples actions conditioned on a task’s language instruction and robot observation.
  • VLAs represent robot actions as text tokens so a vision-language model can be fine-tuned to output them autoregressively.
  • ECoT first generates reasoning text, including high-level subtasks and grounded motions, gripper positions, or object bounding boxes, before action tokens.

4 Why Does Embodied Chain-of-Thought Reasoning Improve Performance?

The paper proposes three explanations for embodied reasoning gains: better representations, a learning curriculum, and greater effective expressivity. These hypotheses guide lightweight recipes and experiments that separate the mechanisms.

  • The paper proposes practical reasoning recipes by formulating hypotheses about why embodied reasoning improves policy performance.
  • Hypothesis 1: Representation learning: Reasoning may improve representations by signaling which objects or features are relevant to the policy.
  • Hypothesis 2: Learning curriculum: Reasoning may provide a learning curriculum by letting policies learn simpler reasoning-to-action mappings before the full observation-to-action task.
  • Hypothesis 3: Effective model expressivity: Reasoning may increase effective expressivity because longer token sequences provide more compute during training and inference.

5 ECoT-Lite: Practical Training Recipes for Embodied Reasoning Policies

ECoT-Lite introduces lightweight training recipes that isolate representation learning, scaffolding, and expressivity as possible mechanisms behind embodied reasoning benefits. These recipes aim to preserve generalization while reducing annotation or inference costs.

  • Design goal: The recipes are designed to retain ECoT’s generalization while mitigating extensive annotation requirements and high-latency inference.Figure 3 summarizes the standard, pre-training, scaffolding, dropout, and thinking-token variants.
  • Representation learning: Reasoning pre-training and co-training use embodied reasoning data to shape VLA representations, either before action training or alongside it.Pre-training separates reasoning and action objectives; co-training alternates reasoning-only and action-only examples.
  • Reasoning dropout: Reasoning dropout sequences reasoning and action tokens during training but randomly removes reasoning, enabling direct observation-to-action inference.The approach explicitly trains the policy to use reasoning information while retaining inference without generated reasoning.
  • Reasoning scaffolding: Reasoning scaffolding supplies reasoning examples in context during training without applying loss to them, optionally dropping the scaffold.This tests whether reasoning can support learning without requiring the policy to predict it.
  • Thinking tokens: Thinking tokens add empty tokens during training and inference to increase effective model capacity without semantic reasoning.Their sequence length is varied to provide additional computational resources to the VLA.

6 Experiments

Experiments compare ECoT-Lite variants with standard VLAs and embodied CoT across simulated LIBERO and real-robot Bridge tasks. Results support representation learning and reasoning use as important mechanisms, while non-semantic thinking tokens hurt performance.

  • Experimental setup: The evaluation uses LIBERO and Bridge tasks designed to test generalization beyond training data, including randomized objects, spatial relations, and unseen objects.The study uses publicly available training datasets and evaluates multiple task distributions in both environments.
  • Experimental setup: 121,500 simulated and 444 real-robot trials compare ECoT-Lite, standard VLAs, and prior embodied CoT under matched data, compute, and evaluation conditions.The comparisons use the same robot demonstration and reasoning data and comparable initial states, lighting, and camera angles.
  • Performance: ECoT and reasoning dropout reach about 90% on LIBERO-90, exceeding the prior 88.6% state of the art, while reasoning pre-training improves over the standard VLA by 5.4%.Reasoning dropout achieves this without generating test-time reasonings, making it faster than full ECoT.
  • Performance: Both reasoning dropout and pre-training improve significantly over the standard VLA on Bridge, while the proposed approaches are around 3× faster than ECoT.Unlike on LIBERO, reasoning pre-training is more effective than reasoning dropout on Bridge.
  • Representation learning: All representation-learning policies outperform the non-reasoning VLA, with pre-training improving 5.4% and co-training improving 1.9%.The results support learning from reasoning data as useful for action prediction, while suggesting co-training is less effective than pre-training.
  • Test-time reasoning: Test-time reasoning is unnecessary for best LIBERO-90 performance but improves Bridge performance, where diverse tasks expose object-selection and obstacle-collision failures.The authors attribute this contrast to LIBERO’s narrower task distribution and less varied reasoning features.
  • Curriculum and expressivity: Reasoning scaffolding improves performance by 2.9% over baseline, while adding thinking tokens decreases average performance by 3.8%.The scaffolding result weakly supports in-context reasoning as a curriculum aid; the thinking-token result does not support expressivity as the main benefit.

7 Which Robot Reasoning Approach is Best for My Problem?

The preferred reasoning strategy depends on the tradeoff between peak performance, task diversity, inference speed, and data availability. Full ECoT maximizes performance, while reasoning dropout and pre-training target narrower and more diverse settings respectively.

  • Performance and speed: Full ECoT is the most performant approach but runs at about 1–1.2 Hz, compared with 3–4 Hz for same-architecture VLAs.Both ECoT-Lite variants map observations directly to actions and therefore retain faster inference.
  • Strategy selection: Reasoning dropout matches full ECoT across LIBERO variants and suits narrow task domains or applications needing optional test-time reasoning.It has ECoT’s resource demands because the two methods share training aside from reasoning dropout.
  • Strategy selection: Reasoning pre-training outperforms dropout on Bridge in all but one split, but requires more gradient steps and does not require paired reasoning-action data.Its sequential objectives also reduce the memory required per training datapoint.
  • Practical prescription: Use full ECoT to maximize performance, reasoning dropout for narrower domains or optional test-time reasoning, and pre-training for diverse domains or unpaired reasoning data.The recommendation accepts slower inference for full ECoT and additional training for pre-training.

8 Discussion

The study isolates three hypotheses for why embodied chain-of-thought improves robot policies and identifies faster alternatives that preserve performance while avoiding slow inference. It validates reasoning pre-training and test-time reasoning dropout in real-world manipulation and gives prescriptions for when to use each approach.

  • The analysis tests whether robot reasoning improves representation learning, provides learning curricula, or increases policy expressivity.
  • Reasoning pre-training and test-time reasoning dropout maintain robot-reasoning performance improvements while avoiding slower inference.
  • The proposed approaches are validated in real-world manipulation experiments and accompanied by prescriptions for when each is appropriate.
  • The approach can in principle learn from arbitrary embodied robot reasoning data, including data from other embodiments, but this remains future work.

9 Limitations

The authors identify unresolved limitations involving data requirements, incomplete understanding of learning dynamics, and controlled design choices. Their reported evaluation evidence includes numerical results for simulated LIBERO and real-world Bridge tasks.

  • Users still need robot reasoning training data, which can be difficult or expensive to extract.
  • The study does not investigate reasoning’s impact on actual model learning dynamics, leaving representational transfer and grounding mechanisms for future work.
  • Many design choices, including policy architecture, training hyperparameters, and reasoning corpora, are held constant for controlled comparisons, leaving room for further optimization.
  • Numerical performance values are reported for simulated LIBERO and real-world Bridge evaluations in Tables 1 and 2.

A.1 Further Discussion: Pre-training vs. Co-training

The paper compares reasoning pre-training with co-training and links reasoning benefits to how action prediction uses learned reasoning representations. It also reports that reasoning helps selectively across Bridge generalization settings, while LIBERO’s narrow task distribution may permit internalization of reasoning features.

  • Pre-training vs. Co-training: Reasoning pre-training and co-training optimize reasoning and action objectives differently, motivating comparison of their effects on policy learning.The paper visualizes co-training as jointly optimizing mixed losses, whereas pre-training optimizes reasoning and actions consecutively.
  • Pre-training vs. Co-training: Reasoning representations matter when action predictions are grounded in them, whereas preserving those representations alone is insufficient.The authors connect this interpretation to results showing reasoning pre-training matches or exceeds full ECoT on nearly all semantic generalization tasks.
  • Dropout vs. Full ECoT: Reasoning-enabled policies succeed on Bridge examples involving out-of-distribution grasp objects and collision-prone behaviors, while disabling reasoning tends to fail.The cited examples compare reasoning dropout or absent reasoning with full ECoT behavior.
  • Dropout vs. Full ECoT: Reasoning dropout does not appear to affect spatial generalization, possibly because numerous left/right labels let non-reasoning policies internalize those concepts.This explanation is explicitly presented as a suspicion rather than a demonstrated mechanism.
  • Dropout vs. Full ECoT: Reasoning dropout performs well on semantic generalization except reaching tasks, indicating that test-time reasoning benefits vary across task categories.The passage reports this pattern without attributing a cause.
  • Evaluation settings: LIBERO-90 uses 3,917 successful filtered trajectories from 4,500 demonstrations, while challenge evaluations introduce perturbed object positions and distractors.The challenge settings are intended to test distribution-shift generalization.

C.1 LIBERO-90 Embodied Chain-of-Thought Pipeline and Dataset Details

The LIBERO-90 ECoT pipeline generates structured reasoning features from demonstrations and trains VLAs to predict discretized action representations. Its data-generation pipeline adapts the original ECoT setup using simulator information and foundation models.

  • Reasoning data: The LIBERO ECoT pipeline extracts plans, subtask and movement rationales, object bounding boxes, and gripper coordinates.These features correspond to high-level planning, current subtask selection, low-level motion, and visual grounding.
  • Reasoning data: LIBERO reasoning data uses simulator segmentation masks and object names for bounding boxes and gripper positions, while Molmo and Llama2 generate subtasks and rationales.The pipeline runs over all 3,917 demonstration trajectories.
  • Pipeline differences: The LIBERO pipeline differs stylistically from Bridge because it excludes the TASK feature, uses simulator ground truth, and applies different foundation-model generation procedures.The authors attribute the resulting stylistic disparities partly to more verbose Molmo-generated LIBERO reasonings.
  • Policy architectures: OpenVLA discretizes 7D manipulator actions into 256 bins, whereas MiniVLA compresses ten-step 7D action chunks into seven VQ-VAE tokens.OpenVLA supports real-robot experiments, while MiniVLA supports large-scale simulation experiments.
  • Action execution: MiniVLA decodes its seven predicted tokens into a ten-step action chunk, and executing all ten actions before re-querying reduces VLA queries tenfold.The authors report this execution strategy worked better in all LIBERO cases.

D.2 Training Hyperparameters

The training setup compares reasoning pre-training, co-training, dropout, and thinking-token variants under largely shared VLA configurations. It also defines how reasoning is generated, omitted, or supplied during training and inference.

  • Training variants: Reasoning pre-training first predicts embodied reasoning with a VLM objective, then fine-tunes the resulting checkpoint for action prediction.The two stages use separate reasoning and standard VLA objectives.
  • Training variants: Reasoning co-training predicts reasoning and actions simultaneously on separate samples, with equal batch probability for each objective.Actions cannot attend to reasonings in this setting because the objectives use different data points.
  • Reasoning dropout: Reasoning dropout randomly removes zero to all reasoning steps during training, allowing some examples to train direct observation-to-action mappings.At test time, the same policy can operate with or without generated reasoning.
  • Reasoning dropout: The Bridge comparison uses a released policy because a significant portion of its transitions lack reasoning annotations, permitting reasoning dropout at test time.The paper uses this policy for its Bridge comparison.
  • Thinking tokens: Thinking tokens add filler tokens before answers to increase fixed-network expressivity, but they do not provide autoregressively generated intermediate computations.The paper contrasts them with ECoT, where generated reasoning tokens can encode intermediate computations for later steps.
  • Thinking tokens: The relationship between the thinking-token argument and robot reasoning remains unclear when exact sequential computations are difficult to represent.The authors therefore treat thinking-token and ECoT policies as not perfectly identical despite controlling token counts.

E.5.2 Practical Implementation Details of Thinking Token Policies

Thinking token policies replace explicit ECoT reasonings with dedicated tokens during training, while the action policy attends to those intermediate representations at test time. Related prompts provide task, subtask, scene, pose, and prior-action context for generating robot reasonings.

  • Thinking-token policy: Thinking token policies replace ECoT reasonings with a similar number of dedicated thinking tokens.The chosen token is a rarely used “ .” token that does not appear in training data or semantically describe the robot tasks.
  • Training procedure: Training appends sampled thinking tokens to each prompt without assigning loss to them, preserving the standard VLA training process.For each datapoint, the number of appended tokens is uniformly sampled from 50-350.
  • Inference procedure: At test time, the policy receives thinking tokens before the action prefix and generates only seven action tokens.The action policy attends to representations of both the original inputs and the intermediate thinking tokens.
  • Reasoning prompts: Subtask-reasoning prompts combine the task instruction, a numbered subtask plan, the current subtask, and scene-based pose and object information.The prompt asks for a concise explanation of why the current subtask is being performed in the situation.
  • Reasoning prompts: Movement-reasoning prompts combine the task, plan, current subtask, recent language-motion actions, and scene context to explain the current motion.The prompt also accepts the current observation image.
Loading 2505.08243v2…