PaperlayerView exact revision on arXiv

Source-linked AI summary

$R^3$: Training Robots to Reason in Natural Language via Reinforcement Learning

Lehong Wu, Yuxiao Qu, Zheyuan Hu, Ivan Zhang, Limin Wei, Zackory Erickson, Aviral Kumar

arXiv:2608.26053v1cs.ROcs.AIcs.CLcs.LG

TL;DR

The paper studies whether VLMs can use free-form natural-language reasoning to guide low-level policies in long-horizon robotic manipulation. It introduces ℛ3, which combines expert-trace mid-training with single-step offline RL, and reports improved performance and OOD generalization over instruction-only imitation. The results support language reasoning as a useful test-time compute mechanism for steering manipulation policies.

  • Problem

    Whether language reasoning can improve robotic manipulation remains unclear, despite manipulation requiring progress tracking, constraint reasoning, mistake recovery, and future-effect prediction.

  • Method

    ℛ3 mid-trains a VLM on expert reasoning traces, then uses rubric-based single-step RL from offline action data to generate reasoning that guides a fixed low-level policy.

  • Results

    ℛ3 improves performance over instruction-only imitation across Language Table and bimanual grocery packing, with stronger generalization across seen and unseen long-horizon tasks.

  • Takeaways & Limitations

    Explicit inference-time free-form reasoning improves generalization beyond reasoning used only as training-time supervision and can serve as test-time compute for action steering.

  • Takeaways & Limitations

    Experiments use two simulated domains and a fixed low-level policy; the RL stage uses semantic agreement as a surrogate objective rather than final task success.

Abstract

from arXiv · show

Reasoning in language allows foundation models to spend more test-time compute on hard problems, such as those requiring decomposition, constraint tracking, and prediction of future consequences. Whether this mechanism can improve robotic manipulation remains unclear, where long-horizon tasks require tracking partial progress, reasoning about object relations, recovering from mistakes, and steering noisy low-level policies. In this paper, we study whether VLMs can be trained to reason directly in natural language to guide low-level manipulation policies. We introduce $R^3$, a simple post-training recipe that turns off-the-shelf VLMs into robotic reasoners: it first mid-trains a VLM on expert-generated reasoning traces to initialize the desired reasoning style, then improves the reasoner with single-step rubric-based RL from offline action data. Unlike prior robotic reasoning methods that mostly use structured traces as auxiliary supervision, $R^3$ trains free-form language reasoning to produce test-time guidance for action. We instantiate $R^3$ on Language Table and simulated bimanual grocery packing, two controlled testbeds for studying robotic reasoning and long-horizon manipulation. $R^3$ improves exploration and generalization across unseen tasks and significantly outperforms instruction-only imitation learning baselines on both benchmarks. Our analyses suggest that free-form language reasoning can function as a test-time compute mechanism for steering low-level policies. Our project page is available at https://robotic-reasoner.github.io/.

1. Introduction

The paper asks whether free-form language reasoning can help VLMs guide robotic manipulation, where long-horizon tasks demand progress tracking, constraint reasoning, recovery, and action steering. It introduces ℛ3, a two-stage training recipe, and finds improved steering, generalization, and reasoning behavior across manipulation settings.

  • Robotic manipulation requires interpreting scenes, tracking physical constraints and interaction history, anticipating future action effects, and recovering from mistakes.
  • ℛ3 trains a high-level VLM to produce natural-language reasoning and instructions that steer a fixed low-level robot policy.The approach uses expert data to supervise reasoning that leads to instructions semantically similar to expert instructions.
  • ℛ3 first mid-trains on expert-generated reasoning traces, then applies single-step reinforcement learning to match expert instructions in offline data.The trajectories include partial progress, mistakes, recoveries, and alternative action choices.
  • ℛ3 improves high-level steering on seen and unseen Language Table tasks and outperforms instruction-only imitation learning on Language Table and grocery packing.The reported gains include stronger generalization to unseen tasks and improved performance for the RL-only grocery-packing variant.
  • Controlled analyses find that explicit inference-time reasoning improves generalization beyond using reasoning only as training-time supervision.The authors interpret these results as evidence that language reasoning contributes to task performance and serves as useful test-time compute.

2. Related Work

Prior robotic systems use structured intermediate representations, hierarchical planning, semantic interfaces, or reasoning-like signals, but explicit free-form reasoning for long-horizon embodied interaction remains distinct from static multimodal reasoning. Generalist robot policies commonly omit explicit reasoning or use it only during training.

  • Language-model reasoning methods improve difficult static and multimodal reasoning tasks, but static-image reasoning need not transfer to embodied action.
  • Robotic manipulation research uses intermediate structures including symbolic task plans, geometric motion planning, visual foresight, latent skills, and language-based interfaces.
  • Recent generalist robot policies and vision-language-action models generally lack explicit reasoning, while some later methods use reasoning-like signals only as training-time supervision.

3. ℛ3: Robotic Reasoners via Reinforcement Learning

ℛ3 trains a high-level VLM to reason over scenes and interaction history before generating instructions for a pretrained low-level robot policy. Its two-stage recipe combines expert reasoning traces, broader instruction data, and single-step offline RL across Language Table and grocery packing.

  • Problem Setup: The decision process combines visual and proprioceptive states, robot actions, transition dynamics, and a binary long-horizon goal reward.The low-level policy executes short-horizon instructions such as moving or pushing objects toward target relations.
  • Data Collection: ℛ3 collects diverse Language Table trajectories with reasoning traces, partial progress, mistakes, recoveries, and alternative action choices.The suite contains 14 long-horizon block-arrangement task types designed to test relational transfer, compositional generalization, and increasing geometric difficulty.
  • Architecture: ℛ3 uses a hierarchical policy in which a high-level VLM generates reasoning and an instruction, while a language-conditioned actor emits robot actions.The VLM conditions on scene, goal, and previous response; the low-level policy receives the instruction.
  • Data Collection: Expert VLM data are split into reasoning-labeled examples for mid-training and instruction-only examples for reinforcement learning.This reflects the setting that high-quality reasoning traces are expensive while subtask instruction labels are easier to obtain.
  • Interaction History: History raises expert VLM pass@1 from 44.9% to 51.0% on line and from 52.3% to 57.6% on V.The previous response helps the VLM carry forward inferred progress and its plan, so history is used for data collection.
  • Mid-Training: Pretrained sub-10B VLMs often produce natural-sounding but shallow reasoning that fails to track progress, object relations, failed attempts, and constraints.Mid-training exposes the model to expert reasoning traces before RL, teaching it to reason about state and interaction history before issuing instructions.
  • Offline RL: Single-step offline RL improves reasoning behavior without requiring expensive online rollouts and long-horizon credit assignment.The formulation optimizes reasoning from expert interaction data while avoiding direct reward optimization over multi-turn environment interactions.

4. Experimental Evaluation on Language Table

On Language Table, ℛ3 improves performance and out-of-distribution generalization over instruction-only imitation, while analyses indicate that explicit inference-time reasoning contributes beyond representation learning.

  • Main Performance Results: RL alone substantially improves ℛ3 performance on training tasks and most OOD tasks, even without mid-training.The exception is diag_line among the reported OOD tasks.
  • Main Performance Results: Mid-training consistently improves RL post-training and provides a strong warm start, with modest reasoning data recovering much of the OOD benefit.The main exception is diag_line, where behavioral mismatch causes degradation.
  • Main Performance Results: ℛ3 significantly outperforms instruction-only imitation across all five held-out OOD tasks.Instruction-only imitation often yields minor gains or degrades performance on OOD tasks.
  • Inference-Time Reasoning Matters Beyond Representation Learning: ℛ3 improves static perception and action understanding, but static VQA gains alone do not fully explain its manipulation improvements.The best model remains below Gemini on several VQA categories while matching or approaching Gemini on many manipulation tasks.
  • Inference-Time Reasoning Matters Beyond Representation Learning: Reasoning supervision improves imitation-learning generalization in some cases, but ℛ3 consistently outperforms imitation variants on OOD tasks.The comparison includes IL, IL (Pre-train), and IL (Co-train), with reasoning removed at test time for the imitation variants.
  • Inference-Time Reasoning Matters Beyond Representation Learning: Increasing the inference-time reasoning budget improves performance, while harder tasks generally elicit longer reasoning traces.These comparisons hold the learned representations fixed when varying the inference-time budget.
  • Analysis of Learned Reasoning: ℛ3 traces show alternative comparison and self-correction behaviors useful for long-horizon manipulation.These behaviors are qualitatively inspected in Appendix C.
  • Analysis of Learned Reasoning: ℛ3’s mid-training aligns instruction distributions with expert behavior, after which RL selectively updates remaining mismatches.The base model overuses short-horizon primitives, whereas mid-training establishes a stronger behavioral prior.

5. Experiments on Bimanual Grocery Packing

In simulated bimanual grocery packing, ℛ3 transfers beyond Language Table and outperforms instruction-only imitation on held-out task configurations, using success and normalized progress evaluation.

  • Experimental Setup: The grocery-packing setup uses a dual-arm xArm-7 platform in a dual-arm workspace following RaC.The environment, goals, and success criteria are detailed in the appendices.
  • Experimental Setup: The evaluation covers 12 held-out task configurations, with 50 episodes per task and 600 episodes overall.Each task uses 5 environment seeds and 10 rollouts.
  • Experimental Setup: An episode succeeds only when all goal objects are stably packed, designated clutter is cleared, and orientation constraints are satisfied.Normalized task progress measures the fraction of packing stages completed.
  • Results: ℛ3 (RL only) substantially outperforms instruction-only imitation on overall success and normalized progress.The evaluation reports success rate and normalized progress across held-out grocery-packing tasks.
  • Results: The results suggest that Stage I mid-training can be skipped when the base VLM already produces useful reasoning in the target domain.Qualitative rollouts show progress tracking and correction of initially wrong object localization.

6. Discussion and Perspectives on Future Work

ℛ3 trains VLMs to generate free-form, action-oriented reasoning that steers a fixed low-level robot policy. Results across two simulated manipulation domains support language reasoning as test-time compute, while future work targets real robots, joint control, and online multi-turn learning.

  • Discussion: ℛ3 combines expert-trace mid-training with rubric-based single-step RL from offline action data.The reasoner generates language guidance for a frozen low-level policy.
  • Discussion: Experiments on Language Table and bimanual grocery packing improve performance over instruction-only imitation without reasoning.Language Table gains extend to seen and unseen long-horizon tasks, including stronger out-of-distribution generalization.
  • Discussion: The trained reasoner tracks interaction history, resolves visual ambiguity, and performs self-correction.These behaviors support action-oriented guidance during manipulation.
  • Limitations and Future Work: The experiments are limited to two simulated domains with a fixed low-level language-conditioned policy, leaving real-robot evaluation for future work.The hierarchical setup systematically isolates reasoning but does not establish performance under real-world perception and control challenges.
  • Future Work: Future extensions include jointly training reasoning and action prediction, multi-turn RL using task feedback, and online updates from environment feedback or human corrections.The current single-step offline RL optimizes semantic agreement with expert instructions rather than final task success.

Appendices

The Language Table appendix defines a 14-task benchmark of long-horizon block rearrangement. Tasks use eight named blocks and a 2D cylindrical pushing end-effector, with examples documented through successful-execution figures and videos.

  • Task Design: 14 long-horizon Language Table tasks require arranging blocks into task-specific spatial configurations.The benchmark is designed around diverse geometric goals.
  • Environment: Each scene contains eight specified colored-shape blocks, and the robot pushes them with a cylindrical end-effector in a 2D action space.The blocks are red moon, red pentagon, blue moon, blue cube, green cube, green star, yellow star, and yellow pentagon.
  • Success Criteria: The appendix specifies success criteria for grouping blocks, making axis-aligned or diagonal lines, and forming rectangles.The criteria constrain spatial spread, alignment, corners, or shared centers depending on the task.

A.2. Grocery Packing Task Details

The grocery-packing appendix describes a simulated dual-arm benchmark, its held-out evaluation goals, data sampling procedure, and rubric-based reward design. Rewards combine instruction accuracy with a penalty for responses shorter than the target length.

  • Environment: The environment uses two 7-DoF xArm-7 arms to pack six YCB grocery objects into small, medium, and large trays.High-level commands specify packing, removing, or transferring named items and tray sizes.
  • Evaluation: Evaluation uses 12 held-out packing specifications with documented goal descriptions and stage counts.Table 6 summarizes the exact goals for these held-out specifications.
  • Success Criteria: Objects count as packed only when correctly placed, sufficiently still, upright within 15°, and free of designated clutter in trays.The progress metric is K/N, where K is the maximum number of correctly packed goal objects and N is the number of goal objects.
  • Data Construction: Reasoner-training samples oversample subtask onset and pre-completion while skipping the final one-second transition tail.Sampling includes three starting-stage frames, up to four interior frames, and one pre-end frame when available.
  • Reward Function: The scalar reward combines instruction accuracy and response length: R = R_acc + R_len.Accuracy uses judge-based linguistic or semantic matching for Language Table, while grocery packing uses exact instruction matching; R_len penalizes responses shorter than 80 words.
  • Training Analysis: Training reward tends to increase with the amount of mid-training performed before RL.Figure 7 compares full mid-training, one-quarter mid-training, and RL-only initialization.

A.6. Training Hyperparameters and Checkpoint Selection

The appendix reports the training frameworks, model and checkpoint-selection procedures, and the figures and tables used to document them. ℛ3 selects RL checkpoints by validation reward, whereas imitation-learning baselines select checkpoints by validation loss.

  • Training Setup: Training uses LLaMA-Factory for supervised fine-tuning, verl for RL, and Qwen3.5-4B as the base model.The method uses two SFT epochs and the last checkpoint for mid-training.
  • Training Setup: Language Table imitation-learning baselines use four SFT epochs, while grocery-packing baselines use eight.These settings are chosen to provide strong baseline comparisons.
  • Checkpoint Selection: RL evaluates the checkpoint with the highest validation mean reward for each run.Figure 9 visualizes this selection with stars marking the highest ℛ3 validation reward.
  • Checkpoint Selection: Imitation-learning checkpoints are selected using the lowest validation loss.Figure 8 visualizes the selected IL checkpoints.
  • Execution Examples: Figure 10 documents successful task executions used to illustrate long-horizon goals and goal states.Task-related blocks are marked with white dots.

A.7. Our Embodied Chain-of-Thought (ECoT) Implementation

The implementation adapts ECoT to long-horizon manipulation by producing short-horizon instructions from scene state, object states, and textual reasoning rather than low-level actions.

  • ECoT outputs a short-horizon instruction, whereas the original approach outputs the low-level robot action.
  • The implementation omits explicit future-plan supervision because changing scenes, policy failures, and stochastic collisions require closed-loop correction.Textual reasoning can still contain planning.
  • The ECoT input combines the task goal, end-effector state, object states, textual reasoning, and instruction.End-effector and object states use 2D coordinates from Language Table.
  • The example reasons about objects in the top-left quarter before instructing the robot to move the red moon to the board center.

B.1. Visual Question Answering Evaluation.

The VQA evaluation measures both static-scene perception and instruction understanding from manipulation outcomes across five Language Table question classes.

  • The evaluation uses five VQA classes spanning object localization, spatial relations, distances, instruction satisfaction, and transition explanation.
  • Absolute Position: Absolute Position asks which board region contains a queried block or robot arm and requires a fixed region phrase.
  • Relative Position: Relative Position asks for one object’s direction relative to another object or the robot arm.
  • Distance: Distance asks which visible block is nearest to or farthest from a specified block or robot-arm anchor.
  • Table 10 reports accuracy separately for each VQA question class, with Gemini shown as an external reference rather than ranked.

B.2. Qualitative Analysis of Reasoning Traces

The qualitative analysis compares reasoning across training stages and examples, showing progression from unstable exploration to reliable, deliberate, state-aware reasoning after reinforcement learning.

  • Completions from the base, mid-trained, and mid-trained-plus-RL models are compared on the same 30 validation scenes using countable and qualitative signals.
  • Mid-training stabilizes formatting and removes hallucinated objects, while RL adds explicit planning without restoring base-model instability.
  • The qualitative comparison characterizes the base model as exploratory but unstable, mid-training as reliable, and RL as deliberate and state-aware.
  • Judge agreement: Human-majority labels provide the reference for judge agreement on 100 prompt-response pairs, while ambiguous mismatches cluster in low semantic-match tiers.
  • Expert trajectories: Expert trajectories include accidental success, mistaken object movement followed by correction, and clearing an oracle block before target placement.
  • ℛ3 reasoning: ℛ3 examples show scene description, progress assessment, future planning, and executable instruction generation.
  • ℛ3 reasoning: ℛ3 re-examines occluded scenes and considers alternate plans before selecting an instruction.
  • Base-model reasoning: Base-model examples show incomplete scene understanding, weak progress estimation, superficial backtracking, and goal-misaligned instructions.

C.3. Additional Reasoning Examples on Grocery Packing

Additional packing examples show ℛ3 tracking multi-view state and long-horizon progress, revising mistaken interpretations, and producing constrained short-horizon instructions.

  • On held-out packing goals, ℛ3 typically describes current views, tracks progress, issues a short-horizon instruction, and revises incorrect interpretations.
  • Scene and progress tracking: The model uses three camera views to identify remaining transfers and packing stages across trays.
  • Error recovery: After mislocating the foam brick, ℛ3 re-examines the base view, revises its scene description, and issues the correct instruction.
  • Language Table prompt: The Language Table prompt supplies the current image, long-horizon goal, and previous response before requiring reasoning and one short-horizon instruction.
  • Language Table prompt: Language Table instructions are restricted to movement, pushing, separation, touching, and arm-position commands with specified object naming and location formats.
  • Language Table evaluation: Language Table reward scoring distinguishes linguistic matches, adverb mismatches, and semantic matches based on moving the same block to the same final positions.
  • Packing evaluation: Packing instruction-only variants omit reasoning guidelines, while packing RL uses exact instruction-string matching instead of a VLM judge.
  • Packing prompt: Packing prompts allow pack, remove, and transfer commands for named items and tray sizes, with no additional output after the instruction.
Loading 2608.26053v1…