$R^3$: Training Robots to Reason in Natural Language via Reinforcement Learning

• Prior robot methods use structured intermediate representations or language interfaces [64], [21], [32], [28],, while reasoning supervision mainly helps during training [8], [18], leaving free-form test-time reasoning open.Source: §1 · pp. 1–2
• R^3 trains a VLM to turn a scene, interaction history, and expert action into reasoning and a semantically similar instruction for a fixed language-conditioned policy to execute.Source: §1 · p. 2
• Stage I mid-trains on expert multi-turn traces of progress, mistakes, recoveries, and alternatives; Stage II applies single-step rubric-based RL to offline instruction-only data, rewarding semantic match.The RL stage does not require expert reasoning traces.Source: §1 · p. 2, Figure 1 · p. 2
• R^3 improves steering on seen and unseen Language Table tasks, outperforming instruction-only imitation, and its RL-only variant also outperforms instruction-only imitation on grocery packing.The approach is studied in Language Table and simulated grocery packing.Source: §1 · p. 2
• Inference-time reasoning improves generalization beyond training-only auxiliary reasoning; the model compares alternatives, revisits scene and history when uncertain, and selects more incremental next steps.Source: §1 · pp. 2–3

Reasoning and test-time compute

• Static VQA performance can differ substantially from long-horizon manipulation steering, where a VLM must reason through a low-level controller and partial observability.Prior chain-of-thought and test-time scaling methods address harder problems through extra reasoning [31] [56] [59] [62] [63]; VLM rationales and visual traces improve VQA and multimodal reasoning [39] [48] [65] [66], while interactive agents revise behavior from feedback [10] [60] [42] [47].Source: §2 · p. 3

Intermediate structures in robotic manipulation

• Robotic systems use symbolic, predictive, latent, or modular intermediates, whereas this work trains a VLM to produce natural-language reasoning that steers a frozen policy.Prior approaches include task-and-motion planning, visual foresight [16] [20], hierarchical or latent planning [40] [46], and LLM systems such as SayCan [1], Inner Monologue [25], and Code as Policies [35].Source: §2 · p. 3

Reasoning in generalist robot policies

• Generalist robot policies lack explicit reasoning, and this work trains a VLM for free-form language guidance instead of relying on hand-designed intermediates while keeping the low-level policy frozen.Other methods use reasoning-like signals as training supervision [8] [27] or inference-time intermediates [11] [12] [18] [21] [26] [28] [32] [34] [36] [43] [50] [61] [64] [67] [68].Source: §2 · p. 3

3.1. Problem Setup and Environments

• A high-level VLM uses the task goal, observation, and interaction history to produce a reasoning trace and subtask instruction, which steers a pretrained low-level policy.The policy executes a fixed-length action chunk, and the process repeats as the task progresses.Source: §3 · p. 3, §3.1 · p. 4, Figure 2 · p. 4
• Language Table tasks require long-horizon object-movement composition and spatial reasoning; the expert data pairs traces with instructions for mid-training, while broader RL data withholds traces.With interaction history, the expert’s pass@1 rises from 44.9% to 51.0% on line and from 52.3% to 57.6% on V.Source: §3.1 · pp. 4–6, Table 1 · p. 5

3.2. Stage I: Mid-Training Reasoning Behaviors into the VLM

• Mid-training addresses shallow reasoning and failures to track progress, object relations, mistakes, and constraints by training the VLM to predict expert reasoning traces followed by instructions.The traces initialize useful reasoning patterns before reward optimization; successful and unsuccessful traces provide supervision on instructions, partial progress, mistakes, and recovery.Source: §3.2 · p. 6

3.3. Stage II: Rubric-Based Single-Step RL with Offline Data

• Single-step RL trains on offline contexts with expert instructions but no expert reasoning, rewarding generated instructions that match expert intent and reasoning that explains their appropriateness.This uses action supervision without human-written rewards or multi-turn robot rollouts; the approach avoids the cost and credit-assignment difficulty of online long-horizon RL.Source: §3.3 · pp. 6–7
• For Language Table, a VLM judge scores semantic instruction alignment, feasibility, and likely outcome; for grocery packing, an exact-match reward is used because commands are finite and unambiguous.Grocery packing uses the previous instruction as history, avoiding the reasoning-context imputation used for Language Table.Source: §3.3 · pp. 7–8

4. Experimental Evaluation on Language Table

• The Language Table evaluation uses 14 long-horizon block-arrangement tasks split across mid-training, RL, and five held-out OOD tasks.Each task is evaluated on 64 held-out scenes with 16 trials per scene; mid-training uses 256 expert trajectories per task, and RL uses 128 successful expert trajectories per task.Source: §4 · pp. 8–9

4.1. Main Performance Results

• ℛ3 (RL only) improves training-task performance and outperforms the base model on every OOD task except diag_line, while mid-training provides a strong warm start for RL.Full ℛ3 consistently improves over both the base model and ℛ3 (RL only); ℛ3 (1/4th mid) slightly underperforms full ℛ3 on mid-training tasks but matches or exceeds it on RL and OOD tasks.Source: §4.1 · p. 9
• ℛ3 outperforms instruction-only imitation on all five OOD tasks while achieving superior or comparable performance on mid-training and post-training tasks.IL yields only minor gains or degrades performance relative to the base model on OOD tasks other than clear_half; ℛ3 better preserves or improves on its reasoning base on diag_line and mid.Source: §4.1 · pp. 9–10

4.2. Inference-Time Reasoning Matters Beyond Representation Learning

• Mid-training and RL improve VQA, especially relative-position, distance, and instruction-inference questions, but VQA gains alone do not explain manipulation improvements.Instruction-execution questions improve less, and even the best model remains below Gemini in several VQA categories while matching or approaching it on many manipulation tasks.Source: §4.2 · p. 10
• Increasing the inference-time reasoning budget generally improves task success, and truncating or removing reasoning from the same ℛ3 checkpoint isolates this effect from representation changes.The reported gains are especially notable on group, line, V, iip, gris, iV, diag_line, rect, and iL; reasoning traces are generally longer on harder, lower-success tasks.Source: §4.2 · pp. 11–12, Table 3 · p. 11, Figure 5 · p. 11

4.3. Understanding Reasoning Behaviors Learned by ℛ3

• Mid-training stabilizes the reasoning interface and aligns instruction distributions with the expert; subsequent RL makes reasoning more state-aware and selectively refines behaviors toward higher-reward strategies.The learned traces include self-correction, uncertainty resolution, constraint and progress tracking, and incremental next-step selection; RL also drops rare instructions misaligned with the expert.Source: §4.3 · pp. 12–13, Figure 6 · p. 12

4.4. Comparison with Approaches that Use Structured CoT Templates

• Adding ECoT components slightly degrades overall performance relative to free-form reasoning, while post-hoc reasoning performs comparably to reasoning recorded during data collection.The authors suggest structured visual grounding may be less useful for tasks requiring long-horizon progress tracking, failure recovery, and closed-loop replanning; the post-hoc comparison uses Gemini-generated traces for Gemini-collected trajectories.Source: §4.4 · pp. 13–14, Table 4 · p. 13

Experimental setup

• For bimanual packing, the authors fine-tune π0.5 on human teleoperation data and skip Stage I because the base VLM already reasons usefully and no reasoning annotations are available.Stage II training examples use sampled frames, oversampling subtask onset and pre-completion, and condition on three current views and the long-horizon goal to supervise the current instruction.Source: §5 · p. 14

Comparisons

• Comparisons include the Qwen3.5-4B base model with and without reasoning, instruction-only imitation, and ℛ3 (RL only) initialized from the base model.The RL-only variant uses a string-match reward.Source: §5 · p. 14

Evaluation protocol

• The study evaluates 600 episodes across 12 held-out configurations, measuring success against packing, clutter-clearing, and orientation constraints, plus normalized task progress.Each task has 5 environment seeds and 10 rollouts; success requires every goal object to be stably packed in its assigned tray, designated clutter cleared, and orientation constraints met.Source: §5 · p. 14

Results

• ℛ3 (RL only) achieves substantially higher overall success and normalized progress than instruction-only IL without reasoning on grocery packing.The result is reported for 12 held-out tasks; qualitative rollouts show progress tracking from three camera views and re-examination to correct an initially wrong object localization.Source: §5 · p. 14, Table 5 · p. 15

Discussion

• ℛ3 combines expert-trace mid-training with single-step rubric-based RL to generate action-oriented reasoning that improves over instruction-only imitation on Language Table and bimanual grocery packing.On Language Table, it also improves performance on seen and unseen long-horizon tasks and out-of-distribution generalization; learned behaviors include interaction-history tracking, visual disambiguation, and self-correction.Source: §6 · p. 15

Limitations

• The evaluation is limited to two simulated domains and a fixed low-level language-conditioned policy, leaving real-robot transfer untested.Stage I uses expert-generated reasoning and Stage II a VLM judge for semantic rewards, optimizing a surrogate objective; multi-turn online RL may improve long-horizon behavior.Source: §6 · p. 15

Future work

• Future work could test ℛ3 on real robots and more dexterous, longer-horizon tasks under noisy perception, physical recovery, and unseen environments.These settings would test whether natural-language reasoning remains useful under real-world challenges.Source: §6 · p. 15
• Jointly training reasoning and action prediction could reduce mismatch between high-level intent and low-level execution while retaining language-based interpretability.The paper also proposes studying language reasoning for low-level action generation; the closest approaches condition actions on structured intermediate metadata [28], and joint training raises synchronization challenges.Source: §6 · pp. 15–16
• Future RL could use multi-turn task-completion, progress, recovery, or human-preference feedback, while online updates from environment feedback or human corrections could support broader exploration.The current single-step offline approach rewards semantic agreement with expert instructions rather than final task success; reasoning-guided exploration could expand behavior beyond offline data to new embodiments, objects, and failure modes.Source: §6 · p. 16

A.1. Language Table Task Details

• The 14 tasks use eight distinct blocks on a board, where a cylindrical end-effector pushes blocks through a 2D action space; Figure 10 shows successful executions.The tasks require arranging blocks into task-specific spatial configurations.Source: §A.1 · p. 23
• Group, line, V-shape, and L-shape tasks arrange selected blocks into a compact cluster or specified aligned geometric patterns.The line may be horizontal or vertical; the V apex sits below its upper pair, while the L has upward and rightward arms.Source: §A.1 · p. 23
• Clear-quarter tasks move every block out of a specified board quarter, while isolate-in-place preserves a target block’s position and separates surrounding blocks.The quarter is one of four regions; isolation requires the target to remain within a small drift threshold and other blocks to exceed an isolation threshold.Source: §A.1 · p. 23
• T-shape and group-and-isolate tasks form a four-block T or group two selected blocks while keeping other blocks away.For the T, three blocks form the horizontal bar and one forms a centered stem; group-and-isolate requires selected blocks to cluster while others exceed an isolation threshold.Source: §A.1 · p. 23
• Inverted-V, diagonal-line, and rectangle tasks place three or four selected blocks in the specified apex, straight-line, or axis-aligned corner arrangement.The inverted-V apex is above its lower pair; diagonal lines may slope in either direction, and a rectangle occupies four corners.Source: §A.1 · pp. 23–24
• Midpoint, inverted-L, and clear-half tasks place a block between instructed endpoints, form a downward-and-leftward L, or clear every block from a specified board half.The target half may be top, bottom, left, or right.Source: §A.1 · p. 24

A.2. Grocery Packing Task Details

• The grocery-packing suite uses two 7-DoF xArm-7 arms in MuJoCo to pack six YCB objects into three tray sizes, following pack, remove, or transfer commands.The objects are cracker box, sugar box, tomato soup can, gelatin box, foam brick, and tuna fish can; the task suite is cited as, MuJoCo as, the arms as [37], and objects as [6].Source: §A.2 · p. 24
• Evaluation uses 12 held-out packing specifications, with Table 6 listing their goal descriptions and stage counts.The appendix states that Table 6 gives the exact high-level goal for each held-out specification.Source: Table 6 · p. 24, §A.2 · p. 24
• A goal object counts as packed only when it is in its assigned tray, remains below 0.02 m/s for 30 control steps, and satisfies any 15° orientation constraint; designated clutter must occupy no tray.Progress is K/N, where K is the maximum number of stages completed as correctly packed goal objects and N is the number of goal objects.Source: §A.2 · p. 25
• Reasoner-training examples sample synchronized base and dual-wrist images, oversampling subtask onset and pre-completion while skipping the final transition tail.Sampling takes three frames during the first 1.5 seconds, up to four interior frames, and one frame in the penultimate second; unavailable stages are omitted in short episodes.Source: §A.2 · p. 25

A.3. Comparison of Language Table and Grocery Packing

• Table 7 compares domain-specific components of the shared hierarchical reasoning framework across Language Table and grocery packing.Source: §A.3 · p. 25, Table 7 · p. 25

A.4. RL Reward Function

• The scalar reward combines an instruction-accuracy reward with a response-length penalty; for Language Table, exact linguistic, adverb-mismatch, semantic, and other outcomes receive 1.0, 0.5, 0.25, and 0.0, respectively.Source: §A.4 · pp. 25–26
• The nonpositive log-scaled length term penalizes responses shorter than the default 80-word threshold and applies no penalty at or above it.Here n is response word count, and the threshold is T=80 by default.Source: §A.4 · p. 26

A.5. RL Reward Curve

• Training reward tends to increase with the amount of mid-training performed before RL across the full-, quarter-, and no-mid-training initializations.The plotted reward is a 21-step centered moving average for R^3, R^3 (1/4th mid), and R^3 (RL only), respectively.Source: §A.5 · p. 26

A.6. Training Hyperparameters and Checkpoint Selection

• R³ uses Qwen3.5-4B, with 2 mid-training SFT epochs and the last checkpoint; Tables 8 and 9 list the SFT and RL hyperparameters.SFT uses LLaMA-Factory and RL uses verl.Source: §A.6 · p. 26, Table 8 · p. 27, Table 9 · p. 27
• IL baselines train for 4 Language Table epochs and 8 grocery-packing epochs; the best validation-loss checkpoint is reported because the last performs worse on grocery packing.The last and best checkpoints perform comparably on Language Table.Source: §A.6 · p. 26
• RL selects each run’s checkpoint with the highest mean validation reward, while IL selection uses lowest validation loss; Figures 8 and 9 visualize Language Table selections.Stars mark the selected checkpoints in those figures.Source: §A.6 · pp. 26–27, Figure 8 · p. 27, Figure 9 · p. 27

A.7. Our Embodied Chain-of-Thought (ECoT) Implementation

• Unlike ECoT [64], this implementation outputs short-horizon instructions rather than low-level actions, placing end-effector and object states before textual reasoning.It removes the move-command component and uses 2D coordinates from the Language Table simulator.Source: §A.7 · p. 29
• The implementation omits explicit future plans because varying scenes, low-level policy failures, and stochastic collisions make them noisy, though textual reasoning can still plan.The tasks require closed-loop correction or replanning rather than fixed instruction sequences or open-loop plans.Source: §A.7 · p. 29
• ECoT inputs include the task goal, end-effector and object states, textual reasoning, and instruction; the study compares expert-collected traces with Gemini 3 Flash post-hoc explanations.Gemini generates retrospective explanations for selected instructions without access to its decision-time reasoning.Source: §A.7 · p. 29

B.1. Visual Question Answering Evaluation.

• Absolute Position questions ask models to locate a queried block or robot arm in a fixed set of board regions.The model directly outputs the corresponding region phrase.Source: §B.1 · p. 30
• Relative Position questions test the direction of one object relative to another object or the robot arm, using fixed direction choices.The model directly outputs the relative direction phrase.Source: §B.1 · p. 30
• Distance questions ask which visible block is nearest to or farthest from an anchor block or the robot arm.The answer is the selected block name rather than a multiple-choice option.Source: §B.1 · p. 30
• Instruction Execution questions test whether a scene satisfies a given instruction, requiring the model to identify relevant objects and verify the final visual state.Responses use a correct-versus-incorrect format, and evaluation considers only the final answer.Source: §B.1 · p. 30
• Instruction Inference questions ask models to select the candidate instruction that best explains an observed before-and-after transition.This tests inverse instruction understanding, including distinguishing among similar candidate commands; evaluation considers only the selected option letter.Source: §B.1 · p. 30
• VQA performance is reported as accuracy for each question class; Gemini is an external reference excluded from ranking, and bold values mark the best non-reference model.The accompanying text says reasoning is required for the two instruction-comprehension classes, not the three perception classes; supplied passages do not include accuracy values.Source: §B.1 · p. 30, Table 10 · p. 31

B.2. Qualitative Analysis of Reasoning Traces

• Mid-training removes most formatting failures and hallucinated objects, while RL adds explicit state-aware planning without restoring the base model’s instability.The comparison analyzes traces from the base, mid-trained, and mid-trained + RL models on the same 30 validation scenes.Source: §B.2 · p. 31, Table 11 · p. 31, Table 12 · p. 32

B.3. Judge Validation and Sensitivity

• All tested VLM judges have comparable agreement with human-majority labels, close to inter-human agreement; Qwen3.5 mismatches fall in ambiguous low-match tiers.The evaluation uses 100 prompt-response pairs; Qwen3.5 occasionally confuses “red pentagon” with “red moon.”Source: §B.3 · p. 33, Table 13 · p. 33, Figure 11 · p. 33

C.1. Additional Expert Trajectories

• Expert trajectories include suboptimal actions, including an unintended push that nevertheless succeeds and a mistaken move that the expert later corrects.In another example, the expert clears an oracle block to make the T-shape rather than moving a target block.Source: Figure 13 · p. 34, Figure 14 · p. 34

C.2. Additional Reasoning Examples on Language Table

• R3 examples show a strategy that describes the scene, assesses progress, plans ahead, and then gives an executable instruction, sometimes considering alternate plans first.The illustrated examples include both a general task and the V task.Source: Figure 15 · p. 35, Figure 17 · p. 35
• When the red pentagon is occluded by the robot arm, the R3 model re-examines the scene, task, and history to infer its state correctly.Source: Figure 16 · p. 35
• The Qwen3.5-4B base model’s examples show incomplete scene understanding, weak progress estimates, and under-justified plans that produce a goal-misaligned instruction.Its apparent backtracking can be superficial or based on hallucinated assumptions rather than grounded correction.Source: Figure 18 · p. 36, Figure 19 · p. 36
• On the V task, Gemini uses more diverse instructions, whereas R3 primarily relies on absolute movement instructions.Source: Figure 20 · p. 36

C.3. Additional Reasoning Examples on Grocery Packing

• On held-out packing goals, ℛ3 tracks progress, issues short-horizon instructions, and can revise an initially incorrect scene interpretation.Examples show it identifying items still to move, recognizing completed packing stages, and correcting a mislocated object after re-examining the base view.Source: §C.3 · p. 37, Figure 21 · p. 37, Figure 22 · p. 37

D.1. Prompt for the high-level VLM (Language Table)

• The Language Table template provides the VLM with the scene, long-horizon goal, and previous response, and requests reasoning followed by one short-horizon instruction.The template is used for data collection, training, and evaluation.Source: §D.1 · pp. 38–39

Prompt for the high-level VLM (Language Table)

• Language Table instructions must use supported action formats and visible or explicitly stated information, with concise reasoning that leads causally to the instruction.The prompt also directs the VLM to trust current observations over its previous response and avoid inventing object locations, contacts, or completed subgoals.Source: §D.1 · pp. 38–39

D.2. VLM-as-a-judge Prompt

• For Language Table RL, a VLM judge compares candidate and expert instructions, scoring exact linguistic matches 1.0 or 0.5 when adverbs mismatch.The rubric allows specified paraphrases of verbs, locations, directions, and equivalent expressions.Source: §D.2 · pp. 39–40

VLM-as-a-judge Prompt

• When instructions are not a linguistic match, the judge scores them 0.25 if they move the same block to the same final positions, and 0.0 otherwise.Reversing which block moves is not considered semantically equivalent.Source: §D.2 · p. 40

D.3. Retrospective Reasoning Generation Prompt / Retrospective Reasoning Generation Prompt

• The retrospective-reasoning prompt asks Gemini to explain expert demonstrations from the scene and goal, treating the expert’s chosen instruction as reference rather than premise.It requires specific spatial reasoning and prioritizes the current image over the previous response when judging progress.Source: §D.3 · pp. 40–41

D.4. Prompt for the high-level VLM (grocery packing)

• Packing instruction-only variants omit reasoning guidelines, while packing RL rewards the parsed instruction by exact string match rather than a VLM judge.The packing template is used for training and evaluation of reasoning models; interaction history is the previous instruction, not the previous full response.Source: §D.2 · p. 39, §D.4 · p. 41

Prompt for the high-level VLM (grocery packing)

• The grocery-packing prompt supplies three camera views, the goal, and the previous instruction, and requests grounded reasoning followed by one instruction from the allowed packing, removal, or transfer types.It directs the model to avoid inventing object states and to end after the instruction.Source: §D.4 · pp. 41–42