PaperlayerView exact revision on arXiv

Source-linked AI summary

ROOT: Discovering Rewards for User-Specified Embodied Behaviors

Eren Sadikoglu, Aditya Taparia, Xinyuan Liu, Ransalu Senanayake

arXiv:2610.04250v1cs.RO

TL;DR

Qualitative behaviors such as natural gait and posture are difficult to specify in rewards, and scalar training statistics can miss policies that diverge from user intent. ROOT uses an observation-guided experiment tree to refine rewards, and evaluations report stronger behavioral alignment than existing LLM-based baselines.

  • Problem

    Qualitative properties such as posture, limb coordination, and movement style are difficult to specify with hand-crafted rewards, while high-return policies may still miss intended behavior.

    Source: §I · p. 1

  • Method

    ROOT uses a persistent experience tree linking reward programs, trained policies, rollout observations, diagnostics, and behavioral insights to guide reward refinements.

    Source: §I · pp. 1–2, §III · p. 2

  • Results

    ROOT has the highest or joint-highest VLM alignment score on six of seven tasks, averaging 4.14 versus 3.56 for the strongest baseline.

    Source: B. RQ1: Behavioral Alignment and Task… · p. 5

  • Takeaways & Limitations

    Across multiple tasks and embodiments, ROOT shows improved behavioral alignment, competitive task performance, human preference over baselines, and transfer of simulation-trained policies to a physical Go2.

    Source: §V · p. 8

Abstract

from arXiv · show

Reinforcement learning for embodied control remains constrained by the difficulty of reward specification. Although recent large language model (LLM)-based methods can synthesize reward functions from natural-language descriptions, they often fail to capture subtle behavioral properties that humans care about, such as natural gait, posture, and movement style. This limitation arises because many desired behaviors are easier to recognize visually than to encode in a reward function. We introduce Reward Optimization via Observable Trees (ROOT), a framework for discovering reward functions that align learned policies with user-specified embodied behaviors. Rather than relying solely on scalar training statistics, ROOT casts reward design as an observation-guided search over a persistent experiment tree that stores reward programs, trained policies, and rollout observations, together with behavioral insights distilled by a video-language model, to diagnose behavioral failures and guide subsequent reward refinements. We evaluate ROOT on seven tasks across four embodiments: simulated Hopper, HalfCheetah, Ant, Unitree Go2, and as well as the real-world Unitree Go2. ROOT produces behaviors that better align with user intent than those generated by existing LLM-based reward-generation methods, achieving up to 86.8% locomotion-completeness accuracy and improving Vid-LLM behavioral alignment from 3.56/5 to 4.14/5, a 16.5% improvement over baselines. Human evaluations further support these results, with ROOT preferred in 51-63% of pairwise comparisons.

I. INTRODUCTION

Qualitative embodied behaviors are difficult to encode as rewards, and high-return policies can still miss user intent. ROOT organizes reward refinement around observed behavior and accumulated experiment evidence.

I. INTRODUCTION

  • Qualitative properties such as posture, limb coordination, and movement style are easier to recognize visually than to specify through hand-crafted rewards.The introduction contrasts these properties with directly measurable objectives such as reaching a target or maintaining a speed.Source: §I · p. 1
  • A policy can earn high reward while missing the requested behavior, creating a behavioral gap that scalar training statistics may not reveal.The motivating example is a quadruped that moves forward by dragging its hind legs instead of using the alternating diagonal coordination of a trot.Source: §I · p. 1
  • ROOT stores reward programs, trained policies, rollout observations, diagnostics, and behavioral insights in a persistent tree used to diagnose failures and guide targeted reward refinements.Its reasoning framework can revisit promising rewards as well as propose modifications.Source: §I · pp. 1–2

II. RELATED WORK

  • Prior LLM-based methods automate reward synthesis and refinement, including executable-code generation, and approaches using training statistics or evolutionary refinement,, [13],.Other structured approaches evolve modular rewards or interleave reward selection with policy optimization [15],.Source: §II · p. 2
  • Visual-feedback methods include visual reward models and iterative reward refinement –[21],, [8], while ROOT links critique of its own rollouts to the corresponding reward refinements without requiring task demonstrations.ROOT uses those linked assessments to inform subsequent search decisions.Source: §II · p. 2

III. METHODOLOGY

ROOT treats reward discovery as a search for a reward whose trained policy has a small behavioral discrepancy from the user’s intended task. Its persistent experience tree links reward changes to observed outcomes for later refinement.

III. METHODOLOGY

  • For reward r, ROOT trains a policy and defines the behavioral gap as the expected discrepancy between its rollouts and the intended task, seeking a small gap within a search budget.The discrepancy is an implicit human criterion rather than a quantity directly evaluated during search.Source: §III · p. 2

A. The Experience Tree

  • Each tree node records the reward, trained policy, refinement plan, diagnostics, rollout observations, and distilled findings; failed attempts retain their failure diagnostics.The search state also includes an auxiliary insights document.Source: A. The Experience Tree · pp. 2–3
  • Parent-child links connect a reference reward to a tested refinement and its outcome, while siblings preserve alternative refinements and earlier partial successes remain revisitable after regressions.This tree structure retains both successful and unsuccessful refinement evidence for subsequent search.Source: A. The Experience Tree · p. 3, Figure 2 · p. 3
  • ROOT initializes its tree with varied bootstrap reward candidates guided by an environment-grounded task specification that identifies relevant physical signals and anticipated failure modes.These initial experiment records provide evidence for later refinement.Source: A. The Experience Tree · p. 3
  • Retrieval tools let the reasoning module compare relevant records across branches, while the insights document distills cross-experiment guidance for choosing a node and its refinement.The tree preserves experiment-specific evidence as the insights are revised during reasoning.Source: A. The Experience Tree · p. 3

B. Observation-Guided Search

ROOT conducts reward search through repeated reason, execute, and observe stages, preserving experiment outcomes in a tree that informs later refinements and final policy selection. The experiments examine alignment, performance, search cost, and transfer to a physical robot.

B. Observation-Guided Search

  • Each refinement iteration reasons over prior evidence, tests a candidate reward through fresh policy training, and observes rollout behavior; the budget counts bootstrap and failed attempts.Reason selects a reference and plan; Execute trains; Observe records behavior.Source: B. Observation-Guided Search · pp. 3–4
  • Parent-child and sibling comparisons help the reasoning module choose a reference and refinement plan, preserve successful reward elements, and target remaining behavioral discrepancies.Evidence across branches can also prompt revisiting earlier partial successes after later regressions.Source: B. Observation-Guided Search · p. 3
  • Execute validates candidate reward programs, applies bounded repairs, and trains a fresh PPO policy; failures are recorded with no resulting policy.Successful runs return the reward implementation used, policy, and diagnostic record.Source: B. Observation-Guided Search · p. 4
  • Vid-LLM compares multi-view rollout videos with the requested behavior, while an annotator combines its assessment with diagnostics and prior findings into refinement insights.The resulting record describes achieved behavior, remaining discrepancies, and whether the expected effect was observed.Source: B. Observation-Guided Search · p. 4
  • After the search budget is exhausted, ROOT selects the successfully evaluated policy judged best aligned with the requested behavior; the experiment tree retains regressions and failures for later search decisions.Each experiment is recorded as a child of its selected parent, preserving reward-design lineage.Source: B. Observation-Guided Search · p. 4

IV. EXPERIMENTS

  • The experiments ask whether ROOT improves behavioral alignment while maintaining task performance, how its tree and VLM feedback contribute, and how alignment compares at a given search cost and transfers to a physical robot.The evaluation considers behavioral alignment, task performance, search cost, and physical-robot transfer.Source: §IV · p. 4

A. Experimental Setup

The experiments test ROOT across seven simulated locomotion tasks and deploy a learned trot policy on a physical Go2. They compare reward-design methods using task-specific performance metrics and Vid-LLM behavioral-alignment ratings.

A. Experimental Setup

  • ROOT’s search loop generates and executes reward programs, renders successful policies, gathers rollout observations, annotates experiments, and adds records as tree children.The algorithm initializes the tree and insights, bootstraps initial trials, and then uses reasoning to guide subsequent search.Source: A. Experimental Setup · p. 4
  • The suite covers Ant crab-crawl and spider-creep, Unitree Go2 pace, trot, and canter, plus Hopper and HalfCheetah forward running; a learned trot policy is also deployed on a physical Go2.The tasks are specified in natural language, with qualitative direction, posture, or limb-coordination requirements beyond forward progress.Source: A. Experimental Setup · p. 4
  • Comparisons include Eureka, Text2Reward, L2R, CARD, and LAPP, with shared task descriptions and PPO configurations and method-specific candidate counts and selection rules.Candidate training budgets are 20M environment steps on MJX (30M for Go2 canter) and approximately 98M on Isaac Lab; Claude 4.5 Sonnet is the shared reasoning and reward-generation model.Source: A. Experimental Setup · pp. 4–5
  • Task performance uses native scores for Hopper and HalfCheetah and task-specific locomotion-completeness accuracy for gait tasks, evaluated within tasks.Gait metrics incorporate contact coordination and, for Go2 pace/trot, survival, velocity tracking, and smoothness; tasks use a 0.5 m/s forward command with pushes disabled.Source: A. Experimental Setup · p. 5
  • Behavioral alignment is the mean 1–5 Vid-LLM rating across Seed 2.1 Turbo, Gemma 4 31B, and GLM-4.6V, which assess posture/style and limb coordination from three-view videos.Results report mean and standard deviation across three evaluation seeds using a 1,000-step horizon.Source: A. Experimental Setup · p. 5

B. RQ1: Behavioral Alignment and Task Performance

Across the task suite, ROOT combines competitive task performance with favorable perceived behavioral alignment and human preference. Ablations indicate that experimental history and visual feedback contribute to alignment and task accuracy.

B. RQ1: Behavioral Alignment and Task Performance

  • Human evaluators favor ROOT over pooled baselines on crab-crawl, spider-creep, and canter, including on Ant tasks where ROOT has lower contact-based accuracy than leading methods.This suggests contact agreement alone does not fully capture perceived adherence to the requested behavior.Source: B. RQ1: Behavioral Alignment and Task… · p. 5, Figure 3 · p. 5
  • ROOT ranks first or second in mean task performance on all seven tasks, and has the highest or joint-highest Vid-LLM alignment on six of seven, averaging 4.14 versus 3.56 for the strongest baseline.The alignment passage cautions that Vid-LLM ratings alone cannot establish correct coordination; task metrics and human preferences are considered alongside them.Source: B. RQ1: Behavioral Alignment and Task… · p. 5

C. RQ2: The Role of Tree Memory and Visual Feedback

  • Qualitative rollouts show Hopper hopping, Ant spider-creeping, and simulated Go2 cantering, followed by a simulation-trained trot policy deployed on a physical Go2.Frames progress from left to right.Source: Figure 4 · p. 6
  • Figure 5 plots average behavioral alignment against GPU-hours and LLM/VLM requests for Ant crab-crawling and Go2 canter; ROOT has the highest mean alignment on both tasks while using less training compute than Eureka.Higher alignment and lower cost are preferred.Source: Figure 5 · p. 6
  • Removing history lowers average alignment from 3.98 to 2.80 and task accuracy from 51.22% to 10.74%; additionally removing visual feedback lowers them to 1.84 and 1.91%, respectively.Full ROOT achieves the highest behavioral alignment and task accuracy on all three ablation tasks; the results support the usefulness of both retained history and visual feedback.Source: Table II · p. 6, C. RQ2: The Role of Tree Memory and… · p. 6

D. RQ3: Behavioral Alignment versus Search Cost

ROOT’s alignment and search-cost results indicate that stronger measured alignment does not require the highest training compute or query count. On hardware, the selected trot retains diagonal coordination, with reported improvements in motion smoothness and coordination regularity.

D. RQ3: Behavioral Alignment versus Search Cost

  • On Ant crab-crawl and Go2 canter, ROOT achieves the highest mean Vid-LLM alignment on both tasks while using less training compute than Eureka and less than LAPP on crab-crawl.CARD uses substantially less compute but achieves lower alignment; ROOT uses more model queries for observation and reasoning than reward-code baselines, but fewer than LAPP on both tasks.Source: D. RQ3: Behavioral Alignment versus… · pp. 6–7

E. RQ4: Sim-to-Real Transfer of Behavioral Alignment

  • Trotting on the physical Go2 synchronizes diagonal FL/RR and FR/RL pairs, while simulated pacing synchronizes same-side FL/RL and FR/RR pairs.The distinct contact patterns reflect the requested gait structures beyond forward motion.Source: E. RQ4: Sim-to-Real Transfer of… · p. 7, Figure 6 · p. 7
  • On a physical Go2, ROOT’s transferred trot retains alternating diagonal contacts and improves motion smoothness and coordination regularity over the evaluated controllers.It has higher mean contact smoothness and lower joint jerk than MPC, and the lowest mean cadence entropy and highest mean phase consistency among the evaluated controllers; its regularity occurs at a lower stride frequency than either baseline, while MPC’s mean phase offset is closer to zero.Source: E. RQ4: Sim-to-Real Transfer of… · pp. 7–8, Figure 6 · p. 7, Figure 7 · p. 7

V. CONCLUSION

  • ROOT combines visual feedback, multi-agent reasoning, and a persistent experiment tree to guide reward refinements, and its policies were preferred by human evaluators over baseline behaviors.The conclusion also identifies manipulation and longer-horizon behaviors, and reuse of stored insights across related tasks and embodiments, as future directions.Source: §V · p. 8

APPENDIX A TASK SPECIFICATIONS

The appendix specifies the natural-language behaviors, search and training configurations, ablation comparisons, and evaluation protocol. Hardware deployment evaluates transfer of a selected simulation-trained policy rather than adding another reward-search task.

APPENDIX A TASK SPECIFICATIONS

  • The seven simulated tasks are specified in natural language, with state-based evaluation criteria defined separately; hardware deployment tests whether the simulation-trained policy retains a stable trot rather than conducting another reward search.The behavior specifications and simulator backends are listed in Table III.Source: Appendix A · p. 9

A. Language Models and Reward Search

  • ROOT’s Proposer, Critic, and Strategist use temperature 0.4, while Gemini 3.1 Pro supplies rollout-video feedback during observation.Reasoning agents retrieve individual experiment records, compare refinements across branches, update an insights document, and provide a selected parent’s reward code with a refinement plan; each candidate policy is trained from scratch.Source: A. Language Models and Reward Search · p. 9
  • The search budget counts experiment attempts, including failed execution attempts, whose diagnostics remain available; after search, ROOT selects a successfully evaluated policy using task-alignment descriptions recorded in the tree.This search-time selection is separate from final VLM-panel and human evaluations.Source: A. Language Models and Reward Search · p. 9

B. Policy Training and Baselines

  • Within each simulator backend, methods share task descriptions, environment configuration, PPO settings, and training steps per candidate, but candidate counts and selection rules follow method-specific protocols.MJX candidates receive 20 million environment steps except Go2 canter at 30 million; Isaac Lab Go2 candidates receive approximately 98 million. The LAPP adaptation removes the handcrafted reward, and its environment-step budget matches ROOT’s.Source: B. Policy Training and Baselines · p. 9

C. Ablation Configurations

  • The ablations compare removing history while retaining current-rollout visual feedback, then removing visual feedback when history is already absent.They do not measure removing visual feedback from the full history-enabled method.Source: C. Ablation Configurations · p. 9

APPENDIX C EVALUATION PROTOCOLS

  • Simulation policies are evaluated over a 1000-step horizon with three seeds, and reported task-performance mean and standard deviation describe rollout variability rather than variability across reward searches.Evaluation also measures task performance from simulator states, behavioral alignment with an external VLM panel, and perceived adherence through human preferences.Source: Appendix C · p. 9

A. State-Based Task Performance

State-based task metrics are computed independently of the generated reward, with gait criteria tailored to each embodiment and behavior. The VLM panel provides a complementary assessment of posture, style, and limb coordination.

A. State-Based Task Performance

  • For Hopper and HalfCheetah, task performance is mean forward velocity multiplied by mean survival; for other tasks, locomotion completeness accuracy measures task-specific coordination during locomotion.Ant crab-crawl and spider-creep use contact-coordination criteria during active locomotion, with additional motion checks for crab-crawl; VLM and human evaluations additionally assess posture and movement style.Source: A. State-Based Task Performance · p. 9
  • Ant contact evaluation marks feet below 0.05 m as contacting, uses an adaptive per-foot threshold if fewer than half register, smooths signals over five steps, and treats locomotion as active above 0.1 m/s.Feet are identified from body names, with initial position quadrants as a fallback.Source: A. State-Based Task Performance · p. 9
  • Table III lists behavior specifications and simulator backends, and Table IV summarizes PPO settings; Brax batch size counts rollout sequences, so transitions per minibatch equal batch size times rollout length.For RSL-RL, the stated counts are 24,576 transitions per minibatch and 98,304 transitions per iteration.Source: Table III · p. 10, Table IV · p. 10, A. State-Based Task Performance · p. 10
  • Go2 pace and trot accuracy combines survival, velocity tracking, contact coordination, and smoothness; pace targets synchronized ipsilateral pairs, while trot targets alternating diagonal pairs.Evaluation uses a 0.5 m/s forward command with external pushes disabled.Source: A. State-Based Task Performance · p. 10
  • Go2 canter accuracy is the fraction of detected gait cycles matching the target three-beat footfall pattern.Ndetected counts detected cycles and Nmatch counts those satisfying the evaluator’s canter criterion.Source: A. State-Based Task Performance · p. 10

B. VLM Behavioral Alignment

  • The VLM panel receives task descriptions, three-view rollout videos, and simulator-derived facts, then scores posture and style and limb coordination on integer scales from 1 to 5.Method identities, reward code, and training curves are hidden; the reported score averages both dimensions across judges and evaluation rollouts, and is interpreted alongside state-based metrics because visual plausibility alone need not imply correct contact patterns.Source: B. VLM Behavioral Alignment · pp. 10–11

C. Human Preferences

Human raters compare anonymized ROOT and baseline rollout videos against a behavior description, while search-cost comparisons track training compute and model interactions. The hardware deployment uses a simulation-trained Go2 policy without fine-tuning under restricted straight-line commands.

C. Human Preferences

  • Human raters see a behavior description and anonymized, randomly ordered pairs of ROOT and baseline rollout videos, then choose the closer match or report no clear distinction.ROOT preference is calculated from non-tied judgments, excluding ties and pooling baseline comparisons within each task.Source: C. Human Preferences · p. 11

D. Reward-Search Cost

  • Search-cost comparisons on Ant crab-crawl and Go2 canter measure policy-training GPU-hours and LLM/VLM requests, but method-specific budgets and selection procedures limit direct cost interpretation.Request counts measure interaction volume, not monetary cost, and these comparisons do not isolate the cost of individual ROOT components.Source: D. Reward-Search Cost · p. 11

A. Policy Training and Deployment

  • The Go2 policy is deployed without fine-tuning, with inference at 50 Hz through the Unitree SDK and commands restricted to straight-line locomotion.Forward velocity ranges from −0.6 to 0.6 m/s, with zero lateral and yaw velocity; training randomizes friction and base mass and uses a PD actuator model with torque-speed limits and action-execution delay.Source: A. Policy Training and Deployment · p. 11

B. Controller Comparison

The controller comparison reports hardware trials separately from the three-seed simulation evaluation and characterizes gait quality with seven signal-derived metrics. These metrics cover contact transitions, joint motion, stepping rhythm, and diagonal coordination.

B. Controller Comparison

  • Each controller comparison uses 7–8 hardware trials, starting from standing and ending when the robot leaves camera view or needs operator intervention.Results are reported as trial means and standard deviations, separately from the three-seed simulation evaluation.Source: B. Controller Comparison · p. 11

C. Gait-Quality Metrics

  • Seven gait-quality metrics computed from foot-force and joint-angle signals characterize contact transitions, joint motion, stepping rhythm, and diagonal coordination.The listed metrics include contact smoothness, joint jerk and motion magnitude, stride frequency, cadence entropy, phase consistency, and diagonal phase offset.Source: C. Gait-Quality Metrics · p. 11
  • Contact smoothness scores force-signal change, with higher scores indicating less abrupt measured contact transitions; comparisons require consistent force scaling and temporal sampling.The evaluator uses the mean force-signal slope.Source: C. Gait-Quality Metrics · p. 11
  • Joint jerk is reported as root-mean-square jerk across joints and samples in rad/s3; lower values indicate less abrupt joint motion.Joint-motion magnitude is root-mean-square joint angular velocity in rad/s, but without torque or power measurements it does not establish mechanical work or energy efficiency.Source: C. Gait-Quality Metrics · p. 11
  • Stride frequency describes stepping cadence, while cadence entropy indicates how concentrated stride timing is under a common discretization; neither frequency direction alone establishes a better gait.Cadence entropy is lower when stride timing is more concentrated.Source: C. Gait-Quality Metrics · p. 11
  • Diagonal phase offset estimates average timing lag within diagonal foot pairs, whereas phase consistency measures how stable their relative phase remains over time.Values near 0° indicate synchronized diagonal legs; higher phase-consistency scores indicate less temporal drift.Source: C. Gait-Quality Metrics · p. 12

APPENDIX E REASONING AND EVALUATION PROMPTS

The appendix presents prompt templates for ROOT’s reasoning, reward generation, and behavioral evaluation. Search-time observations and experiment records guide refinement, while a separate evaluator scores selected policies after search.

Behavioral evaluation

  • Search-time observations inform later reward refinement, while the final behavioral evaluator scores selected policies after search; the annotator combines observations, diagnostics, plans, parent findings, and insights.The final behavioral evaluator is distinct from the Vid-LLM that supplies feedback during reward search.Source: Appendix E · p. 12, Figure 12 · p. 13

Proposer prompt

  • The Proposer inspects the experiment tree and records to identify patterns and promising experiments, comparing task-alignment observations with survival, velocity, posture, and training diagnostics.It distinguishes useful coordination without translation from inactivity and uses reward-component curves to inspect convergence and plateaus.Source: Appendix E · p. 12

Strategist prompt

  • The Strategist turns the Proposer–Critic debate into a self-contained reward-refinement prescription grounded in recorded evidence.It specifies parent choice, changes to preserve or make, formulas and coefficient ranges, observation inputs, and expected term magnitudes.Source: Appendix E · p. 12, Figure 9 · p. 12, Figure 8 · p. 12

Reward generation

  • For MJX, the reward generator implements the prescription as a Python function returning a finite scalar reward and named component values; bootstrap generation uses prior insights without a parent experiment.The MJX template includes parent code in the prescription rather than as a separate experiment record.Source: Appendix E · p. 13, Figure 11 · p. 13
  • The Isaac Lab prompt guides reward design using Go2 environment data, gait constraints, and a design archetype, with a Strategist prescription taking priority when provided.Isaac Lab calls the Proposer the Scientist, and its Strategist selects a reward-design archetype.Source: Appendix E · pp. 12–13

Related papers

Loading 2610.04250v1…