PaperlayerView exact revision on arXiv

Source-linked AI summary

InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation

Zhuo Lin, Sirui Xu, Liuyu Bian, Yu-Xiong Wang, Liang-Yan Gui

arXiv:2610.02196v1cs.ROcs.CVcs.GR

TL;DR

Humanoid controllers may already have skills needed for tasks absent from training, but expressing and iterating contact-rich tasks is challenging. InterEvolve evolves staged reward programs using execution feedback on a fixed controller, with an object-aware forward-backward model. The paper reports that evolved programs release capabilities left untapped by human-designed rewards, support simulation tasks and compositions, and run autonomously on a physical Unitree G1.

  • Problem

    Humanoid controllers may encounter tasks not enumerated during training, while existing task interfaces can be difficult to design for nuanced, contact-rich interactions and reward revisions conventionally require policy retraining.

    Source: §1 · p. 1

  • Method

    InterEvolve uses staged reward programs and an object-aware forward-backward model to adapt a fixed controller, with an LLM revising program structure and a numerical optimizer tuning constants.

    Source: §1 · p. 2

  • Results

    Evolved programs unlock behaviors left untapped by human-designed rewards, support diverse and long-horizon tasks in simulation, and run autonomously on a physical Unitree G1 from onboard perception.

    Source: §4.3 · pp. 7–8, §4.4 · p. 9

  • Takeaways & Limitations

    Task knowledge can reside in inspectable, reusable reward programs while the behavioral foundation model remains fixed, allowing the two to improve on separate timescales.

    Source: §5 · p. 9

    Search is bounded by the controller's motor repertoire, reward-inference bank, and available measurements, and its simulation and LLM costs preclude real-time replanning during physical execution.

    Source: E LIMITATIONS · p. 31

Abstract

from arXiv · show

We study test-time evolution for humanoid loco-manipulation: solving tasks that a controller was never trained for by repurposing its existing skills, improving from its own attempts, and retaining what it learns, without retraining. Our key insight is that a broad controller already holds much of the competence a new task needs, and that this competence becomes accessible through an interface between planning and control that is expressive enough to specify contact-rich, multi-stage interactions, yet executable and measurable enough that execution feedback can guide planning from experience. InterEvolve realizes this interface with two components. First, we develop an object-aware forward-backward (FB) behavioral foundation model, whose object residuals on a frozen body prior turn a new reward about the body or objects into loco-manipulation behavior at test time. Second, we specify tasks as reward programs: staged rewards with completion conditions and tunable constants. A large language model (LLM) agent revises the program structure in context, drawing on execution feedback and a skill library of verified programs, while a numerical optimizer tunes its constants. With every candidate verified across parallel simulation scenarios, the program explores new ways to induce, repurpose, and compose the controller's existing motor competence for the task at hand, and thus improves over iterations. Experiments show that human-designed rewards leave much of the FB model's loco-manipulation competence untapped, whereas the programs InterEvolve evolves release it, sometimes through novel strategies. It further produces behaviors for diverse tasks, complex scenes, and long-horizon compositions in simulation, and evolved skills run autonomously on a physical Unitree G1 from egocentric onboard perception.

1 INTRODUCTION

Humanoid controllers can contain the motor skills needed for unseen tasks, but expressing and iterating new contact-rich behaviors remains difficult. InterEvolve searches editable reward programs at test time while keeping its controller fixed.

1 INTRODUCTION

  • New tasks may require motions a controller can already perform, even when that exact behavior was absent from its demonstrations.The paper frames test-time evolution as finding new uses for existing skills, improving through attempts, and retaining experience for later tasks.Source: §1 · p. 1
  • Rewards can specify contact-rich objectives, but conventional reward revisions require training a new policy, making iteration slow.Motion references, goal states, and skill labels instead require a desired motion, target state, or predefined behavior.Source: §1 · pp. 1–2
  • InterEvolve combines staged reward programs with an object-aware forward-backward model that executes new rewards without retraining.Trainable object residuals read object features while the model's body networks remain frozen.Source: §1 · p. 2
  • An LLM revises reward structure from task intent and execution feedback, while CMA-ES tunes program constants and verified programs support later tasks.Parallel simulation supplies batches of execution evidence for comparing revisions.Source: §1 · p. 2
  • The paper evaluates reward design and accumulated experience, and reports unseen tasks, long-horizon compositions in simulation, and autonomous Unitree G1 deployment from onboard perception.Its contributions also include evaluating reference-tracking and goal-conditioned tasks.Source: §1 · pp. 2–3

2 RELATED WORK

InterEvolve differs from prior humanoid-control and reward-design approaches by representing tasks as editable reward programs and testing them on a fixed controller. It combines this search with an object-aware behavioral foundation model and reusable verified programs.

Humanoid loco-manipulation and task interfaces

  • Prior humanoid policies learn whole-body control and loco-manipulation from references, use multimodal prompts, sequence skills, or imitate imagined interactions; InterEvolve instead revises reward programs.Its programs specify relevant contacts and stage transitions.Source: §2 · p. 3

Behavioral foundation models and reward inference

  • Existing forward-backward humanoid models run one policy for new rewards, goals, or references, but body-only observations can collapse rewards with different object goals.InterEvolve's object-aware model addresses this stated limit.Source: §2 · p. 3

Reward design and agents that learn from execution

  • Reward-design agents often train a policy per candidate, while InterEvolve evaluates candidates through rollouts on a fixed controller and stores verified programs for later tasks.Language to Rewards instead uses model predictive control; MotionDisco trains trackers for evolved motions, and ROSETTA builds multi-stage programs from language preferences.Source: §2 · p. 3

Test-time compute and search

  • Where control-time test-time search plans actions or steers a pretrained policy, InterEvolve searches task specifications and verifies reward programs in parallel rollouts.The passage also situates this approach alongside inference-time compute and program-search systems.Source: §2 · p. 3

3 METHOD

InterEvolve adapts a fixed controller by evolving staged reward programs against a fixed verifier, then retains final programs for later tasks. Its nested search separates program structure from numerical calibration.

3.1 PROBLEM SETUP

  • A task pairs a language request and scene context with a fixed controller, a reward program, and criteria that assess outcomes and physical constraints.The verifier remains fixed, and evolution searches programs within a budget of revision rounds.Source: §3.1 · p. 4

3.2 REWARD PROGRAMS AS EDITABLE TASK STRATEGIES

  • A reward program is an ordered sequence of interaction stages, each with reward code and a completion condition, plus tunable constants.The condition advances execution when it holds; constants include weights, tolerances, kernel widths, and thresholds. The structure lets the agent edit objectives and transitions while an optimizer tunes constants.Source: §3.2 · pp. 4–5

3.3 EXECUTING REWARD PROGRAMS WITHOUT RETRAINING

  • The forward-backward model maps a new reward to a latent prompt for the same policy, avoiding policy training for each candidate program.The model estimates the prompt from a fixed bank of body-object states, with reward-weighted states contributing more while the bank retains broad coverage.Source: §3.3 · p. 5
  • Trainable object residuals add object features to frozen body branches, enabling reward-conditioned interaction while retaining the pretrained body prior.The residual branches are trained on human-object interaction data; actor and forward and backward maps receive object access.Source: §3.3 · p. 5

3.4 EVOLVING REWARD PROGRAMS AT TEST TIME

  • An LLM proposes structural revisions from task context, verifier feedback, and the skill library; CMA-ES tunes each valid proposal's constants.Candidates are evaluated in parallel, compared on shared scenarios, and accepted only when improvement exceeds run-to-run evaluation noise.Source: §3.4 · pp. 5–6
  • Evolution stops at its revision budget or when the current best passes every criterion on confirmation scenarios, then adds that program to the skill library.The library records completed tasks, scenes, verifier pass rates, and selected programs with tuned constants so later tasks can adapt or compose them.Source: §3.4 · p. 6

4 EXPERIMENTS

Experiments test reusable control, reward-program evolution, and skill composition across tracking, goal-conditioned tasks, and physical deployment. Evolution improves over fixed programs, while reusable skills support composite and long-horizon behavior.

4.1 EXPERIMENTAL SETUP

  • The evaluation covers held-out reference-tracking clips and eight goal-conditioned task families, with tracking and task success measured using separate criteria.Task success requires all criteria and physical constraints in one rollout; earned tiers measure the mean fraction of criteria passed.Source: §4.1 · p. 6

4.3 REWARD-PROGRAM EVOLUTION FOR GENERAL-PURPOSE TASKS

  • Evolution more than doubles the success of the best calibrated fixed program across eight goal-conditioned task families.The comparison isolates structural edits to what is rewarded and how objectives are staged; calibration alone cannot repair a program with the wrong structure.Source: Table 2 · p. 7, §4.3 · p. 7

4.2 REUSABLE CONTROL FOR REFERENCE TRACKING

  • On reference tracking, object-aware control approaches the dedicated tracker’s success, and an evolved reward raises success above ULTRA while nearly halving final root and object drift.ULTRA retains the lowest tracking errors; object-aware pretraining supplies interaction competence that body-only BFM-Zero lacks.Source: Table 1 · p. 7, §4.2 · p. 7

4.3 REWARD-PROGRAM EVOLUTION FOR GENERAL-PURPOSE TASKS

  • On carrying at chest height, tipping onto a new face, and pushing through gates or around obstacles, evolution raises success above 70% after calibrated fixed programs succeed in at most a quarter of episodes.Kicking to a mark remains the hardest task family; kicks are rare in the training data and require strong whole-body coordination from the reward.Source: §4.3 · p. 8
  • Single-stage programs cause the largest ablation drop, numerical tuning is nearly as important, and most test-time scaling gains arrive in round one.Earned tiers rise monotonically across rounds, but full success can dip when fixing one criterion briefly breaks another.Source: §4.3 · p. 8, Table 4 · p. 9
  • On a physical Unitree G1, programs evolved in simulation run for box kicking and pushing using onboard camera perception and FoundationPose pose estimates.The robot receives filtered object poses from its onboard perception pipeline.Source: §4.3 · p. 8, Figure 4 · p. 9

4.4 REUSE AND COMPOSITION

  • A full skill library helps solve composite relocation, stacking, and carry-place-kick tasks, while a carry-only library helps mainly with relocation and stored programs alone rarely solve the composite task.Reuse depends on whether the library covers the contact modes the task needs, and evolution integrates the stored programs rather than merely retrieving them.Source: §4.4 · p. 9
  • A single reset can run twelve walking-and-pushing phases to arrange six boxes in a ring; in the shown take, every box ends within 11 cm of its cell without later legs disturbing placed boxes.The top-down trace marks each box’s start and target and numbers the push order.Source: Figure 3 · p. 8, §4.4 · p. 9

5 DISCUSSION AND CONCLUSION

InterEvolve separates task knowledge from controller weights: the fixed behavioral model executes editable reward programs, while an agent improves them through test-time search and observed outcomes. Its object-aware motor model uses object and body-link features alongside a frozen body prior.

5 DISCUSSION AND CONCLUSION

  • Task knowledge resides in inspectable, reusable reward programs, while the controller and agent improve on separate timescales through interaction data and test-time search.Physical execution grounds the agent’s revisions in observed outcomes rather than assumptions.Source: §5 · p. 9

A MOTOR MODEL

  • The object-aware motor model is pretrained once before reward-program search and stays fixed during the search.The G1 interaction encoding combines object state with distances from 30 body links to the object surface; the Inspire version adds 24 finger links.Source: A MOTOR MODEL · p. 16, §A.1 · p. 16

A.2 RESIDUAL ARCHITECTURE AND TRAINING OBJECTIVE

The motor model preserves its pretrained body prior while learning object residuals from human-object interaction data. Its actor combines task value with imitation and auxiliary critics, and Inspire-hand control uses a low-dimensional grasp interface.

A.2 RESIDUAL ARCHITECTURE AND TRAINING OBJECTIVE

  • The actor and forward and backward maps keep their body branches frozen and train only object residuals on human-object interaction data.This initializes object-aware learning from the pretrained body prior.Source: §A.2 · p. 16
  • The actor maximizes FB task value together with discriminator and auxiliary critics, with effective regularization weights scaled by the detached mean absolute FB value.The imitation reward is the discriminator probability’s log-odds; auxiliary rewards penalize undesirable motion and physical violations, and the auxiliary critic learns their normalized return.Source: §A.2 · p. 16

A.3 EMBODIMENT AND HAND CONTROL

  • For Inspire hands, a per-hand scalar output is expanded into authored finger targets through software mimic coupling instead of freely exploring all hand actuation.The actor outputs actions for 29 body joints; a separate finger controller supplies actuation in embodiments with actuated hands.Source: §A.3 · p. 16

A.4 MOTOR-MODEL DIAGNOSTICS

The diagnostics examine interaction inputs and residual capacity, then describe reward inference and object-anchored drift correction. Findings show that object and link information matter to task success, while reward tilting and anchoring involve trade-offs.

A.4 MOTOR-MODEL DIAGNOSTICS

  • Removing link geometry leaves object error at the body-only BFM-Zero level, while removing object state yields higher object error but only about a third of the full model’s success.Without link geometry, body error is the lowest among variants; the ablations were retrained on smaller budgets, so main-model rows are reference rather than matched controls.Source: §A.4 · p. 16
  • The residual-capacity ablation varies residual branch width and depth with shared data and body prior, and compares against a from-scratch model trained on the same interaction data.The supplied diagnostics passage specifies the comparison setup but does not report its outcome.Source: §A.4 · p. 17, Table 7 · p. 17

B.1 UNIFORM AND REWARD-TILTED AGGREGATION

  • Reward inference estimates a prompt from reward-weighted backward features in a bank built from replay, and uniform aggregation projects the mean reward-feature product onto the latent sphere.The bank is fixed during evolution, while task rollouts update the skill library rather than adding bank states.Source: §B.1 · p. 17
  • Reward tilting shifts the prompt estimate toward high-reward bank states, but it is a biased heuristic whose concentration depends jointly on reward scale and β.At β = 0 it reduces to uniform aggregation; larger β concentrates weight, potentially making the estimate depend on only a few states.Source: §B.1 · p. 18, Figure 6 · p. 19

B.2 OBJECT-ANCHORED DRIFT CORRECTION

  • Drift correction uses world-frame anchor position error to create a bounded proportional velocity correction, scored over bank states to infer a correction latent blended with the reference latent.The optional threshold gain increases as the offset approaches the 0.5 m termination distance.Source: §B.2 · p. 18
  • Every tested anchor improves tracking without correction; object anchoring uses kp = 3, while anchoring both root and object lowers errors but has lower success rate.The selected plain object anchor therefore differs from the lowest-error dual-anchor variant.Source: §B.2 · p. 18, Table 8 · p. 18
  • 0.29 m at 4 s and 0.32 m by clip end are the median root drifts without correction, versus 0.16 m and 0.18 m with object-anchored correction.The corrected curve levels off after about 4 s; the curves coincide for the first second because of the 0.10 m deadband.Source: Figure 5 · p. 19

B.3 BANK SIZE, REWARD TILT, AND COST

Bank studies expose a trade-off between relevant-state coverage, reward focus, success, and projection cost. The evolution loop edits staged reward programs using rollout feedback, tunes constants, confirms candidates on separate scenarios, and retains verified programs in a skill library.

B.3 BANK SIZE, REWARD TILT, AND COST

  • The reported failure modes are broad uniform weighting across the bank and β = 30 weighting that concentrates nearly all mass on three states; peak memory grows linearly with bank size.For β = 10, weight sits on about ten states in most frames but briefly spreads over thousands during contact.Source: §B.3 · p. 19

C REWARD PROGRAMS AND THE EVOLUTION LOOP

  • The evolution loop prompts an agent to propose programs from task, scene, verifier, library, and rollout feedback; CMA-ES tunes valid candidates before comparison and confirmation.The candidate replaces the current best only when its gain on confirmation scenarios exceeds evaluation noise.Source: §B.3 · p. 20

C.1 PROGRAM SEMANTICS

  • A reward program exposes tunable constants and per-stage rewards, completion conditions, and optional local memory; live context and bank features occupy separate namespaces.The selected lift_rel example places a box on a 0.65 m table and releases it in two stages with seven constants.Source: §C.1 · pp. 20–21, Table 10 · p. 20

C.2 AGENT CONTEXT

  • The agent’s prompt varies search scope by round, moving from stage structures and transitions to reward terms and then small variations as CMA-ES tunes numbers.The verifier remains fixed, reports named criteria, and rollout feedback includes criterion margins and stage-resolved traces; code violating prompt rules is rejected before rollout.Source: §C.2 · pp. 21–22
  • The agent receives reference programs with criterion-level results and is instructed to combine stages from different references when no single program leads on every criterion.Feedback identifies the stuck criterion and stage intended to move it, helping direct revisions to that stage.Source: §C.2 · pp. 22–23

C.3 SELECTION AND THE SKILL LIBRARY

  • Search and confirmation use separate scenario grids to avoid accepting edits on the scenarios that suggested them, and the final program is reevaluated on a different set.The skill library records completed tasks, verifier pass rates, measured success, selected programs, tuned constants, and lessons; eight tasks occupy about 28 KB in the prompt.Source: §C.3 · p. 23

C.4 EVOLUTION COST ACCOUNTING

  • A five-round search executes 6.7–14.2 M environment steps across task families, averaging 9.5 M; simulation takes 16–34 minutes per round on one GPU and dominates wall-clock time.Each rollout runs 700 control steps at 50 Hz, and a three-round run uses 0.13–0.18 M language-model tokens.Source: §C.4 · p. 23

D EXPERIMENTAL PROTOCOLS AND ADDITIONAL RESULTS

The experimental protocols use a verifier to score task outcomes independently of the agent-written reward and define shared rules for replication and reference tracking. Figures illustrate reward-program evolution, basic skills, and composite tasks.

D EXPERIMENTAL PROTOCOLS AND ADDITIONAL RESULTS

  • Figures trace kick-program revisions, eight basic skills, and three composite tasks through sequences of frames.The kick programs change contact strategy across rounds; the basic skills link acquisition, transport, and placement, while the composites show multi-step object tasks.Source: Figure 7 · p. 24, Figure 8 · p. 25, Figure 9 · p. 26

D.1 VERIFIER SCORING AND REPLICATION

  • The verifier independently scores task outcomes and constraints; success requires all criteria to hold in the same rollout, with partial progress recorded as earned tiers.An inaction reference and fixed thresholds support comparisons; success is macro-averaged after averaging test scenarios within each task family.Source: §D.1 · p. 26

D.2 REFERENCE TRACKING

  • The reference-tracking protocol uses shared references, initialization, horizons, and termination rules, with early termination counting against success rate.Episodes terminate for a low root, excessive mean object-surface error, or failed interaction-consistency check.Source: §D.2 · p. 26

D.3 GOAL-CONDITIONED TASK FAMILIES

Selected programs transfer unevenly to new objects, and adapting them raises average success across the new boxes. The same push-to-mark program also generalizes to unsearched goals and starts, with success declining at the widest angles.

D.3 GOAL-CONDITIONED TASK FAMILIES

  • Task families with intermediate contacts or scene constraints are scored on the full verifier rather than terminal position error.This keeps missed contact modes visible in scoring.Source: §D.3 · p. 26
  • Reward inference for rare targets such as kicking performs best with a moderate focus on relevant bank states and enough bank states overall.Uniform averaging washes out the few relevant states, while overly sharp focus also reduces success.Source: §D.3 · pp. 26–27
  • The unchanged push-to-mark program works across unsearched target bearings and box starts, with success dropping only at the widest angles.The target is 2.5 m away; tests vary target bearing or box distance and bearing around the robot.Source: §D.3 · p. 28, Figure 10 · p. 28
  • Adapting programs to new boxes raises their average success from 36–59% unchanged to 79–89%, while transfer varies by task and object.Lifting onto a support retains 70–92% success on all three boxes, whereas push-to-mark reaches at most 25% on other boxes; adapting kicking on plastic and small boxes or pushing the plastic box around an obstacle does not help.Source: §D.3 · p. 28, Table 14 · p. 28

D.4 FIXED-PROGRAM BASELINES

  • Fixed-program baseline comparisons hold feature access, parameter bounds, rollout budget, and reward-tilt parameter β constant.Human and initial agent programs are evaluated both before and after CMA-ES.Source: §D.4 · p. 28

D.5 SEARCH-DESIGN ABLATIONS

The search-design ablations isolate components of program evolution, while library experiments test how stored programs and design lessons contribute to composite-task search. The full library performs better after search than the tested partial or absent-library alternatives.

D.5 SEARCH-DESIGN ABLATIONS

  • The search-design ablations remove one component at a time while keeping task families, controller, and verifier fixed.The variants test CMA-ES constant tuning, targeted edits versus full rewrites, multi-scenario evaluation, scene context, and multi-stage programs.Source: §D.5 · p. 29

D.6 SKILL-LIBRARY REUSE

  • Library experiments freeze the source library before target search, exclude target-task instances, and hold the controller, bank, and target-task budget fixed.The target rollouts inform within-task revision; Table 5 varies library content on composite tasks.Source: §D.6 · p. 29
  • The composite tasks combine contact modes from separate task families within one episode, and none of the tasks or their programs appears in a source library.The tasks include carrying and placing, lifting and lowering onto another object, and pushing followed by kicking.Source: §D.6 · p. 29
  • After three search rounds, the full library reaches 60.9% success on carry-place-kick, versus 48.4% without lessons, 21.9% with lessons alone, and 12.5% with neither.Stored programs without search solve only 6.3% of scenarios; the full library’s gain comes from search adapting stored programs rather than copying one as-is.Source: §D.6 · p. 29

D.7 COVERAGE OF THE PRETRAINING REFERENCES

Several evolved motions form clusters outside the pretraining reference cloud, while others remain within it. The paper also describes the perception and execution pipeline used to run evolved skills on a Unitree G1.

D.7 COVERAGE OF THE PRETRAINING REFERENCES

  • Lifting onto a table, carrying at chest height, kicking, tipping, and carrying-placing-kicking form clusters distinct from pretraining references, unlike ground-height carrying and pushing.Figure 11 compares 11 evolved skills, with 16 rollouts each, against 729 OMOMO large-box clips using body-posture motion descriptors.Source: §D.7 · pp. 29–30, Figure 11 · p. 31
  • The motion comparison uses body posture relative to pelvis and heading, with 29 link positions resampled to 128 steps and summarized by 16 DCT coefficients.Features are standardized and reduced by PCA before joint UMAP embedding.Source: §D.7 · p. 30
  • The median evolved rollout ranks at the 76th percentile in distance from its nearest reference clip; lifting onto a table, chest-height carrying, and carrying-placing-kicking rank at the 98th percentile.Ground-height carrying and pushing around a pillar rank at the 18th and 9th percentiles, respectively.Source: §D.7 · p. 30

D.8 TRANSFER AND PHYSICAL DEPLOYMENT

  • Physical deployment estimates the box pose from camera and LiDAR cues, then replays simulation-generated latent prompts while the frozen controller closes the loop.Execution starts when the object estimate is confident and pauses while it is invalid; the policy commands 29 joints at 50 Hz.Source: §D.8 · pp. 30–31

E LIMITATIONS

InterEvolve's program search is bounded by the controller's motor repertoire, reward-inference bank, and available measurements. It also incurs simulation and LLM costs that prevent real-time replanning during physical execution.

E LIMITATIONS

  • Search cannot elicit behavior the controller never learned, and is bounded by its motor repertoire, reward-inference bank, and available measurements.Source: E LIMITATIONS · p. 31
  • Simulation GPU hours and LLM latency rule out real-time replanning during physical execution.Source: E LIMITATIONS · p. 31
  • Letting accumulated experience also update the controller remains a next step.Source: E LIMITATIONS · p. 31

Related papers

Loading 2610.02196v1…