PaperlayerView exact revision on arXiv

Source-linked AI summary

Beyond Reward Hacking: Proxy Divergence Across Four Layers of a Staged Humanoid Learning Pipeline

Arunabh Bora

arXiv:2610.03196v1cs.RO

TL;DR

Robot-learning pipelines treat rewards, curriculum gates, evaluation statistics, and reference motions as proxies, though each can diverge from its target. The paper analyzes these four layers and proposes reformulations; measurements include a gate accepting absent skill and a push test misranking disturbance magnitudes.

  • Problem

    Robot-learning specification-gaming research has focused mainly on rewards, while curriculum gates, evaluation statistics, and reference motions are also proxies for competence, robustness, and achievable skill.

    Source: §I · p. 1

  • Method

    The paper derives divergence conditions for four proxy layers and proposes reformulations including first-order costs, peak and outcome gates, phase-desynchronised evaluation, feasible references, and function-preserving policy growth.

    Source: A. Second-order kernels permit… · p. 2, B. Phase aliasing in batched evaluation · p. 4, §XI · pp. 5–6

  • Results

    The averaged-error gate accepted a leg-lift with a 59° hip miss, while synchronized resets produced an 88% fall rate for a 0.5 m/s push versus 50% for 2.0 m/s.

    Source: A. Dilution · p. 3, B. Phase aliasing in batched evaluation · p. 4

  • Takeaways & Limitations

    The proposed reformulations support growing a single humanoid policy across stages without retraining while keeping the quantities steering and judging learning honest.

    Source: §XI · pp. 5–6

    All results are simulated and come from a single training seed; the mass model and actuator limits reflect the simulation rather than the intended motors.

    Source: §X · p. 5

Abstract

from arXiv · show

A reinforcement-learning (RL) pipeline for a legged robot is assembled from proxies. A reward stands in for intended behaviour, a curriculum gate stands in for competence, an evaluation statistic stands in for robustness, and a reference motion stands in for an achievable skill. The traditional view treats only the first of these as optimised against, and so locates specification failure (reward hacking) in the reward alone. I argue that all four are proxies in the same formal sense, that each has a characteristic divergence mechanism, and that each admits a reformulation that closes it. For every layer I state the traditional formulation, derive the condition under which it diverges from its target, and give the alternative: first-order (L1) costs where quadratic kernels are flat, peak and outcome statistics where curriculum gates average, gate reachability and information checks, deterministic and phase-desynchronised evaluation, curriculum state treated as part of the model, feasibility-first reference design with residual feed-forward, and function-preserving input widening that lets one policy grow instead of being retrained. The arguments are illustrated by measurements from one continuous lineage of a PPO policy for a simulated 1.91 m humanoid, grown over four stages and 13,500 iterations on a single laptop GPU. Among them, a curriculum gate built on averaged error advanced at its rate limit on every check while the skill it gated was absent, and a batched push test whose synchronised resets aliased the gait phase ranked a 0.5 m/s push as more dangerous than a 2.0 m/s one.

I. INTRODUCTION

The paper extends specification-gaming analysis beyond rewards to curriculum gates, evaluation statistics, and reference motions, treating each as a proxy that can diverge from its target. It connects this analysis to prior work on reward design, evaluation, imitation, and network growth.

II. RELATED WORK

  • Legged-robot pipelines commonly combine weighted heuristic rewards, performance-gated curricula, batched simulation evaluation, and reference-motion tracking, [2],.Related work also includes automated reward design and adaptive command curricula.Source: §I · p. 1, §II · pp. 1–2

A. Contributions

  • The paper analyzes reward, curriculum, measurement, and reference layers by specifying their divergence conditions and reformulations.Its contributions also cover gate reachability and information, evaluation bias, feasibility-first references, and function-preserving policy growth.Source: A. Contributions · p. 1

II. RELATED WORK

  • Robot-learning specification-gaming research has focused mainly on reward proxies, while this paper extends that lens to curriculum gates, evaluation statistics, and reference motions.Prior work examined reward restrictions, exploitation, proxy over-optimisation, and proxy–true-reward correlation –.Source: §I · p. 1, §II · p. 2
  • Prior deep-RL evaluation studies address fragility from seeds, aggregate statistics, and experimental design,, whereas this paper examines systematic biases that do not shrink with sample size.Source: §II · p. 2
  • The paper situates its reference-motion analysis alongside tracking and residual-control methods,, [11],,, noting that hand-authored references may lack the approximate dynamic feasibility of retargeted human motions.Source: §II · p. 2
  • Network-growth work includes function-preserving expansion, zero-output plasticity injection, and training-time growth, while loss of plasticity has also been documented for continual on-policy training such as PPO,.Source: §II · p. 2

III. SETTING

The paper formalizes a proxy as a computable stand-in for a physical target and studies it in a simulated humanoid locomotion setting. The policy and objective operate over staged task distributions and a specified robot-control setup.

III. SETTING

  • The simulated humanoid is 1.91 m and 69.35 kg, with 11 policy-controlled joints and 10 arm and neck joints on external position drives.Training used PPO on 2048 environments for 24 steps per iteration, with 50 Hz control over 200 Hz physics and a single laptop GPU.Source: §III · p. 2
  • The objective is an expectation over task distribution d and policy rollouts, with curriculum level k indexing the task distribution and reward terms defined over state and action.Source: §III · p. 2
  • A proxy is a computable quantity used to steer or judge learning in place of a physical target; divergence occurs when it reports success while the target is absent.Source: §III · p. 2

A. Second-order kernels permit first-order drift

The reward examples show how proxy design can favor drift or joint-limit behavior even when logged performance appears favorable. First-order costs and explicit coordinate support provide alternative formulations.

A. Second-order kernels permit first-order drift

  • The kernel’s zero derivative at zero lets a positive first-order drift incentive move the optimum away from rest; adding an L1 cost makes rest a local maximizer when g′(0) < λ.Source: A. Second-order kernels permit… · p. 2
  • At 0.2 m/s drift, the σ = 0.5 tracking kernel still pays 0.85, and the trained policy crept forward on every zero command.Near zero, the kernel’s loss is second order, so any first-order incentive can dominate it.Source: A. Second-order kernels permit… · p. 2
  • An L1 base-velocity cost of weight 1.5 on zero commands reduced measured drift to 0.01–0.04 m/s.The reformulation pairs saturating kernels for rest states with a first-order cost active only in that state.Source: A. Second-order kernels permit… · p. 2

B. Unconstrained coordinates settle on their bounds

  • When the reward omitted the waist coordinate, the waist stayed at its +45.00° limit for 100% of steps, while logged reward and fall rate improved relative to the corrected policy.A monotone residual dependence can drive an unconstrained coordinate to a feasible bound.Source: B. Unconstrained coordinates settle on… · p. 2
  • Every actuated coordinate must be supported by the reward through a regularizer or commanded target, and must remain inside a limit barrier.The final waist formulation used an observed target in ±0.7 rad, set to zero half the time.Source: B. Unconstrained coordinates settle on… · p. 2

V. CURRICULUM LAYER

A curriculum gate based on tracking error averaged across joints and time can accept large localized failures. Peak-error and outcome checks offer a reformulation that tests the motion’s active joints and physical result.

V. CURRICULUM LAYER

  • The traditional curriculum raises a level when a recent-episode statistic crosses a threshold, subject to a rate limit.Source: §V · p. 3

A. Dilution

  • If one joint misses by e* for fraction ρ of the motion while other errors stay at most ϵ, mean error is bounded by ρe*/|J| + ϵ.Thus a gate on mean error accepts failures whenever e* < |J|(τ − ϵ)/ρ.Source: A. Dilution · p. 3
  • With ten tracked joints, a 0.25 rad threshold, and a miss lasting about 30% of the motion, the gate accepted a 59° hip miss while mean error remained 0.06–0.07 rad.The no-fall criterion also accepted the degenerate solution of keeping both feet down.Source: A. Dilution · p. 3
  • The proposed gate checks peak error across active joints and time, averaged over episodes, and conjoins it with a contact-level outcome predicate.For the leg lift, the predicate requires the lifted foot to carry less than 1 N for a minimum swing fraction and reach a minimum reference-relative height.Source: A. Dilution · p. 3

B. Information and refusal

A curriculum gate can convey no competence information, be unreachable, or silently change the training objective when its level is not preserved across stages.

B. Information and refusal

  • Raises every 41–44 iterations against a roughly 41.7-iteration rate limit, with zero refusals, made the gate transmit no information about competence.The paper recommends logging refusals and treating a curriculum with none as unvalidated.Source: B. Information and refusal · p. 3

C. Reachability

  • A threshold is useful only if some policy can meet it; the reference’s ≈30% air-time at half amplitude fell short of the 50% gate requirement.The proposed check sets each threshold no higher than the open-loop reference’s measured performance.Source: C. Reachability · p. 3

D. Curriculum state is model state

  • Restarting a stage at 1.0 m/s after reaching 1.7 m/s changed the objective, and speed was not raised again for 2000 iterations.Terrain level also fell from 5.6 to 3.5; the paper recommends checkpointing curriculum level with model and optimizer state and comparing stages under a fixed evaluation distribution.Source: D. Curriculum state is model state · p. 3

VI. MEASUREMENT LAYER

Training-rollout outcomes can reflect exploration noise rather than policy skill, while synchronized resets can bias batched disturbance tests by gait phase. The paper proposes deterministic outcome reporting and independently randomized disturbance times.

A. Outcomes manufactured by exploration noise

  • A foot-height metric read 61–69% of target during training although the deterministic policy produced none, because ankle exploration noise tipped the loaded foot onto its toe.The proposed reformulation reports outcomes for the deterministic policy and defines height to count only when contact force is below 1 N.Source: A. Outcomes manufactured by… · p. 3

B. Phase aliasing in batched evaluation

  • Synchronized resets yielded an 88% fall rate for a 0.5 m/s push versus 50% for a 2.0 m/s push from the same side.The bias is independent of robot count; independently drawing disturbance times over a 1 s window made the magnitude series monotonic and removed the asymmetry.Source: B. Phase aliasing in batched evaluation · pp. 3–4
  • With phase-randomized timing, lateral and sagittal pushes had near-equal fall rates: 30% against 26% while walking.The paper draws each robot’s disturbance time independently over a window longer than one step cycle.Source: B. Phase aliasing in batched evaluation · p. 4, Table I · p. 4

C. Quantifier errors in aggregation

The statistic used to support a robustness claim must match its quantifier: a claim about every direction requires the minimum across directions.

C. Quantifier errors in aggregation

  • A harness using the largest magnitude with zero falls in any direction reported a clean result despite a 38% fall rate in another direction.For a claim that the policy survives in every direction, the paper says to compute the minimum over directions.Source: C. Quantifier errors in aggregation · p. 4

VII. REFERENCE LAYER

A reference motion may be infeasible even when it respects joint limits, and reward shaping cannot guarantee that the resulting feasible motion preserves its intended contact sequence. The paper proposes feasibility checks and residual feed-forward for reference-based skill learning.

VII. REFERENCE LAYER

  • If a reference lies outside the set of trajectories the robot can execute without falling, reward optimization selects a feasible trajectory that need not preserve its contact sequence.Additional shaping changes the reward metric but cannot make an infeasible reference feasible.Source: §VII · p. 4
  • A hand-authored single-support leg-lift pose placed the centre of mass 0.21 m beside the support foot despite satisfying every joint limit.Seven reward, initialization, and exploration changes did not produce single support in the deterministic policy.Source: §VII · p. 4
  • The proposed feasibility-first check tests the centre of mass against the support polygon at each keyframe, then executes the repaired reference open-loop.A 20° support-leg roll reduced the offset to 0.02 m, within the 27° limit; the open-loop check tests whether the intended contact sequence is reproduced for part of the motion.Source: §VII · p. 4
  • Open-loop feed-forward on the repaired reference kept the foot clear throughout, while reward-driven discovery held the support-leg joint at −1 to −3° against a −20° target.The reference is fed forward so the policy learns a correction rather than rediscovering the motion from reward.Source: §VII · p. 4

VIII. POLICY GROWTH

Function-preserving input widening lets a policy accept new observations without restarting from scratch. Zero-padding preserves its behavior at handover while keeping the added inputs learnable.

A. Function-preserving input widening

B. Split authority with observed exogenous motion

The policy lineage illustrates how observed exogenous motion, staged training, and proxy audits relate to policy growth. The discussion argues that validation should follow the channels engineers trust, while noting the study’s simulation and single-seed scope.

B. Split authority with observed exogenous motion

C. Empirical lineage

  • S3 reduced peak tilt during the fastest arm sequence from 0.34 to 0.21 and base drift from 0.37 m to 0.14 m.Source: C. Empirical lineage · p. 5

IX. TRADITIONAL VERSUS PROPOSED: SUMMARY

  • Table III serves as a proxy-audit checklist: for each quantity steering or judging learning, identify its traditional formulation, divergence mechanism, and proposed alternative.Source: §IX · p. 5

X. DISCUSSION

  • Curriculum dilution and noise-manufactured outcomes were selected for by optimization, while phase aliasing and quantifier error required no optimization.Source: §X · p. 5
  • The reformulations require no additional training, but results remain limited to simulation and one training seed, with simulated mass and actuator assumptions.The stated costs include one open-loop rollout per reference and curriculum level, one deterministic replay per reported outcome, randomized evaluation timing, and checkpoint-format changes.Source: §X · p. 5

XI. CONCLUSION

The conclusion treats reward, curriculum, evaluation, and reference motion as proxies that can diverge from their targets, and presents reformulations for each. Together with split authority and function-preserving input widening, these support growing one humanoid policy across stages without retraining.

XI. CONCLUSION

  • Curriculum gates, evaluation statistics, and reference motions can diverge from their targets like rewards, and the conclusion specifies reformulations for each proxy layer.Source: §XI · p. 5
  • Table III presents traditional formulations, divergence mechanisms, and proposed alternatives.Source: Table III · p. 6
  • Function-preserving input widening and split authority let a single humanoid policy grow across stages without retraining.Source: §XI · p. 6
  • Figures are regenerated from original training logs; environment configurations and reward terms are available from the author on request.Source: REPRODUCIBILITY · p. 6

Related papers

Loading 2610.03196v1…