PaperlayerView exact revision on arXiv

Source-linked AI summary

Active Reachability Learning for Humanoid Recoverability Monitoring

Zeyuan Feng, Baljeet Singh, Sarthak Ranjeet Kaingade, Somil Bansal, Bryan Trinh

arXiv:2610.33317v1cs.ROcs.SYeess.SY

TL;DR

Reliable humanoid failure monitors need to identify unrecoverable falls early, but mapping recoverable states in contact-rich systems is difficult. The paper combines uncertainty-guided reach-avoid exploration with TD+MC value learning; coverage improves on stochastic benchmarks, and the humanoid monitor performs robustly on aggressive OOD pushes.

  • Problem

    Reliable monitors must detect impending failures early enough to trigger a fallback, but uniform sampling or teleporting can corrupt simulation data when mapping recoverable states in contact-rich systems.

    Source: §I · p. 1

  • Method

    The framework combines TD+MC value learning with uncertainty-guided reach-avoid exploration that targets OOD states while avoiding confidently unrecoverable samples.

    Source: §I · p. 2

  • Results

    Reach-avoid exploration achieves higher coverage than baselines on both stochastic benchmarks; the humanoid monitor reaches F1 0.918±0.009 on aggressive OOD pushes.

    Source: A. Case Studies: DI and 3D Dubins · p. 5, B. Unitree G1 Humanoid: Open-Loop… · p. 7

  • Takeaways & Limitations

    The framework scales to a 29-DoF humanoid and qualitatively demonstrates zero-shot performance on G1 hardware.

    Source: §V · p. 8, C. Runtime Monitor and Hardware… · p. 8

    The framework currently reacts rather than proactively to OOD inputs, its learned solutions remain uncalibrated, and maximum coverage is bounded by the explorer’s action space.

    Source: §V · p. 8

Abstract

from arXiv · show

Deploying humanoid task policies requires reliable runtime monitors that anticipate unrecoverable falls in time to trigger a fallback. We formulate this task as learning the stochastic, finite-horizon recoverability envelope of a given fallback policy. Synthesizing this monitor is challenging: existing methods assume training data naturally covers the recoverability envelope, yet acquiring such data remains a major bottleneck for contact-rich systems. Consequently, they recover only a fraction of the true envelope, leaving them prone to optimistic hallucinations for out-of-distribution (OOD) inputs. We resolve this via a unified framework that iterates between value learning and active exploration. Data exploration is reframed as a reach-avoid problem, where an uncertainty-guided explorer systematically uncovers unseen states while avoiding confidently unrecoverable regions. This loop is further supported by a combined Temporal Difference and Monte Carlo objective for accurate in-distribution value prediction and reliable uncertainty estimation. By explicitly treating the remaining high-uncertainty regions as unrecoverable, the monitor mitigates optimistic OOD hallucinations. Evaluated on stochastic benchmarks and a 29-DoF Unitree G1 humanoid, our approach drastically improves monitor coverage and accuracy and successfully transfers zero-shot to real-world hardware.

I. INTRODUCTION

Reliable humanoid monitoring requires anticipating unrecoverable falls, but contact-rich dynamics make recoverability difficult to map from rollout data. The paper frames monitoring as learning the finite-horizon recoverability envelope of a fixed fallback policy.

I. INTRODUCTION

  • Contact-rich, underactuated humanoids need runtime monitors that detect impending unrecoverable falls early enough to trigger a fallback policy.The monitor is framed as learning an empirical finite-horizon recoverability envelope for that policy.Source: §I · p. 1
  • Rigid-body contact makes uniform sampling and teleportation unreliable, while manually seeded episodic rollouts leave the recoverability envelope undercovered and encourage optimistic OOD predictions.Minor kinematic offsets can cause penetration and corrupt simulation data; slipping and firm contacts can also produce different outcomes from nearly identical states.Source: §I · p. 1
  • Existing CBFs and stochastic reachability solvers face whole-body scaling limits, while PINN reachability relies on uniform sampling that produces irrelevant or corrupted contact-rich samples.RL-based rollouts respect physical constraints but nominal initial-state distributions cover only a narrow fraction of the envelope, limiting operation space and worsening OOD optimism.Source: §I · pp. 1–2

A. System Dynamics

  • The system is modeled as stochastic under a fixed fallback controller, with data collected through policy rollouts from a manually defined initial-state distribution.During exploration, an explorer may select controls and apply artificial disturbances; subsequent rollouts may optionally start from replay-buffer states.Source: A. System Dynamics · p. 2

B. Liveness Value Function and Safe-Set Coverage

The section defines recoverability through a liveness value and focuses learning on the recoverable envelope rather than deeply unrecoverable states. Its humanoid instantiation uses a balancing policy and a finite recovery horizon, with reach-avoid formulations available for additional deployment constraints.

B. Liveness Value Function and Safe-Set Coverage

  • The liveness value is the expected minimum target function along future trajectories; with a binary target indicator, it equals the probability of failing to reach the target set.Lower values indicate more recoverable states.Source: B. Liveness Value Function and… · p. 2
  • Accurate mapping of the ϵ-liveness set supports early alerts before states drift OOD, while unexplored regions should remain conservatively classified as unrecoverable.The section notes that episodic rollouts alone may not fully cover this set for underactuated systems.Source: B. Liveness Value Function and… · p. 2

C. Humanoid Balancing Instantiation

  • For humanoids, the fallback policy balances under stochastic observation noise and domain randomization, and the target set consists of near-stationary states with bounded velocities and canonical torso height.If recoverable trajectories attain their running minimum within K steps, the infinite-horizon and fixed-horizon values coincide.Source: C. Humanoid Balancing Instantiation · p. 2
  • The practical liveness formulation assumes recovery without collapse and allows stationary behavior indefinitely; temporary-collapse recovery or collision-avoidance requirements call for a reach-avoid formulation.The monitor learns the time-indexed value for boundary anchoring and a temporal curriculum, while querying the value at the initial time.Source: C. Humanoid Balancing Instantiation · pp. 2–3

III. METHOD

The method alternates value learning and active discovery because the recoverability boundary cannot be manually specified for a high-dimensional contact-rich humanoid. It returns a value function and uncertainty estimator when exploration stops finding meaningful new states.

A. Overview and Active Learning Loop

  • The loop begins with rollouts of the fallback policy from an initial-state distribution, then fits a value function and uncertainty estimator on the collected data.The value-learning phase aims to represent known states accurately before using uncertainty to guide exploration.Source: A. Overview and Active Learning Loop · p. 3
  • An uncertainty-rewarded disturbance explorer starts from known data states and seeks novel OOD states that are at most marginally unrecoverable.Newly discovered seeds are used for fallback-policy rollouts, which augment the dataset before the next learning round.Source: A. Overview and Active Learning Loop · p. 3
  • Exploration terminates when the explorer fails to find enough meaningful new states, and the resulting monitor triggers the balancing policy when value or uncertainty exceeds its threshold.The fallback is run until a stationary state is reached.Source: A. Overview and Active Learning Loop · p. 3

B. In-Distribution Value Function Learning

This section trains a finite-horizon recoverability value with multi-step temporal-difference targets and Monte Carlo supervision. Bootstrapped TD targets can introduce conservative bias, while MC targets provide unbiased anchors at higher rollout cost.

B. In-Distribution Value Function Learning

  • The learned discrete-time value estimates recoverability under the fallback policy over a finite horizon, and its induced deployment liveness set uses the initial-time threshold ϵV.That deployment set is distinct from the broader ϵ-liveness set mapped during synthesis.Source: B. In-Distribution Value Function… · p. 3
  • Multi-step TD-λ targets combine an observed running minimum with a bootstrap, using a truncated-geometric window capped by M steps or the horizon boundary.The TD loss is a mean-squared prediction residual.Source: B. In-Distribution Value Function… · p. 3
  • Bootstrapped m-step targets overestimate the value even with an exact bootstrap, shrinking the liveness set; this bias vanishes at the full horizon, where the MC estimate is unbiased but high-variance.The bias comes only from the bootstrapped tail.Source: B. In-Distribution Value Function… · p. 3
  • Long-horizon MC targets are expensive, so training uses a small set of unbiased MC anchors alongside abundant TD targets for local temporal consistency.The initial dataset combines TD rollouts from the initial-state distribution with MC rollouts initialized from that distribution and TD-visited states; the time curriculum expands sampled steps toward the full horizon.Source: B. In-Distribution Value Function… · pp. 3–4

C. Uncertainty- and Reachability-Based Explorer

The explorer frames data augmentation as a backward reach-avoid problem: from replay-buffer states, it seeks high-uncertainty states while avoiding confidently unrecoverable ones. Alternating exploration and fallback-policy rollouts adds novel seeds and fresh value-learning data.

C. Uncertainty- and Reachability-Based Explorer

  • From replay-buffer states, the explorer targets high epistemic uncertainty while avoiding states predicted to be unrecoverable with low uncertainty.The exploration policy is trained as a Backward Reach-Avoid Tube problem.Source: C. Uncertainty- and Reachability-Based… · p. 4
  • A value-function ensemble or Random Network Distillation estimates epistemic uncertainty and defines the high-uncertainty target set.Source: C. Uncertainty- and Reachability-Based… · p. 4
  • The avoid condition requires both predicted failure above ε and uncertainty below εσ; the monitor trigger εV must be substantially smaller than the exploration stopping value ε.Continuous target and avoid functions provide dense gradients, even when the high-uncertainty target set is effectively empty.Source: C. Uncertainty- and Reachability-Based… · p. 4
  • The explorer uses a time-independent policy with a tuned horizon: short horizons are myopic, while long horizons can concentrate seeds at distant uncertainty peaks.The time-independent policy simplifies exploration at a slight cost to optimality.Source: C. Uncertainty- and Reachability-Based… · p. 4
  • Rollouts can exceed the training horizon, but seed extraction is capped at five per trajectory; each seed launches a fallback-policy rollout that adds TD and MC data.Learning and exploration alternate until a round yields fewer than Nmin novel seeds or the round budget is reached.Source: C. Uncertainty- and Reachability-Based… · p. 4

IV. RESULTS

The evaluation uses stochastic planar benchmarks for ground-truth comparisons and a Unitree G1 humanoid for high-dimensional synthesis and deployment tests. It specifies coverage, accuracy, and sensitivity comparisons, but the supplied passages do not report their outcomes.

IV. RESULTS

  • The framework is evaluated on a stochastic 4D double integrator and 3D Dubins car, then scaled to a Unitree G1 and tested with a walking policy in simulation and on hardware.Uniform grid sampling is used only for offline benchmark evaluation; synthesis begins from a narrow initial distribution.Source: §IV · p. 4
  • Evaluation reports recall, precision, F1, MSE, cumulative exploration seeds, confident coverage, and GPU wall time against empirical liveness labels from 1,000 Monte Carlo rollouts.Confident coverage counts true recoverable states covered by predictions with low epistemic uncertainty.Source: §IV · p. 4
  • The explorer comparison includes unbiased and biased random-action exploration, each paired with an ensemble or RND uncertainty estimator.The TD+MC objective is compared with TD(λ)-only and time-agnostic TD; sensitivity sweeps vary the MC anchor count and explorer horizon.Source: §IV · pp. 4–5
  • Fig. 2 shows value heatmaps and level-set contours for DI and Dubins in the x-y plane, with projected initial-distribution boxes and target-set circles.Source: Figure 2 · p. 5

A. Case Studies: DI and 3D Dubins

Across the DI and 3D Dubins studies, reach-avoid exploration with RND improves recoverability coverage over random exploration, while achievable coverage depends on the explorer’s control authority. The section also examines TD+MC value learning and the sensitivity of exploration to MC sample count and horizon.

Q1: Liveness-set coverage and design choices

  • 0.656 coverage on Dubins and 0.941 on DI with reach-avoid exceed random exploration, which reaches 0.267 and 0.408 on Dubins and about 0.86 on DI, respectively.The reach-avoid results are obtained in fewer active-learning rounds; RND + reach-avoid is the default configuration.Source: A. Case Studies: DI and 3D Dubins · p. 5

A. Case Studies: DI and 3D Dubins

  • Residual coverage gaps remain and are attributed to inherent system underactuation rather than explorer inefficiency; in the DI system, default-configuration misses are primarily recoverable states moving rapidly toward the target within the control-flip zone.The stochastic PD controller output is flipped at px ≥ 1; both systems use a 2 s horizon with ∆t = 0.02 s, and metrics aggregate over 5 random seeds.Source: A. Case Studies: DI and 3D Dubins · p. 5

Q1: Liveness-set coverage and design choices

  • 0.921 coverage and 0.970 F1 with added lateral-velocity control raise Dubins3D-Explore coverage from 0.656, showing the explorer’s action space bounds achievable coverage.The standard Dubins system leaves part of the recoverable set unreachable from the initial distribution without crossing unrecoverable regions.Source: A. Case Studies: DI and 3D Dubins · p. 5

Q2: Accuracy—TD+MC vs. TD-only

  • The TD+MC objective outperforms TD-only across all reported DI metrics, while incomplete coverage can make TD-only extrapolate overly optimistic OOD values.Under weaker ensemble + unbiased exploration, TD-only instead collapses to an extremely noisy and conservative approximation; MC anchoring stabilizes training.Source: A. Case Studies: DI and 3D Dubins · pp. 5–6, Table I · p. 5

Q3: Sensitivity to NMC and explorer horizon

B. Unitree G1 Humanoid: Open-Loop Metrics

The G1 study evaluates finite-horizon recoverability monitoring in a 69D state space, using active exploration and validation against both nominal and disturbance-generated OOD states. The learned value estimator performs strongly on both sets, with substantial OOD recall gains across active-learning rounds.

B. Unitree G1 Humanoid: Open-Loop Metrics

  • The study certifies a Unitree G1 balancing policy in a 69D state space over a 3-second liveness horizon.The representation includes torso state and full-body joint configurations and velocities; hardware estimates bodyframe velocities and height using a lightweight LSTM.Source: B. Unitree G1 Humanoid: Open-Loop… · p. 6
  • The active-learning loop uses an RND plus reach-avoid explorer over a 0.4-second training horizon, with NMC = 32 and a compact 7D action space.Explorer actions are learned hip-joint offsets combined with a longitudinal torso push bounded by ±50 N; the study trains both a continuous liveness value and a failure-probability estimator.Source: B. Unitree G1 Humanoid: Open-Loop… · pp. 6–7
  • OOD test states follow a 1-second, 100 N push to a random body link, and each captured state is labeled using 100 Monte Carlo rollouts.The study bins and equalizes states by estimated post-disturbance difficulty, ∥v∥+ 0.25 ∥ω∥.Source: B. Unitree G1 Humanoid: Open-Loop… · p. 7
  • F1 reaches 0.957 ± 0.001 on validation and 0.918±0.009 on aggressive OOD pushes, with confident coverages of 0.944 and 0.815, respectively.After 30 active-learning rounds, OOD recall rises from 0.672 to 0.893 overall and from 0.425 to 0.871 in the [3.0, 4.0) bin.Source: B. Unitree G1 Humanoid: Open-Loop… · p. 7, Table III · p. 7

C. Runtime Monitor and Hardware Deployment

Closed-loop tests assess monitored walking under impulses for healthy and degraded policies. The TD+MC monitor reduces failures while avoiding excessive triggering, and the paper reports qualitative zero-shot transfer to G1 hardware subject to stated dynamics assumptions.

C. Runtime Monitor and Hardware Deployment

  • Tests apply multidirectional impulses of tiered magnitudes to a random body link during continuous walking, varying maximum duration from 0 to 0.30 seconds.Each sweep includes 4,096 eight-second trials, and the monitor triggers on predicted unrecoverability or uncertainty.Source: C. Runtime Monitor and Hardware… · p. 7
  • The TD+MC monitor has the lowest failure rates across disturbance bands, avoids false alarms during undisturbed walking or gentle pushes, and predicts 100% of eventual falls in advance.Even under the most severe disturbance band, high-confidence failure predictions precede torso-ground contact by an average of 0.58 s.Source: C. Runtime Monitor and Hardware… · p. 7, Figure 6 · p. 7
  • The monitor consistently reduces failure rates across disturbance tiers, especially for the degraded walker.Table IV reports closed-loop failure and trigger rates for healthy and degraded policies; residual monitored failures are attributed to stochastic simulation conditions.Source: C. Runtime Monitor and Hardware… · p. 7, Table IV · p. 7
  • The paper qualitatively demonstrates zero-shot performance on G1 hardware, with lower empirical deployment failure probability expected if hardware dynamics remain within randomization bounds and sensor noise is more lenient.The expectation is conditional on those hardware and sensor assumptions.Source: C. Runtime Monitor and Hardware… · p. 8

V. CONCLUSION AND LIMITATIONS

The conclusion presents TD+MC value learning with uncertainty-guided reach-avoid exploration for stochastic policy recoverability, while identifying calibration, reactive OOD handling, and explorer action space as limits. The authors also call for further study on harder task classes.

V. CONCLUSION AND LIMITATIONS

  • The framework combines TD+MC learning with uncertainty-guided reach-avoid exploration to learn policy recoverability under stochastic dynamics.The authors report scaling to a 29-DoF humanoid and describe the work as the first effort to comprehensively map the liveness-set envelope for high-dimensional, contact-rich systems.Source: §V · p. 8
  • The framework currently responds reactively rather than proactively to OOD inputs, and its learned solutions remain uncalibrated.Maximum achievable coverage is bounded by the explorer’s action-space choice.Source: §V · p. 8
  • Efficacy on complex dexterous manipulation, long-horizon tasks, and extremely dynamic tasks requires further investigation.The authors also identify extending the method to general reach-avoid settings and using learned reachability values to improve policy synthesis as future directions.Source: §V · p. 8

APPENDIX

The appendix specifies two stochastic control benchmarks: a planar double integrator and a Dubins car. Both use target-reaching settings, with the Dubins3D-Explore variant adding lateral exploration control.

APPENDIX

  • The planar double integrator has position and velocity state, bounded state space, noisy saturated P-control, and outward-biased, distance-scaled disturbances.Its target is the set of states with position norm at most 0.25 m; the controller is flipped for states with px ≥1.0.Source: APPENDIX · p. 8
  • The Dubins car is a constant-speed unicycle with heading control and biased disturbances; Dubins3D-Explore adds a bounded lateral push perpendicular to heading.Its target set matches the double-integrator benchmark.Source: APPENDIX · pp. 8–9

Related papers

Loading 2610.33317v1…