PaperlayerView exact revision on arXiv

Source-linked AI summary

Arrive and Survive: Scaling Safe Goal-Conditioned Policy Learning from One-Bit Failure Signals

Guopeng Li, Yiyang Duan, Yiru Jiao, Chengcheng Xu

arXiv:2608.26571v1cs.LGcs.RO

TL;DR

Failure termination removes discounted future-goal occupancy, causing established CRL to overestimate goal-reaching values and reinforce near-failure behavior. Safe-CRL corrects critic and actor objectives using survival mass estimated from one-bit failure signals. Across twelve failure-prone navigation and locomotion tasks, it improves survival and goal-reaching performance while preserving depth scalability.

  • Problem

    Established CRL renormalizes pre-failure future goals after failure termination, omitting removed survival mass and inducing overestimation bias.

  • Method

    Safe-CRL combines mass-weighted InfoNCE with log-survival-mass actor scoring, using a one-bit failure signal to estimate survival mass.

  • Results

    Safe-CRL achieves higher mean time at goal and mean survival time than Scaling-CRL in all 12 tasks.

  • Takeaways & Limitations

    Safe-CRL preserves CRL’s depth scalability while enabling failure-aware self-supervised goal-conditioned policy learning.

  • Takeaways & Limitations

    Safe-CRL targets failure-terminated goal-conditioned control and does not directly address non-terminating safety costs or constrained-RL objectives.

Abstract

from arXiv · show

Contrastive reinforcement learning (CRL) scales effectively in goal-conditioned tasks by casting policy learning into a self-supervised contrastive objective. However, in a failure-terminated Markov decision process, established CRL considers pre-failure future goals only when constructing positive samples, without accounting for the probability mass removed by failure termination. Our theoretical analysis shows that this omission induces a systematic overestimation bias in goal-reaching values. Consequently, near-failure trajectories provide disproportionately strong supervision of success despite retaining little future occupancy. Unsafe actions can thereby be reinforced through catastrophic failure bootstrapping, leading to failed policy learning and unsustainable goal-reaching behaviours. To address this problem, we introduce two minimal yet strong corrections: mass-weighted InfoNCE corrects the overweighting of short surviving futures in critic learning, and a log-survival-mass score restores the missing survival mass in policy optimization. The resulting method, Safe Contrastive Reinforcement Learning (Safe-CRL), requires only the one-bit signal provided by failure termination to scale safe goal-conditioned policy learning. Across twelve failure-prone robot navigation and locomotion tasks, Safe-CRL consistently improves survival and substantially outperforms the Scaling-CRL baseline in goal-reaching performance. Additionally, deep Safe-CRL policies exhibit complex failure-avoidance behaviours. This study completes the CRL theory under failure termination and provides a scalable safe RL framework. The code is available via https://github.com/RomainLITUD/safe-crl.

1 Introduction

CRL provides scalable self-supervision for goal-conditioned RL, but failure termination removes future occupancy that established CRL renormalizes away. Safe-CRL addresses the resulting bias using two corrections and improves survival and goal-reaching performance.

  • CRL converts sparse goal-reaching rewards into dense self-supervision through goal relabelling and contrastive learning.
  • Failure termination removes future states for relabelling, while established CRL renormalizes the remaining pre-failure goals to unit mass.
  • The missing survival mass causes systematic value overestimation, catastrophic failure bootstrapping, and unsustainable goal reaching.
  • The central question is how CRL can learn scalable, safe goal-conditioned policies using only a one-bit failure signal.
  • Safe-CRL combines mass-weighted InfoNCE with log-survival-mass correction and improves survival and goal-reaching performance across failure-prone environments.

2 The missing survival mass in CRL

Under failure termination, valid future-goal occupancy becomes a sub-probability measure, but established CRL normalizes away its missing survival mass. This creates critic and actor distortions that overvalue short near-failure trajectories and can mis-rank actions.

  • CRL preliminaries: CRL relabels discounted future achieved goals as positives and uses other minibatch goals as contrastive negatives.
  • Failure-aware occupancy: Failure-aware future-goal occupancy excludes post-failure states and is equivalent to the goal-reaching Q-function.
  • Overestimation bias: Established CRL renormalizes each trajectory’s remaining pre-failure occupancy to unit mass, discarding the discounted occupancy removed by failure.
  • Overestimation bias: The resulting trajectory-level amplification factor 1/Ẑτ grows as valid trajectory length decreases, producing non-negative overestimation bias.
  • Critic and actor effects: The two distortions arise from trajectory-wise normalization in critic learning and contrastive normalization in actor evaluation.
  • Critic and actor effects: Short near-failure trajectories receive too much critic weight, while actor scoring omits action-dependent survival mass and can similarly score unsafe actions.

3 Safe-CRL by restoring the survival mass

Safe-CRL restores the survival information lost by established CRL with mass-weighted critic training and log-survival-mass actor scoring. A lightweight Z-encoder estimates survival mass from one-bit failure outcomes while retaining the original achieved-goal space.

  • Critic-side correction: Mass-weighted InfoNCE multiplies each complete row-wise contrastive loss by its realized survival mass.
  • Critic-side correction: At the population level, mass-weighted InfoNCE recovers the normalized conditional distribution associated with failure-aware occupancy.
  • Actor-side correction: Adding log Z to the calibrated contrastive score restores missing survival mass and recovers the action ranking of the failure-aware Q-value.
  • Implementation: Safe-CRL trains a lightweight Z-encoder with independently sampled discounted horizons and binary survival labels.
  • Implementation: The implementation adds log Zψ to actor scoring while retaining Scaling-CRL’s score regularization and other architecture components.
  • Design choice: Safe-CRL preserves the achieved-goal outcome space instead of modeling failure as an explicit contrastive outcome.

4 Experiments

The experiments evaluate Safe-CRL on 12 failure-prone robot navigation and locomotion tasks, using Scaling-CRL as the primary controlled baseline. They compare goal attainment, persistence, and safety under matched training settings.

  • Benchmark and environments: Safe-CRL is evaluated on 12 failure-prone Brax robot navigation and locomotion tasks.Point Goal and Car Goal use Safety-Gymnasium environments; Ant and Humanoid tasks are adopted from Scaling-CRL.
  • Benchmark and environments: Failure is triggered by task-specific hazards, obstacles, gremlin regions, falls, or pitfalls, while goal respawning occurs only in Point Goal and Car Goal.Maze-wall contact itself is not a failure, and the remaining locomotion environments use fixed commanded goals per episode.
  • Baselines and implementations: Scaling-CRL is the primary baseline because Safe-CRL directly extends its contrastive objective under failure termination.An additional SAC-HER baseline is reported in the appendix, while the main comparison controls other training settings.
  • Evaluation metrics: The evaluation reports time at goal, survival time, and goal coverage as complementary metrics.Time at goal reflects attainment and persistence; survival time measures steps before failure or the time limit; goal coverage measures episodes reaching the commanded goal.

5 Results and discussion

Safe-CRL improves goal persistence and survival across the benchmark while retaining depth scalability, learning survival-mass estimates with low overhead, and producing diverse failure-avoidance behaviours. The largest benchmark gains occur in the more difficult Humanoid settings.

  • Benchmark results: Safe-CRL achieves higher mean time at goal and survival time than Scaling-CRL in all 12 tasks.Time at goal rises from 276.8 to 684.0 steps on Ant Goal, from 22.2 to 266.4 on Humanoid Goal, and from 8.2 to 474.1 on Humanoid Big Pitfall.
  • Benchmark results: Goal coverage increases in seven tasks and decreases in five, while time at goal still increases substantially in the affected environments.This indicates that the time-at-goal gains are not simply obtained by sacrificing goal-reaching coverage.
  • Failure modes: Scaling-CRL often reaches goals but fails to remain safely, whereas Safe-CRL mitigates this unsustainable goal reaching and catastrophic failure bootstrapping.The contrast is especially pronounced in Humanoid tasks, where Scaling-CRL survival remains low and goal metrics grow more slowly.
  • Depth scalability: Time at goal and survival time generally improve with network depth, with harder Humanoid tasks benefiting up to 256 layers.Eight-layer Safe-CRL matches or outperforms 64-layer Scaling-CRL across the evaluated depth-scaling tasks.
  • Survival-mass estimation: Predicted survival mass and evaluated survival time show closely aligned trends during training on four Humanoid tasks.The alignment indicates that the Z-encoder tracks failure-dependent trajectory quality.
  • Computational overhead: The default Z-encoder adds approximately 4.2% parameters and 2.8% average wall-clock training time.The reported increases are measured with 64-layer actor and critic networks.
  • Failure avoidance: Deep Safe-CRL policies learn diverse behaviours including braking, replanning, recovery, orientation adjustment, detours, and slowing near hazards.These behaviours are observed in representative Point, Humanoid, and Ant rollouts despite the one-bit failure signal.

6 Ablations

Ablations show that the log-survival-mass correction drives most of the improvement, while the theory-derived coefficient β = 1 increases survival without universally maximizing goal-reaching metrics.

  • Core component ablations: The log-Z-only variant drives the dominant ablation improvement, while MW-InfoNCE alone produces modest changes relative to Scaling-CRL.On Humanoid Goal, log-Z-only increases time at goal from 13.8 to 288.7 steps and survival time from 113.9 to 397.0 steps; MW-InfoNCE alone reaches 28.3 and 146.1 steps, respectively.
  • Core component ablations: Table 1 isolates the contributions of mass-weighted InfoNCE and the log-survival-mass correction.Results are averaged over the final 10% of training across three random seeds, using 16-layer actor and critic networks, or 8 layers for Car Goal.
  • Log-Z coefficient: Increasing β generally increases survival time, while its effect on goal-reaching performance depends on the environment.The tested values are 0.2, 1.0, and 5.0.
  • Log-Z coefficient: Safe-CRL fixes β = 1 because the failure-aware occupancy derivation determines it, rather than tuning it as a safety–goal trade-off parameter.The ablation does not assume this value maximizes every empirical metric in every environment.

7 Conclusion

The paper identifies missing survival mass as the source of overestimation bias in failure-terminated CRL and introduces two corrections forming Safe-CRL. Across failure-prone navigation and locomotion tasks, Safe-CRL improves survival and goal-reaching performance while retaining CRL’s depth scalability, but its scope remains limited to failure-terminated goal-conditioned control.

  • Safe-CRL combines mass-weighted InfoNCE for critic learning with log-survival-mass correction for policy optimization.
  • Across failure-prone navigation and locomotion tasks, Safe-CRL improves survival and goal-reaching performance while preserving CRL’s depth scalability.
  • Safe-CRL is designed for failure-terminated goal-conditioned control and does not directly address non-terminating safety costs or constrained-RL objectives.

8 Related work

Related work spans scalable contrastive goal-conditioned learning, methods addressing stochastic or horizon-dependent reachability, survival-based formulations, and constrained safety objectives. The paper positions its contribution as identifying a previously unreported overestimation bias from discarding survival mass in failure-terminated CRL and derives corrections for it.

  • Goal-conditioned RL uses hindsight future-state relabelling to transform sparse goal-reaching experience into dense self-supervision, while CRL learns through contrastive density-ratio estimation.
  • DDC separates horizon-dependent goal-reaching quantities from reachability probability, while USHER addresses hindsight-selection bias under stochastic dynamics.
  • SVL and SRL apply survival-analysis ideas to goal-conditioned learning, including right-censoring and dwell-time for stable long-horizon behaviour.
  • CMDP-based safe RL methods optimize task reward under explicit expected-cost constraints, unlike this study’s failure-termination setting.
  • The paper identifies a systematic overestimation bias caused by normalizing pre-failure futures and discarding survival mass, then derives mass-weighted critic learning and log-survival-mass actor correction.

A.2 Proof of Proposition 2

The proof shows that trajectory-wise normalization of valid pre-failure futures distorts the population occupancy distribution. Weighting each contrastive row by realized survival mass cancels this distortion, while the resulting minibatch estimator requires no within-batch weight normalization.

  • Established CRL normalizes valid pre-failure future goals separately for each trajectory before averaging them across trajectories.
  • Mass-weighting each trajectory’s complete InfoNCE row by its realized survival mass cancels trajectory-wise normalization and recovers the conditional shape of failure-aware occupancy.
  • Mass weighting corrects the critic’s trajectory-wise normalization distortion but leaves total survival mass absent because the contrastive distribution remains normalized.
  • The unnormalized minibatch average is a direct Monte Carlo estimator of the population mass-weighted contrastive risk, so within-minibatch weight normalization is unnecessary.
  • The self-normalized alternative is not the same finite-minibatch estimator as Safe-CRL’s objective.
  • Negative goals remain sampled from the ordinary anchor-independent minibatch goal marginal and are not reweighted.

A.3 Proof of Corollary 1

The proof derives the calibrated contrastive score and shows that adding log survival mass restores the action-dependent occupancy factor omitted by normalized contrastive learning. It also specifies the survival-target estimator and assumptions governing replay, support, censoring, and zero-mass trajectories.

  • Row-wise InfoNCE scores can contain action-dependent offsets, so calibration is required before comparing candidate actions.
  • Adding log survival mass to the calibrated score restores the total mass removed by contrastive normalization, with coefficient 1.0.
  • The correction follows from the multiplicative factorization of failure-aware occupancy rather than from an introduced trade-off term.
  • The Z-encoder estimates survival mass using a one-sample binary target Y = 1{Tf > H} with H sampled from the discounted geometric distribution.
  • Under time-limit truncation, horizons without an observed failure are treated as survived, yielding a finite-support approximation without explicit right-censoring correction.
  • The theoretical analysis uses a replay-induced trajectory distribution consistently across definitions and requires a supported anchor-independent negative proposal.
  • Zero-mass anchors receive zero weight in the population mass-weighted objective, and implementation masks rows with no valid future states.

B Environments

The evaluation covers 12 failure-prone robot-control tasks in Brax, using one-bit failure termination signals and task-specific navigation and locomotion failure mechanisms. Safe-CRL is compared primarily with Scaling-CRL and additionally with SAC-HER.

  • Evaluation setting: Safe-CRL is evaluated on 12 failure-prone goal-conditioned robot-control tasks implemented in Brax with MJX.Failure immediately terminates episodes, while time-limit truncation is not treated as failure.
  • Navigation tasks: Navigation tasks use respawning goals, lidar observations, and failure termination from entering designated obstacles, hazards, or moving-gremlin regions.Point Goal and Car Goal are adapted from level-2 Safety-Gymnasium navigation tasks.
  • Locomotion tasks: Locomotion tasks terminate on falls, with Pitfall variants also terminating when robots enter designated hazard regions.Maze-wall contact itself is not treated as failure; agents receive 2D lidar observations for walls and pitfalls.
  • Baselines and implementations: SAC-HER is included as an additional reference for conventional hindsight-based goal-conditioned reinforcement learning in the same failure-terminated setting.The comparison is based on JaxGCRL and is secondary to the controlled Scaling-CRL comparison.

D Supplementary results and ablations

Supplementary analyses examine benchmark performance, goal respawning, model depth, survival-mass estimation, computational overhead, and spatial success patterns. Safe-CRL retains advantages across goal settings, while direct Monte Carlo estimation provides more consistent results than TD-based estimation.

  • Effect of goal respawning: Safe-CRL retains a survival advantage when goals are fixed or respawned, with larger gains under respawning.The final-10%-of-training survival advantages are +13.8% versus +5.0% for Point Goal and +18.3% versus +2.2% for Car Goal.
  • Main benchmark: The supplementary benchmark reports final performance as mean ± standard deviation over five random seeds.The corresponding numerical results are provided in Table D.1 for the main benchmark figures.
  • Z-encoder depth: Increasing the Z-encoder from 4 to 64 layers provides no consistent improvement in time at goal or survival time when actor and critic depths are fixed.The ablation suggests that a lightweight survival-mass estimator is sufficient in these experiments.
  • TD-based survival-mass estimation: TD-based survival-mass estimation substantially degrades goal-reaching and survival performance in Car Goal and Humanoid Goal, while remaining competitive on selected Ant tasks.Direct Monte Carlo survival-mass estimation is used by default because it is simpler and more consistent across evaluated environments.
  • Computational overhead: Adding the 4-layer Z-encoder increases average training time from 9.07 to 9.32 hours per 100M environment steps, a 2.8% average overhead.Wall-clock measurements compare Scaling-CRL and Safe-CRL on the same hardware configuration for each environment.
  • Spatial success patterns: Success-rate maps compare goal locations across the playground, with darker colors indicating lower goal-reaching success rates.The robot initially faces the +x direction, and the diamond marks its initial position.
Loading 2608.26571v1…