Source-linked AI summary
SCoCaT: Success Conditioned Constrained Reinforcement Learning for Spacecraft Docking
Aman Arora, Ricard Marsal I Castan, Matteo El-Hariry, Miguel Olivares-Mendez
TL;DR
Terminal-navigation tasks require precise goal attainment while satisfying safety constraints that become active along the approach, but survival-weighted termination can favor hovering outside the goal. The paper augments CaT with a per-step success signal and auxiliary critic; results show improved task engagement while retaining substantial constraint compliance across spacecraft platforms.
Problem
Termination-based constrained RL had not been sufficiently demonstrated for terminal navigation, where safety constraints can become active near a precise goal configuration.
Source: §1 · p. 1Method
SCoCaT adds a per-step success signal and auxiliary value critic decoupled from CaT’s survival weighting to a termination-based constrained-RL approach.
Source: §1 · p. 2, §3.3 · p. 5Results
SCoCaT reaches 0.906±0.066 declared success on the Floating Platform versus 0.000 for both PPO and CaT, and mitigates collapse on the 6-DoF CubeSat with 0.613 declared success at 0.945 compliance.
Source: §4.1 · pp. 7–8Takeaways & Limitations
The paper presents SCoCaT as a principled resolution to feasibility collapse for terminal-navigation tasks, with zero-shot transfer supporting generalization beyond simulation.
Source: §5 · p. 9On the 6-DoF CubeSat, SCoCaT’s compliance is 2.6 points below CaT’s, with the gap attributed to line-of-sight errors.
Source: §4.1 · p. 7, §6 · p. 9
Abstract
from arXiv · showhide
Termination-based constrained reinforcement learning is attractive for safety-critical robotic deployments: it avoids online optimization at inference, scales easily to many constraints via a single scalar per constraint, and is simpler to implement than commonly used Lagrangian methods. Instead of pricing violations through summed cost penalties, this approach makes violations structurally unprofitable by shortening the effective horizon for each violation. We identify a structural failure mode of this method class on terminal-navigation tasks: reaching a precise goal configuration while satisfying safety constraints that tighten along the final approach. When the goal sits inside the region close to where the constraints become active, the survival-weighted objective makes dwelling outside the goal region strictly preferable to entering, producing high constraint compliance with low task completion. We formalize this pathology and show that a minimal augmentation to off-the-shelf RL algorithms like PPO resolves this ``feasibility collapse''. We empirically demonstrate that adding a dense per-step success signal via an auxiliary value critic improves the task completion rate while maintaining safety-critical constraint compliance. Validation across two representative spacecraft platforms: a 6U-CubeSat spanning the mass and degree-of-freedom envelope of operational proximity operations, and a floating platform testbed for zero-shot sim-to-real transfer in our laboratory, supports the generality of these findings.
1 Introduction
Terminal navigation combines precise terminal goals with safety constraints active along the approach, a difficult fit for resource-constrained spacecraft. The paper identifies a failure mode of termination-based constrained RL and introduces SCoCaT to address it.
1 Introduction
- Termination-based constraints shorten the effective horizon after violations, avoiding online optimization at inference while retaining a deployable policy network.The introduction contrasts this structural approach with Lagrangian methods that price violations through dual variables.Source: §1 · p. 2
- In terminal navigation, survival weighting can make hovering outside a constraint-active goal corridor preferable to entering, producing high compliance but near-zero task completion.The paper names this failure mode feasibility collapse.Source: §1 · p. 2
- SCoCaT augments CaT with a per-step success signal and an auxiliary critic, decoupling task-success learning from CaT’s survival weighting.The introduction presents this as a minimal extension that retains the deployment-time simplicity of CaT.Source: §1 · p. 2
- The paper formalizes feasibility collapse and evaluates SCoCaT on two free-flying platforms, including sim-to-real transfer on an air-bearing testbed.The platforms span 3-DoF and 6-DoF settings.Source: §1 · p. 2
2 Related Work
Prior constrained-RL approaches trade off safety, computation, and deployment complexity, while termination-based methods had not been demonstrated for terminal-navigation tasks. The paper positions SCoCaT at the intersection of termination-based safety and a task-success signal.
2 Related Work
- Across both platforms, the figure contrasts SCoCaT’s high task engagement and compliance with CaT’s corridor-avoidance collapse and PPO’s lower compliance.For the CubeSat, CaT enters the corridor in 3.7% of episodes and never completes the task; SCoCaT reaches 79.4% TCR at 0.945 compliance.Source: Figure 1 · p. 3
- Primal-dual methods price violations, whereas termination-based approaches make them structurally unprofitable; the latter had been validated on dense tasks without terminal configurations.The related-work discussion contrasts their trade-offs, including computation and state-visitation concerns.Source: §2 · pp. 2–3
- Direct Behavior Specification uses a success critic with Lagrangian enforcement; SCoCaT combines a task-success signal with termination-based safety in an on-policy setting.The paper identifies this combination as novel to its knowledge.Source: §2 · p. 3
- Prior spacecraft docking studies used unconstrained dense-reward policies or external safety mechanisms, and constrained-RL sim-to-real docking had not been demonstrated.The cited alternatives include run-time assurance and control barrier functions, which add online computation.Source: §2 · pp. 3–4
3 Methodology
The methodology formulates safety as bounded constraint-violation occupancy and identifies why survival-weighted termination can favor hovering outside a goal. It then adds an independently trained success signal to restore learning toward the goal while retaining constraint-based terminations.
3 Methodology
- Safety is formulated as bounded discounted occupancy probability for each constraint and handled through CaT’s termination-based unconstrained objective.The state space includes a compact goal region satisfying pose and velocity tolerances.Source: §3 · p. 4, §3.1 · p. 4
3.1 Problem Formulaton
- Feasibility collapse occurs when a policy hovers just outside the goal corridor while goal-entry probability remains approximately zero.Under the stated absorbing, zero-reward goal and active-entry constraint assumptions, hovering is a strict local optimum; in the non-absorbing rewarding case, it is preferred when R > κ rG.Source: §3.1 · p. 4
3.2 Constraints as Terminations
- CaT converts constraint violations into termination probabilities that survival-weight future rewards, creating a hovering local optimum for terminal-navigation tasks.The termination probability is derived from constraint violations, with pmax_i encoding per-constraint severity.Source: §3.2 · p. 4
3.3 Success-Conditioned Constraints as Terminations (SCoCaT)
- SCoCaT adds a per-step success signal and auxiliary critic whose returns use environment-only terminations, preserving success gradients inside the goal corridor.The success indicator gates on pose and velocity tolerances at every in-tolerance timestep; the policy combines standardized reward and success advantages by equal summation, without an extra weighting hyperparameter.Source: §3.3 · p. 5
3.4 Implementation Details
- Evaluation requires 100 consecutive in-corridor timesteps plus terminal-velocity gates; constraint severity tiers distinguish mission-fatal, hardware-critical, and softer limits.The strict tier covers workspace boundaries and collisions; moderate covers docking velocity, angular rate, and fuel; soft covers line-of-sight and jerk smoothness.Source: §3.4 · pp. 6–7
4 Experiments
Experiments compare SCoCaT with unconstrained PPO and CaT on a Floating Platform and 6U CubeSat, then evaluate zero-shot transfer and ablations. SCoCaT improves Floating Platform docking while maintaining high compliance, and partially mitigates collapse on the harder CubeSat.
4 Experiments
- The experiments use two platforms with a 100-timestep terminal-pose hold criterion, five independent training seeds, and comparisons against PPO, CaT variants, and PEGrad-CaT.Training uses a success-tolerance curriculum over the first 5 × 10^4 environment steps; PPO is the unconstrained baseline and CaT-PPO the direct predecessor.Source: §4 · p. 7
4.1 Simulation Results
- Floating Platform simulation: SCoCaT reaches 100% TCR and 98.8% constraint compliance, versus CaT’s 0.4% TCR and 99.4% compliance—a 250× task-engagement gain at a 0.6-point compliance cost.Under the declared docking criterion, SCoCaT scores 0.906±0.066 against 0.000 for both PPO and CaT.Source: §4.1 · p. 7
- On the CubeSat, SCoCaT improves every constraint over PPO, but PPO has higher declared success: 0.931 versus 0.613.SCoCaT has 12× lower docking-velocity and 3× lower angular-rate violation, with compliance 0.945 versus 0.839.Source: §4.1 · pp. 7–8
- On the Floating Platform, dense and terminal-proximity rewards yield near-identical GCO and compliance, while binary SCoCaT fails with sparse reward and a continuous success signal recovers TCR=49.2%.For the CubeSat, the continuous signal enters more reliably (Ent. 1.000 vs. 0.794), while the binary signal more often converts entry into declared success (0.613 vs. 0.450).Source: §4.1 · p. 8
- On the 6-DoF CubeSat, SCoCaT reaches 0.613 declared success at 0.945 compliance, while CaT enters the corridor in 3.7% of episodes and never completes the task.SCoCaT’s success is 66% of PPO’s at 97% of CaT’s compliance; the seed distribution is bimodal, with four of five SCoCaT seeds entering and holding the corridor on essentially every episode.Source: §4.1 · p. 8
- For hardware deployment, policies are trained with randomized mass and constant disturbance wrenches, then tested on an air-bearing platform with motion-capture pose sensing and 5 Hz binary thrusters.The sim-to-real table covers four runs per method; violation rates are fractions of timesteps, while vG and ωG are measured inside the 10 mm goal corridor.Source: §4.1 · p. 8, Table 1 · p. 9
- In zero-shot hardware transfer, SCoCaT succeeds in 3/4 runs versus CaT’s 0/4, but the four-run comparison (Fisher exact p = 0.14) is not a powered comparison.The angular-velocity gap across methods is attributed to thruster discretization combined with the platform’s 5 Hz maximum operational frequency.Source: §4.1 · p. 9
5 Conclusion
SCoCaT is presented as a principled resolution to feasibility collapse in termination-based constrained RL for terminal-navigation tasks. Validation on two representative spacecraft platforms finds that it alone meets the Floating Platform docking criterion, with a 0.6-point compliance cost, and partially breaks collapse on the harder 6-DoF CubeSat.
5 Conclusion
- SCoCaT is the only method that satisfies the declared docking criterion on the Floating Platform, at a 0.6-point compliance cost, and it partially breaks collapse on the harder 6-DoF CubeSat.The work validates the approach on two representative spacecraft platforms and reports zero-shot sim-to-real transfer.Source: §5 · p. 9
- The paper establishes SCoCaT as a principled resolution to feasibility collapse in termination-based constrained RL for terminal-navigation tasks.Source: §5 · p. 9
- Zero-shot sim-to-real transfer supports generalization beyond simulation within the reported spacecraft-platform evaluations.Source: §5 · p. 9
6 Limitations
The evaluation leaves several robustness and generalization questions open, and the 6-DoF CubeSat shows a modest safety-compliance trade-off for SCoCaT.
6 Limitations
- Hardware validation is limited to n = 4 runs per method, platform float time per charge, and docking-pose-only scores.The study does not vary observation noise, so robustness to degraded state estimation is untested; broader embodiment and task-family validation remains open.Source: §6 · p. 9
- On the 6-DoF CubeSat, SCoCaT loses 2.6 compliance points versus CaT, entirely from line-of-sight errors, while entering the corridor.The seed distribution is bimodal rather than degrading evenly across runs.Source: §6 · p. 9
- The angular-velocity sim-to-real gap appears across all methods and is not closed by constraint enforcement.Source: §6 · p. 9
- The equal-weight advantage scalarization has no extra hyperparameters, but the optimal safety-success trade-off depends on the mission.Learned or adaptive weighting is proposed as a next step.Source: §6 · p. 9
A Proofs
The proofs show when hovering can dominate goal entry under termination-weighted returns, and identify conditions under which that preference becomes specific to CaT.
Proof of Proposition 1
- For an absorbing, zero-reward goal, hovering strictly dominates every policy that enters the goal at a finite step under a discounted objective.The proof uses conditions (i) and (iii), so this result is not specific to CaT.Source: Proof of Proposition 1 · p. 13
- With non-absorbing, rewarding goal states and positive termination at the goal, hovering is strictly CaT-preferred exactly when the stated threshold on R holds.The threshold compares the continuing reward R outside the goal with goal reward rG discounted by the goal termination rate.Source: Proof of Proposition 1 · p. 13
- At docking-velocity termination rate ¯δ = 0.3 and γ = 0.99, κ = 0.023; the paper's R/rG = 0.99 therefore loses to the goal without CaT but beats it under CaT.The docking-velocity constraint fires at every approach step when ∥vt∥> 0.05 m/s and dp ≤ 0.5 m, compressing the value of goal-corridor states.Source: Proof of Proposition 1 · p. 13
Proof of Proposition 2
Proposition 2 analyzes how a separate success signal changes the advantage of entering the goal; the reward-bonus comparison explains why a separate critic better preserves constraint enforcement.
Proof of Proposition 2
- With the paper's reward weights, R ≤ 0.099 and R ≤ 0.198 for the two dense-reward configurations, clearing the binary, continuous-floor, and finite-horizon bounds by 5.0×, 3.4×, and 3.2×, respectively.The finite-horizon bound is R < 0.64 for γ = 0.99, T = 100, and H = 598 steps.Source: Proof of Proposition 2 · p. 14
- Any arbitrarily small mixture of the hovering policy with a goal-reaching policy improves the combined objective, so hovering is not a local optimum.Source: Proof of Proposition 2 · p. 14
- A positive combined advantage for entering the goal occurs once Vs(sG) exceeds R/(1 −γ), breaking the hover equilibrium before the success value fully converges.This argument requires Vs to be trained on environment dones, not CaT termination dones, which could lower it below the threshold.Source: Proof of Proposition 2 · p. 14
B Reward Bonus vs. Two-Critic: Why They Differ
- At the same GCO, SCoCaT-bonus has 37× higher docking-velocity and 13× higher LoS violation rates than SCoCaT: 0.037 versus 0.001 and 0.826 versus 0.066, respectively.These values are for the dense-reward, obstacle-free condition.Source: B Reward Bonus vs. Two-Critic: Why… · p. 15
- A reward bonus shares the CaT reward stream and can offset its survival penalty, whereas the separate success advantage leaves the CaT penalty in the reward advantage unchanged.SCoCaT computes the success advantage from a critic trained on environment-only dones; success requires all tolerances, including velocity, to be met.Source: B Reward Bonus vs. Two-Critic: Why… · p. 15
C Training Details
The appendix specifies randomized training starts, constraint settings, observation inputs, network configuration, and reward variants. Dense reward is used for the main results, while a declared-criterion evaluation notes that TCR credits PPO where neither baseline meets the criterion.
C Training Details
- Training starts sample the vehicle’s position uniformly from 0.01–4.0 m from the goal with zero initial linear and angular velocity; attitude or heading is randomized by platform.The goal pose is randomly placed within 2 m of the environment origin each episode.Source: C Training Details · p. 15
C.1 Constraints
- Fuel violation rates fall from 63% to 3% as severity increases, docking velocity stays below 1.1%, and line-of-sight violations remain above 84% at maximum severity.The persistent line-of-sight rate is described as geometry-limited and motivates its soft classification; the severity sweep uses an earlier constraint implementation, so only its trend is comparable to Table 9.Source: §C.1 · p. 16, Table 2 · p. 16
C.2 Observations
- Docking observations include body-frame linear and angular velocities, goal-relative distance and bearing, and position and orientation errors.Angles use cosine-sine pairs to avoid discontinuities at ±π.Source: §C.2 · p. 16, Table 4 · p. 16
C.3 Hyperparameters
- The policy, value, and successor-value models use independent weights and a shared two-hidden-layer topology of size [64, 64] with ELU activations.The policy uses a MultiCategoricalMixin; the value and successor-value functions use a DeterministicMixin.Source: §C.3 · pp. 16–17
- The active-constraint table covers both platforms; pmax denotes maximum per-step termination probability at full violation, with hard constraints triggering immediate resets.Termination probability scales linearly with violation magnitude.Source: Table 3 · p. 17, §C.3 · p. 17
C.4 Reward Formulations
- Three reward formulations were evaluated, and dense reward is used for all main results; safety requirements enter exclusively through termination probabilities.Terminal-proximity and end-of-episode binary variants are evaluated in Appendix D.Source: §C.4 · p. 17, Table 6 · p. 18
- Under the declared-criterion evaluation, TCR credits PPO where neither baseline meets the criterion.The table reports five seeds with dense reward and includes TCR and compliance alongside Hold and Decl criteria.Source: Table 7 · p. 18
D Full Ablation Study: Floating Platform (3-DoF)
The Floating Platform ablation compares reward-shaping and success-signal variants, alongside a gradient-projection alternative. Success conditioning improves completion relative to CaT with similar corridor entry, but lowers line-of-sight compliance.
D.1 CaT-PEGrad.
- CaT-PEGrad maintains separate reward and success advantage streams, with the success stream using environment-level truncation rather than CaT terminations.A dedicated success critic can provide value estimates for the success stream.Source: §D.1 · p. 18
- CaT-PEGrad projects the reward gradient onto the orthogonal complement of the success gradient, then combines the directions as Δθ = gS + gR⊥S.The adaptation makes success primary and task reward secondary, unlike the original PEGrad priority ordering.Source: §D.1 · pp. 18–19
- Success conditioning improves task completion relative to CaT while corridor entry remains similar, but line-of-sight compliance worsens; SCoCaT still outperforms CaT-PEGrad on both reported metrics.Tables 8 and 9 cover reward-shaping and success-signal ablations on the Floating Platform; their captions define task-performance and constraint-compliance measures.Source: D Full Ablation Study: Floating… · p. 18, §D.1 · p. 19
E Full Ablation Study: 6U CubeSat (6-DoF)
The 6U CubeSat evaluation covers task-performance and constraint-compliance configurations across five seeds using dense reward.
E Full Ablation Study: 6U CubeSat (6-DoF)
- Tables 10 and 11 report 6U CubeSat configurations across five seeds with dense reward, scoring task performance and constraint compliance at the goal corridor over the 0.01–4 m spawn range.Task criteria include corridor entry, sustained engagement, consecutive dwell, and declared completion; the compliance table summarizes mean and per-constraint violation rates.Source: E Full Ablation Study: 6U CubeSat… · p. 19
F Obstacle Avoidance Results
On the Floating Platform, the obstacle-avoidance evaluation adds a collision constraint alongside existing constraints. SCoCaT maintains corridor entry and docking precision comparable to the obstacle-free setting while keeping collision violations near zero.
F Obstacle Avoidance Results
- With a randomly placed cylindrical obstacle activating collision constraints, SCoCaT maintains comparable corridor entry and docking precision while per-timestep collision violations remain near zero.The passage attributes the retained hard-termination enforcement to the decoupled success critic.Source: F Obstacle Avoidance Results · p. 19