Source-linked AI summary
Reachability-Informed Reinforcement Learning for Multi-Impulse Interplanetary Transfers
Yashdeep Chaudhary, Roberto Armellin, Harry Holt
TL;DR
Interplanetary rendezvous requires coordinating maneuver authority with transfer geometry, while reusable policies could generate trajectories across different departures. RARL selects intermediate waypoints within local reachability sets and reconstructs maneuvers with deterministic astrodynamics; in the Earth–Mars case study, multi-state policies complete every tested departure, with a small nominal-cost increase.
Problem
Interplanetary rendezvous couples maneuver allocation to transfer geometry and terminal requirements, and reusable mappings are motivated by the need to generate transfers across different initial states.
Source: §1 · pp. 1–2Method
RARL centers policy decisions on intermediate waypoints defined by local reachability geometry, then uses deterministic reconstruction to connect selected waypoints to nonlinear arcs.
Source: §1 · p. 2, §3.1 · p. 4Results
1.72% above the SCP reference, mean maneuver cost across three runs; each multi-state policy completes all 104 held-out departures, with a 0.61% increase in mean nominal maneuver cost over single-state training.
Source: §6 · p. 20Takeaways & Limitations
The Earth–Mars study demonstrates benchmark-quality nominal performance and empirically broader sampled feasibility when policies are trained over dispersed initial states.
Source: §6 · p. 20Evidence is limited to two-body Earth–Mars transfers with launch-time-induced initial-state dispersion, a fixed target and duration, and a local reachability approximation.
Source: §6 · pp. 20–21
Abstract
from arXiv · showhide
Reinforcement learning offers the prospect of a reusable sequential decision-making mechanism for spacecraft trajectory design, motivating policy interfaces that connect learned decisions to the underlying maneuver geometry. This paper develops Reachability Analysis-Informed Reinforcement Learning (RARL) for deterministic multi-impulse interplanetary transfers, placing intermediate waypoint selection at the center of the learned decision process. Local first-order reachability maps bounded velocity perturbations into an ellipsoidal set of next-node positions, within which the policy selects its waypoint. Lambert reconstruction then determines the corresponding maneuver to reach this selected waypoint along a dynamically consistent ballistic arc, coupling learned transfer-geometry selection with classical astrodynamics. A terminal two-impulse reconstruction completes the rendezvous, supported by a linear maneuver-demand assessment used for reward shaping. Numerical studies characterize this interface on a two-body Earth-Mars benchmark. Across three independent training runs, RARL achieves a mean maneuver cost of 10.23 km/s, 1.72% above a validated local sequential convex programming reference. Training over dispersed initial states extends policy reuse across a departure family with fixed target state and transfer duration. Each of the three independently trained multi-state policies completes all 10,000 held-out Monte Carlo departures without impulse-cap violations, compared with a mean feasibility rate of 6.49% for single-state policies. This broader sampled feasibility is accompanied by a 0.61% increase in mean nominal maneuver cost, without further training across departures. These results demonstrate that a reachability-informed decision interface supports benchmark-quality trajectory construction and policy reuse across dispersed departure conditions.
1 Introduction
Interplanetary rendezvous couples maneuver allocation to transfer geometry and terminal requirements, motivating reusable policies with decisions tied to physically attainable waypoints. RARL uses local reachability geometry and deterministic reconstruction to make waypoint selection the policy’s central role.
1 Introduction
- Interplanetary transfers couple maneuver allocation to complete-transfer geometry and terminal requirements, while mission analysis also assesses families of departure conditions.This motivates reusable decision mappings that can generate transfers from different initial states without resolving the complete optimization problem for every departure.Source: §1 · p. 1
- RL learns state-to-action policies through simulated interaction, and prior work motivates testing whether such policies can accommodate transfer-condition variation while meeting maneuver and terminal requirements.Earlier applications include interplanetary trajectory design under uncertainty and autonomous cislunar transfers.Source: §1 · pp. 1–2
- Rendezvous decisions must account for how each maneuver changes the next ballistic arc and the demand required to meet the terminal state.The paper frames the design question as giving the policy a geometrically meaningful set of next-node targets reflecting available impulsive authority.Source: §1 · p. 2
- RARL organizes policy decisions around intermediate waypoint selection, using local reachability geometry to define targets and deterministic astrodynamics to reconstruct maneuvers.The reachability description is local, and nonlinear transfer feasibility is assessed empirically.Source: §1 · p. 2
- The framework separates geometric decision making from nonlinear maneuver reconstruction and adds a terminal two-impulse proxy to assess remaining rendezvous maneuver demand against available authority.Analytical and numerical studies examine the action mapping, local approximation, and terminal proxy.Source: §1 · p. 2
- Reachability-based RL is evaluated on a deterministic Earth–Mars benchmark against a separately computed sequential convex programming reference, across nominal and launch-time-dispersed initial states.Three independent runs per regime are assessed on a common held-out set of 104 departures; the target state and transfer duration remain fixed.Source: §1 · p. 2
2 Multi-Impulse Transfer Problem Formulation
The problem models a spacecraft traversing a fixed transfer interval as deterministic ballistic arcs separated by bounded impulses. A terminal impulse closes the velocity rendezvous while preserving terminal position.
2 Multi-Impulse Transfer Problem Formulation
- Between maneuvers, the spacecraft follows deterministic dynamics through a flow map, using a two-body heliocentric model for the Earth–Mars study.The force model is autonomous and deterministic.Source: §2 · p. 3
- The transfer spans a fixed interval partitioned into N ballistic arcs at prescribed nodes, with each departure impulse applied at an arc’s start.The nodes run from t0 to tf, with tN = tf.Source: §2 · p. 3
- A terminal arrival impulse leaves position unchanged and closes velocity rendezvous with the prescribed target state.The formulation specifies an initial state and terminal target state over the fixed transfer interval.Source: §2 · p. 3
- Each impulse shares a magnitude limit, and an N-arc transcription has N departure impulses plus one terminal impulse in its objective and feasibility assessment.The stated objective is the impulsive transfer objective subject to the dynamics and boundary constraints.Source: §2 · p. 3
3 Reachability Analysis-Informed Reinforcement Learning Methodology
RARL centers policy decisions on selecting intermediate waypoints from local reachability sets, then reconstructs physically consistent transfer maneuvers with deterministic astrodynamics. The methodology combines a moving target reference, a terminal maneuver-demand proxy, and PPO training around this interface.
3 Reachability Analysis-Informed Reinforcement Learning Methodology
- RARL lets the policy select intermediate waypoints and uses deterministic reconstruction to connect them to transfer segments and complete the rendezvous.The framework couples waypoint selection to Lambert reconstruction and a deterministic terminal solve.Source: §3 · p. 3
3.1 Moving-Target Representation and Local Sensitivity Map
- The terminal target is propagated backward to each node as a time-aligned reference, while a local state-transition map relates velocity perturbations to next-node position deviations.The position-from-velocity sensitivity matrix describes the directions and amplification of those deviations.Source: §3.1 · pp. 3–4
3.2 Reachable-Waypoint Action Interface
- Bounded perturbations define a guarded ellipsoidal waypoint set centered on the nominal coast position, with geometry determined by local segment conditions.The policy action is radially mapped into the unit ball; the guard does not guarantee that every reconstructed Lambert impulse meets the physical cap.Source: §3.2 · pp. 4–5, Figure 1 · p. 4
3.3 Deterministic Segment Reconstruction and Terminal Closure
- Lambert solves reconstruct dynamically consistent intermediate arcs; terminal prograde and retrograde branches are compared by total impulse when available.A successful terminal solve closes the rendezvous under the adopted dynamics, while physical impulse-cap compliance is assessed separately.Source: §3.3 · p. 5
3.4 MDP Observation and Reward Construction
- The reward combines maneuver expenditure, impulse-cap exceedance, final-segment proxy shaping, and truncation penalties for Lambert failure or unbound states.The proxy uses a first-order two-impulse assessment and is active only on the final transition.Source: §3.4 · pp. 6–7
3.5 PPO Training Workflow
- A stochastic actor–critic policy is trained with PPO using rollouts that alternate waypoint selection, deterministic reconstruction, reward evaluation, and policy updates.The workflow collects a prescribed number of transitions per rollout before each PPO update.Source: §3.5 · pp. 7–8
4 Analysis and Validation of the Reachability Interface
The section evaluates the action mapping, local linear reachability approximation, and terminal two-impulse proxy in a deterministic Earth–Mars configuration. The sampled studies report low linearization error and a strong monotone association between the proxy and nonlinear terminal demand in infeasible cases.
4.1 Action-Mapping Properties
- 47.64% theoretical and 47.33% empirical boundary probability for the centered L2 map contrasts with no boundary samples for either alternative mapping.The centered map leaves inputs inside the unit ball unchanged and projects the remaining cube volume onto its boundary.Source: §4.1 · p. 9, Figure 2 · p. 9
4.2 Accuracy of the Local Linear Map
- 9.52 × 10−5 mean relative error at the physical impulse cap, with a 1.81 × 10−4 maximum, supports the local position approximation in the tested Earth–Mars case.The diagnostic sampled 20 coast nodes and 200 directions at each of five impulse fractions; reconstructed intermediate impulses had no cap violations in confirmation runs, but the inner guard does not guarantee containment.Source: §4.2 · p. 10, Figure 3 · p. 10
4.3 Analysis of the Terminal Two-Impulse Proxy
- ρs = 0.9991 between the two-impulse proxy and nonlinear terminal demand in 104 sampled infeasible cases supports its ranking of demand in that regime.All samples had η2imp > 1, so the one-sided reward penalty was active.Source: §4.3 · pp. 10–11, Figure 4 · p. 11
5 Earth–Mars Transfer Case Study
On the two-body Earth–Mars benchmark, the study compares nominal RARL with an SCP reference and tests policies across launch-time offsets. Multi-state policies retain feasibility on all sampled departures, with a small increase in nominal cost.
5.1 Benchmark and Numerical Configuration
- The Earth–Mars benchmark uses two-body heliocentric dynamics, with 20 equal-duration ballistic segments and 21 impulse slots for rendezvous.The benchmark is based on Marmo et al..Source: §5.1 · pp. 11–12
5.2 Training and Evaluation Protocol
- Multi-state training samples 100 launch offsets in [−2, 2] days; checkpoint selection uses 10 interleaved states, followed by evaluation on 104 shared MC departures.The held-out evaluation grid does not overlap the training grid.Source: §5.2 · pp. 12–13
5.3 Deterministic SCP Reference
- SCP optimized all 21 impulses under affine dynamics and impulse bounds; the reference converged in four iterations and passed nonlinear propagation validation.Independent propagation gave terminal position and velocity errors of 2.19 × 10^-4 km and 2.20 × 10^-11 km/s.Source: §5.3 · pp. 13–14
5.4 Single-State Benchmark Performance
- 10.2318 ± 0.0218 km/s was the mean total maneuver cost, 1.72% above the 10.0585 km/s SCP reference.The SCP reference was validated by independent propagation and met the stated terminal tolerances.Source: Table 5 · p. 14, §5.3 · p. 14, §5.4 · p. 15
- At nodes 10–15, selected waypoints lie near the projected set center, with ηxy from 0.014 to 0.064 and preceding impulses totaling 0.1347 km/s.The waypoints then move outward through nodes 16–17 and return near the boundary at nodes 18–19.Source: Figure 7 · p. 16, §5.4 · p. 17
5.5 Multi-State Performance under Launch-Time Dispersion
- 100% of held-out MC transfers were feasible for each multi-state policy, versus (6.49 ± 2.49)% for single-state policies.The comparison uses the common 104-departure validation set.Source: Table 6 · p. 17, §5.5 · pp. 17–18
- 0.61% higher mean nominal maneuver cost accompanied the broader departure feasibility: multi-state cost was 10.2946 ± 0.0225 km/s versus 10.2318 km/s for single-state policies.All three multi-state policies remained feasible at nominal departure.Source: Table 6 · p. 17, §5.5 · p. 18
- Across departures, multi-state allocations retain an approximate bang–off–bang structure, with near-cap impulses at nodes 0–7 and median impulses below 0.05 km/s at nodes 10–14.Figure 10 summarizes impulse and cost distributions across MC departures.Source: §5.5 · p. 19, Figure 10 · p. 20
6 Conclusion
RARL centers the policy on waypoint selection defined by local reachability and uses Lambert solutions to reconstruct transfer segments. In the Earth–Mars study, it approaches the SCP reference for nominal cost and reuses multi-state policies across sampled departures, within a limited two-body setting.
6 Conclusion
- RARL uses a local first-order reachability map to define waypoint choices and Lambert solutions to reconstruct nonlinear transfer segments.Diagnostics examine action geometry, local approximation error, and terminal maneuver-demand assessment.Source: §6 · p. 20
- A mean maneuver cost 1.72% above the SCP reference was achieved across three independent training runs; each multi-state policy completed all 104 Monte Carlo departures without truncation or impulse-cap violations.The multi-state policies had a 0.61% increase in mean nominal maneuver cost relative to single-state training and operated without further training across departures.Source: §6 · p. 20
- The findings are limited to a two-body Earth–Mars transfer with launch-time-induced initial-state dispersion, a fixed target state, and a fixed transfer duration.The reachability map remains a local approximation; broader transfer families and longer or more nonlinear arcs are proposed for further assessment.Source: §6 · pp. 20–21
- The reusable state-feedback mapping is proposed for potential use as an onboard high-level guidance layer with deterministic reconstruction or as a structured initializer for trajectory optimization.These applications build on a policy interface coupling learned transfer-geometry selection with deterministic maneuver reconstruction.Source: §6 · p. 21
A Secondary Full-State Terminal-Difficulty Proxy
This section introduces a supplementary full-state terminal-correction proxy based on a single linearized impulse. Its unit convention affects how position and velocity residuals are weighted, while the reward uses a separate two-impulse assessment.
A Secondary Full-State Terminal-Difficulty Proxy
- The supplementary proxy estimates a single impulsive correction from the complete linearized terminal mismatch, rather than separate departure and arrival corrections.It provides a complementary measure of terminal correction demand.Source: A Secondary Full-State… · p. 21
- Using kilometers and seconds as base units fixes the relative weighting of position and velocity residuals in the least-squares fit.Source: A Secondary Full-State… · p. 21
- The reward uses the two-impulse assessment to match the terminal rendezvous structure; the single-correction proxy remains available for diagnostics or alternative formulations.Source: A Secondary Full-State… · p. 21