Source-linked AI summary
Where Planning Changes the Outcome of Orbital Games
Duncan Eddy, Mahdi Al-Husseini, Mykel J. Kochenderfer
TL;DR
Close-proximity orbital maneuvers can be adversarial and costly, but it remains unclear when policy choice matters relative to vehicle capability. OrbitalGym compares controllers and planners across two close-range games and varied conditions, finding limited policy effects in pursuit-evasion but larger effects in lady-bandit-guard.
Problem
Prior studies often assess one orbital-game method in one scenario, often against passive or random defenders, leaving unclear when policy choice matters relative to vehicle capability in costly close-proximity encounters.
Source: §1 · p. 1, §7 · p. 14Method
OrbitalGym compares feedback controllers and planning policies in pursuit-evasion and lady-bandit-guard across varied capability and encounter conditions.
Source: §1 · p. 2Results
Policy changed capture fraction by 10% or more in at most 15 of 90 configurations against five of six evaders, but the best MPPI guard raised defense by 10% or more in 56 to 74 of 90 configurations against four frequent breachers.
Source: Policy Choice Decides Lady-Bandit-Guard · p. 10, §7 · p. 14Takeaways & Limitations
Comparisons of orbital-game decision methods should report relevant parameters and test whether results hold when those parameters change.
Source: §7 · p. 14Far-range encounters, which call for nonlinear dynamics, remain outside the study’s scope.
Source: §7 · p. 14
Abstract
from arXiv · showhide
This paper introduces OrbitalGym, a simulation environment for studying when planning algorithms or system design determine the outcome of adversarial orbital encounters. In an orbital game, two non-cooperative teams of satellites maneuver during rendezvous and proximity operations to win a differential-reward objective. Advances in terrestrial game playing have prompted many methods for orbital games, but studies typically evaluate one method in one scenario, often against a passive or random defender. It is unclear whether better planning matters once vehicle capability (maneuverability, sensing, communications) and adversarial responses are considered. OrbitalGym is a JAX-based environment supporting varying numbers of agents, state representations (position, velocity, attitude, fuel), explicit or implicit communication, and proximity or conical field-of-view sensors. It compiles games to batched GPU code, enabling large-scale parallel simulation of policies across many configurations. This identifies regimes where policy choice affects outcomes and regimes where system capability dominates. We study two close-range games: pursuit-evasion, where one spacecraft tries to capture another, and lady-bandit-guard, where a guard prevents an attacker from reaching a protected asset. We compare PD, LQR, model predictive path integral control, and Monte Carlo tree search over ratios of thrust and propellant, initial separation, game duration, sensor capability, and the definition of capture. In pursuit-evasion the propellant ratio decides most encounters: against five of six evader policies, pursuer policy changes capture probability by 10 percent or more in at most 15 of 90 capability configurations. In lady-bandit-guard, policy matters more: against bandits able to reach the asset, a better guard policy raises defense probability by at least 10 percent in 56 to 74 of 90 configurations.
1. INTRODUCTION
The paper introduces OrbitalGym2 to test when policy choice changes orbital-game outcomes and when vehicle capability, information, or game design dominates. It addresses a gap left by studies that evaluate isolated configurations, often against passive or scripted opponents.
1. INTRODUCTION
- Rendezvous and proximity operations can serve benign or adversarial purposes; uncertainty about an opponent can make conflict more likely when conflict is costly.The introduction cites more than 6 instances of Chinese and Russian satellites maneuvering near other nations’ satellites without warning or explanation between January 2024 and March 2025.Source: §1 · p. 1
- Orbital-game research spans differential-game solutions, feedback guidance, tree search, and reinforcement learning, while existing environments support selected games.Prior studies varied some parameters but assumed perfect information shared by all actors, leaving uncertain whether algorithmic improvements persist under more realistic sensing.Source: §1 · pp. 1–2
- Scalable simulation enabled planning advances in terrestrial games, including MCTS, AlphaGo, AlphaZero, MuZero, and AlphaStar, alongside standardized training and evaluation interfaces [24].The paper presents orbital-game studies as a setting where scalable, comparable evaluation can clarify how planning decisions and system design interact.Source: §1 · pp. 1–2
- Orbital-game outcomes depend on vehicle capability, encounter geometry and duration, information, and opposing policies, so isolated-configuration studies cannot establish when planning matters.The paper asks where policy choice changes outcomes and where capability, information, or the game's definition decides them.Source: §1 · pp. 1–2
- OrbitalGym2 evaluates five orbital games and compares PD, LQR, and MPPI, plus MCTS in pursuit-evasion, across varied configurations using identical initial encounters.The paper studies pursuit-evasion, where one spacecraft tries to capture another, and lady-bandit-guard, where a guard prevents a bandit from reaching a protected asset.Source: §1 · p. 2
2. ORBITAL GAMES
OrbitalGym models adversarial spacecraft encounters in which each side acts from its belief, the environment propagates both sides’ commands, generates observations, and evaluates game-ending events and rewards. It studies pursuit-evasion and lady-bandit-guard, with configurable orbital dynamics, vehicle capabilities, sensing, communications, and objectives.
2. ORBITAL GAMES
- OrbitalGym studies pursuit-evasion, where the bandit pursues and the guard evades, and lady-bandit-guard, where bandits seek a protected point and guards prevent them.The environment provides five games with shared state, dynamics, actions, and observation models but different objectives and termination rules.Source: §2 · p. 2, Games · p. 2
Orbit and Attitude Dynamics
- OrbitalGym models relative spacecraft motion in a rotating RTN frame and supports numerical orbital propagation with configurable perturbing forces.For close separations, linear HCW dynamics admit an exact discrete-time state transition, enabling planners to predict maneuver sequences efficiently.Source: Orbit and Attitude Dynamics · pp. 2–3
Thruster Modeling
- Propellant limits total maneuvering, while a constant-mass model lets thrust and total delta-v budgets vary independently.Guard-to-bandit thrust and budget ratios describe relative capability; ratios above one favor the guard, and those below one favor the bandit.Source: Thruster Modeling · pp. 3–4
Observations and Beliefs
- Sides act on beliefs updated from relative-position measurements, using radial or body-fixed cone sensors and an Extended Kalman or particle filter.Under full information, the belief is the true relative state; otherwise, the filter accounts for the opponent’s unknown maneuver as process uncertainty.Source: Observations and Beliefs · p. 4
Communication
- Teammates can exchange and fuse beliefs through configurable distance, cone, or groundstation link models, but communications are not exercised in the paper’s single-agent experiments.OrbitalGym also supports explicit transmissions that can have a cost, including revealing the communicator’s position in lady-bandit-guard.Source: Communication · p. 4
Captures, Outcomes, and Rewards
- Pursuit-evasion ends in capture or horizon survival, while lady-bandit-guard ends in capture, breach, or horizon survival; reported outcomes are capture and defense fractions.Capture and breach can require velocity matching as well as proximity. Planners optimize denser lookahead rewards based on objective-closing distances and event bonuses; potential-difference shaping is also available for learning.Source: Captures, Outcomes, and Rewards · pp. 4–5
3. ORBITALGYM
OrbitalGym is a JAX environment that separates orbital-game components for independent variation and compiles stateless game transitions for batched simulation. Its experiments use this shared execution path to compare policies across many configurations and record reproducible episode-level results.
3. ORBITALGYM
- OrbitalGym separates truth, planning, observation, and belief models, allowing state fields and planning dynamics to vary independently.Vehicle state can include position, velocity, attitude, body rates, and propellant mass; planning and belief models can use different dynamics.Source: §3 · p. 5
- A stateless step function returns the next state, observations, rewards, and termination flag; JAX vectorizes it across episodes and scans it over each game.Gymnasium and PettingZoo adapters wrap the same function for single-agent and multi-agent learning libraries.Source: §3 · p. 5
- A batch of 11,520 encounters of 180 steps takes about 1 s with feedback policies on both sides, 28 s with MPPI on one side, and 55 s with MPPI on both.The timing uses a 10 s step on one NVIDIA H100 GPU.Source: §3 · pp. 5–6
- Each experiment compiles one program per policy-type combination and evaluates configurations on shared initial conditions, recording seeds, source revision, and per-episode results.Scripts regenerate result figures and quoted statistics from those records.Source: §3 · pp. 5–6
4. POLICIES
OrbitalGym compares policies ranging from fixed actions and feedback control to planners that optimize against forecasts of future play. Each policy acts on its side’s belief about the game state.
4. POLICIES
- The compared policies span nonreactive baselines, feedback laws, and planners; policies use belief-state estimates, and commanded actions are clipped before application.The section presents the policy summaries, with full formulations and settings in Appendix A.Source: §4 · p. 6
Non-Reactive and Feedback Control Policies
- Coast and random provide baselines, while direct, PD, LQR, and HCW intercept provide fixed-direction or state-feedback control.Direct thrusts along or away from the target without a velocity term; PD uses relative position and velocity, LQR uses u = −Kx, and HCW intercept recomputes a single-impulse solution assuming the opponent coasts. Pursuers and guards apply these laws relative to their opponent; LBG bandits apply direct, LQR, and HCW intercept relative to the lady.Source: Non-Reactive and Feedback Control… · p. 6
- PE evaders use flee or transverse thrust, while the LBG bandit’s evade policy approaches the lady and turns away from guards within 300 m.Flee is motivated by the optimality of fixed-direction thrust in Wong’s constant-gravity, constant-thrust, propellant-limited games.Source: Non-Reactive and Feedback Control… · p. 7
Model Predictive Path Integral Planning
- MPPI samples and simulates impulse sequences, weights them by negative cost, then executes the first impulse and shifts its nominal plan.Variants are named for their assumed opponent policy; the experiments use shared default settings without per-configuration tuning, except in deliberate lookahead and sample-count experiments.Source: Model Predictive Path Integral Planning · p. 7
Monte Carlo Tree Search
- MCTS searches nine impulses held for three steps, models opponent-policy uncertainty with chance nodes, and executes the most-visited action.In PE, opponent policies are sampled uniformly from coasting or fleeing in pursuer search, and coasting, direct pursuit, or HCW intercept in evader search. New leaves are valued by rollouts to the end of the lookahead, using HCW intercept for pursuers or fleeing for evaders.Source: Monte Carlo Tree Search · p. 7
5. EXPERIMENTS
The experiments use controlled orbital-game simulations to identify when policy choice changes outcomes and when system capability determines them. They compare win fractions across configurations while controlling encounters and varying selected scenario parameters.
5. EXPERIMENTS
- Each configuration uses 128 encounters with shared initial conditions and seeds; results are capture or defense fractions, and a meaningful policy gain is at least 10 percentage points.Every counted gain in Section 6 differs from zero at the 95% confidence level based on observed standard error.Source: §5 · pp. 7–8
- Both games use one vehicle per side, HCW dynamics, 10-second steps, and a default 5 m/s delta-v budget for the pursuer or guard.PE defaults to 300 m initial separation; LBG places the bandit 1,000 m and guard 200 m from the lady. Both use a 10 m capture radius, and LBG uses a 50 m breach radius.Source: §5 · pp. 7–8
- The full-information baseline removes estimator error; information experiments instead use Kalman-filter estimates from radial or cone sensors, which can lose targets outside range or cone angle.The cone sensor points along the estimated direction to the opponent and has unlimited slew rate.Source: §5 · p. 8
- 512-encounter replications changed average win fractions by 0.3% in PE and 0.9% in LBG, and meaningful-gain counts from 15 to 17/90 against MPPI-LQR evaders and 74 to 75/90 against MPPI-LQR bandits.Each replication used 512 new encounters per configuration.Source: §5 · p. 8
6. RESULTS
Results compare the best feedback-control and MPPI policy in each configuration, with policy gain measuring their difference. Propellant budgets govern most pursuit-evasion outcomes, while guard policy choice often changes defense against bandits that threaten the asset.
Maneuverability Decides Most Pursuit-Evasion Encounters
- Against a full-thrust fleeing evader with a 5 m/s pursuer budget, policy gain reaches 10% or more in only 9 of 90 configurations.The largest gain is 64% at ρu = 1.1 and ρ∆v = 0.75, where MPPI-Flee captures 77% and the best feedback policy 13%. At equal thrust limits and ρ∆v = 0.75, the best pursuer captures 1%, 84%, and 27% with budgets of 1, 5, and 20 m/s, respectively.Source: Maneuverability Decides Most… · pp. 8–10, Figure 3 · p. 11
Policy Choice Decides Lady-Bandit-Guard
- The best MPPI guard improves defense by at least 10% in 56–74 of 90 configurations against four frequently breaching bandits, and never falls that far below feedback control.At equal thrust limits and delta-v budgets against the MPPI-LQR bandit, the idle, best feedback-control, and best MPPI guards defend 0%, 1%, and 83% of encounters, respectively. Direct and evasive bandits rarely breach even an idle guard, so guard policy matters little against them.Source: Policy Choice Decides Lady-Bandit-Guard · p. 10, Figure 4 · p. 12
The Capture Definition Changes the Winner
- At the contested operating point, increasing capture radius from 5 m to 50 m raises feedback-pursuer capture from 13% to 84%, while MPPI capture remains 84% to 88% and policy gain falls from 70% to 3%.With velocity matching at ve = 0.5 m/s, no pursuer captures more than 9% of fleeing evaders; applying the same condition in LBG reduces the largest MPPI-guard advantage from 91% to 28%. Testing capture only at the end of each 10 s step changes feedback-control capture fractions by at most 8%, but reduces MPPI captures by up to 69%.Source: The Capture Definition Changes the… · pp. 12–14
Distance and Time Set the Reach of Each Policy
- From 800 m, the best feedback-control pursuer captures about half of coasting-evader encounters, while the best MPPI pursuer captures every encounter; from 1.6 km, feedback captures none and MPPI most.At 800 m, MPPI-Flee captures every encounter in a median of 450 s while spending 4.0 m/s on average; LQR captures 57% in a median of 620 s. From 100 m against a fleeing evader, direct pursuit captures 62% within 600 s, while HCW intercept, LQR, and MPPI-Flee capture at most 24%; those three capture nearly every encounter within 900 s.Source: Distance and Time Set the Reach of… · pp. 10–11, Figure 5 · p. 13
Information Changes Which Policy Wins
- With a 10° cone sensor on both sides, the best pursuer captures 77% of coasting evaders, versus all with full information; policy choice still changes capture by at least 10% in at most 15 of 90 configurations against five of six evaders.Against a coasting evader 1,000 m away, MPPI-Flee captures every encounter with full information, 68% with a 10° cone sensor, and 40% with a 500 m-range radial sensor. At ρu = 1 and ρ∆v = 0.75, a radial sensor lowers MPPI-Flee capture from 84% to 5%, while HCW intercept remains near 30%; pursuer estimate error is the main cause.Source: Information Changes Which Policy Wins · pp. 11–12, Figure 6 · p. 13
Policy Settings Interact with the Scenario
- With full information, increasing MPPI-Flee lookahead from 90 s to 900 s raises capture against a fleeing evader from 9% to 89%.With a radial sensor, the best lookahead shortens to 360 s with 0.1% noise and 210 s with 1%. Forecasting the opponent changes outcomes more than increasing evader samples from 64 to 1,024, which changes capture by at most 15%; MCTS results fall between feedback-control and MPPI policies.Source: Policy Settings Interact with the… · p. 12, Figure 7 · p. 14, Table 6 · p. 14
7. CONCLUSIONS
OrbitalGym shows that policy choice affects outcomes differently across pursuit-evasion and lady-bandit-guard, while evaluation results depend on game and system parameters. The conclusions identify future extensions for orbital-game research.
CONCLUSIONS
- 10% or more improvement in defense occurred in 56 to 74 of 90 configurations with the best MPPI guard against each of the four most frequent-breaching bandits.Feedback-control guards missed accelerating bandits even with unlimited propellant, while the MPPI guard forecast their approach and intercepted them.Source: §7 · p. 14
- 10% or more change in capture fraction from tested MPPI versus feedback-control policies occurred in at most 15 of 90 configurations against five of six evaders with a 5 m/s pursuer budget.Propellant-budget ratio organized pursuit-evasion outcomes; MPPI anticipated propellant limits over its lookahead, unlike feedback-control policies.Source: §7 · p. 14
- Initial separation, horizon, sensing, opponent forecasts, and capture definitions can change policy reachability or rankings, so evaluations should report these parameters and test robustness to their changes.A short horizon can reverse policy rankings; sensing lowered capture fraction and changed which policy group won, while lookahead and opponent forecasting mattered more than sampling effort.Source: §7 · p. 14
- Future work could study far-range nonlinear encounters, teammate communication, power and environmental resource models, and whether learned-policy gains persist beyond training configurations.Far-range work could draw on reachable-set analysis; semidecentralized formulations make teammate communication explicit, while power, battery, and eclipse models add resources competing with sensing and thrust.Source: §7 · p. 14
APPENDIX A: POLICY FORMULATIONS
Appendix A gives the full policy formulations, including command limits, belief-state substitution, and the relative-state conventions used to apply pursuit laws across the games.
APPENDIX A: POLICY FORMULATIONS
- Each command is limited by Equation (7) before application, and policies that plan on a belief use the estimate ˆxk in place of xk.Source: Appendix A · p. 14
- The pursuit laws are written for the PE pursuer, the bandit, using x = sB −sG as the pursuer’s state relative to the evader.Here r and v are the position and velocity components of x, respectively.Source: Appendix A · p. 14
- The LBG guard applies the same laws to its state relative to the bandit, −x, with its own thrust limit; the LBG bandit applies them relative to the lady.Source: Appendix A · p. 14
Non-Reactive and Feedback Policies
The section specifies baseline, feedback, intercept, and planning policies for orbital encounters, including their control assumptions and implementation choices. It describes both reactive control laws and lookahead-based planners.
Non-Reactive and Feedback Policies
- Baselines include random impulses sampled uniformly within the admissible ball, direct pursuer and fleeing-evader impulses, transverse evasion, and bandit impulses rotated away from a nearby guard.The bandit rotates its direct impulse when the guard is within 300 m; transverse evasion switches its reference axis when line of sight is within 25.8° of cross-track.Source: Non-Reactive and Feedback Policies · pp. 14–15
- The PD pursuer applies proportional and derivative feedback to relative position and velocity, with Kp = 10^-4 s^-2 and Kd = 0.02 s^-1.For the unsaturated continuous-time double integrator in free space, these gains give damping ratio one and a 100 s time constant.Source: Non-Reactive and Feedback Policies · p. 15
- The LQR pursuer uses discrete Riccati-equation gains and assumes a coasting evader; constant evader thrust causes steady-state offset because the regulator lacks integral action.Its SI-unit weights scale position by 100 m, velocity by 1 m/s, and impulse by 0.1 m/s.Source: Non-Reactive and Feedback Policies · p. 15
- The HCW intercept pursuer recomputes a fixed lookahead TL each step, producing roughly exponential closing and reaching capture radius after about TL ln(d0/rc).This coasting-target behavior assumes unsaturated commands and remaining propellant; the command imposes no terminal velocity and is not fuel-optimal.Source: Non-Reactive and Feedback Policies · p. 15
- MPPI perturbs nominal action sequences, forecasts the opponent, and weights standardized costs; its default uses N = 256 samples over H = 60 steps (600 s).Noise is held for m = 3 steps, with β = 0.5, λ = 0.1, γ = 0.99, and event bonus b = 10; these values were not tuned per configuration.Source: Non-Reactive and Feedback Policies · p. 15
- Stochastic MuZero uses 256 simulations per decision and a 600 s lookahead, averaging action value over uniformly drawn opponent policies at chance nodes.Each simulation evaluates two rollouts of up to 60 steps, about 32,000 simulated steps per decision; the policy executes the most-visited root action.Source: Non-Reactive and Feedback Policies · p. 15
BIOGRAPHY[
The biographies profile three Stanford researchers whose work spans decision-making systems, communication-constrained multi-agent systems, and robust decision-making.
BIOGRAPHY
- Duncan Eddy is a SISL staff researcher focused on decision-making in safety-critical, climate, and space systems.He earned a B.S. in Mechanical Engineering from Rice in 2013 and a Ph.D. in Aerospace Engineering from Stanford in 2021; he led space operations at Capella Space and founded Amazon Project Kuiper’s Constellation Management and Space Safety Organization.Source: BIOGRAPHY[ · p. 17
- Mahdi Al-Husseini is a SISL Ph.D. candidate developing models and algorithms for semidecentralized multi-agent systems under communication constraints.He holds degrees from Georgia Tech and Stanford and is a U.S. Army captain and experimental test pilot in training.Source: BIOGRAPHY[ · p. 17
- Mykel J. Kochenderfer is a Stanford associate professor who directs SISL, researches robust decision systems, previously worked at MIT Lincoln Laboratory, and earned degrees from Edinburgh and Stanford.He is also an associate professor of computer science by courtesy; at Lincoln Laboratory he worked on airspace modeling and aircraft collision avoidance.Source: BIOGRAPHY[ · p. 17