Source-linked AI summary
Adversarial Reinforcement Learning for Robust Planetary Landing
Miguel Leiva-Vélez, Nicolas Gaston-Rozado, Andrea Scorsoglio, Hodei Urrutxua, Víctor Rodríguez-Fernández
TL;DR
Stochastic perturbation sampling may miss mission-relevant disturbances with spatial or temporal structure. This work trains a landing controller with an adversarial disturbance-generating agent, and preliminary results show that adversarial sequences expose vulnerabilities missed by stochastic training, alongside a loss of precision in milder conditions.
Problem
Independent stochastic sampling may miss mission-relevant disturbances with spatial or temporal correlation, leaving some challenging scenarios unexplored.
Source: §1 · p. 1Method
An adversarial agent generates physically constrained, performance-degrading perturbations in an asymmetric sequential zero-sum game with the landing controller.
Source: §1 · p. 1Results
Takeaways & Limitations
Adversarial agents can serve as active disturbance discoverers, but uniform performance across the perturbation envelope may require broader training schemes.
Source: §6 · p. 5The initial adversarial campaign shows partial specialization to its regime, with some loss of precision in milder conditions.
Source: §6 · p. 4
Abstract
from arXiv · showhide
Deep reinforcement learning (DRL) policies for planetary landing and other aerospace guidance, navigation, and control tasks are commonly trained under randomized disturbances to improve robustness. However, typical stochastic perturbation sampling assumes independent variations, whereas real mission disturbances often exhibit spatial and temporal correlation and cannot be reproduced through independent random draws. This work investigates adversarial reinforcement learning to discover challenging off-nominal conditions. A secondary agent generates physically constrained perturbations that degrade the performance of a landing controller, forming an asymmetric sequential zero-sum game. Training proceeds in three stages: baseline controller learning, robustness enhancement with stochastic perturbations, and an iterative adversarial phase in which the controller and disturbance generator are alternately optimized. Preliminary experiments in a three-degree-of-freedom landing environment show that adversarial disturbances reveal vulnerabilities missed by conventional randomization and provide early indications that, with a more extensive training campaign, the approach can yield improved robustness and stronger generalization under previously unseen perturbation regimes.
1 Introduction
Planetary landing is a safety-critical GNC task requiring robust performance under environmental and parametric uncertainty. This work proposes adversarial reinforcement learning to find challenging disturbances, with preliminary results indicating vulnerabilities missed by stochastic randomization.
1 Introduction
- Planetary landing is safety-critical, with nonlinear dynamics, limited control authority, and environmental and parametric uncertainty.The vehicle must satisfy terminal constraints while maintaining safety and fuel efficiency under adverse conditions.Source: §1 · p. 1
- DRL approaches commonly use stochastic perturbations or domain randomization during training to improve robustness.DRL supports learning closed-loop policies in simulation for nonlinear dynamics and long-horizon objectives.Source: §1 · p. 1
- Stochastic sampling may miss mission-relevant disturbances because they evolve sequentially, interact with vehicle dynamics and flight phase, and exhibit spatial or temporal correlation.Such structured scenarios occupy a small subset of the disturbance space and are unlikely to be found through independent random sampling.Source: §1 · p. 1
- A secondary agent generates temporally correlated perturbation sequences to degrade controller performance in an asymmetric sequential zero-sum game.The adversary searches for difficult but physically plausible scenarios.Source: §1 · p. 1
- Adversarially generated disturbances expose vulnerabilities missed by conventional stochastic randomization in a three-degree-of-freedom lunar landing environment.The authors report that this approach may also help policies remain effective under previously unseen and more severe perturbation regimes.Source: §1 · p. 1
2 Related Work
Planetary landing research includes classical model-based control and newer DRL approaches. This work situates adversarial reinforcement learning as a way to search sequentially for performance-degrading disturbances that stochastic sampling may undersample.
2 Related Work
- Classical guidance and model-based optimal control remain dominant in practice because of their interpretability and reliability.ZEM/ZEV-type laws provide compact nominal solutions [1], while convex optimization-based powered descent offers real-time trajectory generation with explicit constraints and theoretical guarantees.Source: §2 · p. 1
- Prior DRL studies span powered-descent control, algorithm comparisons, adaptive recurrent policies, hybrid guidance, transfer, and landing-site selection [1] [4] [6] [8].These approaches remain sensitive to reward design and distribution shift.Source: §2 · pp. 1–2
- Stochastic perturbation sampling may undersample rare mission-relevant sequences, whereas adversarial reinforcement learning models disturbance generation as a sequential search for performance-degrading scenarios [6].The paper investigates this approach for DRL-based planetary landing.Source: §2 · p. 2
3 Problem Formulation
The paper formulates landing as a three-degree-of-freedom point-mass problem with uncertainty and partial observability. A controller and adversary have opposing objectives, with the adversary selecting physically admissible disturbances over time.
3 Problem Formulation
- Landing is modeled in three degrees of freedom with a point-mass vehicle in a target-centered inertial frame.The state includes position, velocity, and mass.Source: §3 · p. 2
- The translational dynamics include effective thrust and external disturbances; the model also accounts for internal actuation imperfections.The passage identifies commanded thrust, local gravity, and specific impulse, and notes examples of internal imperfections such as thrust bias, misalignment, or degradation.Source: §3 · p. 2
- The controller seeks soft-landing conditions while respecting mission and control constraints throughout descent.The reward is designed to promote safe, accurate, and fuel-efficient descent.Source: §3 · p. 2
- The controller operates under uncertainty and partial observability, receiving noisy observations through an observation model.The task is modeled as a partially observable Markov decision process.Source: §3 · p. 2
- A second agent selects physically admissible disturbances over time, while the controller seeks successful landing and the adversary degrades performance through bounded perturbation sequences.Their interaction is formulated as an asymmetric zero-sum game.Source: §3 · p. 2
4 Methodology
The methodology uses staged training, then alternates controller and adversary optimization within the stochastic-training disturbance bounds. The recurrent adversary generates smooth disturbance sequences, and robustness is evaluated across in-distribution and unseen perturbation regimes.
4 Methodology
- Training progresses from a nominal baseline to stochastic perturbation fine-tuning and then iterative adversarial training against the robustified controller.The environment is a simplified 3-DOF setting used as an intermediate step toward full six-degree-of-freedom dynamics.Source: §4 · p. 2
- The adversary controls external-force magnitude and direction, thrust misalignment, and specific-impulse degradation using bounded increments that allow smooth temporal evolution.Its perturbations remain within the bounds used during stochastic robustness training.Source: §4 · p. 2
- The lander reward uses progress along a position-dependent reference-velocity field that transitions to near-vertical descent in the terminal phase.The progress term penalizes motion beyond the admissible local rate, while the adversary uses a sparse terminal objective proportional to final landing error.Source: §4 · pp. 2–3
- The adversary receives the lander’s information through a recurrent state to capture temporal correlations across the trajectory.The observation channels include vehicle and guidance variables, internal perturbation states, and external perturbation variables, all normalized before network input.Source: §4 · p. 3
- Robustness evaluation distinguishes in-distribution perturbations within training magnitudes from out-of-distribution tests using larger unseen perturbations.The protocol sweeps external force, thrust misalignment, and specific-impulse efficiency loss independently and in combination, with randomized disturbance directions and 100 Monte Carlo rollouts per scenario.Source: §4 · p. 3
- Both the main policy and adversarial policy use Proximal Policy Optimization (PPO) [10].The lander outputs a continuous four-dimensional action whose first three components specify normalized thrust direction and whose fourth sets thrust magnitude.Source: §4 · pp. 2–3
5 Results
The results show that stochastic fine-tuning compensates for perturbation-induced degradation, while adversarial training exposes difficult disturbance sequences and shows stronger robustness at higher disturbance levels. Nominal evaluation also shows stable descent to the landing region.
5 Results
- Under nominal conditions, the baseline policy converges quickly and stably, and representative trajectories reach the landing region with a stable descent geometry.The authors attribute nominal-state generalization to the reward formulation's consistent training signal.Source: §5 · p. 3, Figure 2 · p. 3
- Stochastic fine-tuning rapidly compensates for perturbation-induced performance loss, but increasing disturbance levels still reduce landing precision, particularly during terminal descent.The relatively small variation in subplot b) of Figure 1 reflects this compensation.Source: §5 · p. 3, Figure 1 · p. 3
- Adversarial training exposes sequences that degrade the frozen stochastic policy even within its training bounds, followed by alternating adversary-exploit and controller-adapt behavior.The adversarially hardened policy shows better landing precision than the stochastically fine-tuned policy against adversarial disturbances.Source: §5 · pp. 3–4, Figure 3 · p. 4
- The adversarially trained controller shows stronger robustness at higher disturbance levels, while stochastic fine-tuning improves performance across most tested perturbation conditions.Some degradation remains in lower-perturbation regions; the adversarially trained policy also shows better landing precision than the stochastically fine-tuned policy against adversarial disturbances.Source: §5 · p. 4, Figure 3 · p. 4
6 Discussion
Adversarial training exposes failure modes missed by stochastic training and improves performance under stronger perturbations, but can reduce precision in milder conditions. The study is preliminary, and uniform performance across the perturbation envelope may require broader training schemes.
6 Discussion
- The nominal PPO policy learns stable, generalizable descent under randomized initial conditions, providing a baseline for later robustness analyses.The discussion says the reward formulation shapes landing behavior without additional guidance heuristics.Source: §6 · p. 4
- Stochastic perturbation training improves robustness over the nominal policy, yet temporally correlated sequences within the same admissible bounds can still cause substantial degradation.Random-sampling-only assessments may overestimate resilience, particularly during terminal descent.Source: §6 · p. 4
- Adversarial training exposes failure modes stochastic training misses and improves performance under stronger perturbations, but shows some loss of precision in milder conditions.The discussion identifies partial specialization to the adversarial regime and calls for interleaving nominal, stochastic, and adversarial scenarios.Source: §6 · p. 4
- The study is preliminary; uniform performance across the full perturbation envelope will likely require broader schemes such as mixed-distribution curricula or league-based training.The authors describe adversarial agents as active disturbance discoverers and mention league-based training inspired by AlphaStar.Source: §6 · p. 5