PaperlayerView exact revision on arXiv

Source-linked AI summary

Curriculum-Based Deep Reinforcement Learning for Robust Autonomous Rocket Landing under Atmospheric Uncertainty

Atin Chowdhury, Piyush Joshi

arXiv:2610.38636v1cs.AI

TL;DR

Far-range rocket landing requires coordinating lateral approach and braking with descent and attitude control amid uncertain dynamics and imperfect observations. The paper trains a curriculum-driven PPO policy in a stochastic 3D simulator and evaluates it across 100,000 episodes, finding high aggregate success alongside reduced reliability in the most severe weather band.

  • Problem

    Far-range landing couples lateral approach and braking with descent and attitude control, while atmospheric and vehicle uncertainty changes command effects and noisy measurements obscure the physical state.

    Source: §1 · p. 1

  • Method

    The paper combines a three-stage PPO curriculum with transfer, critic warm-up, earlier-task rehearsal, and progressively widened weather severity in a stochastic environment.

    Source: §1 · p. 2

  • Results

    96.656% success across 100,000 evaluation episodes, declining from 98.84% at zero stochastic-weather severity to 88.07% in the highest severity band.

    Source: §8 · p. 19, §9 · p. 20

  • Takeaways & Limitations

    Severity-conditioned outcomes expose a measurable reliability loss under strong compound disturbances despite high aggregate performance within the evaluated distribution.

    Source: §8 · p. 19, §9 · p. 20

    The simulator uses parameterized engineering perturbations and simplified dynamics rather than flight-calibrated distributions or a high-fidelity six-degree-of-freedom vehicle model.

    Source: §8 · p. 20

Abstract

from arXiv · show

Autonomous far-range rocket landing is a challenging control problem because navigation, descent-rate regulation, attitude management, and fuel use must be coordinated in the presence of uncertain atmospheric and vehicle dynamics. This paper presents a curriculum-driven proximal policy optimization (PPO) framework for three-dimensional rocket landing in a stochastic environment incorporating altitude-dependent aerodynamic drag, wind shear, temporally correlated turbulence, discrete gusts, uncertain propulsion and aerodynamic parameters, thrust-vector misalignment, engine lag, actuator-gain variation, noisy observations, and command latency. Learning is organised into far-range approach, soft descent, and integrated landing stages. Policy transfer, critic warm-up after reward changes, rehearsal of earlier tasks, and progressive expansion of weather severity are used to support learning across these coupled phases. Across 100,000 evaluation episodes, the policy achieves 96,656 successful landings, corresponding to a success rate of 96.656\% and a 95\% Wilson confidence interval of 96.543--96.766\%. Success decreases from 98.84\% under zero-disturbance conditions to 88.07\% in the highest weather-severity band. Successful touchdowns exhibit small miss distances and low residual horizontal and vertical velocities. An exploratory logistic-regression analysis identifies mean wind speed as the strongest measured correlate of success, with initial distance, initial altitude, initial vertical velocity, and wind-shear exponent also associated with outcomes. The results quantify both the effectiveness of the learned policy over the evaluated disturbance distribution and the degradation that occurs under severe compound atmospheric conditions.

1 Introduction

Far-range rocket landing couples lateral approach, braking, descent, attitude, and fuel control, while uncertain dynamics and imperfect observations make commands difficult to predict. The paper presents a curriculum-driven PPO study that evaluates landing performance across a joint distribution of atmospheric and vehicle perturbations.

1 Introduction

  • The central challenge is coordinating lateral acceleration and braking with descent and attitude control, because locally useful corrections can leave the vehicle unable to recover before touchdown.The paper characterizes the task as nonlinear, hybrid, and constrained.Source: §1 · p. 1
  • Distant landing requires deciding when to accelerate toward the pad, when to brake, and how to coordinate those manoeuvres with descent under changing input-output dynamics and noisy state information.The paper notes that nominal-only training can produce policies that degrade when disturbances change.Source: §1 · pp. 1–2
  • The study presents curriculum-driven PPO for three-dimensional far-range landing with stochastic atmospheric and vehicle uncertainty, including wind, turbulence, gusts, propulsion variation, actuator effects, noisy measurements, and latency.The policy does not observe the atmosphere’s latent state, and training progressively raises maximum weather severity while retaining calm episodes.Source: §1 · p. 2
  • The paper evaluates landing probability and touchdown quality over 100,000 episodes, stratifying outcomes by weather severity and selected disturbances rather than testing only a nominal configuration.It also inspects successful landing quality and remaining failure modes.Source: §1 · p. 2
  • The contributions combine a parameterized 3D uncertain environment, a three-stage curriculum with transfer and rehearsal, stage-dependent reward terms, and severity-conditioned evaluation with exploratory disturbance associations.The reported analysis aims to make performance degradation visible rather than relying on a single average.Source: §1 · p. 2

2 Related Work

Prior work spans constrained optimisation, learned feedback, and curriculum or robustness methods, but comparisons depend on models and evaluation protocols. This study distinguishes itself through a three-stage far-range curriculum, progressively widened uncertainty, and a large evaluation focused on severity-conditioned outcomes.

Model-based and learning-based guidance

  • Optimisation-based guidance structures powered descent around constraints, while deep reinforcement learning offers simulation-trained feedback policies without hand-coding every manoeuvre.The paper cautions that headline success rates across reusable-rocket studies are not comparable without accounting for initial states, touchdown criteria, and test conditions.Source: §2 · p. 2
  • Prior learned-landing studies cover six-degree-of-freedom planetary descent and custom reusable-rocket environments, while newer approaches vary in dynamics, action spaces, training assistance, and evaluation.The passage cites.Source: §2 · p. 2

Curriculum learning

  • The curriculum separates far-range approach, soft descent near the pad, and integrated landing, transferring policy parameters while reinitializing the critic and temporarily freezing the actor after reward-mode changes.Earlier-stage episodes are rehearsed in later stages; these choices are intended to reduce interference, but controlled ablations are needed to quantify their individual effects.Source: §2 · pp. 2–3

Rehearsal and robustness

  • The method uses task-level rehearsal of complete earlier-stage episodes and randomizes joint atmospheric and vehicle uncertainty, so its results describe robustness over the chosen distribution rather than isolated causal effects or sim-to-real transfer.The paper states that disturbance sources are not isolated in controlled ablations.Source: §2 · p. 3

Study distinction

  • The study’s stated distinction is combining the three-stage far-range curriculum, progressively widened atmosphere and vehicle uncertainty, and a 100,000-episode evaluation of success and touchdown quality.It also explicitly discusses simulator simplification and remaining validation requirements.Source: §2 · p. 3

3 Environment and Problem Formulation

The paper formalizes a reduced-order, partially observed 3D landing simulator, with latent dynamics and disturbances distinct from the noisy 29-dimensional policy observation. The task uses stage-specific initial conditions and touchdown criteria, and aggregate success estimates apply to the specified evaluation distribution.

3 Environment and Problem Formulation

  • The simulator is a repeatable reduced-order environment, not a validated model of a particular launch vehicle; its policy receives a potentially noisy 29-dimensional observation rather than the latent state.The augmented simulator state includes hidden atmospheric and episode-level vehicle variables.Source: §3 · p. 3, §3.3 · p. 5

3.1 State, actions, and discrete-time dynamics

  • The vehicle state includes position, velocity, attitude, rates, fuel, engine state, and hidden atmospheric and episode parameters, making the augmented dynamics Markovian while the policy problem remains partially observed.The simulation runs at 60 physics frames per second, with one decision held over three frames except during command latency.Source: §3.1 · p. 3, §3 · p. 3
  • The policy selects thrust, pitch, and roll through 18 joint discrete commands, while the simplified update applies thrust along a perturbed nose direction, gravity, drag, fuel use, and attitude dynamics.The implementation uses a factorized MultiDiscrete action interface.Source: §3.1 · pp. 3–4

3.2 Atmospheric forces and wind field

  • Weather severity controls stochastic perturbations, while deterministic air density and drag remain at zero severity; the model adds simplified anisotropic drag, altitude-dependent wind shear, correlated turbulence, and finite-duration gusts.The aerodynamic weathercock torque is also simplified rather than a full aerodynamic moment model.Source: §3.2 · p. 4

3.3 Episode randomisation and observation model

  • The weather sampler draws calm episodes and severity-conditioned parameter ranges, including episode-level propulsion and actuator uncertainties that are not independently observed by the policy.The listed ranges describe an engineering model, not flight-calibrated distributions.Source: §3.3 · p. 5, Table 1 · p. 5
  • The 29-dimensional observation contains state and derived features but no wind vector; sensor noise varies with severity, and the feed-forward policy maintains no explicit recurrent state or belief over hidden parameters.Reward and touchdown checks use the underlying simulator state rather than noisy observations.Source: §3.3 · p. 5

3.4 Landing task, termination, and evaluation criterion

  • The three stages use different initial-state distributions and reward weights, and success requires touchdown within 16 simulator units of the pad with bounded vertical and horizontal speeds.Stage 3 has no separate uprightness threshold; tilt is descriptive, not part of the Boolean success label.Source: §3.4 · p. 5
  • The reported aggregate metric is empirical success probability over 100,000 episodes, estimating outcomes under the specified evaluation distribution rather than outside it.The objective is expected discounted return under the joint distribution of initial states, weather parameters, and time-varying disturbances.Source: §3.4 · p. 5

4 Proposed Architecture

The architecture links noisy observations and discrete control commands to a delayed flight environment, then stores transitions for PPO updates. Separate actor and critic branches support action selection and value estimation, while hidden disturbances require feedback from available observations.

4 Proposed Architecture

  • The landing controller must infer whether to accelerate laterally, brake, reduce descent rate, or prioritize terminal alignment because these objectives interact under imperfect measurements and variable vehicle response.Figure 1 links the policy to a flight environment with atmospheric and vehicle variation.Source: §4 · p. 6, Figure 1 · p. 6

4.1 Actor–critic policy and rollout optimisation

  • The actor maps the 29-dimensional observation to factorized thrust, pitch, and roll choices, while a separate critic estimates value; PPO updates both branches from stored rollout transitions.The action space contains 18 joint commands, with 4096 transitions gathered across eight parallel environments per rollout.Source: §4.1 · p. 7

4.2 Environment feedback and disturbance injection

  • The environment applies command latency, engine lag, thrust and control variation, atmospheric forces, and evolving wind disturbances before calculating rewards from the simulator state.The episode manager selects stage-specific initial conditions and may sample rehearsal episodes from earlier stages; wind and hardware perturbations are not exposed to the actor.Source: §4.2 · p. 7

5 Reward Function: Mathematical Formulation and Rationale

The reward combines stage-dependent guidance with local safety penalties and touchdown rewards to make long-range landing learnable without prescribing a fixed trajectory. Its shaping supports approach, braking, and touchdown, but the total return is not strictly bounded and the distance-progress term is not policy-invariant shaping.

5.1 Shared geometric quantities and directional alignment

  • Thrust-direction shaping favors pointing toward the pad below the horizontal speed envelope and counter-thrusting when approach speed exceeds it.The directional preference is continuous, and the stage-one and stage-three alignment weights differ.Source: §5 · p. 7, §5.1 · p. 8

5.2 Stage-one navigation reward

  • Stage one rewards reaching and settling inside the hover region at altitude 300, with success requiring all hover tolerances for 15 consecutive decisions.The tolerances cover distance, height error, horizontal and vertical speeds, and tilt.Source: §5.2 · p. 8

5.3 Stage-two and stage-three landing rewards

  • The landing rewards combine descent-speed tracking, near-pad position and horizontal-speed penalties, attitude and angular-rate penalties, control costs, radial progress, and thrust alignment.The vertical reference follows a braking-distance relation at altitude and enforces a minimum descent-speed magnitude near the surface.Source: §5.3 · p. 9
  • Stage two uses nonzero attitude-penalty weights, whereas stage three sets both to zero to permit tilt for lateral acceleration and counter-thrust.Attitude remains in terminal shaping.Source: §5.3 · p. 9, Table 2 · p. 9

5.4 Terminal rewards and theoretical interpretation

  • A successful touchdown receives a softness bonus, while failed touchdowns are penalized for excessive downward speed, tilt, pad miss distance, and horizontal speed.Stage three disables the upright-touchdown requirement, although terminal reward terms remain tilt-dependent.Source: §5.4 · p. 10
  • Without discounting, distance-difference rewards sum to net horizontal distance closed; with PPO’s γ = 0.995, the sum is not exactly telescoping or policy-invariant shaping.The term still provides local directional feedback at each decision.Source: §5.4 · p. 10
  • Dense progress and directional feedback, touchdown discrimination, time and thrust costs, and a 0.1 scale shape the reward, but total return has no strict global bound.Only saturated components are bounded; the progress term is not clipped.Source: §5 · p. 7, §5.4 · p. 10

6 Curriculum and PPO Training

Training combines a three-stage rocket-landing curriculum with PPO, progressive weather exposure, and rehearsal of earlier tasks. The training curves are comparatively stable in the far-approach and integrated stages but less stable in near-pad descent, and critic fit is only a training diagnostic.

6.1 Three-stage task curriculum and parameter transfer

  • Three stages sequence far approach, near-pad soft descent, and integrated landing; actor parameters transfer between stages, with critic warm-up after reward changes and rehearsal of earlier tasks.Rehearsal samples complete episodes from earlier configurations: stage two uses 15%, while stage three uses 10% from each earlier stage.Source: §6.1 · p. 11

6.2 Weather-severity curriculum

  • Weather severity expands linearly from calm conditions to the stage cap over the first half of training, then remains at that cap; 15% of episodes retain calm conditions.Because one severity level jointly scales multiple perturbations, their effects are statistically dependent, so grouped results are associations rather than isolated disturbance effects.Source: §6.2 · p. 11

6.3 PPO objective, network, and optimisation schedule

  • PPO uses a clipped policy surrogate alongside value loss and an entropy bonus, with separate actor and value branches.The network has two 128-unit tanh layers per branch; training uses eight parallel environments and stage-specific learning rates that decay linearly.Source: §6.3 · pp. 11–12

6.4 Training dynamics and interpretation of the learning curves

  • Stage 1 rewards improve rapidly and remain mostly positive later, while mean episode length eventually falls to roughly 650–850 decisions.The combination is consistent with more efficient far-approach learning, but individual trajectories still include poor returns and do not prove every episode reaches the hover region.Source: §6.4 · p. 13, Figure 2 · p. 13
  • Stage 2 reward and episode-length curves are non-monotonic, with late negative reward trends and increasing episode length; the curves alone do not establish convergence or identify the cause.Checkpoint-wise evaluation and controlled comparisons are needed to distinguish possible effects of task distribution, disturbance severity, and rehearsal.Source: §6.4 · p. 13, Figure 3 · p. 13
  • After an initial transient, Stage 3 returns remain broadly around 9–10 and episode length fluctuates mainly around 450–550 decisions, consistent with a relatively stable aggregate training regime.Individual low-return outliers remain, so final success and touchdown quality are assessed using the separate 100,000-episode evaluation.Source: §6.4 · pp. 13–14, Figure 4 · p. 14
  • Critic explained variance remains near one in Stage 1 and high through most of Stage 3, while Stage 2 has more pronounced dips; it measures critic fit, not landing success.Independent evaluation is required to assess policy success and touchdown quality.Source: §6.4 · p. 14, Figure 5 · p. 14
  • Across stages, training curves are diagnostics of the training distribution and reward function, not evidence of robustness across independent training seeds.The far-approach and integrated traces are comparatively stable, whereas Stage 2 is less stable and warrants further checkpoint evaluation.Source: §6.4 · p. 14

7 Evaluation Protocol and Results

Across 100,000 evaluation episodes, the policy’s success rate was high under the specified evaluation distribution but declined in the highest weather-severity bands. Results also describe touchdown quality, failure modes, marginal disturbance associations, and exploratory regression findings.

7.1 Overall outcomes

  • 96,656 of 100,000 episodes were successful, for a 96.656% success rate (Wilson 95% CI [96.543%, 96.766%]).The interval reflects evaluation-sampling uncertainty, not variation across independently trained policies.Source: Table 4 · p. 15, §7.1 · p. 15

7.2 Success versus weather severity

  • Success was 98.84% in the level-zero subset and 88.07% in the (0.8, 1.0] weather band, with rates near 99% through level 0.6 before dropping at higher severity.The highest band contained 105 timeouts and 1,907 touchdown failures.Source: Table 5 · p. 15, §7.2 · p. 15, Figure 6 · p. 16

7.3 Touchdown quality among successful episodes

  • Among successful episodes, mean miss distance was 6.896 ± 3.439 units, mean horizontal touchdown speed was 0.0573 ± 0.0310 units/frame, and mean absolute vertical speed was 0.1085 ± 0.0252 units/frame.These touchdown-quality statistics are conditional on the 96,656 accepted landings.Source: Table 6 · p. 16, §7.3 · p. 16
  • The success label does not establish upright landing: the maximum tilt among successful episodes was 50.71°, and no joint estimate using an added θ ≤10° condition was reported.Marginal tilt summaries cannot determine the proportion meeting all four predicates simultaneously.Source: §7.3 · p. 16

7.4 Failure modes

  • Miss distance was the most frequently violated touchdown criterion, affecting 3,179 crashes (98.2%); 491 (15.2%) violated vertical-speed limits and 452 (14.0%) horizontal-speed limits.These criterion-violation percentages overlap; 107 episodes timed out, including 105 in the highest weather band.Source: Table 7 · p. 17, §7.4 · p. 17

7.5 Sensitivity to wind, turbulence, latency, and initial state

  • Success declined across disturbance bins: it was 68.10% at mean wind speed (1.2, 1.5] units/frame, 84.95% at turbulence intensity (0.2, 0.25], and 88.27% at two-frame command latency.These are marginal associations; each grouping mixes other correlated effects.Source: Table 8 · p. 17, §7.5 · p. 17, Figure 7 · p. 18
  • Initial-state patterns were smaller: success was 97.24% in the 1,000–1,100-unit distance bin versus 95.83% in the 1,400–1,500-unit bin, and 95.89% at initial vertical velocities −1.5 to −1.1 versus 97.34% at −0.4 to −0.2 units/frame.These marginal summaries do not isolate initial-state effects from co-sampled disturbances.Source: §7.5 · p. 18

7.6 Exploratory logistic regression

  • Mean wind speed was the strongest measured correlate of success in the exploratory logistic regression (OR = 0.244, 95% CI [0.228, 0.260]); initial distance had OR = 0.839.The model was assessed in-sample (AUC 0.827), and its coefficients are conditional associations rather than causal estimates.Source: §7.6 · pp. 18–19, Table 9 · p. 19

8 Discussion

Landing success is high overall but declines under severe compound weather, while measured disturbance associations and conditional touchdown statistics help characterize where reliability degrades. Reward components and training diagnostics offer context, but the study's model and evaluation design limit attribution and generalization.

8 Discussion

  • 96.656% success across the evaluated distribution falls to 88.07% in the highest weather-severity band, from 98.84% under zero stochastic-weather severity.The decline is concentrated in severe compound conditions involving large wind, turbulence, sensor perturbations, and response delays.Source: §8 · p. 19
  • Radial progress, braking envelopes, and thrust alignment provide a plausible multi-phase learning mechanism, but no reward-term ablation isolates their causal contributions.Under discounted PPO, progress shaping is not strictly policy-invariant potential-based shaping.Source: §8 · p. 19
  • Mean wind speed is particularly strongly associated with success, but jointly sampled disturbances complicate attribution of individual effects.The regression is exploratory, and controlled single-family and factorial experiments are needed to estimate effects more cleanly.Source: §8 · p. 19
  • Successful touchdowns have moderate miss distances and low residual translational velocities, but these conditional statistics exclude unsuccessful-attempt tails.Pad miss is the most common violated criterion; excessive vertical and horizontal speed also contribute and may co-occur.Source: §8 · p. 19
  • Stage 1 and Stage 3 have comparatively stable rolling rewards, whereas Stage 2 is more non-monotonic and explained-variance dips occur particularly during Stage 2.Explained variance reflects fit to sampled return targets, not policy optimality or robust landing performance.Source: §8 · pp. 19–20
  • The findings are bounded by uncalibrated parameterized perturbations, simplified dynamics, coupled disturbances, one trained policy, and absent controlled ablations.Episode-sampling confidence intervals do not capture variation across training runs; future work proposes independent seeds, controlled and factorial tests, matched flat-PPO comparisons, and higher-fidelity evaluation.Source: §8 · p. 20

9 Conclusion

The paper trains a curriculum-driven PPO policy for far-range landing under atmospheric and vehicle uncertainty. Across 100,000 episodes it achieves 96.656% success, while the conclusions remain limited to the evaluated simulator and disturbance distribution.

9 Conclusion

  • Three curriculum stages combine policy transfer, critic warm-up, rehearsal, progressive weather severity, and stage-dependent rewards for approach, braking, descent, and terminal correction.The policy operates from normalized observations that do not expose latent wind or all episode-level hardware parameters.Source: §9 · p. 20
  • 96,656 of 100,000 evaluation episodes succeed, for 96.656% success; success falls from 98.84% at zero weather severity to 88.07% in the highest band.Mean wind speed is the strongest measured correlate of success, with additional associations involving turbulence, command latency, and initial-state variables.Source: §9 · p. 20
  • The findings are bounded by simplified dynamics, uncalibrated and coupled perturbations, single-policy evaluation, and absent controlled ablations.The paper identifies independent seeds, isolated uncertainty tests, matched flat-PPO comparisons, and higher-fidelity or hardware-in-the-loop evaluation as future work.Source: §9 · p. 20
Loading 2610.38636v1…