PaperlayerView exact revision on arXiv

Source-linked AI summary

Curriculum-Driven Deep Reinforcement Learning for Autonomous Far-Range Rocket Landing

Atin Chowdhury, Piyush Joshi

arXiv:2610.35954v1cs.LGcs.AI

TL;DR

Rocket-landing RL studies often use flat training or simplified, near-pad starts, limiting evidence on far-range approaches. This paper trains a three-stage curriculum-driven policy with warm-up, rehearsal, rotation-matrix observations, and redesigned rewards; it reports a 99.98% success rate versus 68.31% for the baseline in its evaluation.

  • Problem

    Prior RL approaches often use flat training or simplified dynamics and near-pad initial conditions, leaving far-range lateral approach less addressed.

    Source: §1 · p. 1

  • Method

    A three-stage curriculum with critic warm-up and rehearsal trains a policy using rotation-matrix observations and a compact O(1)-magnitude reward.

    Source: §1 · pp. 1–2

  • Results

    99.98% versus 68.31% success rate is reported for the proposed policy and baseline, respectively, on the evaluation distribution.

    Source: §10 · p. 15

  • Takeaways & Limitations

    The evaluation reports that the proposed policy satisfies all four touchdown-quality conditions in 99.98% of episodes.

    Source: §10 · p. 15

    The deterministic simulator does not fully represent real-world uncertainties, so the reported success rate applies to the tested simulation setup rather than real-world landing reliability.

    Source: §9 · p. 15

Abstract

from arXiv · show

Autonomous vertical landing of reusable launch vehicles is a critical yet formidable challenge in aerospace engineering, primarily due to highly nonlinear dynamics, stringent precision require- ments, and inherent environmental uncertainties. While reinforcement learning (RL) has emerged as a promising alternative to traditional model-based control, existing approaches frequently rely on flat training paradigms and highly simplified initial conditions, drastically limiting their real-world applicability. This paper proposes a comprehensive curriculum-driven deep reinforcement learning framework designed to master both far-range navigation and precision landing. By employing a three-stage curriculum augmented with a critic warm-up procedure and a rehearsal mechanism to prevent catastrophic forgetting, alongside a high-dimensional observation vector utilizing a full 3 X 3 rotation-matrix, the agent is trained to navigate from an initial horizontal distance of 1,000 to 1,500 units vastly farther than prior benchmarks before executing a soft touchdown. Crucially, we intro- duce a redesigned reward function that reduces unnecessary computational overhead and addresses limitations of earlier formulations, including poorly calibrated reward components and potentially conflicting optimization objectives. The redesigned function bounds the per-step reward magnitude by (O(1)), while its progress term accumulates the total distance closed over the trajectory. The available 100,000-episode results report a 99.98% success rate for the proposed policy and 68.31% for the baseline under the evaluation conditions. In the evaluation, the proposed policy retained 69.1% of its initial fuel capacity and achieved a mean final distance from the pad centre of 7.07 units, compared with 13.10 units for the baseline. Landing visualizations are available in Video 1 and Video 2.

1 Introduction

The paper targets rocket landing beyond close-pad starting conditions with a learning-driven curriculum and redesigned observations and rewards. It reports a 99.98% success rate for the proposed policy under its evaluation conditions.

1 Introduction

  • Model-based landing systems can be reliable, but rely on accurate dynamics models and extensive manual calibration that may be unavailable in uncertain environments.RL offers an alternative through policy discovery by simulator interaction without a hand-designed control law.Source: §1 · p. 1
  • Prior RL landing studies often use simplified dynamics or start near the pad, making lateral approach a trivial part of the task.Source: §1 · p. 1
  • A three-stage curriculum, critic warm-up, rehearsal, rotation-matrix observations, and a compact O(1)-magnitude reward support far-range navigation and landing.The curriculum covers far approach, soft descent, and full landing; the initial horizontal reach is 1,000–1,500 units.Source: §1 · pp. 1–2
  • 99.98% success rate is reported for the proposed policy under the evaluation conditions.Source: §1 · p. 2

2 Related Work

Prior rocket-landing research spans model-based guidance and several RL strategies, but often uses simplified conditions or controller-assisted bootstrapping. This work distinguishes itself with a learning-driven curriculum and a much farther initial approach.

2 Related Work

  • Powered-descent optimization relies on accurate models and solves a convex optimization problem at each guidance cycle, while deep RL studies have reported comparable or better accuracy and faster inference.Source: §2 · p. 2
  • Earlier 3D RL approaches used controller or synthetic-curriculum scaffolds, whereas this curriculum is learning-driven and needs no classical controller.Source: §2 · p. 2
  • Prior work also explored adaptive PPO, integrated guidance and control, model-based RL, hierarchical guidance, and FPGA-accelerated inference.The paper places its curriculum in a broader literature that includes rehearsal and transfer mechanisms across curriculum stages.Source: §2 · p. 2
  • Chowdhury et al. [1] reported 84% success over 100 episodes from near the pad, while the present work identifies flat training, coarse success criteria, and limited evaluation as gaps.Source: §2 · p. 3
  • The proposed work uses a multi-stage curriculum, critic warm-up, rehearsal, full rotation-matrix observations, and 1,000–1,500-unit initial displacements, with evaluation over 100,000 episodes.Source: §2 · p. 3

3 Environment and Problem Formulation

The environment is a deterministic 3D rocket simulator with discrete agent decisions and explicit touchdown-quality criteria. The state representation encodes attitude and body-frame position and velocity projections, while actions jointly select thrust, pitch, and roll.

3 Environment and Problem Formulation

  • A decision aggregates three physics frames in a deterministic simulator running at 60 Hz; lengths use simulator units and the fuel capacity is 1,000 units.Source: §3 · p. 3
  • Touchdown quality requires tilt ≤10°, pad distance ≤16 units, vertical velocity ≥−0.15 units/frame, and horizontal speed ≤0.25 units/frame simultaneously.Source: §3 · p. 3
  • Attitude uses a full orthonormal 3 × 3 rotation matrix, whose columns represent the ship’s body X axis, nose direction, and body Z axis in world coordinates.Source: §3 · p. 3
  • The 29-dimensional observation includes body-axis projections of horizontal position and velocity that indicate which tilt opposes drift, alongside horizontal distance and radial velocity.Source: §3 · p. 3
  • The stopping-margin calculation uses net deceleration of 0.013 units/frame2, derived from thrust force 0.020 minus gravity magnitude 0.007.Source: §3 · p. 4
  • The joint discrete action space has 18 simultaneous choices: binary thrust and three choices each for pitch and roll.Source: §3 · p. 4

4 Reward Function Design

The reward is designed to guide far-range navigation and soft landing with stage-specific shaping. Its progress term accumulates distance closed, while per-step penalties provide local shaping and remain bounded.

4.1 Stage One Reward (Far Approach)

  • Stage one rewards progress toward a hover point from 1,000–1,500 units away, combining velocity caps and alignment incentives.Success requires meeting the hover conditions for 15 consecutive decisions; leaving workspace bounds terminates with failure.Source: §4.1 · p. 4

4.2 Stage Two and Stage Three Reward (Soft Descent and Full Task)

  • Stages two and three share a landing reward anchored on a soft-descent reference, but use different initial distributions and tilt-penalty weights.Stage two includes tilt penalties, while stage three sets them to zero and uses tilt for navigation and braking.Source: §4.2 · pp. 4–5, Table 1 · p. 5
  • The stage-three progress term accumulates distance closed, contributing about +9 scaled return for a traverse from roughly 1,131 to 7 units.Pad-distance and horizontal-velocity penalties are fine-grained shaping terms active only near the pad.Source: §4.2 · p. 5

4.3 Terminal Adjustments

  • Terminal adjustments reward touchdown softness and penalize leaving the workspace; successful stage-three returns are about +10, while crash penalties make unsuccessful returns strongly negative.Per-decision rewards and terminal adjustments are multiplied by 0.1.Source: §4.3 · pp. 5–6

4.4 Comparison with the Baseline Reward

  • The redesigned reward has O(1) per-step magnitude versus O(10^2) for the baseline, with per-episode returns within ±15 instead of O(10^4).The paper reports that actor and critic gradients remain balanced under the new reward.Source: §4.4 · p. 6

5 Proposed Architecture

The proposed architecture closes the loop between a 6-DoF flight simulator and a PPO actor-critic agent. The simulator produces observations and rewards, while the agent selects actions and updates its policy from collected rollouts.

5 Proposed Architecture

  • Figure 1 depicts the closed-loop interaction between the PPO agent and physics simulator, including observation, action, reward, and update flows.Solid arrows mark interaction, dashed arrows gradient or parameter updates, and dotted arrows rollout data.Source: Figure 1 · p. 7, §5 · p. 7

5.1 The 6-DoF Flight Simulator (Environment)

  • The custom simulator advances rocket state using rigid-body, thrust, gimbal, and gravity dynamics, then computes normalized, clipped observations and stage-specific rewards.Its observation includes position, velocity, a 3 × 3 orientation matrix, angular velocity, fuel, and braking margins.Source: §5.1 · p. 7

5.2 The PPO Actor-Critic Agent

  • The PPO agent encodes observations and uses separate actor and critic networks to select among 18 joint actions and estimate state value.Collected transitions are stored in a rollout buffer for PPO updates using clipped surrogate and value losses.Source: §5.2 · pp. 7–8

6 Training Methodology

Training uses PPO in three curriculum stages with stage-specific timestep budgets and learning rates. Critic warm-up and rehearsal episodes support transitions between stages and retention of earlier-stage learning.

6 Training Methodology

  • PPO training spans three stages, using 4 × 10^6, 2.5 × 10^6, and 8 × 10^6 timesteps with learning rates 3 × 10^-4, 1.5 × 10^-4, and 1 × 10^-4, respectively.All stages use a 0.995 discount factor and linear learning-rate decay to 10%.Source: §6 · p. 8, Table 3 · p. 8
  • After stage one, the actor is frozen for 100,000 timesteps while the reinitialized critic warms up; rehearsal mixes earlier-stage episodes into later training.Stage two includes 15% stage-one episodes; stage three includes 10% each from stages one and two.Source: §6 · p. 8

7 Training Diagnostics

Training diagnostics trace learning across far approach, soft descent, and full landing, showing how critic conditioning and rehearsal shape each stage. Reward and episode-length curves document the agent’s progression toward the composite task.

7 Training Diagnostics

  • The section examines reward, episode-length, and critic-conditioning curves; smoothing is used only for readability, with raw values preserved.The diagnostics are used to assess the curriculum and critic warm-up across the training stages.Source: §7 · p. 8

7.1 Stage One: Far Approach

  • In stage one, reward rises as the agent learns the far approach, while episode length settles as it finds an efficient route to the hover zone.The critic’s return estimate becomes exact before transfer, avoiding critic noise during stage two’s warm-up window.Source: §7.1 · p. 9, Figure 2 · p. 9, Figure 3 · p. 9

7.2 Stage Two: Soft Descent

  • Stage-two value loss spikes after the reward change, then critic recovery during warm-up accompanies a steady reward climb; rehearsal also adds variance through occasional stage-one episodes.Episode length stabilises as the agent learns the vertical braking profile.Source: §7.2 · p. 10, Figure 5 · p. 10, Figure 4 · p. 10

7.3 Stage Three: Full Task

  • The full-task reward stabilises as the policy integrates far approach and soft descent, while rehearsal makes episode length variable and the inherited critic requires no warm-up.The critic remains well-conditioned from timestep zero because stage three inherits the reward and critic from stage two.Source: §7.3 · p. 11, Figure 6 · p. 11, Figure 7 · p. 11

8 Experimental Setup and Results

The evaluation compares the proposed policy with the Chowdhury et al. baseline over 100,000 test episodes, after sampling the proposed policy’s initial horizontal distance from 1,000 to 1,500 units. It reports success rates, landing measurements, and touchdown-condition compliance.

8.1 Evaluation Protocol

  • The evaluation reports sample means and standard deviations over 100,000 episodes, with initial horizontal distance sampled uniformly from 1,000 to 1,500 units.Coordinate-wise absolute offsets |x0| and |y0| are distinguished from radial horizontal distance; observed tilt compliance is assessed separately from the Stage 3 success flag.Source: §8.1 · pp. 11–12, Table 4 · p. 12

8.2 Results

  • The proposed policy retained 691 of 1,000 units of initial fuel capacity, approximately 69.1%.This result is reported for the far-range evaluation.Source: §8.2 · p. 13
  • 99.98% success for the proposed policy versus 68.31% for the Chowdhury et al. baseline on the 100,000-episode test set.The proposed policy had 99,980 successful episodes.Source: §8.2 · pp. 12–13, Table 6 · p. 13
  • 7.07 ± 1.99 units was the proposed policy’s mean final distance from the pad centre, versus 13.10 ± 5.1 for the baseline.The proposed method began at a radial horizontal distance sampled uniformly from 1,000 to 1,500 units.Source: §8.1 · p. 11, §8.2 · p. 13, Table 7 · p. 14
  • All touchdowns met the tilt, vertical-speed, and horizontal-speed thresholds; the 20 episodes failing the joint criterion exceeded the pad radius.The four-condition assessment checks measured touchdown tilt even though the saved Stage 3 configuration does not enforce an upright-touchdown constraint in its success flag.Source: §8.1 · p. 12, §8.2 · p. 14, Table 8 · p. 14

9 Discussion

The discussion reports stronger far-range landing performance than the baseline and describes how curriculum stages, observation design, and reward scaling support training. It also identifies limits on the interpretation and generality of the evaluation.

9 Discussion

  • The three stages decompose far approach, soft descent, and full landing; critic warm-up protects the transferred actor, while rehearsal preserves earlier skills during composite-task training.Stage one converges in 4 × 10^6 timesteps; stage two uses 100,000 timesteps of critic warm-up.Source: §9 · p. 14
  • The rotation-matrix observation exposes horizontal distance, radial velocity, and braking margins, allowing a two-layer 128-unit network where the baseline used three 256-unit layers.The supplied body-relevant quantities let the agent plan its approach.Source: §9 · p. 14
  • O(1) per-step rewards keep episode returns within ±15 and value loss small, unlike the baseline’s fifteen-constant reward clipped to [−500, +500].Figures 3, 5, and 7 show smoothly declining value loss and explained variance stabilising near 1.0.Source: §9 · p. 14
  • 99.98% success vs 68.31% for the baseline; mean final distance was 7.07±1.99 units vs 13.10±5.1, despite starting ≈1250 units away.The proposed policy retained 69.1% of its fuel capacity in the far-range evaluation.Source: §9 · p. 14
  • γ = 0.995 gives an ≈200-decision credit-assignment horizon, shorter than the ≈985 decisions per evaluation episode; the study does not establish that this discount is optimal.A sweep over γ ∈ {0.99, 0.995, 0.999} is proposed as future work.Source: §9 · p. 15
  • Generality remains unestablished beyond PPO or the deterministic simulator, and the eight hand-tuned reward constants lack systematic sensitivity and transferability evaluation.The authors also call for a flat-PPO ablation, multiple random seeds, and perturbation studies of initial conditions, vehicle parameters, propulsion, actuators, and disturbances.Source: §9 · p. 15

10 Conclusion

The conclusion presents a three-dimensional rocket-landing framework combining curriculum learning, observation and action design, and a redesigned reward. In the tested distribution, it reports higher success and closer landings than the baseline, while identifying robustness to real-world disturbances as future work.

10 Conclusion

  • 99.98% success for the proposed policy vs 68.31% for the baseline; mean final distance was 7.07 ± 1.99 units vs 13.10 ± 5.1, despite the proposed method starting over 1,000 units away.The proposed policy satisfies all four touchdown-quality conditions, including the 10° tilt threshold, in 99.98% of episodes.Source: §10 · p. 15
  • The framework combines a far-approach, soft-descent, and full-task curriculum with critic warm-up, rehearsal, full rotation-matrix observations, a joint multi-discrete action space, and O(1) per-step rewards.The reward design uses eight constants rather than the baseline’s fifteen.Source: §10 · p. 15
  • The evaluation is simulation-based, and future work targets sensor noise, actuator faults, and real-world aerodynamic disturbances.The paper characterizes the far-range benchmark as a step up in task difficulty that more closely reflects returning-booster conditions.Source: §10 · p. 15
Loading 2610.35954v1…