PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for Six Degree-of-Freedom Planetary Powered Descent and Landing

Brian Gaudet, Richard Linares, Roberto Furfaro

arXiv:1810.08719v1eess.SYcs.SY

TL;DR

Powered descent demands accurate state estimation and control that can convert the lander’s state into engine commands for pinpoint landing. The paper learns an integrated guidance and control policy with reinforcement learning, then reports pinpoint-landing performance and robustness in simulation.

  • Problem

    Powered descent requires guidance and control that can use estimated lander state to achieve pinpoint landing while addressing actuator-command mapping in a 6-DOF setting.

    Source: §I · p. 5

  • Method

    Reinforcement learning learns a global integrated policy mapping the navigation system’s estimated state directly to thrust commands for each engine.

    Source: §I · p. 6

  • Results

    The authors report pinpoint accuracy, soft landings, large divert capability, and robustness to noise and parameter uncertainty in simulation.

    Source: §V · pp. 32–33

  • Takeaways & Limitations

    The policy does not require a cone-shaped glide-slope constraint, allowing targets such as the bottom of a deep crater.

    Source: §V · p. 33

    The RL approach lacks local or global stability guarantees, and hard state constraints may require manual reward tuning.

    Source: §I · p. 8

Abstract

from arXiv · show

Future Mars missions will require advanced guidance, navigation, and control algorithms for the powered descent phase to target specific surface locations and achieve pinpoint accuracy (landing error ellipse $<$ 5 m radius). The latter requires both a navigation system capable of estimating the lander's state in real-time and a guidance and control system that can map the estimated lander state to a commanded thrust for each lander engine. In this paper, we present a novel integrated guidance and control algorithm designed by applying the principles of reinforcement learning theory. The latter is used to learn a policy mapping the lander's estimated state directly to a commanded thrust for each engine, with the policy resulting in accurate and fuel-efficient trajectories. Specifically, we use proximal policy optimization, a policy gradient method, to learn the policy. Another contribution of this paper is the use of different discount rates for terminal and shaping rewards, which significantly enhances optimization performance. We present simulation results demonstrating the guidance and control system's performance in a 6-DOF simulation environment and demonstrate robustness to noise and system parameter uncertainty.

I. Introduction

Pinpoint Mars landing requires accurate state estimation and guidance that can translate estimates into feasible engine commands during a short powered descent. The paper motivates an integrated learned policy against the limits of separately optimized systems and prior approaches that did not demonstrate generalized 6-DOF landing control.

I. Introduction

  • A 6-DOF integrated policy is aimed at the actuator-mapping challenge left open by prior 3-DOF pinpoint-landing simulations.Earlier navigation work demonstrated pinpoint accuracy in 3-DOF Monte Carlo simulations, but did not address mapping commanded acceleration to actuator commands.Source: §I · pp. 2, 9
  • Separate guidance and control can reduce fuel efficiency, limit maneuvers, encounter actuator saturation, and require redesign after combined-system simulations.The reference trajectory must also remain implementable by the lander’s thrusters.Source: §I · p. 5
  • Prior Mars IGC work lacked evidence of generalization across deployment conditions and relied on simplified dynamics or unrealistic thrust and torque capabilities.Prior lunar IGC also required a separate controller to map guidance outputs to individual engine commands and was demonstrated on a single trajectory.Source: §I · p. 6
  • The proposed RL policy maps estimated lander state directly to each engine’s thrust command and is trained over randomly generated initial conditions across the deployment ellipse.The reward can incorporate thrust, glide-slope, attitude, rotational-velocity, and terrain-avoidance constraints; the authors do not guarantee optimality.Source: §I · p. 6
  • RL can be optimized and verified with high-fidelity dynamics, but the approach lacks local or global stability guarantees and hard state constraints may require reward tuning.The authors contrast this with model-based control guarantees that depend on the system model accurately representing actual dynamics.Source: §I · p. 8
  • The paper identifies this as the first published demonstration of an RL-derived integrated guidance and control system for planetary landing in a 6-DOF environment.The authors report that solutions trained over limited initial-condition ranges often fail when those ranges are extended.Source: §I · p. 9

A. Problem Statement

The problem statement casts powered descent as a 6-DOF landing task in which a policy must guide the lander to a target state while satisfying flight and system constraints. It frames this as reward minimization, with hard constraints enforced differently from soft constraints.

A. Problem Statement

  • The RL formulation seeks a 6-DOF closed-loop policy for quasi-fuel-optimal trajectories that reach the landing site with pinpoint accuracy while satisfying flight and system constraints.The stated objectives include terminal position and velocity, attitude, and control effort.Source: A. Problem Statement · pp. 10–12
  • The baseline minimum-fuel formulation minimizes thrust-related cost and flight time subject to equations of motion, boundary conditions, glide-slope, and thrust constraints.The lander is modeled in a target-centered frame with position and velocity vectors; thrust magnitude has minimum and maximum bounds.Source: A. Problem Statement · pp. 10–11
  • Hard attitude constraints terminate an episode with a negative reward, while soft constraints are encouraged through rewards without premature termination.The listed constraints include terminal glide slope and pitch and roll limits; the equations of motion are set by the environment.Source: A. Problem Statement · p. 12

B. Equations of Motion

The motion model combines thruster-generated body-frame forces and torques with translational and rotational dynamics. It also specifies disturbance and lander assumptions, including a four-thruster configuration that does not directly control yaw rate.

B. Equations of Motion

  • The model sums each thruster’s body-frame force and torque, then uses attitude to relate body-frame thrust to inertial-frame motion.Each thruster is described by its direction and position vectors, and commanded thrust is bounded by its minimum and maximum values.Source: B. Equations of Motion · pp. 12–13
  • Attitude is represented by a quaternion, rotational velocity follows Euler rotational dynamics, and translational motion is modeled in the inertial frame.The attitude is updated by integrating its differential kinematics.Source: B. Equations of Motion · pp. 13–14
  • The model treats the lander as a uniform-density ellipsoid and includes normally distributed environmental disturbances; a 100 m/s wind is estimated to exert 162 N.The dynamics are integrated with a 0.05 s Runge–Kutta time step, while navigation updates arrive every 0.2 s.Source: B. Equations of Motion · pp. 14–15
  • The simulated lander has four throttleable thrusters rated from 1000 N to 5000 N, but the configuration provides no direct control of rotation about the z-axis.Yaw can still change through coupling with pitch via roll; the authors note that angled thrusters could provide direct yaw control.Source: B. Equations of Motion · p. 14

C. Policy Gradient Method

The policy-gradient framework learns actions through episodic interaction, using a policy that maps observations to actions. Its gradient updates favor actions associated with higher returns, with value-function baselines used to reduce variance.

C. Policy Gradient Method

  • An episode consists of observations, actions, and rewards; the agent uses each observation to choose an action, and the environment returns a next observation and reward.Episodes may end when the task is completed or a constraint is violated.Source: C. Policy Gradient Method · pp. 15–16
  • When observations do not satisfy the Markov property, concatenating recent observations or using recurrent networks can help infer motion from observation history.The paper notes that recurrent networks can be used for both policy and value-function implementations.Source: C. Policy Gradient Method · p. 16
  • The policy-gradient objective is an expectation over trajectories, and the value-function parameter w denotes a function parameterization.The trajectory return discounts rewards over time.Source: C. Policy Gradient Method · pp. 16–17
  • For a stochastic policy, actions are sampled from a distribution conditioned on the observation, enabling a policy-gradient expression for updating policy parameters.The policy-gradient estimate can use Monte Carlo rollouts; replacing immediate rewards with state-action values and subtracting a state-value baseline yields the Advantage-Actor-Critic formulation.Source: C. Policy Gradient Method · pp. 16–17

D. Proximal Policy Optimization

Proximal Policy Optimization approximates trust-region policy updates with a clipped objective. The method uses advantage estimates and updates its policy and value function from batches of trajectories.

D. Proximal Policy Optimization

  • PPO approximates the trust-region update by clipping its objective to constrain policy adjustment and help prevent drastic changes between updates.The trust-region approach uses a KL-divergence constraint on policy changes.Source: D. Proximal Policy Optimization · pp. 17–18
  • PPO’s advantage estimate is the empirical return minus a state-value baseline, while the value-function parameters are learned with a cost function.Policy and value-function updates use batches of episode trajectories, with gradient ascent on the policy and gradient descent on the value function.Source: D. Proximal Policy Optimization · p. 18
  • The implementation uses a Gaussian action distribution with diagonal covariance and adjusts the clipping parameter to target a policy-update KL divergence of 0.001.Policy and value function are learned concurrently, and exploration variance affects action log probabilities.Source: D. Proximal Policy Optimization · p. 18

E. Implementation Details

Implementation choices include adaptive optimization and input scaling intended to support policy and value-function learning. The clipping parameter and optimizer step size are adjusted during training.

E. Implementation Details

  • The implementation uses ADAM for both networks and approximates Gaussian KL divergence with the mean-square difference between pre- and post-update action log probabilities.The approximation is described as close to exact KL divergence and is used in the PPO2 baseline.Source: E. Implementation Details · p. 19
  • The clipping parameter is adjusted based on policy-update KL divergence, while the ADAM step size is scaled by a parameter ζ.The stated bounds are ϵ_max = 0.01, ϵ_min = 0.5, ζ_min = 0.1, and ζ_max = 10.Source: E. Implementation Details · p. 19
  • Inputs are scaled using statistics updated incrementally over optimization, and policy outputs are scaled so maximum engine thrust corresponds to one.For value-function learning, accumulated episode rewards are multiplied by 1 − γ.Source: E. Implementation Details · p. 19

F. RL Problem Formulation

The formulation trains a policy in a simulated landing environment using shaped rewards to guide the lander toward a soft pinpoint landing. It also separates discount rates for terminal and shaping rewards, which the authors report significantly improves optimization performance.

F. RL Problem Formulation

  • Each episode ends when altitude falls below zero or an attitude constraint is violated, and the agent's actions become engine-thrust commands through the thruster model.Initial conditions are randomized within the ranges in Table 3, using a reduced set to speed simulation and iteration.Source: F. RL Problem Formulation · p. 20, Table 3 · p. 20
  • The policy and value functions use four-layer neural networks with tanh activations on each hidden layer.The network architecture specifies observation and action dimensions and the number of units in each layer.Source: F. RL Problem Formulation · p. 20, Table 4 · p. 21
  • Shaping rewards address sparse feedback by guiding the agent toward a soft pinpoint landing through a target-velocity field.The field targets a location 15 m above the landing site and a vertical velocity of -2 m/s; below 15 m, its downrange and crossrange target velocities are zero.Source: F. RL Problem Formulation · pp. 20–22
  • The reward function balances target-velocity tracking, control effort, attitude constraints, and a bonus for landing within specified terminal limits.Constraint violations carry a large negative reward and terminate the episode; the authors state that these consequences are sufficient to prevent violations once learning has converged.Source: F. RL Problem Formulation · pp. 22–23
  • Optimization introduces uncertainty in initial mass, gravity, and the inertia tensor to train a robust policy.The inertia-tensor perturbation has no physical significance and is used only to introduce parameter uncertainty.Source: F. RL Problem Formulation · pp. 23–24, Table 6 · p. 24
  • Separating discount rates for terminal and shaping rewards significantly improves optimization performance.The authors report that including the terminal reward without multiple discount rates worsened performance.Source: F. RL Problem Formulation · p. 24

A. Policy Optimization

Policy optimization proceeds through repeated rollouts, with trajectories used to update the policy and value function. Training stops after 300,000 episodes, while performance is still improving and exploration noise continues to affect attitude and rotational-velocity statistics.

A. Policy Optimization

  • Each update uses trajectories from 120 episodes to compute advantages and update the value and policy function approximators.Steps count agent-environment interactions and convert to trajectory duration using the 0.2 s navigation period.Source: A. Policy Optimization · pp. 24–25
  • Optimization stopped after 300,000 episodes while performance was still improving and action standard deviation remained around 4% for exploration.The exploration noise still had a significant impact on the policy.Source: A. Policy Optimization · p. 25
  • Attitude and rotational-velocity standard deviations during learning partly reflect parameter uncertainty and policy exploration, while testing bounds these magnitudes more tightly.The reported learning statistics are accumulated over the 120 episodes used for policy and value-function updates.Source: A. Policy Optimization · p. 25

B. Policy Testing

Policy testing evaluates the 6-DOF agent under force noise and mass uncertainty, then compares its fuel use and trajectories with other policies. The same trained policy is also tested over wider deployment ellipses and during a divert maneuver.

B. Policy Testing

  • The policy was tested for 10,000 episodes with Gaussian force noise and wet mass varied uniformly by up to ±5% from nominal.The force-noise mean was drawn uniformly from −100 to 100 N at each episode’s start, with a standard deviation of 100 N.Source: B. Policy Testing · p. 25
  • 4%: the 6-DOF RL agent used more fuel than 3-DOF DR/DV, while its fuel efficiency was close to the 3-DOF RL agent.DR/DV was used as a proxy for optimal performance; its glide-slope adjustment added about 20 kg to its fuel consumption.Source: B. Policy Testing · pp. 25–27
  • An 800 m downrange and crossrange divert at 1,500 m altitude increased average fuel consumption by 30 kg without otherwise impacting performance.Source: B. Policy Testing · p. 27
  • The 3-DOF and 6-DOF position and velocity trajectories were almost identical, while thrust differed where the 6-DOF policy had to rotate the lander.Attitude adjustments were needed to redirect thrust during descent and reduce crossrange velocity.Source: B. Policy Testing · p. 28
  • At 9 and 12 km² deployment ellipses, the same policy trained on a 4 km² ellipse retained similar landing statistics, but average and maximum fuel consumption increased.The 12 km² case required raising the deployment altitude to 3,000 m.Source: B. Policy Testing · p. 29

IV. Comparison with GPOPS Solution

The comparison reports higher fuel consumption for the RL policies than for the GPOPS optimal-control solution. The authors identify reward shaping as an important target for improving fuel efficiency.

IV. Comparison with GPOPS Solution

  • 18%: the 6-DOF and 3-DOF RL policies consumed 290 kg of fuel, versus 250 kg for the GPOPS solution.The GPOPS trajectories were open loop, so fuel consumption would be higher when combined with a trajectory-tracking controller.Source: §IV · p. 30
  • The authors attribute the fuel-efficiency difference between the RL policies and the optimal solution to the effective but non-optimal reward shaping function.They suggest learning a neural-network-parameterized shaping function from optimal trajectories as future work.Source: §IV · p. 31
  • Zero-divert fuel consumption over the 9 km² deployment ellipse was around 240 kg, which the authors describe as close to MSL fuel use after excluding sky-crane fuel.Source: §IV · p. 32

V. Conclusion

The paper applies reinforcement learning to integrated guidance and control for Mars powered descent and validates the approach through Monte Carlo simulation. The policy offers increased landing precision and robustness to noise and parameter uncertainty, with modest computational requirements.

V. Conclusion

  • The integrated reinforcement-learning guidance and control system was validated through Monte Carlo simulation and provides a significant increase in landing precision over current practice.Coupled with a navigation system such as [3], it can achieve pinpoint accuracy.Source: §V · pp. 32–33
  • The policy is robust to noise and parameter uncertainty and does not require a cone-shaped glideslope constraint, allowing targeting locations such as deep-crater bottoms.Source: §V · p. 33
  • The trained policy is expected to run on current-generation flight computers.Source: §V · p. 34

Related papers

Loading 1810.08719v1…