PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Reinforcement Learning for Safe Landing Site Selection with Concurrent Consideration of Divert Maneuvers

Keidai Iiyama, Kento Tomita, Bhavi A. Jagatia, Tatsuwaki Nakagawa, Koki Ho

arXiv:2102.12432v1cs.ROcs.AIcs.LGcs.SYeess.SY

TL;DR

Existing landing-site methods do not account for how future observations and divert maneuvers can change the safest target. This paper uses reinforcement learning to jointly select landing sites and guidance strategies during descent. The trained agent achieved a 94.8% landing-to-safe-site ratio, though it used more fuel than the comparison agents.

  • Problem

    Earlier landing-site assessment methods do not consider future target changes, although observation quality and the best landing site vary with spacecraft state.

    Source: INTRODUCTION · p. 2

  • Method

    A model-free reinforcement-learning agent successively selects landing sites and guidance strategies, learning from simulator interactions.

    Source: INTRODUCTION · p. 3

  • Results

    94.8% landing-to-safe-site ratio was achieved by the trained memory-less agent, whose safe-landing ratio exceeded that of the fixed control policy.

    Source: Comparison with other strategies · p. 19

  • Takeaways & Limitations

    The framework achieved over 90% successful landing at difficult sites where over 80% of terrain near the initial target was hazardous.

    Source: CONCLUSION · p. 20

    The memory-less trained agent used more fuel than the two comparison agents, due to large thrust magnitudes and frequent target changes.

    Source: Comparison with other strategies · p. 20

Abstract

from arXiv · show

This research proposes a new integrated framework for identifying safe landing locations and planning in-flight divert maneuvers. The state-of-the-art algorithms for landing zone selection utilize local terrain features such as slopes and roughness to judge the safety and priority of the landing point. However, when there are additional chances of observation and diverting in the future, these algorithms are not able to evaluate the safety of the decision itself to target the selected landing point considering the overall descent trajectory. In response to this challenge, we propose a reinforcement learning framework that optimizes a landing site selection strategy concurrently with a guidance and control strategy to the target landing site. The trained agent could evaluate and select landing sites with explicit consideration of the terrain features, quality of future observations, and control to achieve a safe and efficient landing trajectory at a system-level. The proposed framework was able to achieve 94.8 $\%$ of successful landing in highly challenging landing sites where over 80$\%$ of the area around the initial target lading point is hazardous, by effectively updating the target landing site and feedback control gain during descent.

INTRODUCTION

Landing-site selection must account for future observations and divert opportunities because the best target can change with spacecraft state. The paper proposes jointly learning target selection and guidance, while modeling a soft lunar landing scenario with 3-DOF dynamics.

INTRODUCTION

  • Future observations can change which landing site is best, so target choice must account for the descent trajectory and possible later diverts.The target and control plan are coupled, and trajectory and target choices affect the quality and quantity of terrain observations.Source: INTRODUCTION · pp. 2–3
  • Earlier landing-site assessment methods do not account for future target changes, while ALHAT selects discrete points using local static terrain information.Other methods consider divert feasibility or scientific value, but the cited approaches do not jointly address evolving target choice and control.Source: INTRODUCTION · p. 2
  • The proposed learning-based method successively selects landing sites and guidance trajectories during hazard detection and avoidance.The agent seeks to maximize safe-landing probability while minimizing fuel consumption, landing error, and final velocity.Source: INTRODUCTION · p. 3

Dynamics

  • The modeled scenario is a soft lunar landing with 3-DOF dynamics, constant gravity, and planetary rotation ignored.The glide-slope constraint helps prevent low-altitude terrain impacts and maintains an inclination intended to support navigation and hazard detection.Source: Dynamics · p. 4

Terrain Generation

The simulator builds synthetic terrain and evaluates its safety, then models how descent geometry and LIDAR conditions shape the safety map available to the lander.

Terrain Generation

  • The generated terrain is 1000m x 1000m at 1m resolution, using crater and boulder distribution models.The terrain generator adapts the cited boulder model to lunar topography by adjusting parameters.Source: Terrain Generation · pp. 4–5

Safety Map Generation

Observation Data Generation

  • To reduce simulation computation, pseudo-observation safety maps are produced by cropping the full-field stochastic safety map and adding noise.This avoids generating a new DEM and running the safety algorithm at each observation time.Source: Observation Data Generation · p. 6
  • The observation-safety-map model represents LIDAR field of view, increasing range-measurement error and sampling distance that can overlook small boulders, and labels pixels at slant angles below 70 degrees unsafe.DEM observations were generated every 5 seconds at a fixed 64x64 size.Source: Observation Data Generation · pp. 6–7

Modeling the Problem as POMDP

The landing problem is represented as a partially observable decision process with spacecraft and terrain state, observations from LIDAR and state estimates, and rewards tied to landing outcomes. The paper learns policies from simulator transitions, testing both memory-less and history-dependent approaches.

Modeling the Problem as POMDP

  • The POMDP state comprises spacecraft position, velocity, and mass plus the true DEM; actions include thrust output and target landing position.The reward accounts for consumed fuel, final landing-point safety, and final velocity.Source: Modeling the Problem as POMDP · pp. 7–8
  • The observation space contains LIDAR DEM data and predicted spacecraft state, generated through observation and state-observation models.The transition function represents stochastic spacecraft dynamics.Source: Modeling the Problem as POMDP · p. 8
  • Because partial observability makes optimal policy learning difficult, the paper learns from simulator transitions without modeling the transition or observation functions.It tests a memory-less policy based on the latest observation and a memory-based policy that uses history through recurrent neural networks.Source: Modeling the Problem as POMDP · pp. 8–9

Observation Data Interpretation

The observation map is compressed into a latent representation before reaching the reinforcement-learning agent. The agent also adjusts the landing target and ZEM-ZEV guidance parameters during descent.

Observation Data Interpretation

  • The autoencoder compresses 64x64 safety-map observations into a latent vector, trained separately on 5000 random rollouts to reconstruct the observed maps.The agent receives the compressed representation rather than the full image data.Source: Observation Data Interpretation · p. 9

General Concept of Control

  • The agent controls both the target landing position and adaptive gains and time-to-go in the ZEM-ZEV feedback guidance law.The target position changes during descent, distinguishing this approach from prior learning of adaptive ZEM-ZEV parameters alone.Source: General Concept of Control · pp. 9–10

Controller

  • The controller's gains are KR and KV, and tgo denotes time-to-go; without thrust-magnitude or trajectory constraints, KR = 6 and KV = 2 yield an energy-optimal trajectory.The agent is trained to decide KR, KV, tgo, and target position rf at each time step.Source: Controller · p. 10

TRAINING

The training section introduces reinforcement learning's policy and action-value framework, then describes DDPG for continuous actions. Replay data, exploration noise, and target networks support the training procedure.

Reinforcement Learning

  • The agent learns a policy to maximize expected discounted future reward in a Markov decision process, with the Q-function representing expected return after an action.The Bellman equation gives a recursive relation for the Q-function.Source: Reinforcement Learning · p. 11

Deep Deterministic Policy Gradient (DDPG)

  • DDPG is an off-policy, model-free actor-critic method for continuous action spaces, with a critic estimating Q(s, a) and an actor producing actions.The policy is parameterized as a deterministic mapping from state to action.Source: Deep Deterministic Policy Gradient… · p. 11
  • DDPG reuses transitions from different behavior policies through a replay buffer, while exploration adds noise to the actor's policy.Target networks slowly track learned networks to stabilize training and support convergence.Source: Deep Deterministic Policy Gradient… · p. 12

Twin Delayed Deep Deterministic Policy Gradient (TD3)

TD3 modifies DDPG to address Q-value overestimation and sensitivity to hyperparameter settings. Its changes stabilize target-value estimation and slow policy and target-network updates.

Twin Delayed Deep Deterministic Policy Gradient (TD3)

TD3 with memory

The framework uses recurrent networks to preserve observation-action history for partially observed control, and trains policies to choose landing targets and guidance actions. The actor’s target and timing outputs affect both descent guidance and future observations.

TD3 with memory

  • RDPG replaces state-based policy and action-value functions with functions of observation-action history when the system is partially observed.The implementation adds LSTM networks to the actor and critic to retain limited history information.Source: TD3 with memory · p. 13
  • The LSTM actor and critic combine current inputs with history represented by the recurrent network’s output.The policy and critic architectures distinguish current observation inputs (and current action for the critic) from information stored in the LSTM.Source: TD3 with memory · p. 13, Figure 8 · p. 14, Figure 9 · p. 14
  • Replay buffers store transition sequences by episode length so minibatches can use episodes of the same length.Episode lengths in the environment ranged from 4–10 time steps.Source: TD3 with memory · p. 13

Overall Framework and Training Process

  • The agent observes lander state and terrain and outputs actions including thrust and a target landing position.The observed-state and action elements are summarized in Tables 1 and 2.Source: Overall Framework and Training Process · p. 13, Table 1 · p. 15, Table 2 · p. 15
  • The actor shifts the next target within the latest DEM’s field of view and outputs the time-to-go decrease until the next observation, scheduled five seconds later.The target offset parameters affect both guidance and observation; the reward penalizes fuel use, constraint violations, final velocity, and error from the final target.Source: Overall Framework and Training Process · pp. 15–16

Initial Condition and Hyper-parameter Settings

The true terrain digital elevation model is randomly selected from a set generated before training, and ranged parameters are randomized at each episode’s start.

Initial Condition and Hyper-parameter Settings

  • Ranged lander parameters are randomized at each episode’s start, while the true terrain digital elevation model is randomly selected from the pre-generated set.The initial state and other simulation specifications are summarized in Table 3.Source: Initial Condition and Hyper-parameter… · p. 16

Training Process

The memoryless trained agent achieved the highest safe-landing ratio in simulations where the initial target was in highly hazardous terrain. Training comparisons favored the memoryless agent, while the results also expose trade-offs in fuel use, landing accuracy, and observation limits.

Training Process

  • Both agents reached near-maximum performance after around 250k training iterations, with validation on terrain datasets different from training.Figure 11 plots moving averages over 5,000 episodes.Source: Training Process · pp. 16–17, Figure 11 · p. 18
  • The memoryless agent outperformed the LSTM agent; LSTM training was more sensitive to hyperparameters.The authors suggest memory may be more useful if navigation errors increase partial observability, which was relatively weak with the low-error safety map.Source: Training Process · pp. 17–18

Comparison with other strategies

  • The fixed-control comparison retains agent target selection but fixes control gains and calculates time-to-go with Eq.6; the single-divert policy selects a safety-map maximum at 500 m.The single-divert policy represents a simplified version of existing landing techniques.Source: Comparison with other strategies · p. 18
  • The trained memory-less agent achieved the maximum landing-to-safe-landing-site ratio of 94.8%; the fixed control policy had a lower safe-landing ratio because it could not flexibly change thrust magnitude for long divert maneuvers.Table 4 reports the method comparison over 1000 episodes using mean values.Source: Comparison with other strategies · p. 19, Table 4 · p. 19
  • The fixed-control policy had a lower safe-landing ratio because it could not flexibly vary thrust for long diverts.In the described case, the trained agent reduced vertical velocity to maintain altitude and a wider field of view, while the fixed controller did not thrust as aggressively.Source: Comparison with other strategies · p. 19
  • The memory-based agent had the lowest safe-landing probability and largest landing-position and velocity errors, but the lowest average fuel use.The authors interpret this pattern as a local-optimum policy favoring fuel reduction over other metrics.Source: Comparison with other strategies · p. 19
  • Single-observation selection can fail when safe sites are unavailable within the field of view or at usable slant angles; omitted navigation and attitude errors may worsen map and guidance accuracy.The authors note that these additional error sources were not simulated.Source: Comparison with other strategies · p. 19
  • Changing ZEM-ZEV gains does not guarantee convergence to the agent’s final target position and velocity, and the memoryless agent used more fuel than the two comparison policies.The authors connect its higher fuel use to large thrust magnitude during descent and frequent target changes.Source: Comparison with other strategies · p. 20

CONCLUSION

The proposed HDA framework updates the landing target and control parameters after each observation. In simulation, the trained agent achieved over 90% probability of successful landing at difficult sites where over 80% of the terrain around the initial target was hazardous.

CONCLUSION

  • Over 90% probability of successful landing was achieved at difficult sites where over 80% of the terrain around the initial target was hazardous.The agent gradually updated the target landing point toward safe regions.Source: CONCLUSION · p. 20
  • The framework models the HDA sequence as a POMDP and uses an RL agent with an auto-encoder to interpret maps and output ZEM-ZEV feedback-control parameters.The agent was trained through interaction with the simulator and updates the target landing site and control parameters after observations.Source: CONCLUSION · p. 20
  • Future work targets simulator refinement with accurate LIDAR DEM generation, 6-DOF dynamics, and lander-state measurement errors; increased partial observability may require a memory-based agent.The authors also plan to test other baseline control policies and observation-data interpretation architectures to improve performance and stability.Source: CONCLUSION · pp. 20–22
  • Figure 13 compares the obtained agent with a fixed-gain ZEM-ZEV controller, while Figure 14 shows changes in control gains KR and KV during control.Figure 13 marks the lander trajectory and LIDAR DEM field of view.Source: Figure 13 · p. 21, Figure 14 · p. 22

Related papers

Loading 2102.12432v1…