Adaptive Generalized ZEM-ZEV Feedback Guidance for Planetary Landing via a Deep Reinforcement Learning Approach

• Future missions may require pinpoint landing between 10 and 100 meters, while powered-descent guidance must meet fuel and flight constraints.Source: §1 · pp. 2–3
• Classical ZEM/ZEV is analytically simple and accurate, but generally cannot enforce thrust or flight constraints.Its feedback law generates a closed-loop acceleration command and minimizes the integral of squared acceleration.Source: §1 · p. 4
• Waypoint-based additions to classical ZEM/ZEV report good performance but lack flexibility and real-time adaptation.Source: §1 · pp. 4–5
• The proposed adaptive ZEM/ZEV changes guidance gains and time-to-go to generate quasi-optimal trajectories that satisfy flight constraints.The stated constraints include thrust limits and glide slope.Source: §1 · p. 5
• The guidance structure remains classical ZEM/ZEV, while a state-dependent learned policy determines optimal gains at each time step.An actor-critic method learns the policy; its critic uses an Extreme Learning Machine to approximate the value function.Source: §1 · p. 5
• With an adequate cost function and accurate dynamics simulator, the model-free method can address varied constraint combinations and produce results generally close to constrained offline fuel-optimal solutions.The authors also state that the unchanged guidance structure preserves global stability despite gain adaptation.Source: §1 · p. 5

• The scenario is terminal powered descent for a Mars soft landing, described in an orthonormal reference frame centered on the nominal target.Source: §2 · pp. 6–7
• The guidance policy outputs an acceleration command, from which thrust is recovered using engine specifications and spacecraft mass.Source: §2 · p. 7
• The setup neglects planetary rotation and thin-atmosphere effects, and assumes gravity is vertically aligned.Source: §2 · p. 7
• The lander must remain above ground sloped at 4° except within a 5-meter-radius flat area around the target.Source: §2 · p. 7

1303.1. Classical Zero-Effort-Miss/Zero-Effort-Velocity algorithm

• Classical ZEM/ZEV derives acceleration commands from terminal position and velocity errors relative to their zero-control projections; nonlinear dynamics may require integrating those projections and do not guarantee optimality.The classical formulation assumes no acceleration or state constraints; the paper states its solution is not necessarily optimal when the stated dynamics conditions do not apply.Source: 1303.1. Classical… · pp. 8–9

3.2.1. Markov decision processes

• An MDP represents states, actions, transition dynamics, and rewards, while a policy specifies how actions are selected from states.The paper describes stochastic policies parameterized by θ.Source: §3.2.1 · p. 10

3.2.2. Stochastic policy gradient theorem

• Policy-gradient methods estimate objective gradients from experience samples, while REINFORCE is unbiased but has high variance that can slow convergence.Introducing a critic is presented as a way to reduce variance while keeping error contained.Source: §3.2.2 · pp. 12–13

1553.2.3. Stochastic actor-critic algorithm

• Actor-critic methods pair an actor that adjusts policy parameters with a critic that estimates action value for the policy-gradient update.The paper’s critic estimates an advantage function, and the actor-critic components are represented by single-layer feedforward networks.Source: 1553.2.3. Stochastic actor-critic… · pp. 13–14

3.3. Extreme learning machines

• Extreme Learning Machines use single-layer feedforward networks with randomly assigned, fixed input weights and biases, learning only output weights.The paper describes this as a single-step alternative to iterative backpropagation and says it can require a fraction of the learning time.Source: §3.3 · pp. 14–15

4. Adaptive-ZEM/ZEV algorithm

• The adaptive algorithm learns state-dependent KR, KV, and time-to-go Tf while retaining generalized ZEM/ZEV guidance structure and its closed-loop characteristics.The learning framework combines REINFORCE with an Extreme Learning Machine critic to estimate the value function.Source: §4 · pp. 16–17

4.1. Samples generation

• Each training iteration generates trajectory batches under the guidance policy, sampling Tf at episode start and recording states, actions, costs, and next states.Initial positions are sampled around the nominal position to explore nearby states and avoid singularities in policy evaluation.Source: §4 · p. 17, §4.1 · p. 18

4.1.1. Policy

• The stochastic policy has fixed variance and a learned mean parameterized by radial-basis features of position and velocity, with outputs KR, KV, and Tf.Its deterministic network has position and velocity inputs, a radial-basis hidden layer, and a three-dimensional output layer.Source: §4.1.1 · pp. 18–20

4.2. Critic neural network

• The critic approximates the value function with a single-layer network trained using ELM; the implementation chooses Monte Carlo returns over temporal-difference estimates.The authors cite possible extrapolation errors when successive iterations visit different state regions as the reason for preferring the Monte Carlo version.Source: §4.2 · pp. 21–22

2804.3. Policy update

• Policy updates use a sampled gradient estimate with the critic’s advantage function, then apply gradient descent and stop when recent cumulative-cost differences meet tolerance or iterations are exhausted.The paper notes that the advantage estimate can reduce variance while its approximation introduces bias.Source: 2804.3. Policy update · pp. 22–23

5. Numerical Results

• A positive impact bias makes collision-free solutions cheaper than constraint impacts, leading training to prioritize constraint avoidance before fuel reduction.The authors report discontinuous jumps in the cost profile as training progresses.Source: §5 · pp. 25–26
• The adaptive guidance satisfies the glide-slope constraint while classical ZEM/ZEV violates it, and fuel use remains above the GPOPS optimum.The reported fuel comparison places A-ZEM/ZEV slightly ahead of classical ZEM/ZEV but below GPOPS.Source: §5 · pp. 27–28, Table 4 · p. 32
• The ELM critic’s learning time is short relative to the overall iteration time, leaving most computation for trajectory generation and policy updates.Training tests updated policies on 25 trajectories per iteration in the 3D case.Source: §5 · pp. 26–27
• State-dependent gains let the closed-loop guidance adjust to constraint scenarios, avoiding collisions while steering toward the target.The learned time of flight depends on the initial state and is not modified during descent.Source: §5 · p. 32

3705.1. Monte-Carlo analysis

• Across 1,000 3D trials, the trained policy avoids constraint violations, lands near the target, and keeps final velocity below 5 cm/s.Initial positions vary within ±500 m in x and y, with z fixed at 1,500 m; initial velocity bounds are ±5 m/s.Source: 3705.1. Monte-Carlo analysis · p. 33

6.1. Closed-loop Dynamics

• The guidance acceleration yields a linear, non-autonomous feedback system, allowing classical linear-system methods to be used for stability analysis.The analysis reduces the LTV system to an equivalent LTI system.Source: §6.1 · pp. 33, 35

6.2. Transformation of LTV systems into LTI systems

• An LTV system is termed invariable when valid transformations can convert it to an LTI system with a constant matrix.The paper distinguishes algebraic transformations from transformations of the time scale.Source: §6.2 · p. 35

4256.3. Stability of the A-ZEM/ZEV algorithm

• For the A-ZEM/ZEV algorithm, two conditions provide a low-cost instantaneous stability check that can be added to the control loop.The analysis also proposes taking action if the algorithm enters unstable regions.Source: 4256.3. Stability of the A-ZEM/ZEV… · pp. 38–39
• Along the tested 2D and 3D descents, all eigenvalues’ real parts remain strictly negative, and sampled state-transition matrices remain bounded.The authors conclude stability for the sampled cases based on the bounded STM components.Source: 4256.3. Stability of the A-ZEM/ZEV… · pp. 39–40, Figure 13 · p. 39

7. Conclusions and future efforts

• A-ZEM/ZEV improves fuel consumption and allows path constraints to be implemented directly in the guidance law.The trained guidance neural network can be implemented onboard; with appropriate state-determination sensors, guidance is recovered from the state in real time for autonomous soft landing.Source: §7 · p. 41
• Transforming the LTV system into an LTI system makes instability conditions easier to check, including online in the control loop, where countermeasures can be applied.The paper also describes post-processing checks.Source: §7 · p. 41
• For simple regression tasks, a shallow network is sufficient, and ELM offers good accuracy with computational time negligible relative to total iteration time.The authors report that ELM scales well with input dimension and its one-step learning process suits the task.Source: §7 · p. 41
• The learned guidance parameters extend an existing parametrized policy, with ZEM/ZEV providing precise terminal guidance and reinforcement learning adding flexibility.The authors say the approach can extend to other guidance problems relying on a parametrized law when the state space can be discretized effectively.Source: §7 · p. 42
• The method's convergence is described as guaranteed under unbiased advantage-function estimation and a correctly set-up cost function, but convergence remains slow.Meta-learning is suggested to speed learning across sequential tasks, especially when environmental conditions change, such as actuator or sensor failure.Source: §7 · p. 42
• The authors propose meta-learning across randomized MDP instances as a way to improve adaptation to wider environmental parameter variation and robustness to mis-modelled dynamics.They frame improved performance in a more realistic setup as an expected outcome of that extension.Source: §7 · p. 42