Source-linked AI summary
Adaptive Guidance with Reinforcement Meta-Learning
Brian Gaudet, Richard Linares
TL;DR
Guidance for space missions must handle dynamics that can vary or remain incompletely modeled. This paper uses recurrent reinforcement meta-learning for adaptive guidance and compares it with non-recurrent RL and DR/DV across difficult tasks. The recurrent policy performs best across the four tasks, though the MLP policy comes close in some.
Problem
Space missions can involve complex, time-varying dynamics that are incompletely modeled during mission design.
Source: INTRODUCTION · p. 1Method
The paper trains PPO with recurrent policy and value-function networks using reinforcement meta-learning to develop an adaptive guidance law.
Source: INTRODUCTION · p. 2Results
Across four difficult tasks, the recurrent policy performs best, the MLP policy comes close on some tasks, and DR/DV performs worst overall.
Source: CONCLUSION · p. 19Takeaways & Limitations
The authors conclude that parameter-uncertainty training yields robust policies and that recurrent policies are the best-performing option in these tasks.
Source: CONCLUSION · p. 19Radar-altimeter distance estimates become error-prone at lower elevations, reducing accuracy in the radar-observation landing setting.
Source: Experiment2: Mars Landing using Radar… · p. 16
Abstract
from arXiv · showhide
This paper proposes a novel adaptive guidance system developed using reinforcement meta-learning with a recurrent policy and value function approximator. The use of recurrent network layers allows the deployed policy to adapt real time to environmental forces acting on the agent. We compare the performance of the DR/DV guidance law, an RL agent with a non-recurrent policy, and an RL agent with a recurrent policy in four difficult tasks with unknown but highly variable dynamics. These tasks include a safe Mars landing with random engine failure and a landing on an asteroid with unknown environmental dynamics. We also demonstrate the ability of a recurrent policy to navigate using only Doppler radar altimeter returns, thus integrating guidance and navigation.
INTRODUCTION
The paper targets guidance in missions where dynamics may vary or be incompletely modeled. It uses recurrent reinforcement meta-learning to adapt guidance and compares recurrent and non-recurrent RL policies with DR/DV across variable-dynamics tasks.
INTRODUCTION
- Mission dynamics may be time-varying and incompletely modeled, including changes from fuel transfer, fuel depletion, uncertain asteroid environments, sensor distortion, or actuator failure.These examples motivate guidance that adapts in real time.Source: INTRODUCTION · p. 1
- Training on randomized dynamical systems gives a recurrent policy experience across ground-truth dynamics, requiring its hidden state to adapt to different systems.The authors frame this as meta-learning: learning through experience across MDPs to adapt to novel MDPs.Source: INTRODUCTION · p. 2
- The recurrent policy’s hidden state can adapt to unobserved forces and lander mass from observation-action history, while an MLP policy has no persistent state at test time.The comparison covers asteroid dynamics, noisy radar-altimeter observations, engine failure, and high mass variability.Source: INTRODUCTION · p. 2
- The study compares DR/DV, an MLP RL policy, and a recurrent RL policy across tasks with unknown, highly variable dynamics.DR/DV receives ground-truth gravity and initial lander mass, while RL receives observations that are functions of position and velocity.Source: INTRODUCTION · p. 2
- For radar-only landing, the agent observes returns from four Doppler radar altimeters rather than ground-truth position and velocity; the measurement error becomes extreme at lower elevations.The DR/DV guidance law is not evaluated for this task.Source: INTRODUCTION · p. 3
METHODS
The landing dynamics are modeled in three degrees of freedom, with environmental and rotational terms included in the translational model. Mars and asteroid settings use different stated gravity, thrust, and mass assumptions.
Equations of Motion
- The landing is modeled in 3-DOF, with translational motion specified by the equations of motion.The supplied passage identifies terms for position, thrust, gravity, mass, environmental disturbances, and rotational velocity.Source: Equations of Motion · p. 3
- Environmental disturbances are modeled as normally distributed random variables, while rotational velocity is set to zero in Mars landing experiments.The disturbance examples include wind and variations in atmospheric density.Source: Equations of Motion · p. 3
- Mars thrust is constrained to 2000–15000 N with nominal wet mass 2000 kg, whereas asteroid landing uses pulsed 2 N thrusters along each target-centered axis.Source: Equations of Motion · p. 3
Reinforcement Learning Overview
The overview describes episodic reinforcement learning, where agents map observations to actions and learn from rewards. It also outlines recurrent networks and policy-gradient methods for learning from trajectories.
Reinforcement Learning Overview
- An RL episode provides observations and rewards as an agent maps observations to actions, with the environment updating state and potentially ending the episode on task completion or constraint violation.Observations may be corrupted internal states or raw sensor outputs.Source: Reinforcement Learning Overview · pp. 3–4
- Exploration begins with random actions and decreases as experience accumulates; field deployment typically turns exploration off because it is expensive using an actual lander.Telemetry may be used to update the dynamics model and policy through simulated experience.Source: Reinforcement Learning Overview · p. 4
- A recurrent network can help infer motion from observations that do not satisfy the Markov property, using temporal information from observations and actions.The paper contrasts this with concatenating observations across multiple time steps.Source: Reinforcement Learning Overview · p. 4
- Policy-gradient learning optimizes expected discounted trajectory rewards, and the A2C formulation uses an advantage function formed by subtracting a state-value baseline from the state-action value.The cited discussion describes estimating the gradient using Monte Carlo rollouts.Source: Reinforcement Learning Overview · pp. 4–5
Proximal Policy Optimization
The paper presents PPO as a policy-gradient method related to TRPO and describes its clipped objective, advantage estimate, and concurrent policy and value-function learning.
Proximal Policy Optimization
- TRPO constrains policy-update size, with monotonic-improvement theory when each iteration’s policy change is bounded by a parameter.The text notes that a closed-form bound leads to prohibitively small steps, motivating an approximate constrained optimization approach.Source: Proximal Policy Optimization · p. 5
- PPO approximates TRPO’s policy-adjustment constraint using a clipped objective function.The supplied PPO passages identify clipping as the mechanism used to account for the adjustment constraint.Source: Proximal Policy Optimization · p. 6
- PPO uses an advantage estimate based on empirical return minus a state-value baseline.Source: Proximal Policy Optimization · p. 6
- The implementation adjusts the clipping parameter to target policy-update KL divergence of 0.001 and learns the policy and value function concurrently.It uses a Gaussian action distribution with mean from the policy and diagonal covariance.Source: Proximal Policy Optimization · p. 6
Recurrent Policy and Value Function Approximator
The method uses recurrent policy and value-function layers, with hidden states carried through rollouts so recurrent computation can be parallelized across episodes.
Recurrent Policy and Value Function Approximator
- The value function also needs a recurrent layer because the policy cost uses advantages based on observed returns and expected values.Its hidden state evolves with the past sequence of observations and returns.Source: Recurrent Policy and Value Function… · pp. 6–7
- Parallel unrolling stores policy and value-function hidden states in rollouts and uses them to initialize each unrolled segment.The current recurrent layer then evolves the state across the segment’s remaining steps.Source: Recurrent Policy and Value Function… · p. 7
- Training preserves trajectory order, pads batches for unrolling, and uses GRUs to address exploding or vanishing gradients.The passage illustrates a five-step unroll; padded values are excluded from loss computation.Source: Recurrent Policy and Value Function… · p. 7
RL Problem Formulation
The formulation combines shaped rewards for landing with a recurrent policy and value function, and introduces multiple discount rates to address a conflict between terminal and shaping rewards.
RL Problem Formulation
- Sparse landing rewards would rarely be reached through random actions in a realistic 3-DOF setting, motivating per-step shaping rewards.The authors instead engineer rewards that guide the agent toward a soft pinpoint landing.Source: RL Problem Formulation · p. 9
- The shaping heuristic aligns velocity with the line of sight, reduces target velocity as time-to-go falls, and encourages vertical descent below 15 m.It targets a point 15 m above the landing site and a z-component of velocity of -2 m/s.Source: RL Problem Formulation · pp. 9–10
- The reward trades off target-velocity tracking, fuel conservation, and a landing bonus; constraint violations incur large negative reward and episode termination.The success bonus requires terminal position, velocity, and glideslope to fall within specified limits.Source: RL Problem Formulation · p. 10
- The observation includes velocity error, estimated altitude, and time-to-go, but omits translational coordinates aside from altitude, which the authors say supports generalization across the state space.The policy and value functions use four-layer networks with tanh activations and a GRU recurrent layer.Source: RL Problem Formulation · pp. 8, 10, 11
- The authors report that multiple discount rates significantly improve performance over GAE with or without multiple rates, while adding terminal reward without them worsened performance.They use separate rates for the terminal-reward term and the other reward terms.Source: RL Problem Formulation · p. 11
EXPERIMENTS
The experiments compare DR/DV, MLP, and recurrent policies across randomized dynamics, with Mars initial conditions specified for optimization.
EXPERIMENTS
- Each experiment compares DR/DV, MLP, and recurrent policies unrolled for 1, 20, or 60 steps, then tests trained policies for 10,000 episodes.Policies are optimized using PPO.Source: EXPERIMENTS · p. 11
- For Mars experiments, gravity components vary uniformly by ±5% of nominal and initial lander mass by ±10%.The Mars optimization initial conditions are listed in Table 3.Source: EXPERIMENTS · p. 11, Table 3 · p. 11
Experiment 1: Asteroid Landing with Unknown Dynamics
The asteroid experiment tests guidance under widely varying, initially unknown environmental dynamics. DR/DV fails in this task, while RL policies perform acceptably and fuel efficiency rises with longer recurrent unrolling.
Experiment 1: Asteroid Landing with Unknown Dynamics
- Asteroid angular velocity, gravity, and solar radiation pressure are independently randomized by directional component within Table 4 bounds.The authors frame the wide variation as an unknown-dynamics scenario, while noting near-Earth asteroid parameters are normally known more tightly.Source: Experiment 1: Asteroid Landing with… · p. 12, Table 4 · p. 12
- The lander targets a pole site 250 m from the asteroid center, with initial-condition variation described as much greater than typical for such a mission.The setup defines a landing plane whose normal depends on the target site.Source: Experiment 1: Asteroid Landing with… · p. 12, Table 5 · p. 12
- The asteroid RL setup adapts the Mars reward formulation, sets the minimum glideslope to zero, and discretizes each pulsed-thrust action to negative, zero, or positive maximum thrust.The DR/DV nominal gravity parameter was tuned for best performance and was higher than the actual environmental value.Source: Experiment 1: Asteroid Landing with… · p. 12, Table 6 · p. 13
- DR/DV fails with occasional catastrophic position errors, whereas all RL policies perform acceptably and fuel efficiency increases with more recurrent steps.The reported failures typically occur within fewer than 1,000 test episodes.Source: Experiment 1: Asteroid Landing with… · p. 13, Table 7 · p. 13
Experiment2: Mars Landing using Radar Altimeter Observations
The radar-altimeter task tests landing without access to the terrain map, using observations that do not uniquely identify position and velocity. Recurrent-policy performance improves with longer unrolls, while low-altitude measurement errors and the tested pointing configuration affect outcomes.
Experiment2: Mars Landing using Radar Altimeter Observations
- The task uses four Doppler radar altimeter returns without access to the digital terrain map; multiple positions and velocities can yield the same observation.Consequently, choosing an action optimally can require a history of past readings.Source: Experiment2: Mars Landing using Radar… · p. 13
- The simulated altimeter is about 1000X faster than the prior ray-casting approach but becomes error-prone at lower elevations, sometimes measuring the far side of terrain features.Its accuracy is evaluated against ray-cast readings at random map locations and elevations.Source: Experiment2: Mars Landing using Radar… · p. 16, Table 8 · p. 16
- Target-pointing altimeters markedly improve performance, while a landing site farther south on the map degrades it.The authors suggest terrain diversity and retaining observation–advantage relationships as possible factors.Source: Experiment2: Mars Landing using Radar… · pp. 16–17, Table 11 · p. 17
- Longer recurrent-network unrolls generally improve average performance; the 120-step learning curve initially progresses but later stalls, likely amid highly inaccurate low-altitude readings.The authors infer that relevant temporal dependencies may span a significant fraction of an episode.Source: Experiment2: Mars Landing using Radar… · pp. 16–18, Figure 7 · p. 17
Experiment 3: Mars Landing with Engine Failure
With engine failures occurring in half of episodes, longer-unroll recurrent policies achieve performance consistent with a safe landing, unlike the DR/DV baseline, which fails catastrophically.
Experiment 3: Mars Landing with Engine Failure
- Engine failure is simulated with probability 0.5, reducing downrange or crossrange thrust by half or vertical thrust by a factor of 1.5.The lander's maximum thrust is increased to 24000N for this test.Source: Experiment 3: Mars Landing with Engine… · p. 18
- The 20-step and 60-step recurrent policies achieve performance consistent with a safe landing and improve fuel efficiency; DR/DV fails catastrophically.The MLP and 1-step recurrent policies come close to a safe landing, while the 1-step policy performs near the MLP.Source: Table 12 · p. 18, Experiment 3: Mars Landing with Engine… · p. 18
Experiment 4: Mars Landing with High Mass Variation
In the high-mass-variation landing task, the agent lacks access to actual mass during descent. The 20-step recurrent policy achieves a good landing, with a slight further improvement from the 60-step policy.
Experiment 4: Mars Landing with High Mass Variation
- Fuel use averages around 1200 kg and peaks at 1600 kg, causing substantial mass variation that the agent cannot observe during descent.The task divides engine specific impulse by a factor of 6; the authors compare the mass variability to EKV interception of an ICBM warhead.Source: Experiment 4: Mars Landing with High… · p. 18
- The 20-step recurrent policy achieves a good landing, slightly improved by the 60-step policy; the MLP and 1-step recurrent policy remain unsafe.DR/DV has a large maximum position error and unsafe terminal velocity.Source: Experiment 4: Mars Landing with High… · p. 18, Table 13 · p. 19
CONCLUSION
Across four difficult tasks, recurrent policies achieve the best performance, though MLP policies come close on some tasks, while DR/DV performs worst overall.
CONCLUSION
- Across asteroid landing, noisy radar-altimeter landing, engine failure, and high mass variability, recurrent policies perform best, though MLP policies come close on some tasks.The DR/DV policy has the worst performance across all four tasks.Source: CONCLUSION · p. 19
- The authors conclude that training with parameter uncertainty leads to robust policies and that recurrent policies adapt in real time to unknown or highly variable dynamics.Future work will examine more realistic 6-DOF environments and other sensor models, including simulated camera images and flash LIDAR.Source: CONCLUSION · p. 19