PaperlayerView exact revision on arXiv

Source-linked AI summary

Real-time optimal control via Deep Neural Networks: study on landing problems

Carlos Sánchez-Sánchez, Dario Izzo

arXiv:1610.08668v1eess.SY

TL;DR

The paper addresses whether deep networks can represent optimal control for deterministic nonlinear spacecraft landing when the full state is known. It trains networks on optimal state-action data and finds that the resulting policies support near-optimal, real-time landing control across a broad set of initial states.

  • Problem

    Deep-network approximation of optimal state-action pairs for fully observed deterministic continuous nonlinear systems has been largely neglected.

  • Method

    The paper trains deep neural networks directly on optimal state-action pairs to represent optimal state-feedback for aerospace landing systems.

  • Results

    The trained networks produce optimal state-feedback, preserve final cost without significant impact, avoid catastrophic failures far from nominal conditions, and generalize beyond the training-state subset.

  • Takeaways & Limitations

    The results support designing real-time optimal-control architectures for planetary landing using DNNs.

Abstract

from arXiv · show

Recent research on deep learning, a set of machine learning techniques able to learn deep architectures, has shown how robotic perception and action greatly benefits from these techniques. In terms of spacecraft navigation and control system, this suggests that deep architectures may be considered now to drive all or part of the on-board decision making system. In this paper this claim is investigated in more detail training deep artificial neural networks to represent the optimal control action during a pinpoint landing, assuming perfect state information. It is found to be possible to train deep networks for this purpose and that the resulting landings, driven by the trained networks, are close to simulated optimal ones. These results allow for the design of an on-board real time optimal control system able to cope with large sets of possible initial states while still producing an optimal response.

Nomenclature

The paper examines deep neural networks as approximators of optimal control actions for deterministic nonlinear aerospace landing problems. It uses supervised learning from optimal state-action pairs, targeting real-time onboard control across broad initial-state regions.

  • Deep neural networks are used to approximate optimal state-action pairs for deterministic continuous nonlinear systems with fully observed states.
  • Earlier approaches included value-function approximation, Pontryagin-based trial solutions, and networks retrained for each initial condition.
  • The paper applies this approach to four pinpoint-landing cases: a quadcopter, two mass-varying spacecraft models, and a mass-varying rocket.
  • Training data consist of optimal state-action pairs generated offline through indirect single-shooting and continuation techniques over a large region of interest.
  • The resulting networks require modest CPU effort for onboard descent-guidance generation, while training remains offline.
  • The learned policies generalize beyond the state-space subset used for training and support real-time optimal control without onboard direct or indirect solvers.

II. Optimal control

The paper formulates a free-time deterministic optimal-control problem and uses deep networks to represent its optimal state-feedback policy directly. The HJB framework connects this policy to minimum cost, while avoiding onboard numerical optimal-control solvers.

  • The control problem seeks an admissible policy steering any initial state x0 to a target set while minimizing running and terminal costs.
  • Because final time is free, the value function represents the minimum cost from the initial state without explicit time dependence.
  • The HJB equations are challenging PDEs whose viscosity solution solves the original optimal-control problem and implies a unique optimal state-feedback u*(x).
  • Rather than approximating the value function and differentiating it, the DNN learns the optimal state-feedback directly from the control problem.
  • The trained policy can be embedded in nonlinear model predictive control to provide real-time optimal control capabilities.

III. Optimal landing control problems

The paper considers four landing optimal-control models spanning different spacecraft dynamics, objectives, and control-profile classes. Pontryagin-derived TPBVP solutions provide optimal state-action pairs for training deep networks.

  • The objectives include time-optimal, mass-optimal, and quadratic control formulations across the considered spacecraft models.
  • Four landing models cover quadcopter, simple spacecraft, reaction-wheel spacecraft, and thrust-vectoring rocket scenarios.
  • The resulting test cases represent continuous, discontinuous, bang-off-bang, and saturated control profiles.
  • Pontryagin’s maximum principle is used to derive a two-point boundary value problem for each model.
  • Solutions satisfying the dynamics, boundary conditions, and free-time condition generate optimal state-action pairs for DNN training.

B. Simple spacecraft (SSC)

The simple spacecraft model represents a mass-varying point spacecraft landing under lunar gravity, with thrust magnitude and freely steered pitch as controls. Its quadratic-to-mass-optimal continuation is solved through a TPBVP.

  • The simple spacecraft state comprises position, velocity, and mass, with lunar gravity set to 1.6229 m/s^2.
  • Thrust is bounded by a maximum magnitude of 44000 N, while pitch is freely steerable because rotational inertia is omitted.
  • The target state requires zero position and velocity, while terminal mass remains unconstrained.
  • The parameter α ∈[0, 1] continuously connects quadratic control at α = 0 to mass-optimal control at α = 1.
  • Pontryagin’s maximum principle yields the TPBVP and the corresponding optimal controls, including a necessary expression for pitch control.

C. Reaction wheel spacecraft (RWSC)

The reaction-wheel spacecraft extends the mass-varying landing model with bounded attitude-rate control and terminal attitude constraints. These changes produce different optimal landing trajectories and introduce possible wheel saturation as a scope boundary.

  • The reaction-wheel spacecraft adds a bounded attitude-rate control actuated by a reaction wheel.
  • The model’s changing mass can increase maximum angular velocity and may cause reaction-wheel saturation.
  • Its state includes position, velocity, and mass, with lunar gravity and rocket-engine efficiency matching the simple spacecraft formulation.
  • The target requires zero position, velocity, and attitude, forcing a terminal vertical descent unlike the simple spacecraft case.
  • The objective continues from quadratic control at α = 0 to mass-optimal control at α = 1, while reaction-wheel power is always included.

D. Thrust vectoring rocket (TVR)

The thrust-vectoring rocket model represents a lunar landing vehicle whose thrust direction can tilt relative to the body axis. Its optimal-control formulation includes position, velocity, attitude, angular velocity, and mass.

  • The thrust-vectoring rocket state includes position, velocity, orientation, angular velocity, and mass.
  • Thrust magnitude is bounded by 20 N, while thrust-vector tilt is constrained to φ = 10 deg.
  • Its objective uses a quadratic-to-mass-optimal continuation for thrust, while reaction-wheel power remains part of the attitude-control cost.
  • Pontryagin’s maximum principle produces the TPBVP and optimal control expressions for thrust and vector tilt.

and trajectory (right)

The paper generates optimal landing trajectories by solving control problems and exploring their initialization regions with continuation-guided random walks. This procedure produces large state-action datasets, while solver limitations constrain coverage for the thrust-vectoring rocket.

  • Each optimal trajectory is represented as state-action pairs (x∗, u∗), and random walks perturb initial states to generate additional trajectories.Perturbations use uniformly drawn step sizes bounded by η = 0.02 times each variable’s initialization range.
  • Indirect-method control profiles avoid the chattering caused by numerical instabilities in the prior direct method, which would hinder training state-action pairs.The direct method’s chattering has a minor effect on trajectories but poses a major problem for training data.
  • Continuation from quadratic control to time- or mass-optimal control supplies better initial costate guesses for difficult optimal-control problems.The homotopy parameter α is increased continuously from 0 to 1, producing smooth solution changes.
  • 15,000 optimal trajectories and 1,500,000 optimal state-action pairs are generated for the quadrotor and spacecraft models.Independent random walks create validation data after training/evaluation trajectories are collected.
  • The random-walk procedure achieves uniform coverage of the initialization region for the reaction-wheel spacecraft mass-optimal problem.Figure 6 shows random walks and the resulting initial-state distribution.
  • For the thrust-vectoring rocket, arbitrary initial conditions are too challenging for the solver, so sampling begins near a perfect vertical landing and cannot uniformly fill A.The resulting trajectories are concentrated around nominal descent, excluding high-velocity states directed away from the landing position.

V. Learning the Optimal Control

The paper learns optimal control as a state-only mapping with separate feed-forward deep networks for each control variable. It compares architectures, hidden activations, and output functions designed to represent saturated controls.

  • Deep feed-forward networks learn the optimal state-action relation because the control to be learned is a function of the current state alone.Separate networks are trained for the individual control variables.
  • The study compares shallow and deep networks with different layer counts, units per layer, and hidden/output nonlinearities.The considered architectures include one hidden layer and several hidden layers.
  • Hidden layers use either hyperbolic tangent units or ReLUs, defined as max(0, x).The paper motivates ReLUs by their non-saturation and sparse outputs, which can support learning and regularization.
  • Output layers use tanh, linear, or bounded linear functions that clip predictions to the control-variable bounds.The bounded linear function is intended to exploit saturation in optimal controls, including bang-bang profiles.
  • Inputs and outputs are normalized using the training-data mean and standard deviation, with tanh outputs additionally scaled to [−1, 1].The scaling matches the range of the final tanh activation.

B. Training

The training and evaluation procedure fits networks to optimal actions, then tests their closed-loop trajectories against target-reaching and optimality criteria. Evaluation accounts for tolerance-induced distortions in cost comparisons.

  • Networks minimize squared loss between predicted neural-network actions N(x_i) and optimal actions y(x_i) using stochastic gradient descent.Weights use learning rate η = 0.001 and momentum μ = 0.9.
  • Early stopping monitors a held-out 5% of training data and continues for five patience epochs after the last improvement.Weights are initialized with Xavier’s method.
  • DNN-driven trajectories are obtained by numerically integrating the system dynamics under network control.This closed-loop simulation evaluates whether learned actions remain effective beyond pointwise action prediction.
  • A trajectory succeeds when it reaches a tolerance region around the target, with state tolerances set to roughly 0.5%–1% of training-set variable ranges.The final state is the closest simulated state to the target, measured using tolerance-scaled heterogeneous units.
  • Because target tolerances can make the DNN cost appear slightly better than the optimal cost, the raw cost comparison is not fully accurate.This motivates matching the optimal trajectory’s stopping distance before computing the comparison.
  • Evaluation reports success rate, target distance across relevant state variables, and relative cost error for successful trajectories.Optimality compares costs after stopping the optimal trajectory at the same target distance as the DNN trajectory.

VII. Results

Results compare network architectures across four control profiles and show that depth, hidden-unit choice, and output constraints affect action-prediction accuracy. ReLU hidden layers and saturation-aware outputs are often advantageous, but no activation dominates every profile.

  • Control profiles: The four evaluated control profiles span smooth, saturated, bang-bang, and plateau-rich behavior.They include S-MOC u2, Q-QC u2, RW-MOC u1, and RW-MOC u2.
  • Output functions: For QUAD-QC u2 and RW-MOC u1, bounded-linear outputs perform best, closely followed by tanh outputs.Both functions can reproduce saturated regions, whereas linear outputs perform worst on these cases.
  • Neuron types: ReLU hidden units outperform tanh in most cases, with performance differences reaching up to two times for bounded control profiles.The exception is RW-MOC u2, where tanh performs consistently but only slightly better.
  • Limitations: The study cannot conclude that one hidden-layer nonlinearity is better across all domains.The reported differences depend on the control profile, especially when abundant plateaus are present.
  • Network depth: Five-layer networks with 16 units per layer outperform shallow networks with 515 hidden units despite having nearly four times fewer parameters.The broader architecture comparison favors deeper networks with similar parameter counts.

C. DNN-driven trajectories

DNN-driven trajectories closely reproduce optimal controls and achieve high landing performance across the considered models. After reaching the target, most models exhibit hovering-like behavior, while the directly pitch-controlled model oscillates because the target control is non-unique.

  • Landing performance: 100% success rates are achieved by the quadcopter and simple spacecraft for both objective functions.The reached state is close to the target, with distance D(xf) well below the success bounds.
  • Control reproduction: The DNN accurately reproduces the optimal control profiles and state-control behavior in representative trajectories across the considered problems.Figures 10 and 11 illustrate predictions and DNN-driven trajectories for reaction-wheel and thrust-vectoring models.
  • Landing performance: 100% and 98.3% success rates are obtained for reaction-wheel spacecraft under quadratic and time-optimal control, with relative error in J below 1%.The DNN predictions accurately represent the optimal control, including plateau-rich controls such as RWSC-MOC u2.
  • Landing performance: 99.0% and 95.0% success rates are obtained for the thrust-vectoring models under the QC and MOC objectives.Their relative error in J remains below 0.5% for both objectives.
  • Behavior after reaching xf: The networks cannot generally learn exact target hovering when the training data contain terminal controls that differ from the control required for hovering.For time- and mass-optimal cases, the required thrust mg is absent because target-reaching data use saturated thrust values.
  • Behavior after reaching xf: QUAD, RWSC, and TVR models hover near the target after reaching xf, whereas SSC continuously oscillates above the target.For SSC, averaging the non-unique terminal pitch control can produce nonzero thrust tilt, preventing exact hovering.

E. Generalization

The trained DNNs generalize beyond their initialization and training regions, though performance depends on the model and extension. The conclusion is that deep networks can approximate optimal state-feedback for real-time landing control, with depth materially affecting the approximation.

  • Generalization: 84.4% and 75.0% success rates are obtained for quadrotor extensions A1 and A2.These extensions reach 5 [m] in x and z for A1 and 10 [m] for A2.
  • Generalization: Long after target acquisition, RWSC and TVR trajectories maintain hovering near the target while spacecraft mass decreases.The commanded thrust c1u1 decreases to compensate for propellant loss, and the learned behavior persists at masses below the training-data range.
  • Interpretation: The networks appear to learn aspects of system dynamics and exploit them to maintain the spacecraft close to the target.This interpretation is supported by persistent hovering despite mass reductions beyond the training data.
  • Conclusion: Network depth strongly influences results: shallow networks cannot satisfactorily approximate the complex optimal state-feedback structure.The paper presents DNNs as a route toward real-time optimal-control architectures for planetary landing.
  • Conclusion: The trained DNNs introduce no significant impact on the final cost function and avoid catastrophic failures for conditions far from nominal.The paper reports optimal state-feedback for several deterministic, continuous-time, nonlinear aerospace systems.
Loading 1610.08668v1…