PaperlayerView exact revision on arXiv

Source-linked AI summary

Data-Driven Hamiltonian for Neural Optimal Control

Christopher A. Strong, Aswin Surya, Namhoon Cho, Jason J. Choi, Claire J. Tomlin

arXiv:2610.36378v1eess.SYcs.SY

TL;DR

RL for optimal control can require substantial data, while offline performance depends on dataset quality and coverage. This paper extends the Hamilton-Jacobi framework with a data-driven Hamiltonian and neural value-function training. The reported results include near-optimal LQR performance on random data and a smaller lunar-lander optimality gap than CQL and FQL.

  • Problem

    RL can require substantial data for optimal control, and offline performance remains dependent on the quality and coverage of available data.

    Source: §1 · pp. 1–2

  • Method

    The paper extends DDH to optimal control with running costs and uses it to train neural value functions with PDE residuals and worst-case fictitious rollouts.

    Source: §1 · p. 2, §3 · p. 4

  • Results

    On random-trajectory LQR data, DDH is within 0.1% of optimal cost; on lunar lander, its optimality gap is more than 3× smaller than CQL and FQL.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Neural DDH can train across the state space, including previously unobserved state-action pairs, by using DDH to handle them.

    Source: §5 · p. 8

    DDH requires enough system information to bound dynamic uncertainty and a cost evaluable at any state and action; minimal information can yield prohibitively large uncertainty sets, especially in high dimensions.

    Source: §5 · p. 9

Abstract

from arXiv · show

Reinforcement learning (RL) has emerged as a prominent data-driven approach to optimal control, but often requires substantial amounts of data and provides limited guarantees when encountering out-of-distribution state-action pairs. In this paper, we propose an alternative approach grounded in the well-established Hamilton-Jacobi framework in optimal control theory. Rather than requiring an explicit analytical model of the system dynamics to evaluate the Hamiltonian, we replace this with a data-driven Hamiltonian (DDH), which we generalize to a broad class of optimal control problems. In contrast to standard RL approaches, the DDH provides a principled mechanism, requiring only minimal prior system information such as a Lipschitz constant, for quantifying and accounting for uncertainty in regions not covered by the data, enabling the derivation of controllers with performance guarantees. Additional prior system knowledge, when available, can be readily incorporated to improve performance. For high-dimensional systems, we further introduce a physics-informed neural approximation of the value function, trained using a novel procedure that generates worst-case fictitious rollouts evaluated through the DDH. On an LQR problem, the DDH reaches near-optimal cost on every dataset we test, including random data on which offline RL methods remain far from optimal, and needs far fewer samples than online RL. On a nonlinear lunar lander benchmark, the DDH achieves an optimality gap to a trajectory-optimization reference more than $3\times$ smaller than CQL and FQL, and matches the performance of SAC with approximately $37\times$ fewer samples. These results establish the DDH as a promising framework for data-driven optimal control, particularly in settings where available data are limited in quantity or quality.

1 INTRODUCTION

The paper proposes extending the Data-Driven Hamiltonian (DDH) to optimal control with running costs, addressing the difficulty of applying Hamilton-Jacobi methods when system dynamics are unknown. Its neural approach uses DDH-based training beyond observed state-action pairs, and the reported LQR and lunar-lander results show strong performance across data settings.

1 INTRODUCTION

  • RL can require substantial interaction data, and offline performance remains dependent on the quality and coverage of available data.Source: §1 · pp. 1–2
  • The paper extends DDH to optimal control with running costs and trains neural value functions using PDE residuals evaluated at arbitrary states, including states outside the dataset.Source: §1 · p. 2
  • Worst-case fictitious rollouts provide supervised value targets without an explicit dynamics model or simulator and can start from arbitrary states.The rollouts represent worst-case dynamics uncertainty consistent with the data.Source: §1 · p. 2
  • On random-trajectory LQR data, DDH is within 0.1% of optimal cost, while CQL, IQL, and FQL are 13%, 26%, and 61% worse than optimal.On the lunar lander, neural DDH has an optimality gap more than 3× smaller than CQL and FQL, and matches SAC with approximately 37× less data.Source: §1 · pp. 2–3
  • Against online RL, DDH matches SAC’s best performance on every dataset with at least 25× fewer samples and PPO’s with 100× fewer samples.Source: §1 · p. 2

2 PROBLEM FORMULATION AND BACKGROUND

This section formulates continuous-time optimal control with unknown dynamics and a running cost, under assumptions that provide trajectory data and high-level information to bound dynamic uncertainty. It reviews the HJB framework and motivates replacing the model-dependent Hamiltonian with a data-driven bound.

2 PROBLEM FORMULATION AND BACKGROUND

  • The dynamics are unknown, but the method assumes trajectory data with exact state velocities and high-level information, such as Lipschitz constants, sufficient to bound uncertainty.The authors note that estimating such constants requires less modeling effort than identifying the dynamics, but means the method is not fully model-free.Source: §2 · p. 3
  • The objective is to approximately minimize a discounted cost over admissible control signals, with running and terminal costs.Source: §2 · p. 3
  • The HJB framework characterizes the optimal cost-to-go through a PDE, and the optimal control is derived from its value function.Source: §2 · p. 3
  • Evaluating the Hamiltonian to solve the HJB equation requires the dynamics, making it the key quantity to replace when those dynamics are unknown.Source: §2 · p. 4
  • Prior DDH work bounded the Hamiltonian for reachability problems with zero running cost; this paper generalizes the definition and uncertainty set.Source: §2 · p. 4

3 APPROACH: THE DATA-DRIVEN HAMILTONIAN FOR OPTIMAL CONTROL

The approach bounds the Hamiltonian using data and uncertainty sets, then uses that bound to learn a conservative value function. For higher-dimensional systems, neural value learning combines PDE-based training with fictitious rollouts under worst-case dynamics.

3 APPROACH: THE DATA-DRIVEN HAMILTONIAN FOR OPTIMAL CONTROL

  • The DDH minimizes over a search set of controls and maximizes over data-consistent velocities, upper-bounding the Hamiltonian and yielding a conservative value-function bound.The uncertainty set must contain the true velocity, and the search set is a subset of admissible controls.Source: §3 · p. 4
  • State-only DDH restricts controls to those observed in the dataset, which can cause chattering; state-control DDH searches beyond observed controls for a tighter bound.When computing bounds is expensive, the tightening can be limited to the top k state-only bounds.Source: §3 · pp. 4–5

3.1 NEURAL DATA-DRIVEN HAMILTONIAN-BASED VALUE FUNCTION

  • For high-dimensional systems, a neural value function is trained with PDE and terminal-condition losses, with the DDH residual evaluable at arbitrary states.Grid-based HJB solvers scale exponentially with state dimension.Source: §3.1 · p. 5
  • Because PDE loss enforces only local consistency, fictitious rollouts under DDH worst-case dynamics provide additional value targets for supervising the neural value function.These dataset-dependent adversarial dynamics use the DDH-selected control and worst-case velocity; the rollout loss pulls the value estimate toward the resulting return.Source: §3.1 · p. 5
  • Rollout supervision data are comparatively expensive to compute, so the training procedure can hold them fixed for several iterations before refreshing them.The procedure samples PDE states uniformly and samples rollout start states from a supervision region.Source: §3.1 · pp. 5–6
  • For discounted infinite-horizon problems, the method can learn a static value function; the authors observe more stable training, likely due to contraction.The rollout loss no longer depends on time, while the PDE loss enforces the static HJB equation.Source: §3.1 · p. 5

4 EXPERIMENTAL RESULTS

The experiments test DDH on LQR datasets of varying quality and on a lunar lander benchmark. DDH is near-optimal across the tested LQR datasets and substantially reduces the optimality gap to offline RL on lunar lander.

4.1 VERIFICATION ON AN LQR PROBLEM

  • The LQR comparison uses optimal, noisy-optimal, random-trajectory, and uniform-random datasets; online PPO and SAC collect their own data, so they are sample-efficiency references rather than like-for-like baselines.Each trajectory dataset contains 100,000 transitions, and smaller dataset sizes use prefixes of the larger dataset.Source: §4.1 · pp. 6–7
  • At n = 20,000, rectangular DDH stays within 0.2% of J⋆ on every LQR dataset; on random trajectories CQL, IQL, and FQL remain 13%, 26%, and 61% above it.The dataset-size plot compares average rollout cost over 1,000 test initial conditions against an optimal Riccati reference.Source: Figure 1 · p. 7, §4.1 · p. 7
  • DDH benefits from broad state and control coverage, while offline RL improves fastest on expert data because its policies tend to stay close to recorded actions.For offline RL, this helps when recorded actions are optimal and hurts when they are random.Source: §4.1 · p. 7
  • At n = 500, rectangular uncertainty sets and tightening each lower LQR cost on most datasets, while neural DDH is comparable to the grid solution.The variants isolate uncertainty-set shape, tightening, and function approximation.Source: §4.1 · pp. 6–7
  • Online RL is far less data-efficient: after 50k (SAC) and 526k (PPO) environment steps, neither reaches the 2.98 DDH attains from 1,000 random transitions.Source: §4.1 · p. 8

4.2 LUNAR LANDER CASE STUDY

  • The lunar-lander DDH uses a known sensitivity matrix and single-integrator state relations, and is evaluated against RL baselines and a true-dynamics trajectory optimizer.The evaluation rolls out 15-second trajectories from 20,000 random initial conditions.Source: §4.2 · p. 8
  • The neural DDH reduces the lunar-lander optimality gap by 67% versus FQL and 72% versus CQL; matching its performance takes SAC 37× and PPO 2,300× as much data.The comparison uses a trajectory-optimization reference with access to the true dynamics.Source: §4.2 · p. 8
  • On lunar lander, single-integrator knowledge improves DDH, but removing fictitious-rollout supervision causes a larger deterioration.The rollout targets capture worst-case dynamics uncertainty consistent with the available data.Source: §4.2 · p. 8

5 CONCLUSION AND FUTURE WORK

The paper extends the DDH to optimal control with running costs and introduces a neural value-function approximation trained with worst-case fictitious rollouts. The conclusion highlights benchmark gains, the role of prior system information, and limitations on the required uncertainty sets and cost functions.

5 CONCLUSION AND FUTURE WORK

  • The neural DDH uses worst-case fictitious rollouts for training and handles previously unobserved state-action pairs to train its value network across the state space.The DDH also provides a worst-case bound based on known system properties, defined everywhere, so prior knowledge can be incorporated directly.Source: §5 · p. 8
  • The DDH has more than a 3× lower optimality gap than offline RL algorithms on the lunar lander, while SAC and PPO need roughly 37× and 2,300× more interaction to match its performance.Figure 3 evaluates lunar lander cost on 20k initial conditions from the supervision region; the training data came from a larger region.Source: Figure 3 · p. 9
  • The authors hypothesize that DDH benefits most from broad state-action coverage, whereas offline RL benefits most from optimal demonstrations.They describe DDH as a natural fit when consistent, high-quality expert demonstrations are hard to collect.Source: §5 · p. 9
  • The DDH requires enough system information to form a valid dynamics uncertainty set and a cost function evaluable at any state and action.With only minimal information such as one Lipschitz constant, the uncertainty set can become prohibitively large, especially in high-dimensional spaces.Source: §5 · p. 9
  • The DDH’s trajectories drive the lander toward the origin, while CQL does not and SAC fails to keep the lander within bounds.CQL uses the DDH dataset; SAC uses 30k samples gathered online.Source: Figure 5 · p. 10

A.1 RELATED WORK

The related work contrasts model-free and model-based reinforcement learning with data-driven MPC and the DDH. It positions the DDH as a Hamiltonian-bounding approach that extends prior reachability work to running-cost optimal control and higher-dimensional systems.

A.1 RELATED WORK

  • Model-free RL learns from online interactions or fixed offline transitions, but its value supervision is based on observed transitions, leaving behavior outside dataset support indirectly constrained.CQL penalizes out-of-distribution actions; IQL avoids querying them, while FQL incorporates near-optimal demonstrations.Source: §A.1 · p. 12
  • AWR and TD3+BC also regularize policies toward data; the paper compares CQL, IQL, and FQL as widely used representatives with strong reported performance.Source: §A.1 · p. 12
  • Model-based RL learns dynamics before building a controller, but can face out-of-distribution errors when using its model beyond training data; MOPO penalizes uncertain state-action pairs.Source: §A.1 · p. 12
  • Most direct data-driven MPC work focuses on linear systems to use Willems’ fundamental lemma, restricting generalizability.Source: §A.1 · p. 12
  • The DDH bounds the Hamiltonian using data and minimal system information, extending prior reachability work to running costs and higher-dimensional systems.Earlier DDH work focused on reachability problems, which have zero running cost.Source: §A.1 · p. 12

A.2 MODULAR COMBINATION OF DDHS

Multiple valid DDHs can be combined by taking their minimum, which remains an upper bound on the Hamiltonian. This lets the combined bound draw on different information sources in different regions.

A.2 MODULAR COMBINATION OF DDHS

  • The minimum of individually valid DDHs defines a combined DDH because each constituent upper-bounds the Hamiltonian.Source: §A.2 · p. 12
  • A nominal model with a known maximum error and a data-derived bound can be combined, using the dataset bound near observations and the model-based bound farther away.The associated control and worst-case velocity are selected by the arg min and arg max of the DDH attaining the minimum.Source: §A.2 · p. 13

A.3 ADDITIONAL LIPSCHITZ CONSTANTS AND UNCERTAINTY SETS

The appendix describes four sensitivity notions that produce uncertainty sets of different shapes and tightness. These alternatives encode different amounts and structures of prior knowledge about the dynamics.

A.3 ADDITIONAL LIPSCHITZ CONSTANTS AND UNCERTAINTY SETS

  • A uniform Lipschitz constant treats all directions alike and yields a spherical uncertainty set, while an input-wise vector allows different sensitivities by state coordinate.The input-wise set shrinks along low-sensitivity state directions.Source: §A.3 · p. 13
  • An output-wise vector bounds each velocity component separately, producing a rectangular uncertainty set; a sensitivity matrix combines input- and output-wise structure and is the tightest option.Source: §A.3 · p. 13
  • For the double integrator, only one sensitivity-matrix entry is nonzero, so the rectangular variants perform best in Figure 7.Zero Jacobian entries correspond to zero-width directions of the uncertainty set.Source: §A.3 · p. 13
  • The sensitivity assumptions bound changes in dynamics across state and control, with control analogues defined for the uniform, input-wise, output-wise, and combined forms.Setting the control to a recorded value recovers the earlier sets; experiments use the tightened rectangular variants from the final row.Source: §A.3 · p. 13

A.4 LQR BASELINE TRAINING DETAILS

The LQR baselines were evaluated on fixed test initial conditions, with checkpoint selection separated onto validation conditions. Robustness checks also report seed variation and repeated uniform-random dataset draws.

Datasets

  • Offline-RL baselines used discounted-return training with γ = 0.995, while evaluation used undiscounted finite-horizon cost; the mismatch cost little on optimal data.CQL came within 1% of J⋆ on the optimal dataset.Source: §A.4 · p. 14

Evaluation and epoch selection

  • All methods were scored on the same 1,000 fixed test initial conditions, while offline-RL checkpoints were selected using 500 disjoint validation initial conditions.Evaluation used true dynamics with T = 2 s and ∆t = 0.01.Source: §A.4 · p. 14

Baseline training details

  • CQL, FQL, IQL, PPO, and SAC used their stated implementation settings, with offline methods reported at validation-selected checkpoints and online methods at saved training milestones.FQL’s behavior-cloning coefficient was tuned per dataset; the selected value decreased as action quality declined.Source: §A.4 · p. 14

Robustness checks

  • 3.03 ± 0.04 at n = 50 and 3.00 ± 0.04 at n = 75: grid DDH stayed within 4% of optimal across five uniform-random dataset draws.The reference cost was J⋆ = 2.96.Source: §A.4 · pp. 14–15

A.5 DDH VARIANTS ON THE LQR PROBLEM

The LQR variants compare DDH uncertainty sets, tightening, and function approximation. In the minimal-information setting, scalar Lipschitz constants support comparisons with offline RL, while rectangular bounds can further reduce cost.

DDH variants

  • Table 1 compares all DDH variants at n = 500 by average cost over 1,000 test initial conditions.The variants isolate spherical versus rectangular uncertainty sets, tightening, and neural versus grid approximation.Source: Table 1 · p. 15

Minimal-information DDH

  • The rectangular uncertainty set closes much of the remaining gap to J⋆, reaching 3.11 vs 5.11 on uniform random at n = 50.The minimal-information spherical DDH uses only scalar constants Lx = Lu = 1; its comparisons with offline RL vary by dataset and size.Source: §A.5 · p. 15, Table 2 · p. 15

A.6 SENSITIVITY TO MISSPECIFICATION OF THE LIPSCHITZ BOUND

The appendix tests sensitivity-bound misspecification and details the lunar-lander dynamics, sensitivity bounds, learned value function, and training procedure. It reports gradual cost increases when valid bounds are overestimated, while underestimation is not tested.

A.6 SENSITIVITY TO MISSPECIFICATION OF THE LIPSCHITZ BOUND

  • Cost rises by 4 to 34% at 2× and 10 to 61% at 4× inflated sensitivity bounds; underestimation, which breaks the guarantee, is not tested.Even at 4×, DDH beats correctly configured CQL on uniform random (3.68 vs 7.14) and random trajectories (8.24 vs 17.05).Source: §A.6 · p. 16, Table 3 · p. 16

A.7 LUNAR LANDER DYNAMICS AND SENSITIVITY BOUNDS

  • The lunar lander has six states and three controls, with each control bounded in [0, 1] and experiments using γ = 0.The model specifies mass, inertia, thruster strengths, lever arms, and gravity; Figure 8 depicts the thruster geometry.Source: §A.7 · p. 16, Figure 8 · p. 17

A.7.1 SENSITIVITY

  • Sensitivity is bounded from Jacobians of the dynamics; the IO method uses bounded sensitivity matrices, while sphere-based variants use their maximum singular values as Lipschitz constants.The tighter state-Jacobian bound is used for Lx,io in experiments.Source: §A.7.1 · p. 17

A.8 LUNAR LANDER LEARNED VALUE FUNCTIONS

  • The learned lunar-lander value function has qualitatively reasonable, relatively smooth two-dimensional slices with few training artifacts.The visualization uses the known Lx,io and known single-integrator structure.Source: §A.8 · p. 18, Figure 9 · p. 18

A.9 LUNAR LANDER TRAINING DETAILS (DDH AND BASELINES)

  • Neural DDH training begins with 5,000 gradient steps of PDE loss, then adds rollout supervision from 25,000 fictitious rollouts regenerated every 240 steps.The rollout horizon grows from 0.5 to 4 s; the model-based baseline instead learns dynamics before solving nonlinear MPC, while model-free baselines use CQL, FQL, PPO, and SAC.Source: §A.9 · pp. 18–19

Related papers

Loading 2610.36378v1…