PaperlayerView exact revision on arXiv

Source-linked AI summary

Ground-Truth-First Benchmarking of Reinforcement Learning via Converse Optimality: From Smooth, Fully Observed Systems to Non-Smooth or Partially Observed Tasks

Grégoire Ouerdane, Sinan Ibrahim, Stefan Streif, Pavel Osinenko

arXiv:2610.30954v1cs.LG

TL;DR

Conventional RL benchmarks usually rank returns without revealing distance to an unknown optimum. This paper constructs certified tasks for sensitivity and oracle-referenced evaluation, and extends the framework to non-smooth and partially observed settings. Its conclusion distinguishes continuous-time sensitivity studies from native discrete-time absolute comparisons.

  • Problem

    Conventional RL benchmarks generally leave distance to optimality unknown because they lack a known optimal policy and value.

    Source: §1 · p. 3

  • Method

    The framework constructs tasks with certified optima and evaluates policies against those references, including through sufficient information states for partially observed tasks.

    Source: §8 · p. 37, §10 · p. 42

  • Results

    The framework provides continuous-time generators for sensitivity studies and native discrete-time fixtures for oracle-referenced comparison, with extensions to non-affine, non-Gaussian, non-smooth, and partially observed tasks.

    Source: §10 · p. 42

  • Takeaways & Limitations

    For partially observed evaluation, certifying the oracle in the learner's information set separates learning error from the information penalty.

    Source: §8 · p. 37

    The partially observed construction uses a deliberately designed benchmark with a known finite-dimensional sufficient statistic; it does not claim a nonlinear separation theorem.

    Source: §8.4 · p. 39

Abstract

from arXiv · show

Benchmarking in reinforcement learning is usually relative: algorithms are ranked by returns on environments with unknown optima. Distance to optimality remains unobservable, while conclusions may depend on reward design, stochasticity, hyperparameters, environment selection, and simulator details. We develop a ground-truth-first framework based on converse optimality, constructing each task with a certified optimal value function and policy withheld from learners and used only for evaluation. The framework unifies two modes: sensitivity benchmarking samples certified environments from continuous-time generators to assess robustness to task variation, while oracle-referenced comparison uses native discrete-time generators to evaluate policies via optimality gap, regret, and policy error. Continuous-time models support physically interpretable sensitivity studies but introduce dependence on the simulator step ($\Delta t$) and numerical scheme. Native discrete-time models align the environment, oracle, and learner on the same transition law, eliminating external integration choices as a confounder. We reproduce quantitative results covering optimizer sensitivity and oracle-referenced comparisons on serial planar arms, dynamically extended nonholonomic vehicles, and quadrotor-inspired tasks. The methodology extends beyond smooth affine Gaussian systems: virtual-input pullback handles non-affine raw-action interfaces and finite-covariance non-Gaussian disturbances. A non-smooth converse optimality formulation enables certified benchmarks with kinked value functions and switching surfaces. For partially observed evaluation, an information-state formulation reduces benchmarking to converse optimality on a sufficient statistic and provides an information-penalty reference. Code, fixtures, and diagnostics are available on the public project page.

1 Introduction

Relative-return benchmarks leave distance to optimality unknown, while results can depend on task and simulator choices. This paper constructs certified tasks for sensitivity studies and oracle-referenced comparisons, then extends the framework to non-affine, non-Gaussian, non-smooth, and partially observed settings.

1 Introduction

  • Unknown optima leave conventional benchmarks unable to show whether a higher-return algorithm is near optimal or whether rankings depend on benchmark choices.The paper identifies reward scaling, seeds, task weighting, environment selection, and simulator conventions as possible dependencies.Source: §1 · p. 3
  • Converse optimality starts from a desired value function and policy, then constructs an environment for which they are optimal.This makes optimality gaps, regret, Bellman residuals, and sensitivity across certified task distributions measurable.Source: §1 · p. 3
  • Continuous-time generators support sensitivity studies, while native discrete-time fixtures enable oracle-referenced comparison without an external integration choice.The digital learner's continuous-time environment depends on the integration step and numerical scheme.Source: §1 · p. 3
  • The framework extends to non-affine raw actions and non-Gaussian disturbances, non-smooth values, and partial observability through information states.The extensions retain the certificate logic while varying the learner-facing interface or the regularity of the certified value.Source: §1 · p. 4
  • The methodological contribution includes certified fixture exports and validation and evaluation protocols demonstrated on optimizer sensitivity and robotic control families.The project page provides code, fixtures, results, and diagnostics.Source: §1 · p. 4

2 Related Work

Prior RL benchmark suites support large-scale comparisons but generally lack known optimal policies, while forward optimal control is intractable for general nonlinear systems. The paper builds on converse-optimality work and extends it to non-smooth values, raw-action interfaces with non-Gaussian disturbances, and information-state evaluation.

RL benchmark suites

  • Shared RL suites enable large-scale comparisons but generally lack known optima, so their primary signal is relative return.A method can outperform baselines while remaining far from optimal.Source: §2 · pp. 4–5

Classical optimal control and inverse optimal control

  • Forward optimal control solves for values and policies but is intractable for general nonlinear systems, whereas inverse methods infer costs from demonstrations.Converse optimality instead prescribes a value function and constructs dynamics and costs for which it is optimal.Source: §2 · p. 5

Converse optimality

  • The paper builds on converse-optimality generators and extends prior work to non-smooth values, non-affine raw actions with non-Gaussian disturbances, and information states.Earlier work established continuous-time and discrete-time constructions, parametric RL benchmark generators, and stochastic control-affine oracle metrics [20].Source: §2 · p. 5

Non-smooth optimal control and viscosity solutions

  • Existing converse constructions assume C1 or C2 candidate values, excluding switching-surface and bang-bang structures with kinks or discontinuous feedback.The paper replaces smooth HJB certificates with viscosity inequalities and Clarke subdifferentials.Source: §2 · p. 5

Partial observability and information-state reduction

  • Partially observed benchmarks generally lack a certified reference in the learner's information set, making full-state comparisons conflate learning error with an information penalty.The paper certifies an oracle on a sufficient information state and isolates the penalty analytically in an LQG reference.Source: §2 · pp. 5–6

3 Ground-Truth-First Fixtures

A ground-truth-first fixture pairs an MDP with a certified optimal value and policy, which are withheld from learners and used as evaluation oracles. The certificate is auditable, and extensions preserve the fixture concept while adapting it to different task interfaces and regularity.

3 Ground-Truth-First Fixtures

  • A ground-truth-first benchmark adds a certificate that a specified value function and policy are optimal for the MDP.The reference pair is withheld from learners and supplied as an evaluation oracle.Source: §3 · p. 6
  • The Bellman-residual certificate requires zero residual at the prescribed policy and nonnegative residual for every action.Fixtures record validation domain, tolerance, and maximum residual so an external party can recompute the check.Source: §3 · p. 6
  • A fixture packages the certified MDP with a validation record and may specify time formalism, value regularity, action interface, and observation map.The learner trains on the MDP while the certified value is reserved for evaluation.Source: §3 · p. 6
  • Sampling task parameters yields a distribution over certified MDPs, while a fixed parameter yields a single known-answer task.The paper distinguishes continuous-time sensitivity families from native discrete-time oracle-referenced fixtures.Source: §3 · p. 6, Table 1 · p. 7
  • Extensions preserve the fixture concept while replacing smooth residuals, changing action and disturbance interfaces, or using a sufficient information state.The reference pair remains withheld until evaluation.Source: §3 · p. 7

4 Mode I – Continuous-Time Sensitivity Benchmarking

Mode I uses continuous-time converse generators to create certified environment families for sensitivity studies. Its digital learners experience discretized dynamics, making the integration step and numerical method part of the variation being tested.

4.1 Continuous-time converse generators for sensitivity

  • Continuous-time generators sample smooth fields H and G to produce certified nonlinear control problems spanning different system properties.The generated family serves as the substrate for sensitivity and robustness studies.Source: §4.1 · p. 7, §4 · p. 7
  • The learner experiences a discretized environment whose dynamics depend on integration step and numerical method, making those choices part of the sensitivity test.Native discrete-time fixtures are used for absolute comparisons to remove this confound.Source: §4.1 · p. 8

4.2 Sensitivity evaluation protocol

  • Relative suboptimality normalizes incurred cost by known optimal cost, allowing comparisons across task scales and distributions of environments.The protocol uses an F-test for variance and a t-test for means across sampled tasks.Source: §4.2 · p. 8

4.3 Sensitivity experiment

  • An F-test comparing two relative-suboptimality (RS) distributions for trained policies yielded F ≈ 2.59 (p ≈ 1.7 × 10−5), rejecting equal variance at conventional significance levels.The result shows optimizer choice alters suboptimality spread across environments, not that one optimizer is uniformly better.Source: §4.3 · pp. 8–9

5 Mode II – Native Discrete-Time Oracle-Referenced Comparison

Native discrete-time fixtures let learners, oracles, and evaluation metrics share one transition law, enabling oracle-referenced comparisons without a discretization confound. Results show that algorithm rankings vary by regime, while certified values reveal distance from optimality.

5 Mode II – Native Discrete-Time Oracle-Referenced Comparison

  • Native discrete-time generators make the transition law, oracle, and metrics share the same map, removing discretization as a comparison confound.The framework extends this certificate logic to non-affine actions, non-Gaussian noise, non-smooth values, and partial observability.Source: §5 · p. 9, §5.2 · p. 10

5.1 Deterministic discrete-time generator

  • A deterministic fixture is admissible when V⋆(s) − q(s) ≥ 0 for every state, and a freely chosen orthogonal field can add nonlinear coupling while preserving optimality.For quadratic value and costs, the admissible drift has an explicit form.Source: §5.1 · p. 9

5.2 Stochastic discrete-time generator

  • The stochastic quadratic-Gaussian generator uses an energy equation to construct certified drifts, while a control-strength homotopy varies authority and task difficulty without invalidating the certificate.Larger control authority improves the oracle but amplifies sensitivity to action error.Source: §5.2 · p. 10

5.3 Example families

  • Certified example families vary state-dependent actuation, dynamic-extension depth, coupling, control authority, and disturbance structure.The planar arm has configuration-dependent actuation; vehicle dynamic extension adds delayed actuation and long credit-assignment paths; ConverseQuad adds a non-affine interface.Source: §5.3 · pp. 10–11

5.4 Oracle-referenced evaluation protocol

  • Oracle-referenced evaluation measures optimality gap and trajectory-level regret; matched noise schedules and paired bootstrap intervals support comparisons across algorithms.Under the reward convention, zero gap is optimal and more negative values indicate greater suboptimality.Source: §5.4 · p. 11

5.5 Oracle-referenced experiment

  • On serial-random fixtures, TD3 has the best point estimate in gap and regret; on serial-fixed fixtures PPO is closest to the oracle, while SAC and TD3 show lower-tail failures.On NVDEx, PPO remains near the optimum while the other baselines remain far from zero; the authors attribute the challenge to delayed actuation and long-horizon dynamics.Source: §5.5 · pp. 12–13
  • TD3 leads on serial-random fixtures, while PPO is closest to the oracle on serial-fixed and NVDEx; no algorithm dominates across regimes.NVDEx shows the strongest separation, with the other baselines far from zero.Source: §5.5 · p. 12

6 Extension I: ConverseQuad – Non-Affine Actuation and Non-Gaussian Gusts

ConverseQuad preserves a certified optimum while changing the action interface to nonlinear raw actuator coordinates and allowing finite-covariance non-Gaussian disturbances. Reproduced results rank PPO and SAC as the leading group under the tested configuration, not as universal winners.

6 Extension I: ConverseQuad – Non-Affine Actuation and Non-Gaussian Gusts

  • ConverseQuad tests nonlinear raw-action coordinates and finite-covariance non-Gaussian gusts while constructing the optimum in a latent affine coordinate.The virtual-coordinate optimum is pulled back to the raw-action interface.Source: §6 · pp. 13–14

6.1 Virtual Actuator Coordinates

  • The 12-dimensional error state combines position, attitude, linear-velocity, and angular-velocity errors, with virtual thrust and moment inputs around hover.Thrust couples to translational velocity, while moments couple to angular velocity.Source: §6.1 · pp. 14–15

6.2 Raw Action Interface

  • The agent’s raw command passes through a smooth nonlinear actuator map, whose curvature increases with η and transforms policy errors before they enter the dynamics.The map is nearly linear for small commands and amplifies larger commands.Source: §6 · p. 14, §6.2 · p. 15

6.3 Cost Pullback and Oracle Preservation

  • Pulling back the virtual-action cost makes the raw-action MDP coordinate-equivalent to the virtual affine MDP, preserving the certified optimal action.The construction enforces the Bellman energy identity exactly.Source: §6.3 · pp. 15–16

6.4 Finite-Covariance Non-Gaussian Gusts

  • The certificate allows zero-mean finite-covariance disturbances beyond Gaussian noise, but requires independence across time and excludes state-dependent or temporally correlated noise.The reported Gaussian mixture is illustrative rather than a requirement of the certificate.Source: §6.4 · p. 16

6.5 Reported Results

  • PPO and SAC lead on both ConverseQuad metrics, TD3 is intermediate, and A2C and DDPG are least reliable under the reproduced protocol.The authors frame this ordering as a diagnostic of the raw-actuator interface, not a universal ranking; no new training runs were performed for this manuscript.Source: §6.5 · p. 16
  • The reported results do not resolve how actuator nonlinearity η and mixture parameter ϵ separately affect rankings, because both vary together in the tested configuration.The authors leave a systematic sweep of the two parameters to future work.Source: §6.5 · p. 17

7 Extension II: Non-Smooth Converse Optimality

This section extends converse optimality to non-smooth value functions, developing viscosity–Clarke-based certificates and a generator and validation pipeline for prescribed policies and piecewise-quadratic benchmarks with switching surfaces. It evaluates reinforcement-learning algorithms on discretized instances of these tasks.

7 Extension II: Non-Smooth Converse Optimality

  • The construction is deterministic and continuous-time, and a homotopy parameter selects a generalized-gradient branch to generate systems sharing one value function but differing in non-smoothness sharpness.The section develops a worked construction for a piecewise-quadratic value function with a straight switching line and validates it analytically.Source: §7 · pp. 17–18

7.1 Motivation: why non-smooth values matter

  • Certified non-smooth tasks expose whether a learner struggles with a known kinked optimum or discontinuous optimal policy, separating these challenges from approximation or optimization error.Such tasks address regimes where classical HJB equations and smooth converse constructions do not apply, including bang-bang control and switching systems.Source: §7.1 · p. 18

7.2 Background: viscosity solutions and Clarke subdifferentials

  • The framework uses a viscosity–Clarke condition: generalized gradients replace classical gradients, and HJB inequalities are checked over Clarke subdifferentials; for smooth values, it reduces to the classical HJB equation.At switching surfaces, the Clarke subdifferential contains the convex hull of limiting gradients from adjacent smooth regions.Source: §7.2 · pp. 18–19

7.3 Problem Setup and Assumptions

  • The candidate need only be continuous and locally Lipschitz, not convex or piecewise quadratic; the viscosity HJB certificate and uniqueness of viscosity solutions establish that it is the true optimal value.A measurable optimal feedback is also required, with closed-loop trajectories satisfying the stated regularity condition.Source: §7.3 · p. 20

7.4 Theoretical Results

  • Theorem 7.4.4 characterizes when a locally Lipschitz candidate value function can be realized as the optimal value of a control-affine system through measurable generalized-gradient and policy conditions.When those conditions hold, the policy is optimal and the candidate is the true value function; conversely, if the candidate is the optimal value, a triple satisfying the conditions exists.Source: §7.4 · p. 22

7.5 General Constructive Procedure

  • A six-step construction turns a locally Lipschitz candidate into system dynamics and costs certifying it as viscosity-optimal, with design choices shaped by the value function, switching geometry, and gain matrix.The procedure allows strictly convex, C2 control costs; its first step characterizes Clarke subdifferentials in smooth regions and on switching surfaces.Source: §7.5 · p. 23
  • The six-step construction produces a certified system whose prescribed value function is optimal and whose feedback is optimal, with closed-loop solutions interpreted in the Filippov sense when policies jump.The procedure selects a measurable Clarke generalized gradient, constructs the drift and state cost, verifies viscosity conditions, and checks closed-loop well-posedness.Source: §7.5 · p. 24

7.6 Construction with a Quadratic Control Cost

  • For quadratic control costs, the constructed state cost is automatically nonnegative, while the supersolution condition imposes a gain bound only on switching surfaces.If the scalar-gain bound fails, the construction can change the selected generalized gradient or use a state-dependent gain; in the worked family, cmin(s) = 8(1 − α^2)y^2 > 0 for α < 1 and y ≠ 0.Source: §7.6 · pp. 25–26

7.7 Remarks on Problem Geometry and Multiple Switching Surfaces

  • On switching surfaces, the selected Clarke gradient determines feedback strength while preserving the value function; multiple surfaces yield independent homotopy knobs subject to geometry-dependent certification.For several adjacent smooth regions, the subdifferential is parameterized by barycentric coordinates, but certification depends on switching-surface orientations and the spectrum of g(s)⊤g(s).Source: §7.7 · pp. 27–28

7.8 Problem Definition and Generator Pipeline

  • The generator parameterizes dimension, control authority, coupling, and non-smoothness while deriving certified value-policy pairs; validation checks the nontrivial supersolution residual.In the worked example, the supersolution condition holds for any k > 0, so the gain-enlargement loop is unnecessary.Source: §7.8 · p. 28

7.9 Example: Piecewise-Quadratic Value Function

  • The example family uses two-dimensional piecewise-quadratic value functions spanning smooth to sharply kinked cases, with continuity but a gradient jump on x = 0.Each quadratic piece is strictly convex, but the combined function is not globally convex when β ≠ γ.Source: §7.9 · p. 29
  • The piecewise-quadratic family is certified when its parameters satisfy the supersolution condition, yielding value function V and optimal feedback a* = −p.For the right-hand selection, the condition is (1 + 2k)β ≥ −γ; the configuration (1.4, −1.9, 0.01) fails the certificate and is excluded from certified numerical results.Source: §7.9 · pp. 31–32

7.10 Validation

  • Validation checks the selected-gradient subsolution equality, supersolution inequality across the Clarke subdifferential, and closed-loop well-posedness before exporting a fixture.The supersolution residual is checked at extreme subdifferential points, while bounded trajectories are used to screen for instability.Source: §7.10 · pp. 32–33

7.11 Dataset Construction

  • The dataset samples kink coefficients, passive drift gain, and switching-line gradient selection, rejecting instances that fail the certificate before exporting analytic dynamics and reference evaluators.Exported YAML fixtures include parameters and deterministic evaluation seeds, spanning smooth to sharply kinked tasks with varying passive drift and selection.Source: §7.11 · p. 33

7.12 Numerical Experiments on the Non-Smooth Benchmark

  • Off-policy SAC and DDPG achieve the lowest gaps and regrets, followed by TD3, while on-policy PPO and A2C perform significantly worse on the non-smooth benchmark.The results cover certified configurations C0–C4; the smooth baseline has the smallest on-/off-policy gap, and larger passive gain improves results from C1 to C2 for every algorithm. Unlike the smooth stochastic benchmarks, where PPO leads or shares the lead, this deterministic benchmark favors SAC and DDPG.Source: §7.12 · pp. 34–35, Table 5 · p. 35

7.13 Remark on Higher-Dimensional Extensions

  • The example shows that the non-smooth framework can generate concrete benchmarks, while higher-dimensional value functions and discrete-time dynamics remain future work.The proposed extensions include piecewise-quadratic values with multiple switching surfaces.Source: §7.13 · p. 36

8 Extension III: Partially Observed MDPs

The partially observed extension certifies policies on a sufficient information state, so the reference uses information available to the learner rather than hidden physical state. It builds output-facing benchmarks and separates learning error from the penalty imposed by limited information.

8 Extension III: Partially Observed MDPs

  • The extension certifies an optimal output-feedback policy on a sufficient information state, rather than on a hidden physical state unavailable to the learner.The learner receives observations and rewards while the private evaluator maintains the information state and certified feedback.Source: §8 · p. 37

8.1 Information States and Reduction

  • A sufficient information state reduces the output-feedback problem to a fully observed MDP, whose optimal policy remains optimal among nonanticipative output-feedback policies.Past costs belong in the learner’s history when rewards can carry information about the hidden state.Source: §8.1 · pp. 37–38

8.2 Converse Optimality on the Information State

  • Converse optimality on the information state certifies its candidate value and policy under affine virtual-input dynamics, finite-covariance noise, and separable costs with strictly convex action penalty.The proof transfers optimality from the Markov problem on the sufficient statistic to the output-feedback problem.Source: §8.2 · p. 38

8.3 Quadratic Information-State Construction

  • For quadratic costs and a finite-dimensional sufficient information state, the construction gives a quadratic optimal value and certified output-feedback controller.The construction carries over from the fully observed quadratic specialization by replacing physical state with the information state.Source: §8.3 · p. 39

8.4 Realizing the Information State as an Output System

  • An innovation-based output realization preserves the target information-state transition kernel, but is a deliberately designed benchmark rather than a physically natural hidden-state process.The construction does not claim a nonlinear separation theorem or cover general nonlinear POMDPs.Source: §8.4 · pp. 39–40

8.5 Benchmark Protocol

  • The evaluator reports the output-feedback learning gap separately from the information penalty, avoiding comparison against a full-state oracle that may use inaccessible information.When available, an analytic full-state reference makes the information penalty separately measurable.Source: §8 · p. 37, §8.5 · p. 40

8.6 LQG Reference and Information Penalty

  • In the LQG reference, the information penalty has a closed-form expression that scales with posterior covariance; the quadratic construction applies only to finite-dimensional sufficient states with affine dynamics.The section does not solve arbitrary nonlinear continuous-state POMDPs, whose belief state may be infinite-dimensional.Source: §8.6 · p. 41

9 Discussion

The extensions have specific scope boundaries: the non-Gaussian result excludes several noise structures, the non-smooth experiments retain timestep dependence, and the information-state construction does not solve general hidden-state problems.

Extension I

  • The non-Gaussian certificate covers finite-covariance disturbances but excludes state-dependent, heteroscedastic, and temporally correlated noise; ConverseQuad results use one configuration.A systematic sweep over actuator nonlinearity and mixture weight is left to future work.Source: §9 · p. 42

Extension II

  • The non-smooth theory is continuous-time, while its Euler-discretized experiments inherit Mode I’s timestep confound; a native discrete-time analogue remains open.The scalar-gain version also requires a positive denominator, and the supersolution condition couples gain geometry to the Clarke subdifferential.Source: §9 · p. 42

Extension III

  • The information-state construction certifies a finite-dimensional benchmark, not arbitrary nonlinear continuous-state POMDPs; its sufficient statistic is deliberately known by construction.General hidden-state models can require an infinite-dimensional belief density and a value functional on densities.Source: §9 · p. 42

10 Conclusion

The conclusion presents converse optimality as a ground-truth-first framework with continuous-time sensitivity generators and native discrete-time oracle-referenced fixtures. It extends certification to non-affine and non-Gaussian tasks, non-smooth values, and partially observed evaluation.

10 Conclusion

  • Continuous-time generators support sensitivity studies but inherit simulator-step dependence, whereas native discrete-time fixtures remove timestep ambiguity for absolute comparison.Oracle-referenced metrics include optimality gap, regret, Bellman residual, and policy error.Source: §10 · p. 42
  • The framework makes the optimum part of the benchmark specification rather than an unknown target outside it.This supports comparison by optimality gap, regret, Bellman residual, and policy error.Source: §10 · p. 42
  • The framework extends certification to non-affine raw actions and finite-covariance non-Gaussian disturbances, kinked value functions, and partially observed tasks using a sufficient information state.The information-state formulation also supplies an analytic LQG information penalty separating learning error from information loss.Source: §10 · p. 42

A Reproducibility and Released Assets

The appendices describe reproducibility materials, frame the work as a complement to broader evaluation, and prove the smooth-theory optimality certificate through Bellman residual conditions.

A Reproducibility and Released Assets

  • The released assets include generator code, seeds, fixtures, validation grids, instance metadata, evaluation schedules, baseline configurations, software versions, and reproduction scripts.The oracle enables independent checks of reported returns and each environment’s Bellman consistency.Source: A Reproducibility and Released Assets · p. 45

B Impact Statement

  • Known-answer fixtures can diagnose distance to optimality, but overuse as narrow leaderboards could encourage overfitting or unwarranted deployment confidence.The paper frames them as a complement to broader benchmarks and real-world validation.Source: B Impact Statement · p. 45

C Proofs for the Smooth Theory

  • The smooth-theory certificate is equivalent to zero Bellman residual on the candidate policy and nonnegative residual elsewhere, which makes the policy optimal.For smooth values and unconstrained actions, the condition reduces to the HJB equation and first-order stationarity; locally Lipschitz values use Clarke subdifferentials.Source: C Proofs for the Smooth Theory · p. 46

C.1 Results for the Stochastic Discrete-Time Setting

The stochastic discrete-time results establish when a candidate value and policy certify optimality under stated regularity assumptions. The continuous-time derivation constructs dynamics and a minimizing policy from chosen fields and a candidate value.

C.1 Results for the Stochastic Discrete-Time Setting

  • The expectation-gradient interchange is justified under the assumptions, supporting the stochastic Bellman optimality conditions and the resulting optimal policy.The proof uses a dominated-convergence argument with a growth bound integrable under Gaussian noise.Source: §C.1 · pp. 47–49
  • Under the stated smoothness, convexity, growth, and integrability assumptions, the stochastic converse theorem characterizes when a drift exists for which the candidate value and policy are optimal.The system uses Gaussian noise with positive-definite covariance, and the candidate value is twice continuously differentiable.Source: §C.1 · pp. 47, 49

C.2 Continuous-Time Converse Derivation

  • In the continuous-time derivation, chosen fields define the generator; completing the square yields a unique minimizing policy and a global certificate when the value and cost are positive definite.The drift-dependent terms cancel in the Hamiltonian before the quadratic minimization.Source: §C.2 · p. 50

D Proofs for the Non-Smooth Theory

These proofs establish when a proposed value function and policy satisfy the non-smooth optimality conditions. They also justify existence and well-posedness of the feedback used in the certificate.

D Proofs for the Non-Smooth Theory

  • Strict convexity and smoothness of the control cost make the first-order stationarity condition sufficient for the unique global minimizing action.The objective is coercive, so a global minimum exists; strict convexity makes it unique.Source: D Proofs for the Non-Smooth Theory · p. 51
  • A measurable minimizing feedback exists, and its closed-loop dynamics admit Filippov solutions even when the feedback is discontinuous.The argmin correspondence is measurable with non-empty compact values, enabling a measurable selection.Source: D Proofs for the Non-Smooth Theory · pp. 51–52
  • The sufficiency proof uses the viscosity HJB conditions to certify the proposed value function and feedback as optimal.Under the stated assumptions, uniqueness of the viscosity solution identifies the candidate value function with the true optimal value.Source: D Proofs for the Non-Smooth Theory · pp. 51–52
  • The necessity proof derives the viscosity HJB conditions and stationarity from an optimal value function, completing the stated equivalence.The same drift and measurable selections satisfy all three requirements of the converse condition.Source: D Proofs for the Non-Smooth Theory · pp. 52–53

E Proofs for the Information-State Theory

The information-state proof shows that a sufficient statistic supports dynamic programming and preserves optimality across history-dependent policies. The deterministic discrete-time construction separately builds certified dynamics through a homotopy.

E Proofs for the Information-State Theory

  • The information-state Bellman recursion gives the minimal finite-horizon cost-to-go because each history’s conditional cost depends on it only through the information state.Backward induction establishes the claim for finite horizon T.Source: E Proofs for the Information-State… · p. 53
  • For bounded costs and discount factor γ in (0, 1), the Bellman operator is a supremum-norm contraction with a unique fixed point.Finite-horizon iterates converge uniformly to that fixed point.Source: E Proofs for the Information-State… · p. 53
  • The minimizing information-state policy is optimal among all nonanticipative output-feedback policies, not only policies expressed directly on the information state.The composite policy applies the information-state policy to the statistic of the observed history.Source: E Proofs for the Information-State… · pp. 53–54

F.1 Deterministic Discrete-Time Homotopy Construction

  • The deterministic homotopy construction deforms feasible next-state targets on a value level set while balancing control effort.An orthogonal field can change drift geometry without changing the certificate.Source: §F.1 · pp. 55–56

F.2 Quadratic-Gaussian Energy Identity Derivation

The quadratic-Gaussian derivation expands the expected value under zero-mean noise using its covariance. The reproducibility protocol separates learner access to environments from oracle-based validation and evaluation.

F.2 Quadratic-Gaussian Energy Identity Derivation

  • With quadratic value, state cost, and control cost, zero-mean noise contributes the covariance term tr(PΣ) to the expected next-state value.The resulting Bellman integrand is expanded into quadratic and cross terms in the action.Source: §F.2 · p. 56

G Benchmark Generation, Experimental Implementation, and Reproducibility

  • The released protocol separates continuous-time sensitivity studies from native discrete-time oracle-referenced fixtures, while withholding certificates from learners.The certified value, policy, validation traces, and oracle diagnostics are reserved for validation and post-training evaluation.Source: G Benchmark Generation, Experimental… · p. 57

G.1 Generator Validation and Experimental Implementation

This section describes validation and reproducible implementation for continuous-time optimizer-sensitivity experiments and native discrete-time oracle-referenced fixtures. It covers oracle-metric evaluation and independent audit checks, including distributional results that expose lower-tail failures not shown by mean values alone.

G.1 Generator Validation and Experimental Implementation

  • Fixtures are accepted only after checks of Bellman-metric positive definiteness, Bellman-identity residual, and closed-loop boundedness.The metric is checked on a bounded validation domain; residuals must meet the fixture’s recorded tolerance, and unstable or out-of-domain trajectories cause rejection.Source: §G.1 · pp. 57–58
  • Mode I compares Adam and RAdam PPO pipelines across 80 generated-system runs per optimizer, measuring outcome variability under a shared system schedule rather than a fixed training trace.Neural-network initialization and optimizer stochasticity are not frozen.Source: §G.1 · p. 58
  • Mode II uses native discrete-time fixtures to align transitions, oracle, learner, and metrics, and reports ConverseArm and NVDEx benchmark families.ConverseArm is a stochastic serial planar arm with state-dependent actuation; NVDEx uses velocity-extension chains with delayed actuation while retaining a certified quadratic oracle.Source: §G.1 · p. 60

G.2 Oracle Evaluation, Artifacts, and Reproducibility Checklist

  • Oracle evaluation distinguishes cost and reward conventions: the certified policy has zero optimality gap up to numerical precision, while lower cost regret is better.Reward-convention gaps are zero at optimality and become more negative with suboptimality; plots state the convention used.Source: §G.2 · p. 61
  • All policies use the same evaluation starts and disturbance streams, enabling paired comparisons; confidence intervals are computed by paired bootstrap over trajectory indices.For deterministic environments, the shared schedule reduces to matched start states.Source: §G.2 · pp. 61–62
  • One-step Bellman-consistency diagnostics are zero in expectation for oracle actions under the exact transition law and are used for validation, not learner training.The diagnostic is subject to numerical and Monte Carlo error.Source: §G.2 · p. 62
  • Released artifacts include generator configurations and validation outputs, training scripts and traces, oracle metrics, evaluation schedules, bootstrap summaries, and diagnostic figures.The reproducibility checklist also includes fixture configurations, seeds, accepted/rejected-instance logs, validation tolerances, and CRN schedules.Source: §G.2 · p. 62
  • Reproducibility materials include metric and figure-generation scripts, software and dependency versions, and hardware and wall-clock summaries.The reported hardware was an Intel Core i9-9900 CPU, RTX 3080 12 GB, and 16 GB DDR4 RAM.Source: §G.2 · pp. 62–63
  • Some algorithm–regime pairs have heavy negative tails, with runs or fixtures orders of magnitude farther from the certified optimum; Figure 10 displays this distributional variability.Its violin widths show empirical optimality-gap distributions, not confidence intervals; the dashed line marks the certified optimum at zero.Source: §G.2 · p. 63, Figure 10 · p. 63
  • The benchmark is independently auditable: groups can regenerate or reload fixtures, verify Bellman consistency, train without oracle access, and evaluate with paired start-state and disturbance schedules.Source: §G.2 · p. 63
  • Table 6 reports native discrete-time fixture comparisons using means and paired-bootstrap 95% CIs, with zero optimal for optimality gap and lower regret preferred.Source: Table 6 · p. 63

Related papers

Loading 2610.30954v1…