PaperlayerView exact revision on arXiv

Source-linked AI summary

Dissimilarity-Based Markov Decision Processes for Stabilizing State Distributions

Xun Shen, Yuepeng Wang, Daniyal Sajedi-Hosseini, Akifumi Wachi, Ken Kawano, Yoshihiko Fujisawa, Mehrshad Sadria, Sébastien Gros, Katsuki Fujisawa

arXiv:2610.29024v1cs.LG

TL;DR

Expected-return optimization alone does not ensure that a stochastic system’s state distribution converges to or stays near a target. This paper develops DMDPs and error-aware stability certificates; on HalfCheetah, DMDP-Lag with PPO/SAC brings the velocity distribution closest to a fast and safe target among compared methods.

  • Problem

    Expected-return optimization alone does not guarantee that the evolving state distribution converges to a desired target or remains stable.

    Source: §1 · p. 2

  • Method

    The paper formulates distribution control as a DMDP and uses finite-dimensional representations and Lyapunov certificates to analyze optimal, learned, and deployed policies.

    Source: §1 · pp. 2–3, §2.1 · p. 4

  • Results

    On HalfCheetah, DMDP-Lag (PPO/SAC) brings the velocity distribution closest to a fast and safe target among compared methods and attains the highest evaluation.

    Source: §6 · p. 16

  • Takeaways & Limitations

    HalfCheetah experiments provide empirical evidence that distribution feedback and explicit target matching improve regulation toward a desired velocity distribution.

    Source: §1 · p. 4

    The estimated-decrease theorem fixes the policy and surrogate during deployment and does not establish stability during policy updates or exploration.

    Source: §4.3 · p. 12

Abstract

from arXiv · show

Reinforcement learning for stochastic systems typically optimizes expected cumulative rewards, while reliable and repeatable behavior also depends on how the state distribution evolves over time. Optimizing expected return alone does not guarantee that this distribution converges to a desired target or remains stable. This paper develops a theoretical framework for learning policies that stabilize state distributions. We formulate the problem as a Dissimilarity-based Markov Decision Process (DMDP), in which the policy acts on the current state distribution and the objective measures its dissimilarity to a target distribution. We establish sufficient conditions under which the optimal policy makes the target distribution asymptotically stable. To support policy learning, we introduce a finite-dimensional representation of distributions and show how stability guarantees for its parameter dynamics extend to the underlying state distributions. We further develop certificates for the learned policies, including their deployment using estimated distribution parameters. These results quantify how learning and estimation errors affect the guarantees and identify conditions under which learned policies retain asymptotic stability. For practical implementation, we combine distribution estimates and target-dependent rewards with PPO and SAC. A computable example on a linear system illustrates the proposed theoretical certificates. Furthermore, experiments on a HalfCheetah task demonstrate the benefits of distribution feedback and explicit target matching for regulating the velocity distribution.

1 Introduction

The paper frames stochastic-system control around whether a policy can stabilize the evolving state distribution, not merely optimize expected return. It develops DMDP theory, finite-dimensional Lyapunov certificates, and learning and deployment guarantees, then evaluates distribution feedback on HalfCheetah.

DMDP formulation

  • DMDPs make the current state distribution the decision state and use a target-dependent dissimilarity as the stage cost.The framework connects distribution feedback, optimal-policy stability, and certificates for policies obtained from data.Source: §1 · p. 2

Optimal-policy stability

  • For the undiscounted infinite-horizon problem, sufficient conditions link the stage cost and optimal value to asymptotic stability of the target distribution.The formulation recovers norm-based distributional stabilization when its dynamics are representable by an admissible DMDP policy.Source: §1 · p. 3

Finite-dimensional certificates

  • A finite-dimensional distribution representation and compatibility condition transfer Lyapunov decrease in parameter space to stability of the induced distributions.For the ideal undiscounted problem, dissipativity-based cost rotation connects the certificate to optimal control.Source: §1 · p. 3

Learning and deployment

  • Learned-policy certificates quantify how model, learning, and deployment errors affect Lyapunov decrease and residual sets under stated high-probability error bounds.The paper also gives a computable linear-system example that evaluates certificate error terms and residual sets.Source: §1 · p. 3

Empirical evaluation

  • On HalfCheetah, DMDP-Lag (PPO/SAC) brings the velocity distribution closest to a fast and safe target among compared methods and attains the highest evaluation.The experiments provide empirical evidence that distribution feedback and explicit target matching improve regulation; Lyapunov certificates are not established for the learned HalfCheetah policies.Source: §1 · p. 4

2 Dissimilarity-Based MDPs

DMDPs regulate probability measures through distribution-dependent feedback and a dissimilarity-based objective. The section defines distributional stability, gives sufficient conditions for optimal-policy stability, and explains why direct solution is impractical.

2.1 State-Distribution Dynamics and Dissimilarity Measures

  • The DMDP uses the current probability distribution as its decision state, applies a distribution-dependent policy, and can penalize its dissimilarity to a target distribution.Unlike the norm-based MDP, whose policy acts on a realized state sample, the DMDP policy acts on the distribution itself.Source: §2.1 · pp. 4–5

2.2 Optimal-Policy Stability

  • With γ = 1, an optimal policy makes the target distribution D-asymptotically stable when the stated invariance and stage-cost and value bounds hold.The theorem assumes an invariant finite-value region and a stationary target under the optimal closed-loop dynamics.Source: §2.2 · p. 5

2.3 Challenges in Solving DMDPs

  • Direct solution is impractical because the value function is defined over an infinite-dimensional measure space and the distributions and transition operator are generally unknown.Practical applications instead have finite trajectory data from which distributional dynamics must be inferred.Source: §2.3 · pp. 5–6

3 Finite-Dimensional Representation and Stability Analysis

The paper replaces infinite-dimensional distribution dynamics with finite-dimensional parameter dynamics, then uses Lyapunov arguments to transfer stability guarantees back to distributions. Cost rotation and a compatibility condition support certificates for the ideal optimizer and other admissible policies.

3.1 Parameterized Distributions

  • A finite-dimensional parameter ω represents each distribution, with deterministic parameter dynamics that are exact under closure of the distribution family and otherwise approximate.Closure holds for linear dynamics with additive Gaussian noise and a Gaussian family, but not for the product of uniform families under HalfCheetah dynamics.Source: §3.1 · pp. 6–7
  • The parameterized optimal control problem uses a stationary parameter ω⋆ representing the target distribution, which also normalizes the stage cost through ℓ0.ω⋆ is an invariant parameter whose associated measure is the target ρ⋆, not an additional target.Source: §3.1 · p. 7

3.2 Cost Rotation

  • For the undiscounted problem, cost rotation preserves optimal policies when storage is bounded and infinite-horizon limits are well defined.Under the stated conditions, the rotated optimal value can serve as a Lyapunov function for the optimal parameter dynamics.Source: §3.2 · p. 8, §3.3 · pp. 8–9

3.3 Lyapunov Certificates for Distributional Stability

  • With D-compatibility, a Lyapunov decrease in parameter space transfers to D-asymptotic stability of the induced distributions, including under an admissible policy not assumed optimal.The parameter distance and distributional dissimilarity must be locally comparable on the operating set; the ideal optimal policy also has a stability guarantee under the theorem’s conditions.Source: §3.3 · pp. 8–9

4 Stability Certification for Learned Policies

Section 4 develops certificates for learned policies under model and estimation errors, and studies when those guarantees persist as data increase. The certificates apply to fixed policies; the practical implementation combines distribution estimates and target-dependent rewards with PPO and SAC.

4 Stability Certification for Learned Policies

  • The certificates address model, estimation, and learning errors but do not establish stability during policy updates or exploration.The section’s guarantees concern a fixed learned policy.Source: §4 · p. 9

4.1 Certification with an Approximate Dynamics Model

  • With an approximate model, a high-probability decrease certificate transfers to the true dynamics, with a residual set that shrinks as the error bound vanishes.The transfer requires suitable uniform error bounds and invariance on the operating region.Source: §4.1 · p. 10

4.2 Deployment with Estimated Distribution Parameters

  • Deployment with estimated distribution parameters adds an estimation-error residual; a vanishing uniform estimation error recovers the exact-parameter certificate.The stability certificate holds on the event where the relevant bounds hold.Source: §4.2 · p. 11

4.3 Certification from an Estimated Lyapunov Decrease

  • A candidate policy can be certified directly from estimated parameter-level Lyapunov decreases, without constructing an explicit approximate transition model.The certificate accounts for both decrease-estimation error and error in the policy input, and fixes the candidate during deployment.Source: §4.3 · pp. 11–12

4.4 Asymptotic Recovery and Limiting Certificates

  • As data increase, optimizer recovery requires the stated convergence and invariance assumptions; a limiting policy is stable when its certificate and parameterization compatibility conditions hold.Vanishing residuals and convergence alone are insufficient if the decrease margins also vanish.Source: §4.4 · p. 13

4.5 Practical Implementation

  • PPO and SAC use sliding-window distribution estimates and target-dependent rewards to regulate HalfCheetah velocity distributions, but certificates are not established for the learned policies.Policies also receive noisy benchmark observations; the reward combines benchmark return, distribution dissimilarity, and a threshold bonus.Source: §4.5 · p. 13

5 Empirical Validation

The HalfCheetah evaluation tests whether distribution feedback and target matching regulate velocity and pitch toward a fast, safe target. With SAC, DMDP-Lag leads on target distance, return, and threshold satisfaction, while stability and cost results further distinguish the methods.

5 Empirical Validation

  • The target encodes fast, safe running as a velocity band below the limit and a narrow pitch interval around zero.The evaluation also compares methods against baselines, including distributional and Lagrangian variants.Source: §5 · pp. 13–14

5.1 Results with SAC

  • At both noise levels, each -Lag method has clearly lower cost than methods without a cost constraint; during the second half of training, their costs stay near or below the limit.The unconstrained methods’ costs stay far above the limit.Source: §5.1 · p. 14
  • With the cost constraint, Implicit DMDP-Lag (SAC) ranks second to DMDP-Lag (SAC) in evaluation return and target distance, while DMDP (SAC) is closer to the target than Implicit DMDP (SAC).The former comparison is against Distributional SAC-Lag; knowing the target further improves DMDP’s target distance over its implicit counterpart.Source: §5.1 · p. 14
  • For k ≥ 500, DMDP-Lag (SAC) has the smallest stability distance at both noise levels, and Implicit DMDP-Lag (SAC) has the second-smallest.Both are clearly below the other six methods; stability distances for every method drop within about 100 steps before changing more slowly.Source: §5.1 · p. 14
  • At both noise levels, DMDP-Lag (SAC) has the closest target velocity distribution, highest evaluation return, and largest fraction of steps with D_t ≤ D_thr.Without noise, it also has the lowest cost of all methods.Source: §5.1 · p. 14, Table 2 · p. 15

6 Conclusion

The paper formulates distribution control as a DMDP and develops stability certificates for optimal and learned policies. On HalfCheetah, DMDP-Lag brings velocity distributions closest to a fast, safe target; however, its learned policies were not certified there.

6 Conclusion

  • DMDPs control the state distribution by mapping it to an action and penalizing its dissimilarity from a target distribution.Finite-dimensional parameterization enables certificates for optimal policies, offline learned policies, deployment with estimated parameters, and decrease estimated from transition data.Source: §6 · p. 15
  • DMDP-Lag (PPO/SAC) brings the HalfCheetah velocity distribution closest to a fast, safe target; DMDP-Lag (SAC) achieves the highest evaluation return without noise and both outcomes under moderate noise.Distribution feedback helps even without the target, and knowing the target takes the DMDP methods further.Source: §6 · p. 16
  • The Lyapunov certificates were not computed for the learned HalfCheetah policies; computing them in the linear example requires the true model, and certified residual sets exceed the sets where learned loops settle.The paper identifies learning a Lyapunov function with the policy and bounding error terms from data as a next step toward certifying high-dimensional tasks from data alone.Source: §6 · p. 16

Contents of the Appendices

The appendices prove the main stability results: an optimal-policy condition yields distributional asymptotic stability, parameter-space stability transfers under compatibility, and the result applies to the finite-dimensional optimal policy.

A.1 Proof of Theorem 1

  • Under Theorem 1’s assumptions, including the bounds on stage cost and value, the optimal policy makes the target distribution D-asymptotically stable.The proof establishes stability and attractivity, which together give asymptotic stability.Source: §A.1 · p. 24

A.2 Proof of Lemma 1

  • A parameterization that is D-compatible transfers asymptotic stability of the parameter dynamics to the induced distributions on the invariant set.The lemma’s decrease condition yields parameter convergence, which supports distributional stability and attractivity.Source: §A.2 · p. 25
  • The proof of Lemma 1 establishes parameter stability by bounding the Lyapunov function and using its monotonicity to keep the parameter error within a chosen tolerance.The parameter dynamics are also attractive under the lemma’s decrease condition.Source: §A.2 · p. 25

A.3 Proof of Theorem 2

  • For the finite-dimensional optimal control problem, D-compatibility and the stated invariance, fixed-point, and value-function conditions make the optimal policy’s induced distribution dynamics D-asymptotically stable.The proof applies Lemma 1 using the rotated value function as its Lyapunov function.Source: §A.3 · pp. 25–26

A.4 Proof of Theorem 3

Theorem 3 bounds the learned-policy decrease error on a high-probability event. It certifies strict decrease outside a residual set, with asymptotic stability when the decrease condition holds throughout the invariant set.

A.4 Proof of Theorem 3

  • On an event with probability at least 1 − α, the learned policy’s Lyapunov function strictly decreases outside the residual set Ω̄_Δ, which shrinks toward the target as the error bound shrinks.The event requires the stated learned decrease margin and error bound.Source: §A.4 · p. 26
  • If the error is zero, or state-dependent errors leave a decrease margin throughout the invariant set, the induced distributions are D-asymptotically stable on that event.The result also applies when state-dependent errors are dominated by the decrease margin.Source: §A.4 · p. 26
  • The proof bounds the true closed-loop Lyapunov change by the learned decrease margin plus an error bound, yielding strict decrease wherever the margin exceeds that bound.The true and learned closed-loop maps are compared through a telescoping decomposition.Source: §A.4 · p. 26

A.5 Proof of Theorem 4

The proofs establish how estimation and learning errors affect Lyapunov certificates, when those certificates imply stability of the induced distributions, and why convergent scalar costs alone do not guarantee distributional convergence.

A.5 Proof of Theorem 4

  • With probability at least 1 − α_on, estimated distribution parameters yield a certificate that decreases outside a residual set; zero residual, or a suitable state-dependent decrease, gives distributional asymptotic stability.The residual set shrinks to the target as the offline and online residual terms tend to zero.Source: §A.5 · p. 27

A.6 Proof of Theorem 5

  • With probability at least 1 − α_n, an estimated Lyapunov-decrease certificate for plug-in dynamics yields strict decrease outside a residual set and stability under a suitable decrease condition.When the total residual is zero, the certificate decrease uses σ_n directly.Source: §A.6 · pp. 27–28

A.7 Proof of Proposition 1

  • Every limit policy of the learned sequence is distributionally stable when it satisfies the Lyapunov certificate and the parameterization is D-compatible.The proposition’s proof establishes convergence of the learned objectives before applying the stability certificate to limit policies.Source: §A.7 · pp. 28–29

B Technical Comparison with Related Approaches

  • The comparison distinguishes distribution feedback and distribution-dependent costs from state feedback and expected per-state costs, and notes that optimality or learning convergence alone does not certify learned-policy stability.Distributional stability is not established merely by fixed-chain ergodicity or convergence of learning iterates.Source: B Technical Comparison with Related… · p. 29

B.1 Norm-Based MDPs

  • A norm-based MDP’s nonincreasing scalar cost can converge without the state distribution converging, because the scalar records only an integral of the cost.Dissipativity can instead establish distributional convergence under a strict condition with respect to a dissimilarity measure.Source: §B.1 · p. 30

B.1.1 A Counterexample: Convergent Cost without Distributional Convergence

  • Under a norm-preserving finite-order transition, the expected quadratic cost stays constant while a nonsymmetric initial distribution cycles among distinct measures and fails to converge.The point-reflection example has period two; the nonconvergence holds for any dissimilarity that vanishes only when its arguments are equal.Source: §B.1.1 · p. 30

B.1.2 Recovering Successful Norm-Based Stabilization

A DMDP can reproduce successful distribution-stabilizing dynamics from a norm-based MDP when those dynamics are representable. The related-work comparisons distinguish its controlled state distribution from belief states, return distributions, and other distributional objectives.

B.1.2 Recovering Successful Norm-Based Stabilization

  • A DMDP policy reproduces a norm-based policy’s stable distribution trajectory when the original dynamics are stable on a forward-invariant set and are representable by an admissible DMDP policy.The DMDP formulates distribution stabilization directly through the evolution of the state distribution and its dissimilarity to the target.Source: §B.1.2 · pp. 30–31

B.2 Relation to POMDPs and Belief-State MDPs

  • Unlike a POMDP belief, which tracks hidden-state uncertainty from one realization’s observations, the DMDP state is the across-realization distribution controlled toward a target.The DMDP distribution evolves deterministically under a fixed distribution and action; its role comes from the cost, not missing information.Source: §B.2 · p. 31

B.3 Relation to Measure-Functional MDPs

  • DMDP distribution feedback follows from costs such as target dissimilarity, which can make the optimal action depend on distributional features such as variance, tails, or multimodality.An admissible DMDP policy can also reproduce the measure trajectory induced by a stabilizing state-feedback policy.Source: §B.3 · p. 32

B.4 Relation to Distributional Reinforcement Learning

  • Distributional reinforcement learning models return uncertainty, whereas a DMDP controls the state distribution toward a target in dissimilarity.The distinction matters when reliability, safety, or personalization is specified at the state-distribution level.Source: §B.4 · p. 32

B.5 Further Related Approaches

  • The cited approaches address population distributions, finite-time distribution steering, or time-aggregated visitation objectives, but the passage distinguishes them from establishing distributional stability.The comparison also notes that state-based Lyapunov certificates concern realized states rather than their distribution.Source: §B.5 · p. 32

C.1 Estimation of the Distribution Parameter ˆωk

  • One proposed estimation procedure groups samples by context, estimates a conditional state density, generates samples from it, and estimates the distribution parameter by maximum likelihood.Contexts can be time indices or other groupings of samples from the same state distribution.Source: §C.1 · p. 33

C.2 An Example Pipeline for Model Consistency

The example pipeline reduces model consistency to estimating distribution parameters and learning a transition map from those estimates. Parameter estimation is consistent under stated regularity conditions, while uniform consistency of regression with estimated inputs is assumed.

C.2 An Example Pipeline for Model Consistency

  • Conditional density estimation approximates each context-specific state density with a nonnegative basis expansion and normalizes the fitted estimate.Under LS-CDE regularity conditions, the estimate converges in L2 in probability, and almost surely under strengthened conditions.Source: §C.2 · p. 33
  • Maximum-likelihood parameter estimation is consistent when the parameter family is identifiable, the true parameter uniquely maximizes the population objective, and the stated continuity and integrable-envelope conditions hold.The density estimate must converge in the envelope-weighted L1 sense; L2 convergence alone may miss tail mass where log density is large.Source: §C.2 · p. 34
  • With enough generated samples, the empirical log-likelihood converges uniformly to its estimated-density objective, supporting convergence of the resulting parameter estimate as dataset size grows.The sample count S must grow fast enough for Monte Carlo error to vanish almost surely.Source: §C.2 · p. 34
  • The transition map is learned by Gaussian-process regression from estimated before-and-after distribution parameters, under RKHS membership, dense design points, vanishing input perturbations, and a suitable regularization and noise schedule.The framework assumes uniform consistency of this regression with estimated inputs.Source: §C.2 · p. 34
  • The model-consistency pipeline requires consistent distribution-parameter estimates and uniform consistency of the transition regression trained on estimated inputs.The latter is assumed rather than derived from available regression bounds for exactly observed inputs.Source: §C.2 · p. 34

D A Computable Example

The computable example uses a scalar linear stochastic system with a Gaussian distribution family closed under its transition, allowing the paper’s certificates to be checked. The example illustrates error-dependent residual sets, while its parameterization does not test approximation outside that family.

D A Computable Example

  • The example’s hypotheses for Theorems 2–5 and Proposition 1 are checkable; Theorem 2 applies with ΩP = Ωcom, κ1(r) = r², κ2(r) = 2.56r², and ϕ⋆ reaches ρ⋆ exactly in finitely many steps.The policy ϕ⋆ renders the example D-asymptotically stable; its dissimilarity is the squared 2-Wasserstein distance.Source: D A Computable Example · pp. 34–35
  • The Gaussian family N(µ, ς²) is closed under the scalar linear transition, so the distribution-parameter dynamics are exact in this example.The same action is applied across realizations, and the target is parameterized by ω⋆ = (1, 0.8).Source: D A Computable Example · p. 34
  • The example does not test distribution approximation outside the parameterized family, and learned policies that miss the target’s fixed point settle at positive distance from it.In that case, the asymptotic clauses of Theorems 3–5 do not apply.Source: D A Computable Example · pp. 34, 37
  • For Theorem 3, the model error, certificate residual, and residual-set radius shrink as data grow, while some least-data runs fail and have no certificate.The true decrease stays within the certified bound at every test point, and the deployed run’s limit distance is far below its residual-set radius.Source: D A Computable Example · p. 37
  • For Theorem 4, online estimation adds little to the offline residual set, and plug-in trajectories remain inside it, though estimation noise keeps them farther from the target.The estimation event holds in all but a few runs.Source: D A Computable Example · p. 38
  • For Theorem 5, the certified residual-set radius shrinks with more transitions and is smaller than Theorem 3’s with little data, but about equal with the most data.This comparison is qualified because the methods use different Lyapunov functions and margins, and Theorem 5 uses more samples per step.Source: D A Computable Example · p. 38

E Implementation Details

The HalfCheetah implementation regulates a joint velocity–pitch distribution using target-based dissimilarity rewards and sliding-window estimates. Additional SAC results show that DMDP-Lag is among the least affected by sensor noise in stable-distribution shifts.

E Implementation Details

  • The implementation appendix presents the closed-loop setup, compared methods, HalfCheetah task, and PPO hyperparameters.The cited figures and tables identify these implementation materials.Source: E Implementation Details · p. 38, Figure 7 · p. 39, Figure 8 · p. 39
  • The regulated state is the joint forward-velocity and torso-pitch distribution, parameterized by two uniform location-scale families, with a velocity target band of [0.70 vlim, 0.95 vlim] and an upright-pitch target.The target addresses the benchmark’s free posture, where a flipped gait can have a wide velocity distribution.Source: E Implementation Details · p. 38
  • Training combines summed squared marginal 2-Wasserstein distances with a threshold bonus and βD_t penalty, while a sliding-window method-of-moments estimate uses recent velocity and pitch readings.The penalty supplies a learning signal when the threshold reward is zero; the estimate uses the instantaneous readings at the start of each step.Source: E Implementation Details · pp. 38–39

F Additional Results

  • The appendix reports additional results as five-seed means with standard errors, treating a difference as clear only when it exceeds the larger standard error.Evaluation uses deterministic actions, while end-of-training values use stochastic actions.Source: F Additional Results · p. 40

F.1 Additional Results with SAC

  • With SAC, DMDP-Lag’s stable-distribution shift is among the smallest at η = 0.1 and remains within seed variation of the smallest through η = 0.4; Implicit DMDP shifts most.Distributional SAC-Lag also shifts least, but four of its runs are flipped.Source: §F.1 · p. 40, Table 9 · p. 41

F.2 Results with PPO

PPO results favor DMDP methods on target regulation and reliability, although the leading method varies with noise and metric. DMDP and constrained baselines also differ from unconstrained baselines in velocity-band occupancy and stability behavior.

F.2 Results with PPO

  • Without noise, 10/40 PPO runs are synchronized; at η = 0.2, each unconstrained baseline has one flipped run, whereas DMDP methods and constrained baselines have none.Source: §F.2 · pp. 41–42
  • Feeding back estimated distribution parameters without the target lowers cost at η = 0.2 but does not improve target distance; including the target yields closer distributions in the stated comparisons.DMDP (PPO) is closer than Implicit DMDP (PPO) at both noise levels, and DMDP-Lag is closer than Implicit DMDP-Lag without noise.Source: §F.2 · p. 41
  • Without noise, DMDP-Lag (PPO) is closest to the target and leads evaluation return and threshold-step fraction; at η = 0.2, Distributional PPO-Lag leads those three columns.At η = 0.2, PPO-Lag and DMDP (PPO) are within DMDP-Lag’s seed variation in target distance, while its return and fraction are within seed variation of the DMDP methods and other constrained baselines.Source: §F.2 · p. 41, Table 11 · p. 42
  • The DMDP velocity distributions lie mostly within and spread across the target band, unlike unconstrained baselines mostly above vlim and constrained baselines peaking just below it.DMDP (PPO) has a small tail above vlim.Source: §F.2 · p. 42, Figure 10 · p. 43
  • DMDP methods’ returns rise above RThr and remain there, while baseline returns stay far below; DMDP window dissimilarities are lowest, below Dthr.Constrained baselines and DMDP-Lag costs stay near the cost limit, whereas unconstrained baseline costs remain far above it.Source: §F.2 · pp. 42–43, Figure 11 · p. 43
  • Every method’s stability distance drops by about an order of magnitude within 100 noise-free steps; only Distributional PPO continues decreasing over several hundred steps.For k ≥ 500, Distributional PPO-Lag and Implicit DMDP-Lag have the smallest stability distances at both noise levels, within seed variation of each other.Source: §F.2 · p. 43, Figure 12 · p. 44
  • DMDP-Lag (PPO) reaches RThr most often at both noise levels in training and evaluation, within seed variation of DMDP (PPO), and both outperform the baselines on this measure.A DMDP method has the widest 10–90% range of return in each case.Source: §F.2 · p. 43, Table 12 · p. 44
  • Sensor noise shifts every method’s stable distribution more as η increases; PPO-Lag shifts least, DMDP (PPO) is within its seed variation, and the two DMDP methods shift similarly.Source: Table 13 · p. 44, §F.2 · p. 44

Related papers

Loading 2610.29024v1…