PaperlayerView exact revision on arXiv

Source-linked AI summary

Continuous-Time Critic as a Lyapunov Function: Projected Adaptation and Cart-Pole Stabilization

Pavel Osinenko

arXiv:2610.09988v1eess.SYcs.SYmath.DSmath.OC

TL;DR

Finding a suitable CLF can be difficult when a locally stabilizing quadratic does not describe a desired large excursion. The paper adapts the current critic for a known plant through a projection that enforces parameter invariance and decay when feasible. The adaptive positive critic supports force-limited swing-up and stabilization, while the guarantees remain conditional.

  • Problem

    Finding a suitable CLF can be difficult when a locally stabilizing quadratic does not describe a desired large excursion.

    Source: §1 · p. 1

  • Method

    The method adapts critic coefficients for a known plant by projecting parameter motion onto bounded destination constraints and a critic-derivative decay condition.

    Source: §1 · p. 1, §2.1 · p. 3

  • Results

    The adaptive positive critic supports force-limited swing-up and stabilization without mechanical-energy features, replay learning, waypoints, or controller switching; all 16 selected candidate paths passed reintegration audits.

    Source: §6 · p. 8, §5.3 · p. 8

  • Takeaways & Limitations

    The method supplies an online mechanism for enforcing specified critic decrease, not value-function optimality.

    Source: §6 · p. 8

    Feasibility is conditional: greater instantaneous compensation from faster adaptation need not prevent later infeasibility.

    Source: §2.2 · p. 4

Abstract

from arXiv · show

We study adaptive stabilization of known control-affine systems by treating the current parameterized critic itself as a Lyapunov candidate. Its total derivative contains a parameter-velocity term that can compensate unfavorable state evolution. A projection onto the intersection of a bounded admissible velocity set and one decay halfspace enforces parameter invariance and Lyapunov decrease simultaneously. We give an exact pointwise feasibility test, a conditional state-stability theorem, and a counterexample showing why a positive critic family containing a stabilizing member does not imply persistent feasibility. A convex endpoint constraint provides a distinct sampled implementation. A generic periodic polynomial-factor critic with a local Riccati anchor is evaluated on a force-limited cart-pole without mechanical-energy features, replay training, waypoints, or controller switching. The frozen controller passes stabilization and numerical decay checks on 600 sampled initial states across two draw sets; 400 matched continuous trajectories also pass. Independent reintegration audits and a conservative local feasibility certificate are distinguished from a global or machine-arithmetic guarantee. Compared with nonlinear model-predictive control, the measured implementation uses less CPU but has higher cost, settling time, and cart travel. The results support a conditional adaptive stabilization mechanism.

1 Introduction

Finding a suitable CLF can be difficult when a locally stabilizing quadratic does not describe a desired large excursion [1]. This section presents a known-plant critic adaptation whose decrease is conditional on state-dependent feasibility, and positions it among existing critic-learning, adaptive-control, and Lyapunov-constrained methods.

1 Introduction

  • The critic’s parameter-velocity term can compensate unfavorable state evolution, but the resulting formula has a state-dependent feasibility condition rather than guaranteeing stabilization for every stabilizable system.The construction treats critic coefficients as adaptable parameters for a known plant.Source: §1 · p. 1
  • The contributions are a bounded destination projection with an explicit feasibility budget and conditional stability proof, a convex sampled endpoint formulation, and a reproducible cart-pole study.The paper emphasizes that persistent feasibility remains necessary for theoretical guarantees.Source: §1 · pp. 1–2

1.1 Related work and positioning

  • Earlier work applies continuous critic updates to approximate optimal value functions or establish boundedness with approximation errors, whereas this paper directly constrains the current critic’s total derivative.It does not require convergence to an optimal value representation.Source: §1.1 · p. 2
  • Adaptive-control and real-time-QP precedents already account for parameter derivatives or Lyapunov constraints; here the parameters are critic coefficients for a known plant.The frozen critic is required to be uniformly positive, while bounded parameter motion enforces decrease when the joint constraints are feasible.Source: §1.1 · p. 2
  • Neural Lyapunov Control and related synthesis methods provide precedents; here, the critic evolves along the physical trajectory, with numerical large-excursion cart-pole evidence and a separate local certificate.The contribution is a bounded destination constraint on critic motion, its exact pointwise compensation budget and persistent-feasibility boundary, a convex sampled endpoint formulation, and a reproducible case study.Source: §1.1 · p. 2

2 Projected Critic Adaptation

For a known plant, the critic’s total derivative includes a directly adjustable parameter-velocity term. The method projects a preferred update onto simultaneous parameter-invariance and decay constraints, whose feasibility is conditional rather than automatic.

2.1 Plant and critic

  • The chain rule makes critic-parameter velocity directly adjustable in the decay condition, without estimating unknown plant parameters.The state contribution and the term p⊤v together determine whether the chosen critic decreases.Source: §2.1 · p. 3

2.2 One projection for two requirements

  • The projection selects the closest admissible velocity while requiring both a feasible destination θ + τv and the decay halfspace p⊤v + a ≤ 0.The bounded destination constraint keeps parameters in Θ; when feasible, the strictly convex objective gives a unique optimizer.Source: §2.2 · p. 3
  • The paper gives an exact pointwise test for nonemptiness of the joint constraint set, with the proof minimizing the decay expression over compact Θ.For box parameters, the multiplier procedure checks the test before solving the active decay constraint.Source: §2.2 · pp. 3–4
  • At a fixed state and parameter, the admissible velocities’ maximum decay contribution can be zero, and faster adaptation need not prevent later infeasibility.The parameter boundary or p = 0 can eliminate instantaneous compensation.Source: §2.2 · p. 4

2.3 Why feasibility is not automatic

  • A positive critic family containing a stabilizing member need not be feasible from every admissible initial parameter: at θ = 0.1, invariant-box velocities cannot enforce strict decay in the example.The stabilizing member is θ = 0.75, but at the lower boundary θ = 0.1 every invariant velocity satisfies v ≥ 0.Source: §2.3 · p. 4
  • Sequentially projecting onto parameter invariance and then decay can fail, because the invariance projection may leave the decay halfspace.The paper’s corner example gives a halfspace-feasible point whose positive-orthant projection violates that halfspace.Source: §2.3 · p. 4

3 Conditional Stability and Sampled Implementation

The continuous-time theorem gives state convergence and stability only on a compact region where the adaptation constraints remain feasible and the solution can continue. A separate exact-flow sampled constraint yields decay at sampling instants, with additional conditions needed for intersample convergence.

3 Conditional Stability and Sampled Implementation

  • The theorem requires feasible adaptation constraints throughout the compact region; it does not establish that parameters converge to a particular value.Its conclusions are conditional on the stated assumptions and covered sublevels.Source: §3 · p. 4
  • Feasibility on a compact sublevel, together with continuation, makes solutions forward complete, keeps them in that sublevel, and ensures x(t) → 0.The theorem also gives Lyapunov stability uniformly over initial parameters for sufficiently small initial states covered by the assumptions.Source: §3 · p. 4
  • A unique convex optimizer need not vary continuously with the state: the supplied example has a discontinuous update at x = 0 despite smooth data.The example diagnoses a regularity issue, not instability; projected dynamics require explicit solution and regularity conditions.Source: §3 · p. 4

3.1 A distinct endpoint-constrained realization

  • For the sampled implementation, the held-flow endpoint constraint is additional to the instantaneous constraints, and is convex in the update when V(y, ·) is convex.The predicted physical endpoint is independent of the parameter destination because the input is held during the interval.Source: §3.1 · p. 5
  • Under exact held flows, admissible updates, uniform critic bounds, and a compact sublevel, sampled states remain in that sublevel and converge to zero.The result also gives Lyapunov stability at sampling instants.Source: §3.1 · p. 5
  • Intersample convergence additionally requires the held-flow excursion to vanish uniformly near the target; numerically predicted endpoints alone do not provide a machine-level guarantee.Rigorous prediction-error enclosures or exact flow are needed for that claim.Source: §3.1 · p. 5

4 A Generic Cart-Pole Critic

The cart-pole construction uses a periodic critic family with a local Riccati anchor and a fixed initialization choice, then adapts its parameters under feedback. Its local persistent-feasibility analysis gives a conditional certificate, not a guarantee of entry from a hanging initial condition.

4.1 Physical model and periodic lift

  • The nominal cart-pole study assumes no friction, rail-position constraint, actuator lag, slew-rate limit, measurement noise, or parameter mismatch.The upright equilibrium is uncontrolled and unstable.Source: §4.1 · p. 5
  • The critic family is uniformly positive definite and proper on the cylindrical state space, with a uniform local expansion around the upright equilibrium.The proof uses a lift that vanishes only at the target, a positive factor, and bounded parameters.Source: §4.1 · p. 6
  • The critic family uses the plant model, periodic coordinates, scales, and a local Riccati matrix, but its features omit mechanical-energy error and a predesigned swing-up trajectory.Positive factorization does not certify a globally negative Lie derivative; the stated derivative can vanish away from the target.Source: §4.1 · pp. 5–6

4.2 Initialization and feedback

  • Initialization evaluates two fixed κ candidates with separate factors and retains the one with the smaller initial critic value; only L and η subsequently adapt.The preferred parameter recovery is toward the initialized parameters, and a force-variation penalty is added to the feedback design.Source: §4.2 · p. 6

4.3 Local persistent feasibility

  • A common local invariant state ellipsoid has a feasible zero update for the ideal nominal law, but the proof does not establish entry from a hanging initial condition.For the specified critic, conservative exact-rational checks accept the certificate; its radius is extremely conservative, and the calculation does not certify floating-point simulations or solvers.Source: §4.3 · p. 7

5 Numerical Study

The study evaluates a frozen 2 ms CALF on two cart-pole draw sets, with matched NMPC comparisons, continuous-law checks, and independent numerical audits. The results show strong nominal stabilization evidence and lower CPU use than NMPC, alongside higher cost, settling time, and cart travel in the matched comparison.

5.1 Protocol and implementation

  • The protocol freezes the selected controller before the primary 300-state draw and uses specified independent-coordinate draws, fixed model and force limit, and 45 s simulations.Numerical success requires completion, settling, and instantaneous and normalized endpoint residuals at most 10−6; these finite tolerances are audit criteria, not theorem inequalities.Source: §5.1 · p. 7
  • The held implementation retains the instantaneous halfspace and adds an endpoint constraint, solved with a conic formulation and optional refinement or verified-witness repair.Neither refinement nor repair can make a genuinely empty feasible set nonempty.Source: §5.1 · p. 7

5.2 Validation and comparison

  • The unchanged 2 ms controller passes all 300 starts in the second draw and its unchanged continuous law passes the corresponding 200 box starts; the draw sets contain 600 distinct states.These are distinct evaluations on reused states, not an additional 200 independent draws.Source: §5.2 · p. 7, Table 1 · p. 8
  • The primary candidate/reference median paired command-variation ratios are 0.084829 for the box and 0.022127 near hanging; median paired common-cost ratios are 0.99624 and 0.98584.Box cost is not uniformly improved: the worst ratio is 2.537 and median cart travel increases by approximately 1.1%.Source: §5.2 · p. 7
  • Both methods settle all 100 matched near-hanging starts, while median common cost, settling time, and peak cart displacement are higher for CALF than NMPC.CALF versus NMPC medians are 1208.8 versus 419.83 cost, 8.406 versus 3.8375 s settling time, and 5.9557 versus 1.794 m peak cart displacement.Source: §5.2 · p. 8

5.3 Audits, computation, and a negative result

  • Independent reintegration checks 720000 held intervals: all 16 selected candidate paths pass and 15 of 16 reference paths pass, reproducing the known reference failure.Thirty-two primary full replays reproduce archived arrays exactly, but the selected audits do not validate the full sampled batch; numerical entry into the local ellipsoid is not an exact-flow proof of swing-up entry.Source: §5.3 · p. 8
  • The quiet single-core benchmark gives a median paired total CPU ratio NMPC/CALF of 1.6703, but the compared controllers use 100 ms and 2 ms update periods, respectively.The simulated plant omits computation delays; neither hard-real-time operation nor a worst-case execution-time bound is established.Source: §5.3 · p. 8
  • A 4 ms hold with a 4 ms curvature penalty passed 20 reused development starts but failed the next draw, with a reproduced endpoint violation and a continuous path losing feasibility.The maximum sampled residual was 140.58064; changing the penalty changes feedback, so this is not a discretization-convergence test of one fixed continuous law.Source: §5.3 · p. 8

6 Discussion and Conclusion

The cart-pole results show that an adaptive positive critic can support force-limited swing-up without mechanical-energy error features, replay training, or a separate local controller. The construction remains conditional: persistent feasibility is substantive, and the reported evidence does not establish a useful large-excursion certificate or robustness to several real-world effects.

6 Discussion and Conclusion

  • An adaptive positive critic supports force-limited swing-up and stabilization without mechanical-energy error features, replay training, waypoints, or switching to a separate local controller.The local Riccati anchor and deterministic initialization remain prior structure; the method is neither model-free nor independent of design choices.Source: §6 · p. 8
  • The exact pointwise test identifies compensation available from bounded parameter motion but does not prove that compensation persists along future paths.Uniform positivity, closed-loop solution existence, and persistent feasibility cannot be replaced by critic richness or successful trajectories.Source: §6 · pp. 8–9
  • For the reported plant, the frozen 2 ms recipe has consistent nominal numerical evidence and a conservative ideal local certificate, but no useful large-excursion feasibility or entry certificate.Robustness to disturbances, mismatch, actuator dynamics, and finite computation delay is outside the result; the NMPC comparison is a computational/performance tradeoff, not general superiority.Source: §6 · p. 9

A Endpoint Optimization and Failure Diagnosis

The endpoint formulation constrains parameter updates using a fixed predicted endpoint and a decay budget. A relaxed minimum over the reachable parameter box gives a necessary infeasibility test, while a verified feasible witness can support limited numerical repair.

A Endpoint Optimization and Failure Diagnosis

  • For a fixed predicted endpoint and increment, the endpoint constraint is represented using epigraph variables and exponential- and second-order-cone constraints.The conic form also imposes box bounds and a derivative halfspace, with a velocity objective having the same minimizer as the stated increment objective.Source: A Endpoint Optimization and Failure… · p. 9
  • The relaxed endpoint minimum explicitly minimizes the critic over the reachable parameter box while omitting the derivative halfspace; if it exceeds the decay budget, the fixed-input joint update is infeasible.Floating-point evaluation is diagnostic rather than an interval proof, and the test does not address every alternative input.Source: A Endpoint Optimization and Failure… · pp. 9–10
  • When a box-minimizing witness satisfies both constraints, convexity permits bisection toward a rejected numerical candidate, with acceptance requiring rechecks of the original constraints.No repair is accepted if the witness does not satisfy both constraints.Source: A Endpoint Optimization and Failure… · p. 10

B Sufficient Local Bounds

The appendix supplies sufficient local estimates for the nominal feedback by bounding state-dependent critic and force terms. These bounds are expressed through rational constants and remainder estimates over a local state-radius range.

B Sufficient Local Bounds

  • For r=||s||≤1, the appendix defines norm-bound constants and states that the bounds cover both signs of the fixed initial angle bias.The constants are described as potentially loose upper bounds for the nominal mathematical feedback.Source: B Sufficient Local Bounds · p. 10

B.1 Critic and force remainders

  • The generic critic-gradient bound is cubic in r with coefficient CG=32ε+48BL, while the anchor-coordinate deviations are bounded in terms of κ and r.The anchor-gradient remainder coefficients are given as A2 and A3.Source: §B.1 · p. 10
  • Before the added penalty, the unsaturated force is bounded by quadratic and cubic state terms, and direct differentiation bounds the added penalty gradient.A relative added-penalty bound is also stated in terms of the local radius.Source: §B.1 · pp. 10–11

B.2 Common state bounds

The common bounds give conditions for continuous and held-flow state invariance and decay. The held-flow guarantee is conditional on a bootstrap bound during each hold.

B.2 Common state bounds

  • The sufficient continuous conditions yield invariant s⊤Ps ≤ Pr² and strict quadratic state decrease under arbitrary boxed parameter variation.The force bound ensures saturation is inactive.Source: §B.2 · p. 11
  • During a held flow, the stated bootstrap conditions bound the state norm by β times its initial norm over the hold.The conditions include βr ≤ 1, U₂r + U₃r² ≤ 1, and GₕhC_dβ³r² ≤ β/2.Source: §B.2 · p. 11
  • An additional condition and the same force bound yield common sampled state invariance and decay.The condition uses a positive lower bound qₕ for λ_min(P − Fₕ⊤PFₕ).Source: §B.2 · p. 11

B.3 Feasibility inequalities

The feasibility inequalities bound normalized critic errors and provide sufficient tests for a local joint state–parameter region. A separate exact numerical check verifies stored constants, conditional on the analytic estimates.

B.3 Feasibility inequalities

  • The sufficient tests combine a bound on normalized total derivative with endpoint critic-error bounds to preserve a local joint state–parameter region.With the anchor fixed, zero parameter velocity is a feasible witness, even for arbitrary allowed factor velocities.Source: §B.3 · p. 11

B.4 Exact numerical check

  • Degree-12 Taylor polynomials enclose Fₕ, while rational arithmetic supports the numerical checks of constants and positive scalar exponentials.For positive-definite rational matrices, exact symmetric elimination checks positive pivots, and 1 / tr(Z⁻¹) is a valid lower eigenvalue bound.Source: §B.4 · p. 12
  • The checker recomputes stored constants and rejects changed matrices or arithmetic, but its verification remains conditional on the analytic estimates.It does not formally prove the software or validate the preceding swing-up enclosure.Source: §B.4 · p. 12

C Reproducibility and Claim Boundaries

The reproducibility materials archive the evaluation data, scripts, and local-certificate constants. The authors distinguish these records and checks from guarantees beyond their stated scope, and disclose AI assistance and prior work.

C Reproducibility and Claim Boundaries

  • The ancillary materials include evaluation tables, audit and timing summaries, exact local-certificate constants, and data sufficient to regenerate the figures.A separate checker recomputes the local constants from the archived binary matrix.Source: C Reproducibility and Claim Boundaries · p. 12
  • The 600 sampled initial conditions are distinct, but zero observed failures do not prove a region or a probability-one guarantee.The exact-arithmetic local calculation concerns an ideal nominal controller, not a machine-arithmetic or validated full-swing-up certificate.Source: C Reproducibility and Claim Boundaries · p. 12

D Note on AI Use

  • The author reports AI assistance with the case study and language, while attributing the idea and approach to the author and noting earlier prototypes [22].The author also cites a published actor-critic framework.Source: D Note on AI Use · p. 12
Loading 2610.09988v1…