Source-linked AI summary
Actor-Critic Reinforcement Learning for Control with Stability Guarantee
Minghao Han, Lixian Zhang, Jun Wang, Wei Pan
TL;DR
Model-free RL lacks reliable closed-loop stability guarantees, despite stability’s importance for robotic safety and robustness. The paper combines a data-based Lyapunov theorem with an actor-critic algorithm and reports stable recovery and improved robustness across several control tasks under uncertainties and disturbances.
Problem
Model-free RL can perform well in robotic control but does not by itself guarantee learned-policy stability, which is closely related to safety, robustness, and reliability.
Method
The framework uses a data-based stability theorem and a Lyapunov critic within actor-critic RL to learn controllers for stabilization and tracking tasks.
Results
Across robotic and gene-network control tasks, learned policies stabilized systems and showed robustness to model-parameter variations and external disturbances.
Takeaways & Limitations
The reported evaluations indicate that incorporating a stability guarantee can improve recovery toward equilibria or references under tested uncertainties and disturbances.
Abstract
from arXiv · showhide
Reinforcement Learning (RL) and its integration with deep learning have achieved impressive performance in various robotic control tasks, ranging from motion planning and navigation to end-to-end visual manipulation. However, stability is not guaranteed in model-free RL by solely using data. From a control-theoretic perspective, stability is the most important property for any control system, since it is closely related to safety, robustness, and reliability of robotic systems. In this paper, we propose an actor-critic RL framework for control which can guarantee closed-loop stability by employing the classic Lyapunov's method in control theory. First of all, a data-based stability theorem is proposed for stochastic nonlinear systems modeled by Markov decision process. Then we show that the stability condition could be exploited as the critic in the actor-critic RL to learn a controller/policy. At last, the effectiveness of our approach is evaluated on several well-known 3-dimensional robot control tasks and a synthetic biology gene network tracking task in three different popular physics simulation platforms. As an empirical evaluation on the advantage of stability, we show that the learned policies can enable the systems to recover to the equilibrium or way-points when interfered by uncertainties such as system parametric variations and external disturbances to a certain extent.
I. INTRODUCTION
The paper addresses the lack of stability guarantees in model-free RL by combining Lyapunov analysis with actor-critic learning for stabilization and tracking control.
- Model-free RL achieves strong robotic-control performance but provides limited guarantees for learned-policy reliability.
- Stability matters because it supports convergence to equilibria or references despite parametric uncertainties and disturbances, linking it to safety and reliability.
- Lyapunov’s method uses an energy-like function whose decrease along trajectories establishes convergence, but unknown dynamics make this condition difficult to verify directly from data.
- The proposed method jointly learns a controller and Lyapunov critic with deep neural networks, using a sampled stability condition to obtain an asymptotic stability guarantee.
- The framework targets stabilization and tracking tasks modeled as MDPs, where policies should drive states toward the origin or a reference trajectory.
III. DATA-BASED STABILITY ANALYSIS
The paper develops a data-based Lyapunov stability analysis for stochastic systems, replacing infinitely many state-space checks with one sampling-based inequality under explicit assumptions.
- The analysis assumes the policy-induced Markov chain is ergodic with a unique stationary distribution and that relevant region-of-attraction states can be sampled.
- The proposed Lyapunov approach constructs a positive function whose expected change decreases along trajectories, establishing stability in mean cost.
- The proof connects long-run state-distribution convergence and boundedness assumptions to convergence of expected cost for every admissible initial state.
- The energy decreasing condition is the main criterion used to determine stability.
- The theorem enables data-based verification because stability requires validating one inequality over the sampling distribution rather than infinitely many inequalities over a continuous state space.
IV. ALGORITHM
The algorithm uses a Lyapunov critic within an actor-critic framework to learn policies with stability guarantees for stochastic control systems.
- The method first constructs a Lyapunov critic and then incorporates it into a maximum-entropy actor-critic policy-gradient formulation.
A. Lyapunov Critic Function
The Lyapunov critic is a nonnegative state-action function trained to approximate a Lyapunov candidate, allowing the actor to use stability information during policy learning.
- The critic depends on state and action while its policy expectation defines the Lyapunov function used in stability analysis.
- Its neural parameterization enforces nonnegativity, which is required because the resulting Lyapunov function must be positive definite.
- Lyapunov candidates such as norms or value functions provide supervision because their direct controller gradients are intractable in actor-critic learning.
- The critic is trained toward an approximation target over collected transition pairs, with the value function serving as one valid Lyapunov candidate.
- A finite-horizon cost sum is an alternative candidate that offers an explicit target and lower bias but introduces variance when future costs must be predicted without a model.
B. Lyapunov Actor-Critic Algorithm
Lyapunov Actor-Critic (LAC) learns a policy and Lyapunov critic through a constrained actor-critic objective whose stability condition is evaluated from sampled transitions. Lagrange multipliers balance stability and entropy constraints during training.
- Lyapunov Actor-Critic: LAC learns the controller and Lyapunov critic so the sampled stability inequality is satisfied.The framework uses a constrained optimization formulation and updates the policy, Lyapunov critic, and multipliers from replayed transitions.
- Constrained objective: The objective combines policy entropy with a Lyapunov-difference constraint weighted by Lagrange multipliers.β and λ control the relative importance of the stability and minimum-entropy constraints.
- Critic design: Only one Lyapunov critic is explicitly policy-dependent because direct dependence of the other critic would update the policy toward increasing L(s), contrary to stabilization.The alternative estimator remains unbiased but may have higher variance.
- Training procedure: The algorithm samples transitions into a dataset, repeatedly updates the critic, policy, and Lagrange multipliers, and stops when the prescribed condition is satisfied.The training procedure uses mini-batches of stored transitions and multiplier updates by gradient ascent while keeping multipliers positive.
- Performance comparison: Figure 1 compares total episode cost over training time, with shaded 1-SD intervals computed across 10 random seeds.The caption reports LAC convergence to a stabilizing solution with performance comparable to or better than SAC and SPPO.
V. EXPERIMENT
The experiments evaluate LAC across simulated robotic control and biological gene-network tasks, comparing convergence, performance, stability, robustness, and generalization against SAC and SPPO. The study uses multiple physics platforms and tests uncertainties, disturbances, and altered reference signals.
- Experimental scope: The evaluation covers CartPole, HalfCheetah, FetchReach, Swimmer, Minitaur, and synthetic gene-network control tasks across simulated platforms.The tasks include high-dimensional 3D robots and autonomous cellular systems.
- Evaluation criteria: The experiments assess convergence, task performance, closed-loop stability, robustness to unseen uncertainties and disturbances, and generalization to different reference signals.These criteria test both learning behavior and control behavior after training.
- Baselines: LAC is compared with soft actor-critic and a modified safe proximal policy optimization method using Lyapunov constraints.SAC is used as a strong actor-critic baseline, while SPPO is adapted from constrained MDPs for these tasks.
- Study organization: The study organizes results into performance and convergence, stability, robustness and generalization, and the influence of Lyapunov-candidate choices.Detailed hyperparameters and experiment setup are deferred to the appendix.
A. Performance
Across repeated training runs, LAC converges stably in all experiments, while SAC and SPPO can fail to converge or reach suboptimal solutions in several tasks. In tasks where both methods converge, LAC and SAC often have comparable cost and convergence speed.
- Convergence: LAC converges stably in all experiments despite random initialization.Each method is trained 10 times, with average total cost and variance reported during training.
- Convergence: SAC and LAC perform comparably in total cost and convergence speed across CartPole, HalfCheetah, FetchReach, and Swimmer examples.The comparison concerns tasks where both methods reach convergence.
- Task variation: In GRN and CompGRN, SAC is not always able to find a policy completing the control objective, producing poor average performance.The reported issue appears in the corresponding Figure 1 panels and supplementary material.
- Task variation: In Minitaur, SAC and SPPO converge only to suboptimal solutions, whereas LAC remains stable across random initializations.This result distinguishes stable convergence from merely reaching a converged policy.
B. Evaluation of Stability
LAC is evaluated for stability and robustness by comparing closed-loop trajectories with SAC under varied environments and disturbances. The reported trajectories show convergence to references under LAC, while SAC can diverge, oscillate, or fail under dynamic uncertainty.
- Closed-loop stability: LAC trajectories in GRN and CompGRN converge to the reference signal, whereas SAC trajectories diverge or oscillate around it.The comparison uses state trajectories from policies tested in the two gene-network systems.
- Robustness to dynamic uncertainties: Dynamic-uncertainty tests vary CartPole pole length, gene-expression parameters, and process noise during inference.Figure 3 reports average trajectories with 1-SD regions against reference signals and original-environment dynamics.
- Robustness to dynamic uncertainties: LAC remains robust and tracks precisely under the varied environments, while SAC performs well in the original environment but fails in all varied environments.The reported comparison covers CartPole and gene-network uncertainty settings.
- Robustness to disturbances: Under external disturbances, LAC outperforms SAC and LQR in CartPole and GRN using lower death rate and cumulative cost.In HalfCheetah, LAC is more reliable than SAC for larger disturbances, while both are reliable for smaller ones.
- Robustness to disturbances: SAC and LAC both remain reliable across external disturbances in FetchReach, whereas SPPO agents could hardly sustain any external disturbances.The SAC–LAC difference is reported as most apparent in GRN, Swimmer, and Minitaur.
3) Generalization over different tracking references:
LAC generalizes to unseen reference signals with trajectories close to targets and low variance, whereas SAC fails on some signals and varies more. The study also finds that Lyapunov-candidate horizon affects robustness despite comparable converged training costs.
- Robustness to persistent disturbances: Figure 4 compares LAC, SAC, SPPO, and LQR across disturbance magnitudes using death rate for CartPole and cumulative cost for other tasks.Each trained policy is evaluated for 100 trials per disturbance setting.
- Generalization over different tracking references: LAC follows previously unseen sinusoidal and constant reference signals with low variance, regardless of whether their mathematical form matches training.The tested references include sinusoids with periods 150, 200, and 400, plus constant references 8 and 16.
- Generalization over different tracking references: SAC tracks the original reference well but cannot follow some unseen signals and has larger variance than LAC.The paper specifically reports failure on the period-150 sinusoid.
- Influence of different Lyapunov candidates: Lyapunov candidates with different horizons achieve comparable converged cumulative costs, but robustness decreases as the horizon N increases.The comparison uses N ∈ {5, 10, 15, 20, ∞} and evaluates impulsive-force death rates.
Appendix
The experiments use simulated environments from OpenAI Gym, DeepMind Control Suite, and PyBullet. They cover balancing, locomotion, reaching, and disturbance-robustness tasks with task-specific dynamics and evaluation settings.
- Experiment setup: The experiments use OpenAI Gym, DeepMind Control Suite, and PyBullet physics simulation platforms.Figure S7 provides snapshots of environments using OpenAI Gym.
- CartPole: CartPole keeps a pole vertical at x = 0 using horizontal cart force, with episodes ending when position or angle thresholds are exceeded.The action range is a ∈ [−20, 20], with xthreshold = 10 and θthreshold = 20°.
- HalfCheetah: HalfCheetah controls a two-legged simulated robot to run at 1 m/s through joint torques ranging from -1 to 1.Robustness tests apply impulsive joint torques every 20 seconds with magnitudes from 0.2 to 2.0.
- FetchReach: FetchReach controls a manipulator’s end effector to reach a random goal position using joint torques ranging from -1 to 1.The cost equals the distance between the goal and end effector.
- Swimmer: Swimmer controls a multi-joint snake robot to run at 1 m/s using joint torques ranging from -1 to 1.Robustness tests apply impulsive joint torques every 20 seconds with magnitudes from 0.2 to 1.0.
E. Minitaur
The supplied passages describe the repressilator gene-regulatory-network model and its persistent oscillations, alongside experimental settings for control evaluation.
- Mathematical model of GRN: The repressilator is a synthetic three-gene regulatory network whose mRNA and protein dynamics exhibit oscillatory behavior.
- Mathematical model of GRN: The model includes transcription and translation states, with mRNA concentrations x1, x2, x3 and protein concentrations x4, x5, x6.
- Mathematical model of GRN: Only protein concentrations are observed, while control inputs are implemented through light signals that induce gene expression.
- Mathematical model of GRN: With initial states uniformly sampled from 0 to 5 and no process noise, persistent oscillatory behavior remains evident.
- Mathematical model of GRN: Approximately 150 arbitrary time units characterize the natural oscillation period of the three-gene repressilator.
S3. HYPERPARAMETERS
The supplied passages describe LAC’s Lyapunov-critic implementation and validation, including multiplier convergence to zero and SPPO’s poor performance under unseen conditions.
- LAC implementation: LAC uses a policy network and a Lyapunov critic network, with the critic output constrained to be non-negative through a square term.
- Stability validation: The Lagrange multiplier λ is monitored during training because convergence to zero indicates satisfaction of the stability condition.
- Stability validation: Across all training trials, λ eventually converges to zero, while clipping is activated only early in FetchReach and not in the other experiments.
- Robustness and generalization: SPPO policies could hardly handle previously unseen uncertainty or reference signals and failed in all Repressilator experiments.
- Robustness and generalization: SPPO uses Lyapunov’s method for safety constraints but is not designed to provide a stability guarantee.