Source-linked AI summary
Robust Adversarial Reinforcement Learning with Risk Sensitivity and Critic Consistency Regularization
Jiaxi Wu, Tiantian Zhang, Yuxing Wang, Yongzhe Chang, Xueqian Wang
TL;DR
RL policies can be brittle under uncertainty and distribution shifts, and adversarial training can bring uninformative failures and critic disagreement. RACER regulates adversarial strength by state risk and regularizes critic consistency; it is reported to improve robustness and training stability across environments.
Problem
RL faces dynamic uncertainty and distribution shifts, while RARL can drive agents into uninformative failure states and increase disagreement between double critics, biasing value targets.
Source: §1 · pp. 1–2Method
RACER combines state-dependent regulation of adversarial perturbations with critic consistency regularization to limit excessive disagreement between Q-value estimators.
Source: §1 · p. 2Results
RACER consistently achieves superior robustness scores across environments, with faster improvement early in training and higher robustness ceilings on multiple tasks.
Source: §5.2 · p. 7Takeaways & Limitations
Ablations indicate that risk-sensitive adversarial regulation mainly aids early learning, while critic consistency primarily enhances training stability.
Source: §5.3 · p. 8
Abstract
from arXiv · showhide
Reinforcement learning (RL) achieves strong performance in sequential decision-making but remains brittle under dynamic uncertainty and distributional shifts. Robust Adversarial Reinforcement Learning (RARL) improves robustness via worst-case perturbations, but existing approaches frequently suffer from unstable optimization and degraded value estimation. In particular, overly aggressive adversaries can drive the agent toward uninformative failure states, while adversarial perturbations amplify disagreement between double critics and introduce biased value targets. We propose a unified framework, RACER (Risk-sensitive robust Adversarial critic ConsistEncy-regularized Reinforcement learning), that revisits adversarial RL from a risk-sensitive perspective. First, we introduce a state-dependent adversarial objective that adaptively regulates perturbation strength, suppressing harmful disturbances while preserving informative exploration. Second, we propose critic consistency regularization to reduce disagreement between Q-value estimators and stabilize learning. Comprehensive experiments on challenging continuous control benchmarks demonstrate that RACER consistently improves performance, robustness, and training stability over strong robust RL baselines.
1 Introduction
RARL addresses RL’s vulnerability to uncertainty and distribution shifts through adversarial perturbations, but unconstrained attacks and critic disagreement can undermine learning. RACER combines state-dependent adversarial regulation with critic consistency regularization.
1 Introduction
- Unconstrained attacks can push agents into unrecoverable states and uninformative interactions, while double-critic disagreement can bias value targets downward and destabilize optimization.The introduction links these failure modes to reduced exposure to useful trajectories and to the minimum operator selecting the lower critic estimate.Source: §1 · pp. 1–2
- RACER combines a state-dependent risk-sensitive adversarial objective with critic consistency regularization to address adversarial design and value-estimation challenges.The framework is presented as a unified approach to robust adversarial reinforcement learning.Source: §1 · p. 2
- The risk-sensitive objective adjusts perturbation strength by state risk, suppressing harmful disturbances in unstable regions while retaining exploration pressure in safer regimes.The resulting Walker-Run state distribution is described as more concentrated in stable locomotion while retaining variability for exploration.Source: §1 · p. 2
- Critic consistency regularization constrains disagreement between Q-value estimators; the contribution summary describes it as scale-invariant and aimed at stabilizing double Q-learning under adversarial perturbations.RACER is reported to improve robustness across challenging continuous-control benchmarks.Source: §1 · p. 2
2 Related Work
Prior work frames robust adversarial RL as a response to uncertainty, while identifying instability and risk modeling as continuing challenges. Proposed remedies target optimization, objective design, or adversary scheduling.
2.1 Robust Adversarial Reinforcement Learning
- RARL casts robustness as a protagonist–adversary zero-sum game [23], but prior studies report instability, difficult nonconvex–non-concave optimization [11], and insufficient explicit risk modeling [20].These challenges motivate work on stability, convergence, and robustness in adversarial RL.Source: §2.1 · p. 2
2.2 Holistic Optimization in Robust Adversarial Reinforcement Learning
- Related methods target optimization dynamics, risk-aware objectives, or adversary curricula, including DI-CARL, RARARL [20], CAT, QARL [24], and A2P [15].Examples include dissipation constraints, ensemble Q-value variance, and gradual or adaptive adjustment of adversarial strength.Source: §2.2 · p. 2
3 Preliminaries
The preliminaries formulate robust adversarial RL as a zero-sum game and introduce SAC as the off-policy actor–critic algorithm used in the paper.
3.1 Robust Adversarial Reinforcement Learning
- RARL is a two-player zero-sum Markov game in which the protagonist maximizes expected discounted return and the adversary minimizes it through worst-case perturbations [22] [27].The game is specified by state and agent action spaces, transition dynamics, reward, and discount factor.Source: §3.1 · p. 3
3.2 Soft Actor-Critic
- The paper adopts SAC, an off-policy actor–critic method that combines return maximization with entropy regularization to encourage exploration [8].Its coupled updates are described as a stable and sample-efficient learning procedure.Source: §3.2 · p. 3
4 Methodology
RACER combines a state-dependent risk functional for adaptive, informative adversarial perturbations with critic-consistency regularization to suppress excessive critic disagreement.
4.1 Risk-Sensitive Adversarial Formulation
- RACER regularizes the adversary with state-dependent risk, suppressing destructive perturbations at high risk while allowing strong, informative disturbances at low risk.Unlike RARARL [20], which uses ensemble Q-value variance as a risk proxy, RACER models risk explicitly in the state representation.Source: §4.1 · pp. 3–4
- On Walker-Run, weakening attacks in high-risk states yields higher returns and lower variance than enhancing attacks or a risk-agnostic strategy.Enhancing attacks in high-risk states destabilize training and degrade asymptotic performance.Source: Figure 2 · p. 4, §4.1 · p. 4
4.2 Critic Consistency Regularization
- Its critic penalty measures pairwise Q disagreement, normalizes it by value scale, and regularizes both protagonist and adversary critics.The scale normalization makes the penalty approximately scale-invariant, reducing sensitivity to task-specific reward scales.Source: §4.2 · p. 4
- RACER maintains more controlled, stable critic disagreement than QARL while preserving critic diversity, alongside consistently higher robustness scores during training.Figure 3 tracks disagreement by episode and sampled batch position; darker colors indicate greater disagreement.Source: Figure 3 · p. 5, §4.2 · p. 5
5 Experiments
Experiments test RACER on four MuJoCo tasks against robust adversarial baselines, measuring robustness, component contributions, sensitivity to critic regularization, and transfer to other SAC-based methods.
5.1 Experimental Setup
- The study uses Walker-Run, Hopper-Stand, Cheetah-Run, and Cartpole-Balance, introducing environment-specific high-intensity adversarial disturbances.Comparisons use SAC-based methods, five random seeds, and 200 alternating training iterations per seed.Source: §5.1 · p. 5
5.2 Performance Comparison
- Across four tasks, RACER achieves superior robustness scores, faster early improvement, more stable training, and narrower variance bands than the baselines.The reported early improvement occurs during the first 50–80 iterations; RACER also performs better on Hopper-Stand despite high variance across methods.Source: Figure 4 · p. 6, §5.2 · p. 7
- At final evaluation, RACER maintains strong performance across environmental-property perturbations and shows superior adaptability to varied dynamics.The evaluation includes properties such as friction, body mass, and joint damping.Source: Figure 5 · p. 6, §5.2 · p. 7
- Against a newly optimized adversary facing a frozen protagonist, RACER outperforms all baselines across environments and shows low performance variance.The evaluation tests resilience to stronger, previously unseen adversaries.Source: Table 1 · p. 7, §5.2 · p. 7
5.3 Ablation Study
- Removing either RACER component degrades performance, with the largest drop when both are removed; risk sensitivity mainly aids early learning, while critic consistency mainly smooths training.The full model’s performance gap over ablated variants increases over time.Source: Figure 6 · p. 7, §5.3 · p. 8
5.4 Hyperparameter Analysis
- The effect of the consistency coefficient is non-monotonic; λ = 1 × 10−4 gives the best or near-best robustness across tasks, while values outside the moderate range may degrade performance.Even relatively large λ values do not produce catastrophic critic failure.Source: §5.4 · p. 8, Table 2 · p. 8
5.5 Scalability Analysis
- RACER builds on SAC optimization and transfers to RARL and SAC MixedNE-LD without changing their default hyperparameters, improving returns and robustness in both.The authors characterize it as a plug-and-play enhancement for SAC-based robust adversarial methods.Source: §5.5 · pp. 8–9, Figure 7 · p. 8
6 Conclusion
RACER combines state-dependent risk-sensitive adversarial strength with critic consistency regularization. On standard MuJoCo continuous-control benchmarks, it improves performance, training stability, and adversarial robustness under simulated disturbances.
6 Conclusion
- RACER adaptively adjusts perturbation strength using state-dependent risk and constrains divergence between Q-value estimators to improve value-target reliability.The paper attributes instability to uneven adversarial effects on critics.Source: §6 · p. 9
- Experiments on standard MuJoCo continuous-control benchmarks show improved performance, training stability, and adversarial robustness under simulated adversarial disturbances.Source: §6 · p. 9
A.1 Environments
The experiments use four MuJoCo control tasks with adversarial dynamics, and assess protagonist risk using environment-specific state thresholds.
A.1 Environments
- The four tasks are Walker-Run, Hopper-Stand, Cheetah-Run, and Cartpole-Balance, with protagonist and adversary agents in each environment.Adversarial actions and force magnitudes vary by environment; maximum force is limited in highly sensitive environments to prevent complete policy collapse.Source: §A.1 · p. 10
- A protagonist state value above its environment-specific risk threshold signals risk of losing control in that dimension; deviation size indicates severity.Source: §A.1 · p. 10
A.2 Hyperparameter Setting
The appendix reports the comparison settings and computing environment used for the experiments.
A.2 Hyperparameter Setting
- Baseline comparisons follow QARL settings to help attribute performance differences to the methods rather than hyperparameter tuning.Except for CAT’s adversary, agents use SAC or derivative variants, supporting integration without changing baseline core structures.Source: §A.2 · p. 10
A.3 Hardware and Software
- Experiments ran on a TITAN Xp system with 12GB GPU memory, 15GB system memory, and an Intel Xeon E5-2680 v4 CPU.The implementation uses QARL’s open-source code and the MushroomRL library.Source: §A.3 · p. 10
B Additional Experimental Results
Under adversarial training, RACER maintains steadily increasing protagonist returns while other baselines often fluctuate or improve more slowly. The authors report that experiments were confined to controlled simulations, without real-world safety-critical deployment.
B.1 Competing Process against Adversary
- RACER maintains stable, steadily increasing protagonist returns as adversarial strength grows, while baselines often fluctuate or improve more slowly.The supplemental curves cover Walker-Run, Hopper-Stand, Cheetah-Run, and Cartpole-Balance.Source: §B.1 · p. 10
C Ethical Considerations
- All experiments were conducted in controlled simulations, not real-world safety-critical deployments.The authors caution against misuse of adversarial robustness techniques to attack deployed AI systems.Source: C Ethical Considerations · p. 10
D Broader Impact
RACER may support more stable and reliable learning under adversarial and uncertain conditions, but the authors caution that real-world validation is needed to avoid false confidence in safety. They also note that adversarial training techniques could be misused to attack learning systems.
D Broader Impact
- Risk-aware, state-adaptive perturbations may encourage useful interactions and support safer policy learning in dynamic environments.The authors identify robotics, autonomous systems, and control as potentially relevant safety-critical applications.Source: D Broader Impact · p. 10
- Adversarial training techniques could be misused to design attacks, especially if risk modeling is inverted or exploited to identify vulnerabilities.The authors describe this as a potential misuse.Source: D Broader Impact · p. 10
- Realistic validation is needed to avoid a false sense of safety; future work should emphasize transparent risk modeling and rigorous real-world evaluation.The authors also call for alignment with safety standards, particularly in human-interacting domains.Source: D Broader Impact · p. 10