PaperlayerView exact revision on arXiv

Source-linked AI summary

Adversarially Trained Actor Critic for Offline Reinforcement Learning

Ching-An Cheng, Tengyang Xie, Nan Jiang, Alekh Agarwal

arXiv:2202.02446v2cs.LG

TL;DR

Offline RL must learn from limited-coverage data collected under safety or performance constraints, while existing guarantees and scalable implementations are incomplete. ATAC uses relative pessimism in a Stackelberg actor-critic game, achieving robust policy improvement and strong D4RL performance. Its main cost is a potentially harder adversarial optimization problem and stronger computational requirements.

  • Problem

    Offline RL needs reliable learning from historical data with limited state-action coverage, including safe improvement over behavior and effective use of covered policies.

  • Method

    ATAC formulates offline RL as a relative-pessimism Stackelberg game with a policy actor and adversarially trained critic, implemented through two-timescale stochastic optimization.

  • Results

    ATAC has theoretical robust policy-improvement guarantees for nonlinear function approximators and consistently outperforms state-of-the-art baselines across D4RL continuous-control problems.

  • Takeaways & Limitations

    Relative pessimism gives ATAC safe improvement over behavior across pessimism hyperparameters while retaining strong empirical performance on offline RL benchmarks.

  • Takeaways & Limitations

    ATAC’s adversarial optimization can be harder than fixed-pessimism dynamic programming, and its no-regret policy-optimization oracle is provably efficient only for some linear-function and policy settings.

Abstract

from arXiv · show

We propose Adversarially Trained Actor Critic (ATAC), a new model-free algorithm for offline reinforcement learning (RL) under insufficient data coverage, based on the concept of relative pessimism. ATAC is designed as a two-player Stackelberg game: A policy actor competes against an adversarially trained value critic, who finds data-consistent scenarios where the actor is inferior to the data-collection behavior policy. We prove that, when the actor attains no regret in the two-player game, running ATAC produces a policy that provably 1) outperforms the behavior policy over a wide range of hyperparameters that control the degree of pessimism, and 2) competes with the best policy covered by data with appropriately chosen hyperparameters. Compared with existing works, notably our framework offers both theoretical guarantees for general function approximation and a deep RL implementation scalable to complex environments and large datasets. In the D4RL benchmark, ATAC consistently outperforms state-of-the-art offline RL algorithms on a range of continuous control tasks.

1. Introduction

Offline RL must learn from limited-coverage historical data collected under safety or performance constraints. ATAC addresses this challenge with relative pessimism, theoretical guarantees for nonlinear function approximation, and a scalable deep-learning implementation.

  • Real-world data collection can be expensive or infeasible, especially where behavior policies must satisfy safety or baseline-performance requirements.
  • Offline RL ideally should improve behavior policies safely and learn policies competitive with the best data-covered alternatives.
  • Robust policy improvement supports reliable deployment by maintaining behavior-policy performance across broad pessimism hyperparameter choices and enabling online tuning.
  • Existing methods trade off robustness, conservatism, limited hyperparameter guarantees, simplified theory, or weak empirical evaluation.
  • ATAC provides robust policy-improvement and learning-consistency guarantees for nonlinear function approximators alongside scalable deep-neural-network training.
  • ATAC uses a Stackelberg game in which an actor faces an adversarially trained critic seeking data-consistent scenarios where the actor underperforms behavior.
  • ATAC’s DQRA loss improves optimization stability, while D4RL experiments report consistent outperformance of state-of-the-art baselines across continuous-control problems.

2. Preliminaries

The paper formulates offline RL in an MDP using discounted returns, occupancy measures, offline transition data, and actor-critic function classes. Its guarantees rely on approximate realizability and completeness assumptions measured on the data distribution.

  • The MDP is defined by state and action spaces, transitions, rewards, and discount factor, with policies evaluated by expected discounted return.
  • A policy’s normalized discounted state-action occupancy measure weights expectations used in the offline RL analysis.
  • Offline RL computes policies from pre-collected data consisting of independent state-action-reward-next-state tuples sampled from a behavior-policy occupancy distribution.
  • Function approximation uses a value-function class F for policy Q-functions and a policy class Π for candidate decision rules.
  • Approximate realizability requires each policy’s Bellman fixed point to be approximable in F under admissible data-distribution-weighted norms.
  • Approximate completeness requires Bellman-transformed functions to be approximable in F, with the required accuracy only on the data distribution.

3. A Game Theoretic Formulation of Offline RL with Robust Policy Improvement

The paper formulates offline RL as a Stackelberg game driven by relative pessimism: an actor seeks improvement over behavior while a critic enforces data-consistent pessimistic evaluation. This yields safe improvement for every β ≥ 0 and supports outperforming covered policies at suitable β.

  • A Game Theoretic Formulation of Offline RL with Robust Policy Improvement: Relative pessimism is introduced at the population level to represent uncertainty caused by missing state-action coverage in offline data.
  • A Stackelberg Game Formulation of Offline RL: A Stackelberg game is a sequential leader-follower bilevel optimization that generalizes earlier minimax formulations.
  • A Stackelberg Game Formulation of Offline RL: Offline RL assigns the learner policy as leader and the value critic as follower in the proposed bilevel game.
  • A Stackelberg Game Formulation of Offline RL: The critic remains approximately Bellman-consistent on data, while β controls the relative contribution of pessimism in evaluating candidate policies against behavior.
  • Relative Pessimism and Robust Policy Improvement: Relative pessimism is motivated by robust policy improvement, which supports behavior-level safety across hyperparameters and potentially enables direct online performance tuning.
  • Relative Pessimism and Robust Policy Improvement: Under Assumption 1, the game solution is no worse than behavior for every β ≥ 0 because its lower-bound objective is tight at the behavior policy.
  • Relative Pessimism and Robust Policy Improvement: The guarantee follows from comparing the learned policy’s lower bound with the behavior policy’s zero relative-performance bound.
  • Relative Pessimism and Robust Policy Improvement: Unlike absolute pessimism, relative pessimism compares J(π)−J(µ) and therefore preserves robust improvement for all β ≥ 0 rather than only tuned choices.

4. Adversarially Trained Actor Critic

ATAC combines a Stackelberg-game formulation with no-regret policy optimization and pessimistic policy evaluation, yielding guarantees for robust policy improvement and competition with covered policies. Its practical implementation uses a double-Q residual loss to stabilize deep offline RL optimization.

  • Algorithm: ATAC combines no-regret policy optimization with pessimistic policy evaluation in a Stackelberg-game framework.The theoretical algorithm alternates between a pessimistic critic update and an actor update using a no-regret policy optimization oracle.
  • Theoretical guarantees: The analysis measures coverage through C(ν; µ, F, π) and accounts for finite-sample deviation, off-support bias, and optimization regret.The guarantee can use ν=dπ to remove the off-support term or exploit better-covered distributions with small off-support mass.
  • Theoretical guarantees: With sufficiently large data, small optimization regret, and adequate coverage of dπ, ATAC can compete with any policy π.The guarantee is stated for policies whose state-action distributions are sufficiently covered by the data.
  • Theoretical guarantees: The reported statistical comparison gives ATAC an N^-1/3 finite-sample scaling, while a constrained version can achieve N^-1/2 but is less practical.The N^-1/3 scaling is attributed to regularization, whereas the constrained alternative is described as unfriendly to practical implementation.
  • Theoretical guarantees: β=o(N) guarantees robust improvement over the behavior policy across a wide range of pessimism settings, including β=0.At β=0, ATAC still guarantees a policy no worse than the behavior policy under the stated assumptions.
  • Practical implementation: The practical implementation uses a double-Q residual algorithm loss to improve off-policy optimization stability.The DQRA loss combines double-Q and residual-algorithm ideas; its residual-error component supports stable optimization with higher β values and less pessimism.

5. Experiments

Experiments evaluate ATAC and its variants on D4RL, finding strong baseline comparisons and robust policy improvement across pessimism settings, with identifiable limitations on some datasets and hyperparameters.

  • Setup and hyperparameter selection: ATAC variants are evaluated against CQL, COMBO, TD3+BC, IQL, and behavior cloning, with ATAC scores summarized by medians over 10 seeds.Table 1 marks scores within ϵ = 0.1|J(µ)| of the best domain result in bold.
  • Comparison with offline RL baselines: ATAC and ATAC∗ consistently outperform model-free offline RL baselines and mostly outperform COMBO across D4RL continuous-control domains.Performance is especially improved on walker2d-medium, walker2d-medium-replay, hopper-medium-replay, and penexpert, while halfcheetah-rand is an exception.
  • Robust policy improvement: ATAC robustly improves upon the behavior policy for almost all β values, except very large ones, matching Proposition 6’s prediction.The experiment varies β on hopper to test the practical algorithm’s robust policy improvement.
  • Robust policy improvement: ATAC does not improve over behavior policies on human and cloned datasets even with well-tuned β, plausibly because non-Markovian demonstrations violate µ ∈Π.The passage reports that no offline RL algorithm improves on these datasets and identifies realizability as a suspected cause.
  • Ablation of DQRA loss: For w ∈(0, 1), the DQRA loss usually provides better stability and performance than either conventional double-Q bootstrapping or residual gradients alone.With w = 1, the Bellman minimization becomes unstable; with w = 0, optimization is numerically stable but policy performance is poor, so experiments use w = 0.5.

6. Discussion and Conclusion

The paper frames ATAC as relative pessimism in a Stackelberg game, connects it to imitation learning, and reports both broad empirical performance and a computational limitation from adversarial optimization.

  • Discussion and conclusion: ATAC combines relative pessimism with a Stackelberg game to obtain robust policy improvement alongside theoretical guarantees for offline RL.The paper presents this as an advantage over prior theoretical approaches and reports state-of-the-art D4RL performance.
  • Connection to imitation learning: At β = 0, ATAC reduces to imitation learning because maximal value uncertainty makes exactly mimicking the behavior policy optimal under relative pessimism.With labeled rewards, reduced uncertainty can instead permit strict improvement over the behavior policy.
  • Limitations: ATAC’s adversarial optimization can be harder to solve than dynamic-programming-based offline RL, and efficient no-regret guarantees are known only for some linear and softmax settings.The same computational difficulty appears in the imitation-learning special case, where ATAC uses IPM minimization or Wasserstein-GAN rather than maximum-likelihood behavior cloning.
  • Motivation: Offline RL without broad coverage is needed because real-world data distributions are constrained by policy quality, safety, and system constraints.These constraints can make coverage substantially narrower than the state-action distributions of candidate policies.
  • Related approaches: Prior methods either regularize policies toward behavior data or use pessimism over data-consistent scenarios, with theoretical and empirical trade-offs across these approaches.The discussion positions ATAC among behavior-regularization and pessimism-based methods, including the maximin approach of Xie et al.

B.1. Concentration Analysis

The concentration analysis uses covering numbers to control empirical Bellman-error estimates uniformly over function and policy classes, then converts these controls into high-probability performance bounds.

  • Covering-number setup: Covering numbers discretize both the function class F and policy class Π under specified metrics for uniform concentration analysis.The analysis defines ε-covers and denotes their sizes by N∞(F, ε) and N∞,1(Π, ε).
  • Bellman-error control: Setting ED(f, π) = εr yields a bound on ||f − Tπf||2,µ, denoted √εb, which is used in later guarantees.Lemma 10 supplies the concentration step, and subsequent high-probability statements follow deterministically from it.
  • Proof strategy: The proof applies Bernstein’s inequality and union bounds over the function and policy covers, while controlling approximation and variance terms through the cover radii.The argument explicitly introduces covered functions and policies before deriving the uniform concentration statement.
  • From concentration to guarantees: The performance decomposition bounds policy-value differences using empirical and population losses, Bellman errors, approximation errors, and statistical errors.The resulting expressions combine terms such as O(√εF), √εstat, and β·O(εstat + εF).

B.3. Performance Guarantee of the Theoretical Algorithm

The theoretical guarantee extends ATAC’s performance analysis to approximate realizability and completeness, while the practical algorithm implements the actor–critic updates with target networks and regularization.

  • General performance guarantee: The general performance theorem incorporates approximation errors from realizability and completeness assumptions into the theoretical ATAC guarantee.The proof uses Lemma 12 and a generalized performance decomposition for an arbitrary competitor policy.
  • Assumptions: The guarantee assumes a distribution ν satisfying a bounded coverage-related condition C(ν; µ, F, πk) ≤ C.The constant C can be selected arbitrarily subject to this condition.
  • Proof ingredients: The proof combines no-regret policy optimization with bounds on statistical and function-approximation errors to obtain the final theorem.The analysis uses √εb = O(√εF + √εF,F + √εstat) together with the no-regret oracle property.
  • Practical implementation: The practical ATAC algorithm updates two critics and an actor using minibatches, target networks, Adam steps, projections, and clipping.The actor objective uses the pessimistic critic estimate and an entropy constraint enforced through a Lagrange multiplier.

C.1. Implementation Details

ATAC uses neural-network actors and critics with two-timescale optimization, entropy control, and a DQRA loss. Experiments use dataset-dependent β selection, multiple seeds, and both last-iterate and best-checkpoint reporting.

  • Network architecture: ATAC uses separate three-layer fully connected networks for the Gaussian policy and critics, with 256-unit ReLU hidden layers.Critic weights satisfy an l2 norm constraint of 100.
  • Optimization: ADAM optimization uses minibatches of 256 and two-timescale stepsizes ηfast = 0.0005 and ηslow = 10^-3ηfast.The stepsizes were selected offline using an imitation-learning error heuristic.
  • Optimization: The implementation sets w = 0.5, τ = 0.005, and γ = 0.99; endpoint choices w = 0 and w = 1 have poor stability or performance.The DQRA ablation finds intermediate mixing weights balance stability and performance.
  • Training protocol: Training evaluates β across 10 choices, uses 100 BC warm-start epochs plus 900 ATAC epochs, and runs 10 seeds per choice.One epoch contains 2K gradient updates.
  • Evaluation protocol: Because last-iterate convergence is not guaranteed, results report both the final iterate and the best checkpoint among nine checkpoints.The checkpoints are collected every 100 epochs.
  • Evaluation protocol: Online β selection and checkpointing are justified by ATAC’s theoretical robust-policy-improvement guarantees, although those assumptions may not hold for the practical implementation.The authors report empirical robust improvement on the tested D4RL benchmarks.

C.2. Detailed Experimental Results

Experiments evaluate ATAC on D4RL Mujoco and Adroit datasets using rollout returns, confidence intervals, and ablations. ATAC generally shows stable, improved performance, while DQRA’s intermediate mixing weight provides the best stability-performance balance.

  • Evaluation setup: D4RL evaluation uses the mean Gaussian-policy action for five rollouts and reports statistics over 10 random seeds.The benchmark includes Mujoco v2 and Adroit v1 datasets.
  • Evaluation setup: Table 2 reports median scores with 25th and 75th percentiles, providing confidence intervals and selected β values for each method.The intervals quantify deviations from the median.
  • Main results: ATAC’s confidence intervals are generally small, and its performance improvement over baselines and behavior policies is reported as significant.Larger variations occur in hopper-rand, pen-human, and hammer-human.
  • Main results: ATAC most often selects β = 64 on Mujoco datasets, while selecting smaller β values as datasets include more expert trajectories.The authors relate this pattern to the higher performance of expert behavior policies.
  • DQRA ablation: For DQRA, w = 1 is unstable, w = 0 is stable but under-performing, and intermediate weights balance stability and performance.Among the tested settings, w = 0.5 has the best performance and numerical stability across three hopper datasets.
  • Additional results: Mujoco-v0 results are included for completeness, with hopper-v0 results grayed out because those datasets contain a bug.The main results use v2 partly for this reason and for baseline compatibility.

C.3. Robust Policy Improvement

ATAC’s relative-pessimism formulation robustly improves over behavior policies across broad β ranges, while absolute pessimism requires tuning. This robustness is evaluated across hyperparameters, iterates, datasets, and the method’s maximin-based design.

  • Across-iterate evaluation: ATAC’s robust-policy-improvement score compares a policy’s return with the behavior policy’s return.Table 4 reports score percentiles over β choices, seeds, and iterates from epochs 100 through 900.
  • Across-iterate evaluation: In most datasets excluding *-human and *-clone datasets, more than 50% of ATAC iterates outperform the behavior policy across experiments.For the remaining datasets, more than 60% of iterates remain within 80% of behavior-policy performance.
  • Across-iterate evaluation: The robustness analysis includes incompletely converged iterates and poor β choices, rather than only selected final checkpoints.This broadens the empirical test beyond best-case optimization outcomes.
  • Empirical robust improvement: Across Mujoco domains, relative-pessimism ATAC improves over behavior policies across a wide range of pessimism-controlling hyperparameters.Absolute pessimism does not show this property and requires well-tuned hyperparameters for safe improvement.
  • Empirical robust improvement: Across Adroit *-exp datasets, relative-pessimism ATAC likewise improves over behavior policies across a wide β range.Robust improvement is not observed empirically on *-human and *-cloned datasets, likely because human demonstrators are not modeled by Markovian Gaussian policies.
  • Comparison with CQL: ATAC uses a maximin formulation that decouples policy and critic, whereas CQL uses a minimax formulation deriving policy from a critic.The two solution concepts generally produce different policies.
  • Comparison with CQL: Turning off ATAC’s Bellman term recovers behavior cloning, a property that supports its robust-policy-improvement guarantee and is absent from CQL.ATAC’s policies are updated through a no-regret routine using critics generated over past iterations.

D.3. Robust Policy Improvement

The section contrasts ATAC with CQL at β = 0: ATAC retains a safe policy improvement guarantee, whereas CQL can reduce to arbitrary policies and lose that guarantee.

  • ATAC provably improves the behavior policy across a wide range of β choices, including β = 0.This guarantee holds when the behavior policy µ belongs to Π.
  • For CQL, setting β = 0 cannot guarantee improvement over µ, even when µ ∈ Π and Π contains all greedy policies induced by F.The stated condition is that π_f ∈ Π for every f ∈ F and F contains constant functions.
  • With β = 0, CQL updates produce functions constant across actions in every state supported by µ.The argument establishes actionwise constancy for every update iterate on the support of µ.
  • Consequently, CQL with β = 0 can yield arbitrary learned policies and fails to provide safe policy improvement over µ.The per-state constant functions are the stated reason for this failure.
Loading 2202.02446v2…