PaperlayerView exact revision on arXiv

Source-linked AI summary

Role-Adaptive Policy Optimization for Offline Reinforcement Learning

Seonvin Cho, Soohyun Choi, Songnam Hong

arXiv:2609.40149v1cs.LG

TL;DR

Offline RL regularization balances policy improvement against uncertain value estimates, but execution and critic bootstrapping can favor different balances. RAPO learns update coefficients according to each policy’s role, separating TD3+BC’s actors and adapting only IQL’s extraction coefficient. Across fifteen D4RL tasks, both instantiations improve their base algorithms’ domain-average returns, and TD3+RAPO has the highest overall average among compared methods.

  • Problem

    Offline RL regularization must balance critic-guided policy improvement against uncertain value estimates, and execution and critic bootstrapping can favor different balances.

    Source: §1 · p. 1

  • Method

    RAPO differentiates through candidate updates formed with the base actor objective and adapts coefficients by policy role, separating TD3+BC’s execution and bootstrap actors while adapting only IQL’s extraction inverse temperature.

    Source: §1 · p. 2

  • Results

    Across fifteen D4RL tasks, both RAPO instantiations improve their base algorithms’ domain-average returns, and TD3+RAPO has the highest overall average among compared methods.

    Source: §1 · p. 2, §6 · p. 9

  • Takeaways & Limitations

    Coefficient-learning criteria can be chosen according to how policy actions enter the algorithm’s value learning and execution.

    Source: §6 · p. 9

    The execution score remains a local surrogate whose relation to return improvement depends on the learned critic and state distribution.

    Source: §6 · p. 10

Abstract

from arXiv · show

Policy regularization in offline reinforcement learning balances policy improvement against reliance on uncertain value estimates. This balance can differ between selecting actions for execution and supplying actions for critic bootstrapping, yet methods such as TD3+BC couple these roles through a shared policy. We propose Role-Adaptive Policy Optimization (RAPO), which adapts policy-update coefficients according to their roles in value learning and execution. RAPO learns these coefficients by differentiating through candidate policy updates formed using the base algorithm's actor objective. For TD3+BC, RAPO separates bootstrap and execution actors and adapts their coefficients independently: the bootstrap objective penalizes policy-induced changes in target values, while the execution objective evaluates a local policy-improvement surrogate. For IQL, whose value learning is already independent of the execution actor, RAPO preserves the original value updates and adapts only the inverse temperature in advantage-weighted policy extraction. Experiments on D4RL locomotion and AntMaze tasks show improvements over both base algorithms, with larger gains for TD3+BC, whose RAPO instantiation outperforms baselines on average.

1 Introduction

Offline RL must balance critic-guided policy improvement against behavior regularization, and the right balance can differ between execution and critic bootstrapping. RAPO learns role-specific update coefficients; both instantiations improve over their baselines on D4RL, with TD3+RAPO highest overall among compared methods.

1 Introduction

  • In TD3+BC, execution actions and critic-bootstrap actions share an actor, although improving execution values can change target values and limiting target movement can restrict execution updates.The appropriate regularization balance may also change as the actor and critic evolve.Source: §1 · p. 1
  • IQL’s value learning is already independent of its extracted actor, while policy extraction can still affect performance for a given value function.Methods with separate target and evaluation policies also allow different constraint strengths [Kostrikov et al., 2022, Xu et al., 2024]; policy extraction can substantially affect performance [Hansen-Estruch et al., 2023, Park et al., 2024].Source: §1 · p. 1
  • RAPO differentiates through candidate policy updates formed with the base actor objective, choosing coefficients by role while retaining that objective for actual updates.Execution updates are scored for local policy improvement; bootstrap updates are scored for stability of the critic targets they produce.Source: §1 · p. 2
  • TD3+RAPO separates execution and bootstrap actors and adapts their coefficients independently, whereas IQL adapts only the inverse temperature for advantage-weighted policy extraction.IQL’s value updates use dataset actions without the execution actor.Source: §1 · p. 2
  • Across fifteen D4RL tasks, both RAPO instantiations improve their base algorithms’ domain-average returns, and TD3+RAPO has the highest overall average among compared methods.Matched TD3 controls examine execution-score and bootstrap adaptation, while a fixed-coefficient control tests whether learned endpoint values reproduce adaptation during training.Source: §1 · p. 2

2 Background

Offline learning uses a fixed behavior-collected dataset without further environment interaction. The background contrasts TD3+BC and IQL actor-critic updates, then introduces meta-gradients for training an update coefficient through the policy update it controls.

2.1 Offline Learning Setup

  • Offline RL trains from a fixed dataset of behavior-collected state, action, reward, and successor-state tuples, without further environment interaction.The dataset is denoted D = {(s, a, r, s′)}.Source: §2.1 · p. 2

2.2 Base Actor-Critic Updates

  • TD3+BC combines Q maximization with behavior cloning; its coefficient α controls behavior regularization relative to Q maximization.Increasing α weakens behavior regularization relative to Q maximization.Source: §2.2 · p. 2
  • In TD3+BC, the actor selects execution actions and its target copy supplies critic-bootstrap actions; RAPO assigns these uses to separate actors.TD3+BC and IQL differ in their actor objectives and in the actions used for critic bootstrapping.Source: §2.2 · p. 2
  • IQL learns values from dataset actions independently of its actor, then extracts a policy using advantage-weighted log likelihood with inverse temperature β.β controls how strongly policy extraction favors higher-advantage actions.Source: §2.2 · p. 3

2.3 Meta-Gradient Learning

  • Meta-gradient learning updates a coefficient by differentiating through a policy update: the inner loss trains the policy, and the outer loss trains its coefficient.The outer loss scores the updated actor on a separate minibatch while holding the starting parameters fixed.Source: §2.3 · p. 3

3 Method

RAPO adapts role-specific policy coefficients by differentiating through candidate actor updates and scoring them according to their use. Its objectives assess execution updates for local improvement and bootstrap updates for critic-target stability.

3 Method

  • RAPO forms differentiable candidate actor steps and updates each active role’s coefficient using an outer loss evaluated on an independently drawn minibatch.The inner loss remains the base algorithm’s actor objective.Source: §3 · p. 3, §3.1 · pp. 3–4

3.1 Policy Roles and Adaptive Coefficients

  • TD3+RAPO separates execution and bootstrap actors with independently adapted coefficients, while IQL+RAPO adapts only execution because its value target is actor-independent.The execution actor acts in the environment; the bootstrap actor supplies policy actions for TD3+RAPO’s critic targets.Source: §3.1 · p. 3

3.2 Execution Objective: Local Policy Improvement

  • The execution objective scores candidate actions with a fixed target critic, rewarding first-order improvement while discounting updates by an endpoint-gradient curvature penalty.The penalty uses two action gradients without forming a Hessian; its quadratic-correction bound applies under the stated approximation, not to a general critic.Source: §3.2 · pp. 4–5

3.3 Bootstrap Objective: Target Stability

  • The bootstrap objective penalizes the RMS change in unsmoothed Bellman targets induced by a candidate bootstrap-policy update, using the same fixed target critics for comparison.This is a proxy for stability because the actual training target uses a smoothed target actor.Source: §3.3 · p. 5

4 Experimental Results

Across fifteen D4RL tasks, both RAPO variants improve their base algorithms’ domain averages, with TD3+RAPO achieving the highest overall average. Ablations favor the full execution score and continued coefficient adaptation over the tested controls.

4.1 Evaluation Protocol

  • The evaluation covers fifteen D4RL tasks, with four training seeds evaluated over 50 episodes at one million steps and equal-weight environment averages.The final Table 1 row averages all fifteen tasks; both RAPO methods start with coefficients fixed at five.Source: §4.1 · p. 6, Table 1 · p. 6

4.2 Benchmark Performance

  • 91.7 on Locomotion and 70.0 on AntMaze for TD3+RAPO exceed TD3+BC’s 80.8 and 31.7; both RAPO variants improve their respective domain averages, and TD3+RAPO leads overall.The reported averages weight environments equally.Source: §4.2 · p. 6, Table 1 · p. 6
  • From equal initial values, TD3+RAPO’s αE rises above αB on three datasets, while IQL+RAPO’s βE falls on Hopper and rises on AntMaze-medium-play.On Hopper-medium-expert, TD3+RAPO’s coefficients remain close; α and β have different numerical scales and meanings.Source: §4.2 · p. 7, Figure 2 · p. 7

4.3 Execution-Objective Ablation

  • 83.0 overall for the full execution score exceeds 78.8 for Q and 79.4 for Linear; the full score also leads in both domains.The Linear control isolates the endpoint-gradient penalty, and its lower averages suggest this term helps select execution-coefficient updates.Source: §4.3 · p. 7, Table 2 · p. 8

4.4 Adaptation During Training

  • Adapting αB from five raises the overall mean from 75.9 to 83.0 and the AntMaze mean from 55.8 to 70.0, with especially large gains on AntMaze-large tasks.αE is adapted in both conditions; gains on AntMaze-large-play are 20.0 to 48.0, and on AntMaze-large-diverse are 13.0 to 51.0.Source: §4.4 · p. 8, Table 3 · p. 8
  • 83.0 with adaptation exceeds 78.4 with learned endpoint coefficients fixed from the start; the difference is concentrated in Hopper-medium-expert for Locomotion, while four of six AntMaze tasks improve.Fixing the final coefficient values from the start does not reproduce the adaptive domain averages.Source: §4.4 · p. 8, Table 4 · p. 8

5 Related Work

RAPO builds on behavior regularization, adaptive constraints, and policy decoupling, but distinguishes itself by evaluating coefficient updates according to whether an actor supplies execution actions or critic targets.

Behavior regularization

  • Behavior regularization addresses offline value uncertainty through different interventions: TD3+BC penalizes actor deviation, CQL regularizes value estimates toward pessimism, and ReBRAC separates actor and target coefficients [Fujimoto et al., 2018, Fujimoto and Gu, 2021, Kumar et al., 2020, Tarasov et al., 2023].RAPO retains the base value-loss form and adapts actor-loss coefficients, including for the actor supplying bootstrap actions.Source: §5 · pp. 8–9

Adaptive constraints and meta-gradients

  • Prior adaptive-constraint methods vary constraints across actions or states, while meta-gradients learn update parameters by differentiating through optimization [Peng et al., 2023, Liu et al., 2024, Luo et al., 2025, Xu et al., 2018, Franceschi et al., 2018].ASPC applies meta-gradients to policy-constraint scales, including IQL’s inverse temperature; RAPO’s distinction is its role-specific evaluation criterion, not meta-gradients or value-change regularization itself.Source: §5 · p. 9

Decoupling value learning and execution

Value learning and policy extraction

  • Extraction can change performance even with the same value function [Hansen-Estruch et al., 2023, Park et al., 2024]; RAPO instead trains an explicit execution actor and adapts IQL’s advantage-weighting temperature without changing policy class or adding inference-time search.Other approaches use diffusion-model samples or rank behavior-model proposals at inference [Lin et al., 2026].Source: §5 · p. 9

Local policy improvement

  • RAPO’s execution score is a local surrogate for return improvement; its relation to return depends on critic and state-distribution assumptions and additional approximation errors [Schulman et al., 2015].The method assesses a single candidate update while retaining the base actor objective.Source: §5 · p. 9

6 Conclusion

RAPO evaluates candidate policy updates according to their roles, and both instantiations improve over their baselines on fifteen D4RL tasks. The execution score remains a local surrogate whose relation to return improvement depends on the critic and state distribution.

6 Conclusion

  • Both RAPO instantiations improve over their respective baselines in domain-average return across fifteen D4RL tasks.Within two-actor TD3, the full execution score and bootstrap adaptation beat tested controls, while fixed learned endpoints yield lower averages than continued adaptation.Source: §6 · p. 9
  • The execution score remains a local surrogate, and its relation to return improvement depends on the learned critic and state distribution.The authors report that generative AI tools assisted literature discovery, manuscript drafting and editing, and mathematical arguments, with the authors reviewing the work and taking responsibility for the final text.Source: §6 · p. 10, AI use statement · p. 10

A.1 Proof of Lemma 1

The appendix interprets RAPO’s execution score through a local critic approximation and relates it to return improvement using an exact decomposition. It also specifies how coefficient differentiation is carried out while leaving actor parameters and optimizer state unchanged.

A.1 Proof of Lemma 1

  • The smoothness premise can fail at activation boundaries or where IQL’s minimum critic switches between Q-networks.The proof begins by fixing a state and applying the fundamental theorem of calculus under a Lipschitz condition.Source: §A.1 · p. 12
  • For a quadratic critic, the endpoint-gradient change is H_sδa, and Equation (9) subtracts an upper bound on the quadratic correction’s magnitude.For a general critic, endpoint variation need not control interior curvature, so the score is not a certified lower bound and does not quantify critic uncertainty.Source: §A.1 · p. 12

A.2 Relation to Return Improvement

  • The return-change decomposition separates the dataset-state action-value gain from local, critic, and state residuals.The critic residual includes critic error relative to Q^π, including reference-policy mismatch; RAPO does not bound these residuals, and the execution score does not guarantee return improvement.Source: §A.2 · p. 12

B.1 Differentiation and Normalization

  • Coefficient differentiation holds actor and critic parameters, dataset-action references, and incoming optimizer states constant while differentiating through new gradients and optimizer moments.Each active role has its own unconstrained scalar ρ_i; TD3+RAPO maps it through softplus, while IQL+RAPO uses exp.Source: §B.1 · pp. 12–13
  • The bootstrap prefactor scales the coefficient gradient without adding a direct derivative path; α_B affects that gradient through the candidate policy.The bootstrap RMS treats terminal transitions as zeros and is zero, with zero chosen subgradient, when all inputs are zero.Source: §B.1 · p. 13

B.2 Training Settings

The training settings align the TD3+RAPO comparisons’ Q-network architecture with TD3+RAPO and specify distinct details for bootstrap targets, IQL, and coefficient learning rates. Candidate and actual updates differ in selected optimizer and actor-use details.

B.2 Training Settings

  • The TD3+BC, wPC, and ASPC comparisons use TD3+RAPO’s three-layer, width-256 Q-network with ReLU activations and LayerNorm in each hidden layer.The actor-critic implementations follow CORL.Source: §B.2 · p. 13
  • Both actual and candidate TD3 actor updates use Equation (1)’s loss scale, so α_i changes finite updates and their meta-gradients.The training critic uses a target bootstrap actor with clipped Gaussian action noise, while Equation (11) compares unsmoothed online actors.Source: §B.2 · p. 13
  • IQL clips exp(β_EA) at 100 and uses fixed baseline β values of 3 on Locomotion and 10 on AntMaze.IQL’s Gaussian actor means are used in the execution score and at evaluation.Source: §B.2 · pp. 13–14
  • At initialization five, each method’s coefficient learning rate is selected from three candidates by its four-seed mean in each environment.Table 6 lists the selected rates; TD3 uses the listed rate for both coefficients, and its fixed-bootstrap control reuses the rate for α_E.Source: §B.2 · p. 13, Table 6 · p. 14
  • TD3 updates its critics before scheduled coefficient meta-updates and actor updates, using the new α_B for both real actor updates; IQL schedules its execution meta-update after value updates.The actual execution update uses the coefficient cached before that step’s meta-update, and the TD3 bootstrap target actor follows the target critics’ update rate.Source: §B.2 · p. 14
  • Both methods use batch size 256, discount 0.99, normalized observations, and unchanged Locomotion rewards; TD3 clips smoothed target actions to [−1, 1].IQL+RAPO uses expectile 0.7 on Locomotion and 0.9 on AntMaze; AntMaze rewards are shifted by −1.Source: §B.2 · p. 14

B.3 Evaluation and Configuration Selection

Final scores use four independent runs, each evaluated over 50 episodes at one million training steps, with equal weighting of environments in averages. The shared-learning-rate sensitivity test finds TD3+RAPO above TD3+BC’s domain averages at each tested rate, though task-level performance varies.

B.3 Evaluation and Configuration Selection

  • Final scores are measured at one million steps from four runs, each evaluated over 50 episodes; domain and overall averages weight environments equally.Each run’s episode mean gives one seed score, and reported means and sample standard deviations are computed across the four seed scores.Source: §B.3 · p. 14

B.4 Coefficient Learning-Rate Sensitivity

  • At 3 × 10−4, TD3+RAPO’s Locomotion and AntMaze means are 87.4 and 64.2, versus 80.8 and 31.7 for TD3+BC.TD3+RAPO exceeds TD3+BC’s domain averages at all three tested shared initial coefficient learning rates, though task-level performance varies.Source: §B.4 · p. 14, Table 7 · p. 15

B.5 Outer-Objective and Coefficient Ablations

The bootstrap adaptation comparison keeps two actors and adapts α_E from five while testing fixed and adaptive α_B values. Bootstrap adaptation improves both domain averages at either tested initialization, but initialization still matters, especially on AntMaze.

B.5 Outer-Objective and Coefficient Ablations

  • The four bootstrap conditions retain two actors, adapt α_E from five, and use TD3+RAPO’s actor and critic settings and selected coefficient learning rate.The comparison extends the fixed-α_B = 5 analysis to fixed and adaptive values of one.Source: §B.5 · p. 15
  • Adapting α_B from one raises Locomotion from 88.2 to 90.9 and AntMaze from 23.8 to 38.5; from five, AntMaze reaches 70.0 versus 38.5 from one.Bootstrap adaptation improves both domain averages at either initial value, while initialization still matters especially on AntMaze.Source: §B.5 · p. 15, Figure 3 · p. 16, Table 8 · p. 16
  • The endpoint-fixed control trains new actors and critics with coefficient updates disabled, using each environment’s mean final coefficient pair from four TD3+RAPO seeds.The control’s constants are rounded to six decimal places for training; Table 9 displays the constants and per-task results alongside the adaptive main benchmark.Source: §B.5 · p. 17, Table 9 · p. 17

Related papers

Loading 2609.40149v1…