PaperlayerView exact revision on arXiv

Source-linked AI summary

Decoupling Policy Extraction for Offline Reinforcement Learning

Xuyao Lin, Yixiang Shan, Jinru Duan, Tao Yang, Xinyu Zhao, Runyu Lei, Yiming Zhao, Jiaxin Fan, Zongbao Feng, Peng Jia

arXiv:2608.20909v1cs.LGcs.RO

TL;DR

Offline RL’s fixed dataset weakens coupled actor–critic policy improvement and leaves OOD amplification and support–value trade-offs unresolved. The paper decouples behavior modeling from policy improvement by reranking behavior-supported proposals with a separately learned critic at inference time. Across offline goal-conditioned tasks, this paradigm consistently outperforms corresponding coupled methods, including with an IQL critic’s average success rate rising from 47.89% to 68.49%.

  • Problem

    Fixed offline data prevents actor improvements from generating interactions that validate or correct the critic, while coupled updates face OOD amplification and support–value trade-offs.

  • Method

    The method trains a behavior-cloning actor independently and uses a separately learned critic to rank multiple actor-generated candidate actions at inference time.

  • Results

    Across 30 offline goal-conditioned tasks, decoupled extraction consistently improves performance over coupled methods; with IQL, average success rises from 47.89% to 68.49%.

  • Takeaways & Limitations

    Policy improvement can be performed by decoupling critic-based selection from actor training, with reported gains across multiple critic backbones and offline tasks.

  • Takeaways & Limitations

    Proposal coverage limits performance because the behavior-cloned proposer cannot supply missing or extremely rare high-value behaviors; larger candidate budgets increase ranking errors and inference cost.

Abstract

from arXiv · show

Offline RL methods commonly jointly train the actor and critic, where the critic is used to guide the actor toward higher-value actions. This coupled learning process is well motivated in online RL, where an improved actor collects new data that can further update the actor and the critic. However, training data remains fixed in offline RL, making actor-side policy improvement unable to generate new data to validate or correct the critic. Moreover, retaining this coupled paradigm leads to two related challenges. Firstly, actor updates can drift toward high-valued but potentially out-of-distribution (OOD) actions and amplify critic overestimation. Secondly, conservative value estimation or behavior-cloning regularization creates a difficult trade-off between suppressing OOD actions and selecting high-value actions within the data-supported region. Motivated by this observation, we revisit the conventional offline RL paradigm and propose decoupling policy improvement from actor training. Specifically, we train the actor solely to model the behavior distribution and perform policy improvement at inference time by reranking multiple actor-generated proposals with a separately learned critic. We refer to this paradigm as the decoupled policy extraction paradigm. Under such paradigm, the actor provides behavior-supported action candidates, while the critic performs value-based selection within this candidate set. Extensive experiments show that the decoupled policy extraction paradigm outperforms both behavior cloning and jointly learned offline RL methods, while remaining effective even with a naive Q-learning critic.

1 INTRODUCTION

Offline RL retains actor–critic policy improvement despite fixed data, weakening the feedback loop that motivates it online and creating OOD amplification and support–value trade-offs. The paper proposes decoupled policy extraction and reports consistent gains across offline goal-conditioned tasks.

  • Motivation: In offline RL, actor improvement cannot collect new data to validate or correct critic errors, weakening the rationale for coupled actor–critic training.Online RL benefits from a closed loop in which improved actors expand critic data support through new interactions.
  • Motivation: Coupled policy improvement can shift actors toward overestimated OOD actions, amplifying critic errors and distributional drift.This is identified as the OOD amplification loop problem.
  • Decoupled policy extraction: The decoupled paradigm trains a behavior-cloning actor independently and uses a critic to rerank actor-generated candidate actions at inference time.Policy improvement is therefore explicit test-time selection rather than critic-driven actor optimization.
  • Results: 68.49% average success rate versus 47.89% for the coupled method with the IQL critic across 30 offline goal-conditioned tasks.Other critic backbones show similar improvements, indicating the reported benefit is not specific to one value-learning objective.
  • Results: The authors instantiate decoupled extraction with multiple critic backbones and report consistent improvements over corresponding coupled methods across broad offline goal-conditioned tasks.They also identify computational implications for large policy models by avoiding repeated actor updates.

2 PRELIMINARIES

The preliminaries define offline goal-conditioned RL around a fixed dataset and behavior-supported action regions. They separate critic learning from policy extraction, formalizing proposal, ranking, and candidate-budget components for inference-time selection.

  • Offline goal-conditioned RL: A goal-conditioned Markov decision process uses state s, action a, and goal g, with policy π(a | s, g) conditioned on state and goal.The notation also defines x = (s, g) as a state–goal context.
  • Offline goal-conditioned RL: Offline RL provides only a fixed dataset and cannot collect additional interaction data, so weakly supported actions may exploit uncorrected extrapolation errors.The dataset induces a behavioral conditional action distribution used to define action coverage.
  • Value learning and policy extraction: Critic learning is separated from policy extraction, allowing critic rules such as Q-Learning, IQL, or TRL to share a ranking-based extraction interface.Only relative action ordering matters under strictly increasing transformations of the selection score.
  • Value learning and policy extraction: An RL-trained actor uses critic-derived supervision, whereas a behavior-cloned proposer learns only from dataset actions.This distinction separates critic-guided actor training from behavior-distribution modeling.
  • Proposal-and-selection policies: A stochastic proposer q(a | x) samples N candidate actions, after which a critic-based selector chooses among them.The proposal distribution, critic ranking function, and candidate budget N are explicit design factors.
  • Proposal-and-selection policies: When N > 1, critic ranking changes the deployed action distribution at inference time without retraining the proposer.Random selection from the same candidate set serves as a control for the effect of sampling multiple actions alone.

3 DIAGNOSING COUPLED POLICY IMPROVEMENT IN OFFLINE RL

The section diagnoses two weaknesses of coupled offline policy improvement: critic-error amplification through OOD actor drift and a support–value trade-off under conservatism. Experiments show that stronger conservatism can reduce OOD actions initially but eventually harm performance without improving support.

  • 3.1 OUT-OF-DISTRIBUTION AMPLIFICATION LOOP: Coupled actor–critic training can amplify critic errors when actor updates increase probability on overestimated OOD actions.The fixed offline dataset cannot provide new interaction data to correct these errors.
  • 3.1 OUT-OF-DISTRIBUTION AMPLIFICATION LOOP: 6.8% of EDP-BC actions fell outside the same-checkpoint BC95 region, versus 31.3% for EDP-CRR and 16.8% for EDP-TD3.All methods used the same initialization, but coupled policy-improvement variants exhibited greater distribution shift.
  • 3.2 THE SUPPORT–VALUE TRADE-OFF: Conservative or behavior-cloning regularization does not remove the conflict between suppressing unsupported actions and improving value within the supported region.The section frames this as the support–value trade-off.
  • 3.2 THE SUPPORT–VALUE TRADE-OFF: Moderately increasing αCQL initially decreases the Outside BC95 ratio while improving CQL performance by suppressing OOD action values.The CQL diagnostic holds the SAC actor objective fixed, so conservatism acts through the learned critic.
  • 3.2 THE SUPPORT–VALUE TRADE-OFF: Excessive conservatism eventually degrades performance, while the Outside BC95 ratio no longer decreases monotonically.Broad Q-value suppression compresses value-function dynamics and reduces discrimination among candidate actions.

4 DECOUPLED POLICY EXTRACTION

Decoupled policy extraction trains a behavior-cloning actor independently from a critic, then uses the critic only to rerank actor-generated candidates at inference. This freezes the proposal distribution, limits critic selection to behavior-supported actions, and shifts policy-iteration tuning to inference time.

  • 4 DECOUPLED POLICY EXTRACTION: The method uses a frozen behavior-cloning actor to propose data-supported actions and a separately learned critic to rank them at deployment.Critic information does not update actor parameters during training.
  • 4.1 FROZEN BC ACTOR: The actor is trained by maximizing the dataset action log-likelihood, which reduces to supervised action regression for deterministic parameterizations.The objective models the offline behavior distribution rather than directly optimizing critic values.
  • 4.1 FROZEN BC ACTOR: After training, the actor receives no rewards, critic values, advantage weights, or gradients, so improvements arise from critic selection over fixed samples.This prevents critic-derived preferences from changing the proposal distribution during training.
  • 4 DECOUPLED POLICY EXTRACTION: The critic need only estimate relative values accurately among behavior-supported candidates, rather than suppress arbitrary OOD actions across the action space.This relaxes critic-design requirements and alleviates the support–value trade-off.
  • 4 DECOUPLED POLICY EXTRACTION: The candidate budget N controls inference-time search: larger N provides more actions to compare but can increase ranking errors or weakly supported proposals.Adjusting N requires neither actor nor critic retraining.

5 EXPERIMENTS

Experiments evaluate the decoupled approach across OGBench environments, critic backbones, candidate budgets, and inference costs. Results show consistent gains, including with Q-learning critics, while candidate count creates a performance–cost trade-off.

  • Experimental setup: The evaluation covers six OGBench goal-conditioned continuous-control environments and compares Q-Learning, IQL, and TRL backbones.The environments span locomotion, manipulation, contact-rich control, and high-dimensional settings.
  • Aggregate results: 38.4%, 13.9%, and 202.2% relative improvements in macro-average success rate are reported for navigation, manipulation, and logical-reasoning tasks using IQL.
  • Critic backbones: Even naive Q-learning serves as an effective reranking critic, with Puzzle performance improving 212.8% compared with IQL.The paper attributes IQL’s weaker reranking to conservative expectile value learning that can compress value differences between candidates.
  • Ablation analysis: On Puzzle-4x4, critic reranking added to coupled IQL consistently matches or outperforms the original IQL policy across candidate budgets.
  • Ablation analysis: Decoupled + Reranking achieves the best overall performance at N = 16, surpassing both original IQL and coupled IQL with reranking.The comparison indicates that reranking alone remains constrained by the actor learned under coupled training.
  • Candidate budget: Coupled + Reranking peaks at N = 8 and degrades with larger budgets, while increasing N initially helps explore the behavior-supported action region before OOD actions become more influential.The candidate sweep therefore exposes both the benefit and risk of expanding the proposal set.
  • Inference cost: Inference-time policy improvement introduces additional deployment cost as the candidate budget N increases.Candidates are generated in parallel through batched flow evaluations and then reranked with one batched twin-critic evaluation.

6 RELATED WORKS

Related work includes inference-time critic selection, generative value-guided policies, and decoupled structures. This paper positions its contribution as a general framework for systematically analyzing policy extraction components and candidate budgets.

  • Inference-time policy improvement: SfBC and EMaQ generate behavior-model candidates and select among them with a critic, but also rely on the behavior-cloning actor for critic learning.
  • Generative RL: Diffuser, QGF, and GAF perform inference-time value guidance, but their action generation remains coupled with the critic during inference.
  • Decoupled structures: V-GPS and IDQL use similar decoupled structures, whereas this work studies decoupling as a general policy extraction principle rather than a specific application or instantiation.
  • Policy extraction: The framework explicitly controls the proposer, critic, selector, and candidate budget to analyze when behavior-supported proposals and critic-based selection recover performance.

7 CONCLUSION AND DISCUSSION

The paper argues that offline policy extraction should separate behavior modeling from critic-based policy improvement. Its main limitation is proposal coverage, which bounds performance when valuable behaviors are missing or rare in the dataset.

  • Conclusion and Discussion: Decoupled policy extraction separates behavior-supported action proposal, critic evaluation, and inference-time selection.The actor models the offline data distribution, while the critic selects among proposed actions.
  • Implications for Future High-Capacity Policies: A single proposal model can be reused across tasks by pairing it with cheaper task-specific critics.This reduces the need to repeatedly update large policies with critic-derived objectives.
  • Limitations and Future Directions: Proposal coverage imposes a performance ceiling when missing or extremely rare high-value behaviors are absent from the offline dataset.Increasing the candidate budget may improve coverage but can also expose the selector to more critic ranking errors and increase inference cost.

8 AUTHOR CONTRIBUTIONS

The author contributions assign leadership across experiments, study conception, methodology, analysis, visualization, implementation, and manuscript preparation and revision.

  • Author Contributions: Xuyao Lin led the experiments, prepared visualizations, analyzed results, and contributed to drafting and revising the manuscript.
  • Author Contributions: Yixiang Shan conceived the study, designed experiments, led analysis and interpretation, formulated conclusions, and led manuscript writing and revision.
  • Author Contributions: Jinru Duan prepared visualizations and contributed to manuscript drafting, review, and revision.
  • Author Contributions: Tao Yang developed the methodology, contributed to implementation and experiments, and participated in manuscript writing, review, and revision.

A.1 IMPLEMENTATION DETAILS

The implementation details describe shared experimental settings across methods and environments, including six OGBench environments, proposal-based rerankers, jointly trained baselines, and paired seed evaluation.

  • Evaluation Setup: The experiments use six standard OGBench environments with oracle goal representations.
  • Evaluation Setup: The comparison includes FM-BC, three frozen-policy rerankers, FM-IQL, and the official TRL-RPG baseline.The rerankers are BC+IQL-Q, BC+Q-Learning-Q, and BC+Transitive-Q.
  • Evaluation Protocol: Frozen-proposal methods pair each critic seed with the FM-BC proposal trained using the same seed.

A.2 HYPERPARAMETER SETTINGS

The appendix specifies training schedules, goal-sampling ratios, benchmark hyperparameters, actor-drift diagnostics, and CQL conservatism procedures used to evaluate the methods.

  • Training Schedules: Each trainable component is optimized for 10^6 gradient steps, with BC+Q methods training and freezing FM-BC before critic optimization, unlike jointly trained baselines.
  • Goal Sampling: Goal-sampling ratios comprise current, geometric-future, trajectory-future, and random goal components.The passages explicitly define pgeom and ptraj as geometric and uniformly sampled future trajectory goals.
  • Benchmark Hyperparameters: Task-specific candidate counts for BC+IQL-Q and BC+Transitive-Q are selected independently from {1, 2, 4, 8, 16, 32} using five-seed means.
  • Actor-Drift Diagnostic: The actor-drift diagnostic compares methods using common diffusion actors and behavior-cloning losses, with actions generated directly by diffusion actors without reranking or Q-dependent inference.
  • Actor-Drift Diagnostic: BC95 classifies actions relative to a same-checkpoint behavior-cloning distribution in a shared PCA plane rather than providing a formal environment-level OOD test.The procedure uses KDE density and split-conformal calibration; actions with p-values at most 0.05 are outside BC95.
  • CQL Conservatism Diagnostic: The CQL diagnostic holds the SAC actor objective fixed while varying critic conservatism through an importance-corrected log-sum-exp penalty weighted by αCQL.
Loading 2608.20909v1…