PaperlayerView exact revision on arXiv

Source-linked AI summary

Is One Step Enough for Offline Policy Improvement?

Soohyun Choi, Seonvin Cho, Songnam Hong

arXiv:2609.03842v2cs.LG

TL;DR

Offline policy improvement must balance value maximization against anchoring to dataset actions, which can itself restrict useful changes. This paper studies multi-step proximal re-centering and finds that composition affects ideal endpoints and TD3+BC performance across tested horizons. Its results distinguish horizon subdivision from adding stages at a common local horizon.

  • Problem

    Behavior regularization limits reliance on potentially unreliable critic estimates, but strong anchoring can leave useful policy improvements unrealized.

    Source: §1 · p. 1

  • Method

    The paper studies MPI, in which separately trained later actors use the freshly updated preceding actor as their reference, and analyzes subdivision at fixed total horizon.

    Source: §1 · p. 1

  • Results

    On D4RL locomotion, I-MPI-4 retains high return over more horizons; in R2 and R3, the fraction of task–horizon cells with four-seed mean below 20 is 25.0% for I-MPI-4 versus 53.7% for TD3+BC.

    Source: §6.1 · p. 7, §8 · p. 10

  • Takeaways & Limitations

    At a common small local horizon, adding a second stage can raise the nine-task mean as the total horizon grows.

    Source: §8 · p. 10

    The flow analysis assumes ideal updates with a fixed critic and scale, while training changes both the critic and its normalization; depth alone does not select a reliable horizon.

    Source: §8 · p. 10

Abstract

from arXiv · show

Behavior regularization in offline reinforcement learning limits the exploitation of critic errors, but strong anchoring can also restrict policy improvement. We study how policy improvement is composed through multi-step proximal policy improvement (MPI), which re-centers each proximal objective on the preceding policy. We parameterize the procedure by a nominal total horizon $T$ and $K$ stages with local horizon $T/K$, distinguishing subdivision at a fixed total horizon from additional refinement at a common local horizon. Our analysis shows that sequential re-centering can reach endpoints unavailable to any single proximal step and characterizes how subdivision reduces the leading local discretization error of ideal updates under a fixed critic. We consider TD3+BC and IQL-based policy extraction to examine how improvement composition interacts with actor objectives and policy geometry. TD3+BC experiments on D4RL locomotion suggest that subdivision can broaden the range of useful total horizons, while adding refinement stages at a fixed small local horizon can improve return. The results identify improvement composition as a design choice alongside regularization strength, with distinct effects from horizon subdivision and additional policy extraction.

1 Introduction

Behavior regularization balances critic-value maximization against proximity to dataset actions, but strong anchoring can leave useful improvements unrealized. The paper studies multi-step re-centering and evaluates it across idealized updates and offline policy extraction.

1 Introduction

  • Strong anchoring can restrict useful policy improvements, while weak anchoring can expose the policy to poorly supported actions and critic extrapolation.The regularization coefficient sets the trade-off between value maximization and proximity to dataset actions.Source: §1 · p. 1
  • MPI re-centers each later actor on its freshly updated predecessor, testing whether one improvement budget works better as one step or several smaller steps.The actors are separately trained; later actors use the preceding actor as their reference during training.Source: §1 · p. 1
  • With a fixed critic, two ideal stages can reach an endpoint unavailable to one proximal step, and subdivision reduces leading error relative to policy flow for small total budgets.The two-stage endpoint result comes from a smooth strongly convex example.Source: §1 · p. 1
  • On nine D4RL locomotion tasks, deeper TD3+BC chains retain high returns over a wider range of tested total horizons.The paper also tests two deterministic actor updates and Gaussian policy extraction with an IQL critic.Source: §1 · p. 2

2 Background

The background motivates actor-side policy improvement as a way to use learned values while controlling movement toward unsupported actions. It introduces proximal updates as a geometry-dependent way to balance objective improvement and movement from a reference.

2.1 Offline reinforcement learning

  • Offline RL uses a fixed dataset without further environment interaction, and unsupported actions can make value estimates rely on extrapolation whose errors may propagate through Bellman backups.Actor regularization steers improvement toward data-supported actions, whereas conservative value-learning methods modify the critic.Source: §2.1 · p. 2

2.2 Actor-side policy improvement

  • Actor-side extraction converts learned value information into a policy while controlling movement from data-supported actions, and the extraction rule can affect performance even with fixed value learning.The paper considers TD3+BC’s critic maximization with squared behavior deviation and IQL’s advantage-weighted stochastic extraction.Source: §2.2 · p. 2

2.3 Proximal updates and policy geometry

  • A proximal update balances objective optimization against movement from a reference, with the metric specifying policy movement and the step evaluated at the new point in the implicit form.For Euclidean distance, repeated proximal steps form a minimizing-movement construction; KL/Fisher and Wasserstein geometries encode different forms of policy movement.Source: §2.3 · pp. 2–3

3 Single-step proximal improvement

Single-step proximal improvement trades critic-induced policy improvement against displacement from a reference under a chosen policy geometry. The section relates deterministic TD3+BC to Wasserstein proximity and stochastic advantage weighting locally to Fisher–Rao geometry.

3 Single-step proximal improvement

  • An SPI stage maps a reference policy to a policy trading critic improvement against displacement, with the policy class and geometry determining the operator.The policy-space metric aggregates statewise action-distribution movement over the states relevant to the actor objective.Source: §3 · p. 3

3.1 TD3+BC as a Wasserstein proximal objective

  • Under fixed-critic, fixed-scale, and convex-action-space assumptions, TD3+BC has the same minimizers as a Wasserstein proximal objective with dataset conditional-mean actions as reference and h = α/2.For deterministic policies, Wasserstein–2 between Dirac measures reduces to Euclidean action distance; the squared cloning term differs from distance to the conditional mean only by an actor-independent variance term.Source: §3.1 · p. 3

3.2 Advantage weighting and local Fisher–Rao geometry

  • The unrestricted optimizer of KL-regularized advantage improvement is proportional to the reference policy weighted by e^(hA), motivating advantage-weighted extraction and a local Fisher–Rao proximal interpretation.The Fisher–Rao connection is local, based on the second-order term of KL divergence.Source: §3.2 · p. 4

4 Multi-step proximal policy improvement

MPI composes proximal updates by using each preceding policy as the next reference, with equal-step horizon choices distinguishing subdivision from extending the total horizon. The section describes deterministic and Gaussian implementations that use separately trained actors and deploy the endpoint policy.

4 Multi-step proximal policy improvement

  • With equal stages, h = T/K subdivides a fixed nominal horizon, whereas increasing K at fixed h extends the total horizon to Kh.The coefficients do not measure actual action displacement; actors are trained in order and only the final policy is deployed.Source: §4 · p. 4

4.1 Deterministic policy refinement

  • The deterministic I-MPI first references sampled dataset actions, then each stage references its detached, freshly updated predecessor; at K = 1 it is TD3+BC with α = 2T.Implicit updates evaluate the critic at the new actor action, while explicit updates regress onto a target formed using the critic gradient at the preceding action.Source: §4.1 · p. 4

4.2 Gaussian policy refinement

  • Gaussian refinement starts from Q+BC or AWR extraction, then uses a Gaussian proximal loss referenced to the detached, freshly updated predecessor.The later-stage geometry is W2 after Q+BC and Fisher–Rao after AWR; the paper also evaluates Gaussian policy extraction with an IQL critic.Source: §4.2 · p. 5

5 Properties of composed updates

Composed proximal updates can reach endpoints unavailable to a single step, while subdivision reduces the leading local error under stated ideal-update conditions. A separate bound accounts for critic and optimization errors.

5.1 Endpoints beyond one proximal step

  • For some smooth strongly convex energies, two proximal stages reach endpoints unavailable to any single proximal step.The comparison ranges over positive step sizes and does not establish a fixed-total-horizon performance ordering.Source: §5.1 · p. 5

5.2 Local horizon subdivision

  • At fixed K and small T, subdivision reduces the leading error relative to the common flow by 1/K.This result assumes exact statewise maps, inactive action constraints, and a fixed critic and normalization throughout the chain; it does not establish convergence as K grows at fixed T.Source: §5.2 · pp. 5–6

5.3 Critic and optimization errors

  • With a fixed critic and feasible preceding policies, the proximal chain bounds objective change by accumulated displacement minus optimization error, with an added 2ϵ critic-mismatch term.The bound does not assume the errors in a trained actor are small.Source: §5.3 · p. 6

6 Experiments

The experiments compare horizon subdivision and added refinement stages across nine D4RL locomotion tasks, using TD3+BC and Gaussian extraction with an IQL critic. Results vary by horizon, extraction rule, and whether the comparison changes total horizon or holds it fixed.

6.1 Horizon and depth in TD3+BC

  • Across R2 and R3, I-MPI-4 has higher mean returns and fewer low-return cells than TD3+BC: 25.0% versus 53.7% of cells fall below 20.I-MPI-4 exceeds TD3+BC on eight of nine task means; its mean exceeds I-MPI-3 by 8.3 points (95% task-bootstrap interval [2.9, 14.3]).Source: Figure 2 · p. 7, §6.1 · p. 7, Table 1 · p. 7
  • At local horizon h = 0.1, the two-stage chain scores 67.9 versus 62.6 for the one-stage actor, while also extending total horizon.Returns are similar across depths in R1; in R2 and R3, I-MPI-4 has a higher mean than TD3+BC. Its task-level mean is higher on eight of nine tasks, with HalfCheetah medium the exception.Source: §6.1 · p. 7
  • At fixed T, greater depth reduces the first actor’s coefficient; I-MPI-4’s endpoint gains 9.13 points over its first actor on average in R2, but −0.23 in sampled R3.A matched K = 4 fixed-reference control has mixed task-level effects relative to re-centering, with a mean difference of −0.73 points.Source: §6.1 · p. 7

6.2 Implicit and explicit actor updates

  • On the shared 4 ≤ T ≤ 20 grid, increasing depth from K = 2 to K = 3 raises E-MPI’s mean from 38.4 to 46.4 and I-MPI’s from 41.2 to 51.9.TD3+BC averages 25.4 on the same grid.Source: §6.2 · p. 7

6.3 Gaussian extraction with IQL critics

  • AWR–Fisher–Rao refinement helps on some task–horizon settings but is not a uniform depth gain; at T = 1.5, single-stage policies remain strong on Hopper and HalfCheetah expert.Hopper medium-replay improves with depth around T = 2.5, while deeper endpoints collapse on the cited expert settings.Source: §6.3 · pp. 7–8, Figure 4 · p. 8
  • With Gaussian Q+BC–W2, greater depth raises the nine-task mean over intermediate horizons, but all depths lose that gain at the largest T.On Hopper and HalfCheetah expert at T = 1.5, this extraction also fails, including at K = 1; unlike AWR’s single-stage actor, it does not retain high return. Equal nominal T does not calibrate movement scales across these configurations because extraction loss and refinement geometry both change.Source: Figure 5 · p. 8, §6.3 · pp. 8–9

6.4 TD3+BC target and endpoint diagnostics

  • The diagnostic finds the I-MPI-4 first actor closer to observed successor actions than the TD3+BC target actor, while the deployed endpoint moves farther from those actions.This movement diagnostic does not establish that either actor has a more accurate critic.Source: Figure 6 · p. 9, §6.4 · p. 9

7 Related work

Related work connects offline policy improvement to behavior regularization, iterative updates, and policy geometry. MPI is distinguished by composing persistent actor stages under a common critic while varying total horizon and depth separately.

Behavior regularization and policy extraction

  • BCQ, BEAR, and BRAC constrain departure from offline data [Fujimoto et al., 2019, Kumar et al., 2019, Wu et al., 2019], while TD3+BC and ReBRAC use quadratic actor penalties [Fujimoto and Gu, 2021, Tarasov et al., 2023].AWAC and IQL use advantage-weighted extraction [Nair et al., 2021, Kostrikov et al., 2022], and extraction can matter with fixed value learning.Source: §7 · p. 9

One-step and iterative offline improvement

  • Prior iterative approaches include behavior-supported trust regions, constrained one-step and iterative operators, and CPI’s re-centered KL regularization.MPI composes actors under a common critic within each update, varies total horizon T and depth K separately, and uses its first deterministic actor for critic targets.Source: §7 · p. 9

Policy geometry and expressive actors

  • Information-geometric and Wasserstein methods define policy movement costs [Amari and Nagaoka, 2000, Schulman et al., 2015, Zhang et al., 2018, Moskovitz et al., 2020, Terpin et al., 2022, Pfau et al., 2025].Unlike Value Gradient Flow, MPI learns persistent actor stages and deploys the final actor without test-time transport; unlike diffusion or flow actor parameterizations [Wang et al., 2023, Park et al., 2025], it changes the update sequence within an actor family.Source: §7 · p. 9

8 Discussion and conclusion

The discussion distinguishes horizon subdivision from added refinement and cautions that depth alone does not select a reliable horizon. The appendices connect these claims to ideal proximal updates and behavior anchoring.

8 Discussion and conclusion

  • On the measured TD3+BC sweep, I-MPI-4 retains high returns over more horizons and has fewer low-return cells in R2 and R3.The fixed-reference control has mixed task-level effects, so aggregate gains cannot be attributed solely to re-centering; adding a second stage at a common small local horizon can raise the nine-task mean as total horizon grows.Source: §8 · p. 10
  • Depth alone does not select a reliable horizon: retrospective high-return regions and two-family selection produce mixed held-out outcomes.The flow analysis assumes ideal updates with a fixed critic and scale, whereas training changes both the critic and its normalization; training uses extra actors, but inference deploys only the endpoint.Source: §8 · p. 10

AI use statement

  • Generative AI tools assisted with drafting, literature retrieval, proof checking, experiment-design feedback, analysis code, interpretation of logged results, and figure preparation.The authors state that they originated the research idea and are responsible for the final scientific claims and supporting evidence.Source: AI use statement · p. 10

A.1 Behavior anchoring: proof of Lemma 1

  • Without clipping, the sampled first E-MPI hop is b(s) plus a conditional expected action-gradient term, which generally differs from a gradient step at b(s).The distinction concerns sampled explicit targets versus evaluating a gradient step at the behavior anchor.Source: §A.1 · p. 13

A.2 Proof of Proposition 1

  • For the smooth strongly convex example, the two-stage endpoint (1/4, 1/9) is unavailable to any single proximal step.A convergent sequence of single-step endpoints would require coefficients that force the second coordinate to 1/7, contradicting the target endpoint.Source: §A.2 · p. 13

A.3 Proof of Proposition 2

The appendix analyzes local discretization, optimization-error bounds, and Gaussian policy geometry, then specifies deterministic training schedules. Its guarantees and geometric results depend on stated assumptions and policy parameterization.

A.3 Proof of Proposition 2

  • The proof compares K local updates with the exact flow and a single full step using their second-order expansions.The expansion is developed under smoothness, uniform-remainder, and interiority assumptions.Source: §A.3 · p. 14
  • Fixed-T convergence as K grows requires a unique flow, sufficient smoothness around the flow and numerical paths, and stable steps.The smooth expansion does not apply across a ReLU critic activation boundary.Source: §A.3 · p. 14

A.4 Proof of Proposition 3

  • The fixed-objective bound includes endpoint critic errors, while optimization errors accumulate across stages.A uniform action-value error bound implies the assumed objective error under the fixed dataset state marginal; the live TD3+BC sweep changes normalization scales across stages.Source: §A.4 · p. 14

A.5 Gaussian geometry for MPI refinement

  • For local diagonal-Gaussian refinement, Wasserstein and Fisher–Rao geometries produce different first-order mean and variance updates; small variance attenuates the Fisher–Rao mean velocity.With a mean-only energy, Gaussian Wasserstein refinement keeps the reference standard deviation; a diagonal-Gaussian proximal minimizer may differ from the unrestricted distributional solution.Source: §A.5 · p. 15

A.6 Local Fisher–Rao connection to advantage weighting

  • The AWR–Fisher–Rao relation assumes a finite normalizer and support within the reference policy; finite sampling and clipped weights distinguish implemented AWR from an exact distributional update.Later Fisher–Rao stages use the exact ambient distance.Source: §A.6 · p. 15

B.1 Deterministic algorithms and schedules

  • Deterministic chains use T = τ = α/2 and h = T/K, with persistent actors updated in chain order on the same minibatch.The critic updates each iteration; actor updates occur every second iteration, and the reported deterministic sweep scores average ten evaluation episodes per seed.Source: §B.1 · p. 16

B.2 Gaussian IQL implementation

The appendix specifies Gaussian IQL extraction and matched action/target diagnostics, then documents aggregation and evaluation coverage. Comparisons use matched configurations and report incomplete Gaussian grids where applicable.

B.2 Gaussian IQL implementation

  • Gaussian Q+BC uses an unsquashed mean and learned state-dependent standard deviation, while AWR uses a tanh mean and learned state-dependent standard deviation.AWR weights are capped at 100; subsequent refinement evaluates expected Q using reparameterized Gaussian samples and stops gradients through the updated predecessor.Source: §B.2 · p. 16

B.3 Action and target-value diagnostics

  • The matched diagnostic compares first-target and endpoint action distances on identical next states, and target-value perturbations under a shared same-T TD3+BC critic and identical smoothing noise.Its matched comparison uses nine tasks, five horizons, and two seeds, excluding timeout and terminal rows.Source: §B.3 · p. 17

B.4 Aggregation and uncertainty

  • Region means average four seeds and then tasks and sampled horizons with equal weight; uncertainty intervals resample task-level contrasts while retaining within-task structure.The nine datasets share only three dynamics families.Source: §B.4 · p. 17

B.5 Gaussian evaluation and coverage

  • Gaussian Q+BC–Wasserstein covers all nine datasets, whereas AWR–FR covers seven datasets on an incomplete horizon–depth grid.The nine-task Q+BC–Wasserstein mean weights tasks equally at shared horizons; its K = 1 curves use separately trained full-T actors.Source: §B.5 · p. 17

C.1 Task-level horizon responses

Task-level results show broader high-return performance for I-MPI-4 across the tested horizon grid, alongside variation by task and stage. Diagnostics distinguish first-actor targets from deployed endpoints, and Gaussian Q+BC–W2 results are presented across tasks and depths.

C.1 Task-level horizon responses

  • Across R2 and R3, I-MPI-4 exceeds TD3+BC on eight of nine task means and exceeds I-MPI-3 by 8.3 points (95% task-bootstrap interval [2.9, 14.3]).Cells rising from below 20 with I-MPI-3 to at least 20 with I-MPI-4 contribute 65% of aggregate signed score gain.Source: §C.1 · pp. 17–18, Figure S1 · p. 19, Figure S2 · p. 20

C.2 First actor versus endpoint

  • Across fourteen observed horizons through T = 20, the I-MPI-4 endpoint gains 5.76 points on average over its first actor.In R2 the mean rises from 70.43 to 79.56, a 9.13-point gain; over the observed R3 horizons the difference is −0.23 points.Source: §C.2 · p. 18, Figure S3 · p. 18

C.3 Fixed-reference control for re-centering

  • The re-centered-minus-fixed-reference return difference averages −0.73 points, with a task-bootstrap interval of [−10.6, 8.1], and task means range from +24.9 to −32.8.The return effect depends strongly on environment; on two HalfCheetah-expert seeds, the fixed-reference branch recovers after low returns at π2 while re-centering continues to reduce return at T = 7 and 10.Source: §C.3 · pp. 18–19, Figure S4 · p. 20

C.4 Task-level target and endpoint diagnostics

  • For I-MPI-4, the median target-value perturbation reduction under the shared TD3+BC critic is 38%, while the final-to-target RMS action-distance ratio is 1.365.Across I-MPI-3 and I-MPI-4, first target actors have smaller action displacement and target-value perturbation than TD3+BC on most matched configurations.Source: §C.4 · p. 21, Figure S5 · p. 21

D Gaussian Q+BC–W2 results by task

  • Figure S6 presents Gaussian Q+BC–Wasserstein scores for all nine tasks at the ten horizons observed for every depth K = 1–4.The stated selection protocol is in Appendix B.5; the supplied passages do not report task-level winners or values.Source: D Gaussian Q+BC–W2 results by task · p. 22, Figure S6 · p. 22
Loading 2609.03842v2…