PaperlayerView exact revision on arXiv

Source-linked AI summary

Simple Actors and Deep Critics for Scalable Reinforcement Learning

Guhyeon Kang, Jaehwi Lee, Minhae Kwon

arXiv:2608.26659v1cs.LG

TL;DR

Offline RL faces costly deployment from expressive generative actors and instability when critics are deepened. LAC shifts capacity to a stabilized deep critic with a lightweight deterministic actor, matching strong generative baselines at substantially lower per-action latency.

  • Problem

    Expressive generative actors incur recurring deployment cost, while offline RL must also address distribution shift and instability in deep critics.

  • Method

    LAC combines a lightweight deterministic actor with a residual critic, n-step bootstrap targets, and categorical cross-entropy to stabilize deep offline-RL critics.

  • Results

    On OGBench, LAC matches strong diffusion- and flow-matching baselines while achieving substantially lower per-action inference latency.

  • Takeaways & Limitations

    Critic-side capacity is a productive scaling axis, and LAC’s critic ingredients transfer across actor parametrizations, including diffusion and flow matching.

  • Takeaways & Limitations

    The evaluation is limited to state-based OGBench environments and simulated benchmarks rather than pixel-based or real-robot settings.

Abstract

from arXiv · show

Recent progress in offline reinforcement learning (RL) has been driven by expressive generative actors such as diffusion and flow-matching policies, which capture multimodal behavior in offline datasets. However, these actors require multiple denoising or integration steps per action and thus incur substantial overhead at every decision in deployment. In this work, we revisit where capacity should be invested in an offline actor--critic method. Since the critic is used only during training and is discarded at deployment while the actor runs at every decision step, allocating capacity to the critic rather than the actor is more favorable for inference-time efficiency. However, scaling MLP critics in offline RL is known to introduce several distinct instabilities that have, in practice, kept critics shallow. We identify three distinct failure modes that arise when critics are deepened in offline RL---optimization, bootstrap-noise amplification, and value-range drift---and address each with a corresponding ingredient: a residual MLP backbone, n-step bootstrap targets, and a categorical cross-entropy loss. Combining these ingredients with a lightweight deterministic actor, we propose LAC (Light Actor, deep Critic). On OGBench, LAC matches the strongest diffusion- and flow-matching baselines while achieving up to 4x lower inference latency, comparable to one-step distilled policies without distillation. Its critic recipe also transfers across actor parametrizations.

1 Introduction

Generative actors capture multimodal offline-RL behavior but impose recurring deployment costs, motivating an asymmetric design that shifts capacity toward the training-only critic. LAC addresses deep-critic instabilities with a residual critic, n-step categorical value learning, and a lightweight deterministic actor requiring one forward pass.

  • Motivation: Generative actors—diffusion-based [13] and flow-matching policies [25] [41]—capture multimodal action distributions that Gaussian policies cannot, but their deployment cost recurs at every decision.This cost is especially consequential for resource-constrained edge and IoT systems, where lightweight real-time inference matters.
  • Motivation: Because the critic is discarded after training while the actor runs at every deployed decision, allocating capacity to the critic offers an inference-efficient alternative to scaling generative actors.The critic shapes the policy gradient during training, whereas actor computation is paid repeatedly during deployment.
  • Challenges: Deep MLP critics in offline RL exhibit three failure modes: optimization difficulty, bootstrap-noise amplification, and value-range drift.These instabilities help explain why offline RL has traditionally kept critics shallow [6].
  • Method: LAC combines a deep residual MLP critic with layer normalization, n-step bootstrap targets, a bounded-support categorical cross-entropy loss, and a lightweight deterministic actor.The actor uses deterministic policy gradients with a behavior-cloning regularizer, so deployment requires only one forward pass of a small network.

2 Related Work

Prior offline RL work emphasizes expressive generative actors for multimodal behavior and has generally kept offline networks shallow, unlike capacity-scaling efforts in online RL. LAC instead asks whether a sufficiently strong critic can make expressive actors unnecessary, combining several stabilization ingredients because no single ingredient suffices for deep offline critics.

  • Generative Actors in Offline RL: Offline RL commonly uses expressive generative actors, including diffusion-based methods Diffusion-QL, IDQL [25], and SRPO [8], to capture multimodal action distributions.
  • Novelty: LAC pursues a complementary direction by questioning whether expressive actors are necessary when the critic is sufficiently strong.
  • Critic Stabilization and Distributional Value Learning: Residual backbones, n-step bootstrapping, and categorical distributional learning have prior precedents [6] [27] [3], but LAC finds their combination is required to scale critics stably offline.
  • Capacity Scaling in Reinforcement Learning: Unlike online RL capacity-scaling work such as BBF, offline RL has typically used two- or three-layer MLPs and invested capacity mainly in diffusion or flow-matching actors.

3 Preliminaries

Offline reinforcement learning learns a policy from a fixed dataset without further environment interaction, making distributional shift a central challenge. Actor–critic methods address this with a critic that guides actor training, while only the actor is used at deployment, motivating greater attention to actor inference cost.

  • Actor–Critic Methods: Standard deep RL estimates the discounted-return objective with deep-network value functions, using target networks and Polyak averaging to stabilize critic training.
  • Offline RL: Offline RL learns from a fixed behavior-policy dataset without further interaction, creating distributional shift between the learned policy and collected data [23] [40].
  • Actor–Critic Methods: Actor–critic methods use a critic to estimate action values and a parameterized actor to represent the policy [24] [37] [42].
  • Behavior-Cloning Regularization: In offline RL, behavior-cloning regularization limits actor deviation from dataset actions to reduce extrapolation errors, as in TD3+BC [22] and ReBRAC.IQL [33] and AWAC [44] instead use advantage-weighted regression derived from a KL-constrained return-maximization objective.
  • What is Used After Training: The critic shapes the actor during training, but only the actor runs at deployment, so critic cost is paid during training while actor cost recurs at every decision step.This distinction is the design pivot of the proposed method.

4 LAC: Light Actor, Deep Critic

LAC invests capacity in a deep critic that is discarded after training while retaining a lightweight deterministic actor for deployment. Its residual backbone, n-step targets, and categorical loss address three distinct deep-critic failure modes in offline RL.

  • LAC design: LAC pairs a deep critic with a lightweight deterministic actor, concentrating capacity in the training-only network while keeping deployment computation small.The actor uses a deterministic feedforward MLP with the standard offline deterministic policy-gradient objective and a behavioral-cloning regularizer.
  • Failure modes: Deepening plain MLP critics causes optimization failure, bootstrap-noise amplification, and value-range drift, each requiring a distinct remedy.The failures involve unstable deep-network optimization, recursively amplified errors from one-step bootstrapping, and unconstrained Q-value magnitudes under MSE regression.
  • I1: Residual MLP Backbone: A residual MLP with layer normalization restores gradient flow and stabilizes activations, addressing optimization failure in deep critics.Skip connections provide a unit-Jacobian path through residual blocks at initialization, while layer normalization limits magnitude shifts across depth.
  • I2: n-step Bootstrap Targets: n-step targets reduce bootstrap-noise amplification by incorporating n observed rewards and changing the bootstrap coefficient from γ to γ^n.Their benefit emerges as critic depth grows, while shallow critics gain little because bootstrap-noise amplification is dormant there.
  • I3: Categorical Cross-Entropy Loss: Categorical cross-entropy over a bounded fixed support prevents value-range drift by confining predictions and replacing unbounded regression with bounded classification.The scalar n-step target is projected onto the support using C51-style two-hot interpolation, and the three ingredients are complementary.

5 Simulation Results

LAC matches or exceeds generative actor baselines on OGBench while reducing inference cost, and its stabilized critic recipe transfers across actor parametrizations. Experiments show that critic capacity is the productive scaling axis, whereas actor capacity rapidly saturates.

  • Overall benchmark results: LAC-L achieves the highest average OGBench score, while LAC-S recovers most performance with an actor roughly one-tenth the size and exceeds generative baselines on long-horizon locomotion.The comparison aggregates success rates across five task variants per environment; LAC-L and LAC-S differ only in actor architecture, with remaining gaps on ant-giant and puzzle-3x3.
  • Critic scaling: Only the full stabilized recipe converts critic depth into performance, reaching ∼80% at depth 32 and remaining stable thereafter, while plain and residual-only MLPs fail to scale.Plain MLPs fail across the 4–128-layer sweep; residual-only critics reach roughly 20% without further improvement and show increasing seed variance at depth 128.
  • n-Step bootstrap: For deep critics, n=1 peaks at 42% at depth 32 before falling to 14% at depth 256, whereas every n≥2 configuration remains stable and n=4 performs best beyond depth 16.The results show that bootstrap horizon and depth are coupled through bootstrap-noise amplification rather than independent design choices.
  • Actor scaling: With the critic fixed at 32 layers, performance plateaus after roughly 105 actor parameters, showing that additional actor capacity provides little benefit once critic capacity is strong.LAC-S and LAC-L bracket this plateau, supporting the observed asymmetry between critic and actor scaling.
  • Inference efficiency: Both LAC actors produce actions in one forward pass: LAC-S reaches 0.24ms and LAC-L 0.31ms, with up to 4× lower latency than multi-step actors.The advantage over one-step distilled baselines FQL, SRPO, and CAC is marginal.
  • Critic transferability: Replacing any actor-family critic with the LAC critic yields consistent gains of +24 to +59 points, demonstrating transfer across actor parametrizations.The transferred recipe combines a residual backbone, n-step targets, and categorical loss under an identical protocol.

6 Conclusion

LAC allocates model capacity to a deep critic rather than an inference-time actor, addressing three challenges that arise when critics are scaled in offline RL. Its evaluation is limited to state-based simulated OGBench environments, leaving pixel-based and real-robot settings for future work.

  • 6 Conclusion: LAC invests capacity in the critic because it shapes the actor during training but is discarded at deployment, avoiding recurring actor-side inference costs.This design makes critic capacity a one-time training cost rather than a cost paid at every decision.
  • 6 Conclusion: Scaling deep critics in offline RL requires addressing optimization, bootstrap-noise amplification, and value-range drift.
  • 6 Conclusion: The evaluation is restricted to state-based OGBench environments and simulated benchmarks, excluding pixel-based observations and real-robot deployment.Extending the analysis to pixel-based and real-robot settings remains future work.

Appendices · Appendix A: OGBench Benchmark Details

Appendix A defines the OGBench evaluation suite as 35 tasks across seven environments and three categories, spanning locomotion, temporally extended manipulation, and combinatorial-stitching manipulation. The environments vary in embodiment, horizon, action dimensionality, multimodality, sparsity, and planning difficulty.

  • Appendix A: OGBench Benchmark Details: OGBench comprises 7 environments across three categories, with 5 goals per environment for 35 total tasks.Figure 7 visualizes the full benchmark suite.
  • Appendix A: OGBench Benchmark Details: Locomotion tasks use an 8-DoF quadruped for ant-large and ant-giant, or a 21-DoF humanoid for hum-medium and hum-large.All use the same maze layouts; giant variants require longer horizons, while humanoid tasks have higher-dimensional actions and richer multimodal behavior where generative actors are particularly beneficial.
  • Appendix A: OGBench Benchmark Details: The benchmark spans task conditions where longer horizons, higher-dimensional actions, multimodal behavior, sparse rewards, or combinatorial structure materially increase planning demands.These conditions distinguish locomotion, scene manipulation, and puzzle-solving tasks within the evaluation suite.
  • Appendix A: OGBench Benchmark Details: LAC uses a shared hyperparameter configuration across all OGBench environments, with environment-specific settings documented for discounting and rollout horizon.The supplied appendix identifies these hyperparameter tables and footnotes but does not provide their complete values here.
  • Appendix A: OGBench Benchmark Details: The scene environment tests multi-step manipulation through pick-place and tool-use sequences under sparse rewards, requiring temporally extended planning.
  • Appendix A: OGBench Benchmark Details: The puzzle-3x3 and puzzle-4x4 environments combine continuous control with discrete sliding-tile structure and many behavioral modes.They require solving 3×3 and 4×4 grids, and puzzle-4x4 is among the suite’s hardest tasks.

Appendix B: Experimental Details

LAC’s experimental setup uses shared architecture and training hyperparameters across all seven OGBench environments, with specified n-step target computation, target-network updates, and evaluation procedures. Training and inference measurements use separate, controlled GPU protocols.

  • Experimental Configuration: Architecture and training hyperparameters are held constant across all 7 OGBench environments unless otherwise noted.Table 5 lists the complete configuration.
  • Bootstrap Targets: n-step return windows are computed vectorially at sampling time, do not cross trajectory ends, and zero bootstrap terms when terminal states fall inside the window.
  • Target Networks: Critic and actor target networks are Polyak-averaged after every gradient step, with the actor target supplying next-state actions for the n-step bootstrap target [22].
  • Evaluation Protocol: OGBench success rates are averaged over the final three evaluation epochs and 4 random seeds to reduce single-epoch noise.
  • Inference Measurement: Inference latency uses batch size 1, 100 warm-up iterations, and 10,000 measurement iterations on an NVIDIA RTX 3090, with all baselines evaluated identically.

Appendix C: Full OGBench Results

Appendix C reports per-task success rates across all 35 OGBench tasks, spanning locomotion and manipulation domains. Results average four random seeds with 50 evaluation episodes per seed at the last three evaluation epochs, while baseline sources are documented for fair comparison.

  • Evaluation scope: Table 6 covers all 35 OGBench tasks across locomotion and manipulation domains.The locomotion domains are antmaze and humanoidmaze; manipulation domains are scene and puzzle.
  • Evaluation protocol: Each cell reports the mean and standard deviation over 4 random seeds, using 50 evaluation episodes per seed at the last three evaluation epochs.
  • Baseline comparison: Baseline numbers come from the FQL paper except TD3+BC, which was re-run under the authors’ codebase for direct comparability.

Generative AI Usage Disclosure

The authors disclose that generative AI tools supported manuscript polishing and LAC codebase development, while the authors solely conducted the experiments and interpreted the findings.

  • Large language model assistants were used to polish English phrasing and improve manuscript readability.
  • Large language model assistants supported coding and debugging during development of the LAC codebase.
  • The authors solely conducted all training runs, metric computations, and interpretations of the empirical findings.
Loading 2608.26659v1…