PaperlayerView exact revision on arXiv

Source-linked AI summary

FlashSAC: Fast and Stable Off-Policy Reinforcement Learning for High-Dimensional Robot Control

Donghu Kim, Youngdo Lee, Minho Park, Kinam Kim, I Made Aswin Nahendra, Takuma Seno, Sehee Min, Daniel Palenicek, Florian Vogt, Danica Kragic, Jan Peters, Jaegul Choo, Hojoon Lee

arXiv:2604.04539v2cs.LGcs.RO

TL;DR

High-dimensional robot control challenges methods that rely on narrow on-policy data, while off-policy learning can be slow and unstable when fitting critics from broad replay. FlashSAC scales model capacity and data throughput while reducing gradient updates and bounding critic norms. Across more than 60 tasks in 10 simulators it outperforms PPO and strong off-policy baselines in performance and efficiency, and reduces sim-to-real humanoid walking training from hours to minutes.

  • Problem

    In high-dimensional robot control, narrow on-policy data limits policy evaluation, while off-policy critic fitting over diverse replay can require many updates and accumulate errors through bootstrapping.

    Source: §1 · p. 2

  • Method

    FlashSAC reduces gradient updates while scaling model size and data throughput, and bounds critic weight, feature, and gradient norms.

    Source: §1 · p. 2, §4.1 · p. 5

  • Results

    Across more than 60 tasks in 10 simulators, FlashSAC outperforms PPO and strong off-policy baselines in final performance and training efficiency; sim-to-real humanoid walking training falls from hours to minutes.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results demonstrate that off-policy RL can be fast and stable for scalable sim-to-real robot learning.

    Source: §1 · p. 2

    The work focuses on state-based control; extending critic-stabilization principles to tactile-based learning remains future work.

    Source: §7 · p. 13

Abstract

from arXiv · show

Reinforcement learning (RL) is a core approach for robot control when expert demonstrations are unavailable. On-policy methods such as Proximal Policy Optimization (PPO) are widely used for their stability, but their reliance on narrowly distributed on-policy data limits accurate policy evaluation in high-dimensional state and action spaces. Off-policy methods can overcome this limitation by learning from a broader state-action distribution, yet suffer from slow convergence and instability, as fitting a value function over diverse data requires many gradient updates, causing critic errors to accumulate through bootstrapping. We present FlashSAC, a fast and stable off-policy RL algorithm built on Soft Actor-Critic. Motivated by scaling laws observed in supervised learning, FlashSAC sharply reduces gradient updates while compensating with larger models and higher data throughput. To maintain stability at increased scale, FlashSAC explicitly bounds weight, feature, and gradient norms, curbing critic error accumulation. Across over 60 tasks in 10 simulators, FlashSAC consistently outperforms PPO and strong off-policy baselines in both final performance and training efficiency, with the largest gains on high-dimensional tasks such as dexterous manipulation. In sim-to-real humanoid locomotion, FlashSAC reduces training time from hours to minutes, demonstrating the promise of off-policy RL for sim-to-real transfer.

1 Introduction

High-dimensional robot learning makes narrow on-policy data and costly simulation increasingly inefficient. FlashSAC combines fewer gradient updates, larger models, higher data throughput, and bounded critic norms to improve off-policy training speed and stability; it outperforms PPO and strong off-policy baselines across a broad benchmark suite.

1 Introduction

  • High-dimensional control, vision-based tasks, and costly simulation make repeatedly discarding past on-policy experience increasingly inefficient in sample complexity and wall-clock time.Emerging tasks include humanoid locomotion and dexterous manipulation, while complex contact dynamics and larger policies raise rollout costs.Source: §1 · p. 2
  • Off-policy learning reuses diverse replay-buffer experience for greater data efficiency, but often trains slowly and unstably, limiting its adoption for sim-to-real transfer.Fitting a critic over broad replay data takes many updates and repeated bootstrapping can compound estimation errors.Source: §1 · p. 2
  • FlashSAC reduces gradient updates while scaling model size and data throughput, and bounds weight, feature, and gradient norms to prevent critic-error accumulation.The algorithm is built on Soft Actor-Critic and aims to match large-scale simulation pipelines.Source: §1 · p. 2
  • Across more than 60 tasks in 10 simulators, FlashSAC outperforms PPO and strong off-policy baselines in final performance and training efficiency, with the largest gains on dexterous manipulation and humanoid locomotion.For sim-to-real humanoid walking, training falls from hours to minutes while maintaining stable real-world deployment.Source: §1 · p. 2

2 Related Work

On-policy methods are stable and effective when simulation is cheap, but their reliance on fresh data makes high-dimensional coverage costly. Off-policy methods reuse experience, yet speed, stability, and exploration remain coupled challenges; FlashSAC unifies scaling, critic constraints, and lightweight correlated exploration.

2.1 On-Policy Reinforcement Learning

  • PPO is stable and effective for sim-to-real transfer in constrained domains when fresh on-policy data is cheap, but discarded experience makes sufficient coverage increasingly expensive as task dimensionality grows.Importance sampling can in principle enable reuse, but its weights have extremely high variance in high-dimensional continuous action spaces.Source: §2.1 · p. 3

2.2 Off-Policy Reinforcement Learning

  • Off-policy model-based methods plan with learned dynamics but incur added training cost for model learning and repeated planning, which can limit wall-clock scalability.DreamerV3 and TD-MPC2 are cited as strong vision-based examples.Source: Off-Policy Model-Based RL · p. 3
  • Model-free off-policy methods reuse replay data, but face slow critic fitting, bootstrapped error amplification, and insufficient coherent exploration in high-dimensional action spaces.SAC's maximum-entropy formulation alone may not maintain coherent exploration.Source: Off-Policy Model-Free RL · p. 3
  • Prior speed approaches scale throughput but may rely on small networks, while stability approaches enable larger networks but require more updates; exploration methods alone leave both bottlenecks.FastTD3 and FastSAC are described as using networks of about 0.2M parameters; stability constraints can enable higher asymptotic performance but slow training in data-rich simulation.Source: Off-Policy Model-Free RL · p. 3
  • FlashSAC combines fewer updates with larger models and higher throughput, jointly bounds critic weight, feature, and gradient norms, and uses lightweight noise repetition for correlated exploration without per-environment overhead.The combined design addresses speed, stability, and exploration together.Source: Off-Policy Model-Free RL · p. 3

3 Preliminary

The preliminaries frame robotic control as a discounted Markov decision process and introduce SAC as the off-policy foundation for FlashSAC. SAC reuses replay-buffer transitions, optimizes an entropy-regularized policy, and trains clipped double critics with slowly updated targets.

3 Preliminary

  • The section introduces the reinforcement-learning framework and algorithmic foundation used to build FlashSAC.Robotic control is modeled as a discounted Markov decision process with continuous actions and a policy that maximizes discounted rewards.Source: §3 · p. 4, §3.1 · p. 4

3.1 Markov Decision Process (MDP)

  • A discounted Markov decision process specifies state and action spaces, transition dynamics, reward, and discount factor; the agent selects actions from observations to maximize discounted reward.The discount factor γ lies in [0, 1).Source: §3.1 · p. 4

3.2 Soft Actor Critic (SAC)

  • SAC reuses transitions collected under past policies from a replay buffer and adds entropy regularization to encourage exploration and replay-buffer coverage.Insufficient exploration can worsen approximation and extrapolation errors.Source: §3.2 · p. 4
  • SAC's entropy-regularized objective balances policy entropy against the minimum of two critics' action-value estimates, with α controlling entropy's relative importance.Source: §3.2 · p. 4
  • SAC uses the minimum of two action-value estimates in targets to reduce optimistic value errors, and trains critics with bootstrapped Bellman error against slowly updated target networks.The target networks are updated by exponential moving average, with τ as the target update rate.Source: §3.2 · p. 4

4 FlashSAC

FlashSAC targets high-dimensional robotic control with fast, stable off-policy learning. It combines scaled data and models with fewer updates, critic-stability mechanisms, and broad exploration.

4 FlashSAC

  • FlashSAC combines faster training through scaled data and models with fewer updates, critic constraints for stability, and broad exploration for diverse data coverage.Source: §4 · p. 4

4.1 Fast Training

4.2 Stable Training

4.3 Exploration

  • Exploration uses a fixed action-standard-deviation entropy target across embodiments and repeats sampled noise for Zeta-distributed durations, favoring short intervals but occasionally producing long correlated sequences.Noise Repetition is presented as a lightweight alternative to temporally correlated noise methods that require per-environment processes.Source: Unified Entropy Target · p. 6, Noise Repetition · p. 6
  • Figure 3 shows FlashSAC comparable to PPO in low-dimensional tasks and significantly outperforming PPO in high-dimensional tasks.Source: Figure 3 · p. 7

5 Experiments

FlashSAC is evaluated across state-based, vision-based, and sim-to-real robot control, measuring performance and training efficiency. Results span GPU- and CPU-based simulation as well as real-world humanoid locomotion.

5 Experiments

  • The evaluation measures asymptotic performance and wall-clock time across low- and high-dimensional state-based control, vision-based control, and sim-to-real humanoid locomotion.The GPU-based state suite includes 25 tasks across four simulators, with 15 low-dimensional and 10 high-dimensional tasks.Source: §5 · p. 7, Experimental Setup · p. 7

5.1 State-Based RL on GPU-based Simulators

  • On high-dimensional dexterous manipulation and humanoid locomotion tasks, FlashSAC converges reliably to higher asymptotic performance than PPO with substantially less wall-clock time.Against FastTD3, it converges on tasks where that method often fails or underperforms, and performs better when both methods converge.Source: Experimental Setup · p. 8

5.2 State-Based RL on CPU-based Simulators

  • In CPU-based single-environment tasks, FlashSAC consistently outperforms the baselines in the sample-efficient regime and matches or exceeds dedicated sample-efficient methods without task-specific tuning.These benchmarks emphasize sample efficiency rather than wall-clock throughput.Source: §5.2 · pp. 8–9

5.3 Vision-based RL

  • Across vision-based tasks, FlashSAC matches or exceeds baselines in asymptotic performance while converging faster in wall-clock time.DrQ-v2 is unstable in several environments, while MR.Q incurs additional computation from its auxiliary dynamics model; FlashSAC uses one hyperparameter set without task-specific exploration or auxiliary objectives.Source: §5.3 · p. 9

5.4 Sim-to-Real Transfer

  • FlashSAC achieves stable real-world flat-terrain locomotion after approximately 20 minutes of simulation training, versus about 3 hours for PPO.On rough stairs unseen during training, FlashSAC succeeds after approximately 4 hours, while PPO requires nearly 20 hours for similar capability.Source: Experimental Results · p. 10

6 Analysis

The analysis examines replay-buffer coverage and ablates FlashSAC’s scaling, critic architecture, and exploration choices. Across these analyses, broader replay coverage, larger models with fewer updates, bounded critic dynamics, and repeated action noise support more effective training.

6.1 Off-Policy vs On-Policy

  • Replay-buffer data covers a substantially broader state–action region than data collected around the final policy.The comparison uses object y-position and finger actions on the Shadow Hand cube-reorientation task.Source: Figure 7 · p. 10, §6.1 · p. 11

6.2 Scaling Ablation for Faster Training

  • Increasing replay-buffer size improves performance up to 10M transitions, while 50M slows learning as recent high-quality samples are drawn less often.The 50M buffer can reach slightly higher asymptotic performance given sufficient training time.Source: Figure 8 · p. 11, §6.2 · p. 11
  • Increasing batch size and model capacity while reducing the UTD ratio accelerates convergence, consistent with established scaling-law trends.The resulting higher-capacity models converge faster than the small architectures commonly used for stability.Source: Figure 8 · p. 11, §6.2 · p. 11

6.3 Architectural Ablation for Stable Training

  • Adding architectural components keeps parameter, feature, and gradient norms bounded and monotonically lowers the critic-loss condition number.The full FlashSAC architecture reaches the lowest condition number; the components are Residual Blocks, Batch Normalization, Post RMSNorm, Distributional Critics with Reward Scaling, and Weight Normalization.Source: Figure 9 · p. 11, §6.3 · p. 12
  • Improved optimization stability translates to better task performance, while weight normalization alone yields modest gains and improves robustness in sample-limited regimes.Weight normalization is retained in the final design.Source: Figure 9 · p. 11, §6.3 · p. 12

6.4 Exploration Ablation

  • Performance is largely insensitive to entropy targets from 0.05 to 0.25, and repeating action noise improves convergence and aggregate scores.The default entropy target of 0.15 performs well without task-specific adjustment; noise repetition produces coherent exploratory trajectories.Source: Figure 10 · p. 12, §6.4 · p. 12

7 Lessons and Opportunities

FlashSAC combines scaling with critic-update constraints to address off-policy training speed and stability in increasingly high-dimensional robotics. The section discusses potential applications, identifies state-based control as the work’s scope, and describes how wall-clock time and environments are characterized.

7 Lessons and Opportunities

  • FlashSAC combines fewer gradient updates with scaled model capacity and data throughput, while bounding critic weight, feature, and gradient norms; the paper reports up to an order-of-magnitude lower wall-clock time than on-policy methods.The authors identify vision-rich inputs, slower realistic simulators, and mixtures of demonstrations and self-collected experience as opportunities for stabilized off-policy learning.Source: §7 · p. 12
  • The work focuses on state-based control, with extending critic-stabilization principles to tactile-based learning identified as future work.The authors describe tactile-based learning as a promising direction.Source: §7 · p. 13

A Measuring Wall-Clock Time

  • Wall-clock estimates sum environment interaction time and algorithm update time, with update costs profiled on representative state- and vision-based benchmarks.The appendix also says environments are grouped by simulator and characterized by observation space, action space, and score normalization.Source: A Measuring Wall-Clock Time · p. 20

B Environment Details

  • The environment appendix groups evaluation settings by simulator and reports each environment’s observation space, action space, and normalization score.The supplied passage does not specify the individual environments.Source: B Environment Details · p. 20

B.1 IsaacLab

The IsaacLab evaluation uses a GPU-accelerated platform and includes 12 tasks across manipulation and locomotion.

B.1 IsaacLab

  • IsaacLab v2.1.0 is a GPU-accelerated simulation platform built on NVIDIA Isaac Sim.The evaluation includes 12 tasks spanning gripper and dexterous-hand manipulation, quadruped locomotion, and humanoid locomotion.Source: §B.1 · p. 20
  • Scores are normalized per environment using a near-asymptotic reference obtained by running FlashSAC for an extended training duration.Full environment specifications are listed in Table 1.Source: §B.1 · p. 20

B.2 Mujoco Playground

The appendix describes benchmark suites spanning humanoid locomotion, manipulation, and continuous control. It specifies task counts and score-normalization conventions where supplied.

B.2 Mujoco Playground

  • MuJoCo Playground evaluation covers four humanoid locomotion tasks emphasizing robust whole-body control under complex contact dynamics.Normalized scores are scaled to 40, corresponding to asymptotic performance in converged runs.Source: §B.2 · p. 20

B.3 ManiSkill

  • ManiSkill evaluation uses six rigid-body manipulation tasks with a gripper-based robotic arm and fixes the environment snapshot to commit aad75f2.The benchmark uses the SAPIEN physics engine and supports diverse object geometries and physical interactions.Source: §B.3 · p. 20

B.4 Genesis

  • The Genesis evaluation benchmarks three environments and adapts their source code to the Gymnasium API.The simulator version is v0.3.13.Source: §B.4 · p. 20

B.5 Gym - MuJoCo

  • The Gym-MuJoCo evaluation covers five v4 locomotion tasks involving multi-body dynamics and contact-rich interactions.Scores are normalized using TD7 random-policy and 5M-step reference scores.Source: §B.5 · p. 21

B.6 DeepMind Control Suite

  • The DeepMind Control Suite evaluation includes 10 tasks, from low-dimensional sparse-reward exploration settings to high-dimensional DMC-Hard tasks such as Humanoid and Dog.Observation and action space details are reported in Table 6.Source: §B.6 · p. 21

B.7 HumanoidBench

  • HumanoidBench evaluation covers 14 locomotion tasks requiring stable gait generation and balance on a high-dimensional whole-body-control benchmark based on the Unitree H1.Scores use the success thresholds defined by the benchmark authors.Source: §B.7 · p. 21

B.8 MyoSuite

This section describes the MyoSuite task set, benchmark and hyperparameter coverage, and the simulation, control, terrain, observation, and reward choices used for sim-to-real experiments.

B.8 MyoSuite

  • MyoSuite evaluation covers 10 dexterous tasks involving the elbow, wrist, and hand, with fixed-goal tasks labeled easy and randomized-goal tasks labeled hard.The task set is listed in Table 8; normalized scores correspond to maximum success rate, with 100 denoting 100% success.Source: §B.8 · p. 21, Table 8 · p. 23

C Hyperparameters

  • The hyperparameter tables cover GPU-based simulators, CPU-based simulators, and vision-based tasks; the CPU and vision tables list only differences from Table 9.GPU-based benchmarks include IsaacLab, MuJoCo Playground, ManiSkill, and Genesis; CPU-based benchmarks include MuJoCo, DMC, Humanoid Bench, and MyoSuite.Source: Table 9 · p. 24, Table 10 · p. 24, Table 11 · p. 25

D Sim-to-Real Details

  • Blind locomotion policies are trained in simulation and deployed directly on hardware without additional fine-tuning.The sim-to-real experiments describe the simulation environment, control architecture, terrain curriculum, observation design, and reward formulation.Source: D Sim-to-Real Details · p. 25

D.0.1 Simulation.

  • IsaacLab training uses 4096 parallel environments with domain randomization and takes approximately 4 hours on one NVIDIA A100 GPU.The resulting policy networks are directly deployed on the physical robot without additional fine-tuning.Source: §D.0.1 · p. 25

D.0.2 Low-Level Control.

  • The policy outputs target joint positions at 50 Hz for a 200 Hz low-level PD controller, which converts them to torque commands using proportional and derivative gains.A custom Pybind interface connects the Python policy loop to the C++ Unitree SDK implementation for real-time control.Source: §D.0.2 · p. 25

D.0.3 Terrain Configuration and Curriculum.

  • Training covers flat and rough terrain, with a 10-level curriculum that increases difficulty after successful traversal; progressing beyond level 5 is difficult without perception and leads to aggressive motion.Terrain types include pyramid stairs, random grids, random uniform terrain, waves, and pits; the curriculum advances when the policy traverses 50% of the environment.Source: §D.0.3 · pp. 25–26, Figure 11 · p. 26

D.0.5 Reward.

  • The reward structure is shared, but reward weights differ between FlashSAC and PPO; FlashSAC uses only a minimal alive reward and does not require termination-based shaping.The observation includes hardware-accessible proprioceptive signals and command inputs; the actor also uses a history-based context estimator, while the critic receives privileged information.Source: §D.0.5 · p. 26, §D.0.3 · p. 26, Table 13 · p. 27

E Complete Results

The complete-results section presents learning curves for each task against both wall-clock time and environment steps, distinguishing compute and sample efficiency.

E Complete Results

  • Learning curves for each task are plotted against wall-clock time and environment steps to show compute and sample efficiency, respectively.Results are averaged over random seeds; the compute-efficiency curves specify different environment-step budgets for PPO and other methods.Source: E Complete Results · p. 30

E.1 IsaacLab (State-based RL, GPU Simulator)

  • IsaacLab curves show average episode returns against compute time and environment steps, with 50M environment steps for most methods and 200M for PPO in compute plots.The compute and sample plots average random seeds and show 95% bootstrap confidence intervals.Source: Figure 12 · p. 30, Figure 13 · p. 30

E.2 Mujoco Playground (State-based RL, GPU Simulator)

  • MuJoCo Playground curves plot average episode returns against compute time and environment steps; compute plots use 50M steps except for PPO at 200M.The plots average random seeds and display 95% bootstrap confidence intervals.Source: Figure 14 · p. 31, Figure 15 · p. 31

E.3 ManiSkill / E.4 Genesis

  • ManiSkill and Genesis provide paired compute- and sample-efficiency curves, with 50M training steps except for PPO at 200M in the compute plots.Both simulator sets report average episode returns, average across random seeds, and 95% bootstrap confidence intervals.Source: Figure 16 · p. 32, Figure 17 · p. 32, Figure 18 · p. 33

E.5 Mujoco (State-based RL, CPU Simulator)

  • MuJoCo CPU-simulator curves plot average episode returns against compute time and environment steps, using 1M steps except for PPO at 4M in compute plots.Curves average random seeds and show 95% bootstrap confidence intervals.Source: Figure 20 · p. 34, Figure 21 · p. 34

E.6 DeepMind Control Suite (State-based RL, CPU Simulator)

  • DeepMind Control Suite curves plot average episode returns against compute time and environment steps, using 1M steps except for PPO at 4M in compute plots.Curves average random seeds and show 95% bootstrap confidence intervals.Source: Figure 22 · p. 35, Figure 23 · p. 35

E.7 Humanoid Bench / E.8 MyoSuite

  • Humanoid Bench and MyoSuite provide compute- and sample-efficiency curves; compute plots use 1M steps except for PPO at 4M.The figures show average episode returns, averages across random seeds, and 95% bootstrap confidence intervals.Source: Figure 24 · p. 36, Figure 25 · p. 36, Figure 26 · p. 37

E.9 Vision-based RL

  • DMC-Visual curves report average episode returns against compute time and environment steps, with all methods trained for 1M environment steps in compute plots.The curves average random seeds and show 95% bootstrap confidence intervals.Source: Figure 28 · p. 38, Figure 29 · p. 38

E.10 Ablation Study

The ablation figures define the training configurations compared, but the supplied captions do not state comparative outcomes.

E.10 Ablation Study

  • The ablations vary replay-buffer capacity and mini-batch size, with results averaged over configuration seeds and shown with 95% bootstrap confidence intervals.The captions identify the dotted lines as normalized scores.Source: Figure 30 · p. 39, Figure 31 · p. 39
  • Network-width configurations vary critic hidden dimension while scaling the actor proportionally; network-depth configurations vary residual-block count in both networks.The depth learning curves plot average IsaacLab episode returns against environment steps.Source: Figure 32 · p. 40, Figure 33 · p. 40
  • The UTD-ratio ablation compares update-to-data ratios, while the architectural ablation curves compare configurations using IsaacLab episode returns against environment steps.The captions report averages across random seeds, 95% bootstrap confidence intervals, and normalized-score dotted lines.Source: Figure 34 · p. 41, Figure 35 · p. 41
  • The remaining ablations compare target standard deviations for automatic temperature tuning and whether noise-repeat exploration is enabled.Both figures plot average IsaacLab episode returns against environment steps.Source: Figure 36 · p. 42, Figure 37 · p. 42

Related papers

Loading 2604.04539v2…