Regularised RL improves sim-to-real transfer

1 INTRODUCTION

• Prior evaluations of robotic RL have mainly measured task completion under training conditions, while deployment also brings difficult-to-model observation, environment, and dynamics changes.The paper treats robustness to these perturbations as important to reliable sim-to-real transfer.Source: §1 · p. 1
• Regularisation may induce robustness without explicitly enumerating perturbations, unlike domain randomisation, which trains over a user-specified distribution of variations.The paper investigates whether this robustness translates into more reliable real-world deployment.Source: §1 · p. 1
• Figure 1 illustrates simulated Franka Lift deployment with a changed cube center of mass and ANYmal-D stair climbing under observation noise.The caption reports that non-regularised methods fail to climb the stairs with observation noise.Source: Figure 1 · p. 2
• The paper extends soft mellowmax to continuous control with SMM-AC, approximating its intractable reference distribution using a slowly updated reference network.The proposed algorithm is a practical off-policy actor-critic based on SAC.Source: §1 · p. 2
• SMM-AC achieved the strongest sim-to-real performance and robustness across tasks, while regularised methods maintained higher performance under real-world perturbations.The evaluation covered manipulation and locomotion, with perturbations to observations, environment properties, and system dynamics.Source: §1 · p. 2

Regularized Markov decision processes and RL

• Regularised MDPs add a strongly convex regulariser, which is subtracted from the value function to penalise deviation from a desirable policy property.Negative entropy encourages more uniform policies, while KL divergence encourages policies close to a reference.Source: §2 · pp. 2–3

The benefits of regularisation

• Regularisation has been linked to smoother, more stable or sample-efficient learning and to robustness against reward and dynamics perturbations.The section describes a theoretical duality between regularisation and robustness, and frames robotics as a test of its relevance under imperfect modelling.Source: §2 · p. 3

Mellowmax and soft mellowmax value functions

• Soft mellowmax (SMM) is an alternative soft value function developed for discrete-action Q-learning, and prior work reports larger robust sets than entropy-regularised RL in tree-structured state-action spaces.The paper expects SMM to offer sim-to-real and perturbation benefits, not necessarily to outperform SAC in simulation.Source: §2 · p. 3

Soft actor-critic

• SAC is an entropy-regularised off-policy algorithm for continuous actions that avoids directly computing intractable log-integral-exp values through a one-sample estimate.The cited background also describes SAC’s bootstrapped critic target and policy training using a KL-divergence loss.Source: §2 · pp. 3–4

Related work

• Prior robotics work connects policy stability and regularisation with deployment outcomes, while accelerated off-policy methods pursue stability and sample efficiency through parallelism and distributional critics.The paper positions its contribution as testing whether regularisation’s dynamics robustness improves sim-to-real transfer beyond matching baseline simulation performance.Source: §2 · p. 4

3 SOFT MELLOWMAX ACTOR-CRITIC

• SMM’s KL regularisation uses a Q-dependent reference policy, whose continuous-action integral makes exact evaluation intractable.The algorithm replaces intractable quantities with tractable estimators and learns an actor, critic, and reference network.Source: §3 · pp. 4–5
• SMM’s policy follows a greedier softmax than SAC’s uniform-over-actions regularisation, while its reference policy anchors actor updates to a high-performing region.The paper describes SMM-AC as an interpolation between PPO and SAC: its policy can be sharper than SAC’s while still finding multiple high-return trajectories.Source: §3 · p. 5
• The critic bootstraps its temporal-difference target with a one-sample Monte Carlo estimate of the regularised value function.The estimated value serves only to bootstrap the critic target, whose Q-value prediction error is minimised with the target gradient stopped.Source: §3 · p. 5
• Naively training the actor with a SAC-like loss can destabilise learning; a slowly updated reference network provides a low-variance surrogate that insulates it from critic noise.The reference network tracks the softmax distribution over Q-values and is trained at a substantially smaller learning rate than the actor.Source: §3 · p. 5
• SMM-AC trains its critic, actor, and slower-rate reference network while keeping the replay buffer, target networks, and critic architecture identical to SAC.This setup isolates the effect of the regulariser and supports adaptation to FastSAC and FlashSAC by adding the reference network and its loss.Source: §3 · p. 5

4 EXPERIMENTAL RESULTS ON MUJOCO

• SMM-AC is competitive with SAC and TD3 on MuJoCo tasks, except for a performance collapse on InvertedDoublePendulum-v4.Outside that failure case, it reaches returns comparable to SAC at a similar rate, suggesting comparable sample efficiency on the evaluated tasks.Source: §4 · p. 6
• In additional experiments, FlashSMM-AC achieves returns comparable to FlashSAC, and both outperform FastTD3 and PPO.The paper cautions that higher average returns in simulation do not necessarily translate into better sim-to-real transfer.Source: §4 · p. 6

5.1 EXPERIMENTAL SETUP

• The four tasks pair Franka Reach and Lift with ANYmal velocity tracking on flat and rough terrain, using default domain-randomisation ranges.The flat-terrain ANYmal policy is blind to exteroceptive information; the rough-terrain policy receives perceptual observations.Source: §5.1 · p. 7
• The study compares PPO, FastTD3, SAC, and SMM-AC, training each configuration with five seeds and sweeping ω for both regularised methods.For SAC and SMM-AC, α is fixed at 1 and temperature autotuning is not used.Source: §5.1 · p. 7

5.3 ROBUSTNESS TO DEPLOYMENT-TIME PERTURBATIONS

• Policies are held fixed while Franka and ANYmal observations, environment properties, or dynamics are perturbed; PPO and FastTD3 are excluded from ANYmal perturbations after failing the simulation safety check.Figure 4 reports perturbed return as a percentage of the same configuration’s unperturbed return; 100% means no loss.Source: §5.1 · p. 7, Figure 4 · p. 9

5.2 SIM-TO-REAL TRANSFER

• On Lift and both locomotion tasks, SMM-AC’s best configuration achieves the strongest sim-to-real transfer; reducing domain randomisation lowers every method’s return.With limited randomisation, PPO and FastTD3 fail the ANYmal simulation safety check, while SMM-AC has the highest Lift return at every DR level.Source: §5.2 · p. 8

5.3 ROBUSTNESS TO DEPLOYMENT-TIME PERTURBATIONS

• At the same ω, SMM-AC retains more return than SAC under every perturbation and more than PPO and FastTD3 wherever deployed.SMM-AC loses little or none under most perturbations, but observation noise causes the largest losses for every method and the greatest differences between methods.Source: §5.3 · pp. 8–9
• Under every perturbation, SMM-AC achieves the highest absolute return; on Lift and locomotion, every SMM-AC configuration outperforms every deployed baseline.On Reach under observation noise, SMM-AC at ω = 800 has the highest return and PPO the lowest.Source: Table 2 · p. 8, §5.3 · p. 9

• SMM-AC extends soft mellowmax to continuous control by approximating its intractable reference distribution with a slowly updated reference network.It is competitive with SAC and TD3 on MuJoCo tasks.Source: §6 · p. 9
• On Lift and both locomotion tasks, regularised methods outperform PPO and FastTD3, and SMM-AC’s best configuration has the strongest transfer performance.SMM-AC retains more return than SAC in every tested perturbation condition.Source: §6 · p. 9
• Reducing training-time domain randomisation degrades all methods, suggesting regularisation complements rather than replaces domain randomisation.The conclusion notes that optimal temperature varies across tasks and identifies adaptive temperature scheduling as future work.Source: §6 · p. 9

7 AI DISCLOSURE

• The authors report using generative AI for handwritten-note transcription, code checking and implementation assistance, plotting and table-generation code, experiment scripts, and writing review.They state that all AI-assisted work was reviewed and that the core codebase was written by humans and thoroughly checked, tested, and debugged.Source: §7 · p. 10
• The authors state that they reviewed AI-assisted work, treated LaTeX transcriptions as starting points, and take responsibility for the final content.They also say AI-modified code and plotting changes were vetted for correctness.Source: §7 · p. 10

A RELATED WORK CONTINUED

• Related work describes KL-regularised actor-critic as mitigating overestimation bias without two Q-networks, and Tsallis entropy regularisation as improving on SAC in some MuJoCo environments and on a real mobile robot.The passage attributes the KL result to Gong et al. (2025) and the Tsallis results to Lee et al. (2020).Source: A RELATED WORK CONTINUED · p. 14

• A term with zero gradient can be discarded from the loss function.Source: B DERIVATION OF SMM-AC · p. 14
• The derivation drops the last term in the expectation and converts πref to the parameterised reference policy.Source: B DERIVATION OF SMM-AC · p. 14

C ADDITIONAL DETAILS ON THE MUJOCO EXPERIMENTS

• SMM-AC’s sweep was incomplete because it had too many hyperparameters; its policy and Q-function learning rates were fixed at 3e-4 and 1e-3, respectively.The sweep varied the reference policy learning rate and ω while fixing α = 1; a cosine schedule reduced the reference policy learning rate by 10x.Source: C ADDITIONAL DETAILS ON THE MUJOCO… · p. 14
• PPO had 18 experiments per environment, versus 54 for each other algorithm, because useful PPO sweep parameters were difficult to identify.The search spaces are summarized in Table 3, and the best-performing settings in Table 4.Source: C ADDITIONAL DETAILS ON THE MUJOCO… · p. 14, Table 3 · p. 15, Table 4 · p. 15
• For Pusher-v5 and Swimmer-v4, ω = 400 was additionally evaluated and was crucial for matching SAC performance.This setting was outside the general sweep; the total number of experiments per method remained comparable.Source: Table 3 · p. 15

D ADDITIONAL MUJOCO RESULTS WITH FLASH AND FAST ALGORITHMS

• Temperature autotuning often outperformed static temperatures, supporting its importance for maximal average returns.For FlashSAC and FlashSMM-AC, the figure selected the best-performing static temperature at the end of training and also showed FlashSAC with autotuning.Source: D ADDITIONAL MUJOCO RESULTS WITH FLASH… · p. 16
• FlashSMM-AC and FlashSAC performed similarly across five MuJoCo tasks, while FastTD3 was competitive in most environments but more unstable.Figure 5 plots average return against environment steps across five runs of three million steps; shaded regions show standard deviation and solid lines show means.Source: Figure 5 · p. 16, D ADDITIONAL MUJOCO RESULTS WITH FLASH… · p. 16
• FlashSAC and FlashSMM-AC performed well with high inverse temperatures and did not require learning-rate annealing in these experiments.The authors interpret the absence of annealing as evidence that FlashSMM-AC was more stable than in the CleanRL experiments.Source: D ADDITIONAL MUJOCO RESULTS WITH FLASH… · p. 16, Table 5 · p. 16

E LIMITING DOMAIN RANDOMISATION IN ISAACLAB TASKS

• The IsaacLab study tested whether progressively limiting training DR changed simulated and real-world performance; its training-time return curves reflect the limited-DR environments.IsaacLab could not run multiple environments simultaneously, preventing evaluation of limited-DR models in non-limited environments during training.Source: E LIMITING DOMAIN RANDOMISATION IN… · pp. 16–17

E.1 LIMITING DOMAIN RANDOMIZATION IN ISAAC-REACH-FRANKA-V0

• On Isaac-Reach-Franka-v0, DR was narrowed across three settings by limiting target and reset-position ranges; evaluation on the fully randomised task showed performance dropping as the DR index increased.The first setting used the default IsaacLab ranges; by index 3, target and reset positions varied little.Source: §E.1 · p. 17, Table 6 · p. 17, Table 7 · p. 17

E.2 LIMITING DOMAIN RANDOMISATION IN ISAAC-LIFT-CUBE-FRANKA-V0

• Isaac-Lift-Cube-Franka-v0 used three progressively narrower DR settings for target and cube reset-position ranges, starting from the default configuration.The supplied passages describe the settings but do not state an outcome for this task.Source: §E.2 · p. 17, Table 8 · p. 18, Table 9 · p. 18

E.3 LIMITING DOMAIN RANDOMISATION IN ISAAC ANYMAL-D TASKS

• On ANYmal-D, DR levels narrowed dynamics, disturbances, observation noise, and episode-reset states toward nominal or deterministic values.At Level 3, friction and added-mass perturbations were removed, pushes were disabled, observation-noise bounds were one quarter of Level 1, and reset states were deterministic.Source: §E.3 · pp. 17–18

F ADDITIONAL HARDWARE RESULTS

• In additional hardware results, task errors followed returns, and the highest-return configuration also had the lowest error on Reach, Lift, and both ANYmal tasks.The comparison covered task error for each configuration.Source: F ADDITIONAL HARDWARE RESULTS · p. 18

F.1 HARDWARE DEPLOYMENT WITH LIMITED DOMAIN RANDOMISATION

• Every deployed DR 3 configuration had lower hardware return than at DR 1; Reach returns were negative, and no DR 3 flat-terrain configuration passed the safety check.At DR 2, returns fell on Reach and flat terrain and for all but one Lift configuration; on rough terrain they stayed close to DR 1, with several slightly higher, in line with simulation.Source: §F.1 · p. 19, Table 14 · p. 20
• SMM-AC configurations had the highest return at every deployed DR level on Lift and both ANYmal tasks.Under limited DR, seed-to-seed variation was large, and most same-ω SAC–SMM-AC differences were smaller than their combined standard deviations.Source: §F.1 · p. 19
• Regularisation did not replace domain randomisation: under limited DR, regularised policies still passed ANYmal safety checks more often than the unregularised baselines struggled to do.The hardware comparison deployed checkpoints without perturbation after a simulation safety check; some PPO and FastTD3 configurations, and all configurations at DR 3 on flat terrain, were not deployed.Source: §F.1 · pp. 18–19