Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control

1 Introduction

• Continuous-action RL has conventionally used small networks, and naive capacity scaling has been reported to degrade performance.The introduction asks whether parameter and replay scaling can work alongside existing algorithmic improvements.Source: §1 · p. 2
• The study finds parameter scaling can produce larger performance gains than gradient-step scaling while being more computationally efficient in parallelized setups.It evaluates batch size, distributional Q-values, regularization, and optimistic exploration among other design choices.Source: §1 · p. 2
• BRO significantly outperforms model-free and model-based methods on 40 demanding tasks, including near-optimal performance on Dog and Humanoid.Figure 1 summarizes performance across three benchmark suites at 1M environment steps.Source: Figure 1 · p. 1, §1 · p. 2
• BRO combines the BroNet critic-scaling architecture with domain-specific RL enhancements.The algorithm is presented as a sample-efficient model-free approach.Source: §1 · p. 2
• Regularized critic scaling outperforms replay-ratio scaling in performance and computational efficiency, while critic scaling can largely substitute for domain-specific inductive biases.The latter finding suggests simpler algorithms may retain those benefits.Source: §1 · p. 3

2.1 Experimental setup

• The evaluation covers 40 complex continuous-control tasks across DeepMind Control, MetaWorld, and MyoSuite, selected for large prior optimality gaps and run for 1M environment steps.The baseline comparison includes TD-MPC2, SR-SAC, CrossQ, SAC, and TD3.Source: §2.1 · p. 3, Figure 3 · p. 3

2.2 BRO outline and design choices

• BroNet scales more effectively than other architectures, but scaling it within standard SAC yields weak performance without BRO’s additional algorithmic enhancements.BroNet uses Layer Normalization after dense layers, along with weight decay and full-parameter resets for regularization and stability.Source: §2.2 · p. 4, Figure 4 · p. 4
• A 26M-parameter model at RR=1 significantly outperforms a small model at RR=15 while requiring three times less wall-clock time.Critic model scaling has a stronger impact and plateaus at approximately 5M parameters; model size and replay ratio also work well in tandem.Source: §2.2 · p. 5
• Critic scaling, adding pessimistic Clipped Double Q-learning, and using RR=1 are among the most impactful BRO ablations; reducing BRO (Fast) from RR=2 to 1 has no significant effect.BRO removes Clipped Double Q-learning and uses the ensemble mean rather than the minimum for Q-value calculation.Source: §2.2 · pp. 4–5, Figure 7 · p. 6
• Dual-actor optimistic exploration yields around 10% performance improvement in BRO, with optimism gains particularly pronounced early in training and for smaller models.The exploration policy follows an optimistic upper-bound Q-value approximation.Source: §2.2 · pp. 5–6
• Quantile critic representations improve performance particularly for smaller networks, with gains diminishing for over-parameterized agents.The distributional setup also provides epistemic uncertainty estimates used for optimistic exploration.Source: §2.2 · p. 6

• TD-MPC2 required approximately 2.5 times more environment steps than BRO to reach BRO’s performance on seven challenging Dog and Humanoid tasks.The comparison runs the other algorithms for 3M steps on those DMC environments.Source: §3 · p. 6, Figure 8 · p. 7
• Smaller batches, quantile Q-values, and optimistic exploration helped smaller critics but not the largest 26.3M models; full-parameter resets remained beneficial.The largest model without resets nearly matched BRO with resets.Source: §3 · pp. 6–7
• Increasing critic capacity outperformed increasing actor size, which could be detrimental in some off-policy continuous-control settings.The analysis suggests prioritizing critic capacity while respecting memory limits.Source: §3 · p. 7
• Target networks yielded a small but significant performance improvement, though effects varied substantially across benchmarks and environments.Using target networks doubles memory costs.Source: §3 · p. 7
• BroNet outperformed Spectral and Vanilla MLP architectures in the Dog environments, while Vanilla MLP showed instabilities across measured metrics.The table compares architectures combined with BRO; metrics other than return are averaged over time steps.Source: §3 · p. 7, Table 1 · p. 7
• Some tasks remained unsolved within 1M steps, including Humanoid Run and Acrobot Swingup; applying algorithms to single tasks risks overfitting.The authors advocate standardized benchmarks focused on sample efficiency.Source: §3 · p. 8
• Naively applying BroNet improved performance across tested offline BC, offline fine-tuning IQL, and online SAC with offline data settings.The experiments covered AntMaze and Adroit tasks.Source: §3 · p. 8, Figure 11 · p. 9
• BRO exceeds 80% of maximal performance in 33 of 40 tasks, compared with 18 of 40 for model-free SR-SAC.The experiments cover 40 demanding tasks from DeepMind Control, MetaWorld, and MyoSuite.Source: Figure 10 · p. 9

4.1 Sample efficiency through algorithmic improvements

• Value-overestimation methods include lower-bound Clipped Double Q-learning, online tuning of pessimism, and layer normalization that can improve performance without value overestimation.The cited work notes that pessimistic lower bounds can slow learning.Source: §4.1 · p. 9
• Optimistic exploration methods seek sample efficiency by acting greedily with respect to an upper-bound Q-value, using policy transformations or a dedicated exploration actor.The cited approaches include closed-form transformations and a dual actor network.Source: §4.1 · p. 9

4.2 Sample efficiency through scaling

• Prior scaling work often allocates model capacity to world models or encoders, while naive value-model scaling has been reported to deteriorate performance.A cited study found spectral normalization enabled stable training of relatively large ResNets with performance improvements.Source: §4.2 · pp. 9–10
• Replay-ratio scaling increases gradient steps per environment step; high ratios can improve performance when networks are regularized, with layer normalization and full-parameter resets among effective techniques.Source: §4.2 · p. 10

• BRO’s larger models require more memory than traditional baselines, potentially challenging real-time inference in high-frequency control.The experiments primarily cover continuous control with proprioceptive states; discrete settings and image-based critic–encoder scaling remain open research questions.Source: §5 · p. 10

• BRO achieves markedly superior performance within 1 million environment steps compared with state-of-the-art model-based TD-MPC2 and other model-free baselines.Source: §6 · p. 10
• BRO exceeds 90% success rates in MetaWorld and MyoSuite and 85% of maximal returns in DeepMind Control Suite.Source: §6 · p. 10
• BRO achieves near-optimal policies in challenging Dog and Humanoid locomotion tasks, although some tasks remain unsolved within 1 million steps.The authors call for standardized benchmarks focused on sample efficiency.Source: §6 · p. 10

Broader Impact

• The study is academic and based on simulated benchmarks; the authors encourage attention to potential societal consequences as the research is extended.They state that their contributions do not directly cause negative societal impacts.Source: Broader Impact · p. 15

B.1 Base agent

• BRO uses Soft Actor-Critic, a stochastic maximum-entropy algorithm with automatic entropy-temperature adjustment and two critic networks.Its policy is a Tanh-transformed Gaussian modeled by the actor network.Source: §B.1 · p. 15

B.2 Architecture details

• BRO uses 512-unit Dense layers, residual blocks for depth, and a layer-normalization configuration rather than spectral normalization.Dropout is omitted, and linear layers appear at the input and output stages.Source: §B.2 · p. 15

B.3 Scaling

• BRO exceeds 90% of maximal performance, while the highest-performing SAC agent reaches around 25%; layer-normalized ResNets scale better than spectral-normalized and vanilla-dense networks.Vanilla-dense performance can degrade beyond a certain model size; BRO performs similarly at 4.92M and 26.31M parameters, so the smaller model is used to reduce computation.Source: §B.3 · p. 16

B.4 Scaling replay ratio

• Scaling model size and replay ratio together performs best, while a 5M-parameter BRO at RR=1 beats a 1M-parameter model at RR=15 and runs five times faster.The paper attributes the model-size efficiency advantage to parallelized workloads.Source: §B.4 · p. 16

B.5 Batch Size

• Reducing batch size from 256 to 128 marginally improves aggregate performance and reduces memory use, though no single batch size works best across all tasks.The batch-size choice is evaluated in Figures 9a and 15.Source: §B.5 · p. 16

B.6 Risk-Neutral Temporal Difference

• With network regularization, using the ensemble mean instead of clipped double Q yields significant performance gains without increasing value overestimation.BRO disables clipped double Q and uses the ensemble mean for actor and critic targets.Source: §B.6 · pp. 16–17

B.7 Optimistic Exploration

• BRO’s dual-policy setup estimates an optimistic upper-bound Q-value from quantile-ensemble epistemic uncertainty, with gains especially pronounced early in training and for smaller networks.The exploratory policy pursues the optimistic estimate while separate policies handle exploration and temporal-difference updates.Source: §B.7 · p. 17

B.8 Approaches Examined During the Development of BRO

• BRO incorporates LayerNorm, weight decay, removal of clipped double Q, optimistic exploration, quantile distributional RL, resets, and increased replay ratio.Table 2 lists the approaches examined during development and highlights those incorporated into BRO.Source: §B.8 · p. 17, Table 2 · p. 18

C Tested Environments

• The 40-task evaluation spans DeepMind Control, MetaWorld, and MyoSuite, with selected tasks chosen for their difficulty and varied state and action spaces.The selection includes locomotion and manipulation challenges; MetaWorld follows the TD-MPC2 evaluation protocol, with success assessed within a 200-step episode.Source: C Tested Environments · p. 18
• Many easier tasks in these benchmarks can be solved by modern algorithms within 100k environment steps, motivating the focus on harder tasks.The MetaWorld evaluation uses V2 environments and retains the original goal definitions.Source: C Tested Environments · p. 18

D Hyperparameters

• BRO’s hyperparameters were selected from reported building-block values and extensive experimentation with ablations; BRO (Fast) uses replay ratio 2 to speed training with some performance trade-off.BRO also resets all parameters every 250k steps until 1M steps, with additional resets at 15k and 50k.Source: D Hyperparameters · pp. 19–20

E.1 Scaling and Time Experiments

• The largest BRO variant (26.31M parameters) at replay ratio 5 has execution time similar to the smallest (0.55M) at replay ratio 15, but much greater performance.Increasing model size and replay ratio both improve performance, while model-size scaling is more execution-time efficient due to GPU parallelism.Source: §E.1 · p. 21, Figure 12 · p. 21

E.2 Additional BroNet Analysis

• Layer normalization after each dense layer makes a five-layer MLP comparable to BRO in aggregate IQM, but BRO’s skip connections help on complex tasks such as Dog.The architecture analysis covers five DMC and five MetaWorld environments; the full BRO design combines LayerNorm and residual blocks for robustness on challenging tasks.Source: §E.2 · pp. 21–22, Figure 14 · p. 22

Batch sizes

• Using half or a quarter of BRO’s original minibatch size of 256 does not significantly hurt performance across the reported benchmarks.Figure 15 compares BRO and BRO (Fast) across Dogs and Humanoid, DeepMind Control, MetaWorld, and MyoSuite.Source: §E.3 · p. 22, Figure 15 · p. 22

Target network

• For the BRO agent, using a target network yields performance improvements only in specific environments; the authors present these results in Figure 16.Source: §E.3 · p. 23

Longer training

• At 5M steps, BRO almost solves Dog Run, exceeding 80% of possible returns while significantly outperforming TD-MPC2 and SR-SAC on seven Dog and Humanoid tasks.The longer-training experiments used a single-task setup; BRO and BRO (Fast) were trained for 3M and 5M steps, respectively.Source: §E.3 · p. 23, Figure 18 · p. 23

Image-based tasks

• Replacing SR-SPR’s Q-network with BroNet on Atari 100k tasks is described as a promising direction for discrete, image-based RL.The experiments also varied reset interval and shrink-and-perturb settings, which affected BroNet-agent performance.Source: §E.3 · pp. 23–24, Table 8 · p. 24

Multi-Task RL

• The multi-task evaluation compares BRO (Fast) with SAC on MetaWorld’s 50-task MT50 benchmark, using doubled model widths and task-identifier inputs.The figure caption reports three seeds, but the supplied passages do not state a comparative outcome.Source: §E.3 · p. 24, Figure 19 · p. 24

Training curves

• Figure 20 plots aggregated performance over 1M steps on 40 tasks, with IQM of normalized rewards on the y-axis.The figure aggregates DeepMind Control Suite, MetaWorld, and MyoSuite results.Source: Figure 20 · p. 24, F Training Curves · p. 24
• Figures 21–23 report per-task benchmark curves with IQM and 95% confidence intervals: rewards for DeepMind Control and success rates for MetaWorld and MyoSuite.The figures cover 15 DeepMind Control tasks, 15 MetaWorld tasks, and 10 MyoSuite tasks, each run for 1M steps.Source: Figure 21 · p. 25, Figure 22 · p. 25, Figure 23 · p. 26
• BRO achieves better IQM results than other state-of-the-art agents across 40 tasks at 100k, 200k, 500k, and 1M steps, with strong sample efficiency.Table 9 summarizes DeepMind Control Suite, MetaWorld, and MyoSuite results.Source: F Training Curves · p. 24, Table 9 · p. 26
• Figure 24 presents results for different SAC and TD3 variants across 40 tasks.The supplied passage identifies the comparison’s scope but does not state its outcome.Source: Figure 24 · p. 27