PaperlayerView exact revision on arXiv

Source-linked AI summary

SOLAR: A State-Driven Online Learning Rate Scheduler for LLM Pretraining

Qiulin Shang, Binyu Wang, Yongqi Qiao, Songde Rao, Zhoutong Wu, Kun Yuan

arXiv:2609.34681v1cs.LG

TL;DR

Fixed LR schedules cannot adapt to evolving LLM training dynamics, and online learned scheduling remains difficult to operationalize reliably. SOLAR learns bounded, state-dependent corrections around a base schedule; it improves final PPL across dense and MoE settings, with matched controls showing gains from anchoring, bounds, and group-wise control.

  • Problem

    Hand-crafted LR schedules are fixed before training and cannot adapt to evolving optimization dynamics, while online scheduling at LLM scale is difficult to operationalize reliably.

    Source: §1 · pp. 1–2

  • Method

    SOLAR learns bounded, state-dependent group-wise LR corrections that re-anchor to a base schedule, guided by lightweight features and a progress-aware reward.

    Source: §1 · pp. 2–3

  • Results

    Across dense and MoE settings, SOLAR improves final PPL over tuned static schedules and automatic LR tuners; matched controls move global PPO from 27.09 to 23.74 PPL and group-wise control reaches 22.87.

    Source: §1 · pp. 2–3, §6 · p. 10

  • Takeaways & Limitations

    A policy trained on a 60M proxy can be frozen and reused at larger dense scales without target PPO updates or SOLAR-specific retuning, remaining effective across a fourfold base-LR range.

    Source: §6 · p. 10

    Validation is limited to dense models up to 1B parameters and a supplementary 3B MoE setting, leaving behavior at larger scales untested.

    Source: §6 · p. 10

Abstract

from arXiv · show

Learning-rate (LR) scheduling plays a central role in large language model (LLM) pretraining, yet current practice still relies heavily on hand-crafted heuristics such as Warmup-Cosine-Decay and Warmup-Stable-Decay. Because these schedules are fixed in advance, they cannot adapt to evolving optimization dynamics. Online learned scheduling within the Learning to Optimize (L2O) framework offers a dynamic alternative, but remains brittle at LLM scale due to noisy signals, delayed feedback, and the risk of catastrophic divergence. We propose SOLAR (State-driven Online Learning rAte scheduleR), a stabilized framework for reliable online LR adaptation. SOLAR uses a base schedule as a reference and learns bounded, state-dependent residual corrections for individual parameter groups. Each correction re-anchors to the base at every step, allowing the policy to adapt the LR without relearning the warmup-decay profile. A lightweight state representation and progress-aware reward guide online learning, while a Circuit-Breaker restores training after rare unsafe actions. Across autoregressive language-model pretraining, SOLAR improves final perplexity over tuned static schedules and automatic LR tuners for dense models from 60M to 1B, AdamW and Muon, and two MoE settings up to 3B. Matched 130M controls show that adding base anchoring and action bounds improves a global PPO controller from 27.09 to 23.74 final PPL, while group-wise control reaches 22.87 on the same two seeds. A residual policy trained on a 60M proxy can also be frozen and reused at larger dense scales without target PPO updates, remaining effective across a fourfold base-LR range. These results establish SOLAR as a practical learned LR controller for LLM pretraining.

1 INTRODUCTION

SOLAR adapts learning rates during pretraining by learning bounded, state-dependent corrections around a base schedule. Across dense and MoE experiments, it improves final perplexity over static schedules and automatic tuners, while matched controls and proxy transfer test the design’s practical reach.

1 INTRODUCTION

  • Preset schedules cannot adapt to changing optimization dynamics, while online learned scheduling at LLM scale is difficult to operationalize reliably.Training signals are noisy, and scheduling errors can have delayed or abrupt effects.Source: §1 · pp. 1–2

Empirical Demonstration

  • Matched 130M controls move a global PPO controller from 27.09 to 23.74 PPL with base anchoring and action bounds, while group-wise control reaches 22.87 PPL.The comparison uses the same two seeds.Source: §1 · p. 3

Methodological Design

  • SOLAR keeps the base LR profile outside the learned controller and applies bounded, state-dependent group-wise corrections that re-anchor at every step.A progress-aware reward and Circuit-Breaker support policy learning throughout a full pretraining run.Source: §1 · p. 3

Empirical Demonstration

  • SOLAR improves final perplexity over tuned static schedules and automatic LR tuners across dense and MoE models under AdamW and Muon, with repeated gains through 1B.The 1B AdamW result improves PPL by over 10% versus Cosine; the 1B Muon result improves it by about 4% versus Cosine.Source: §1 · pp. 2–3

Practical Scalability

  • A residual policy trained on a 60M proxy can be frozen and reused at larger dense scales without target PPO updates or SOLAR-specific retuning.It remains effective across a fourfold base-LR range.Source: §1 · p. 3

2 RELATED WORK

The paper distinguishes SOLAR from preset schedules, hyperparameter searches, adaptive optimizers, and learned LR controllers. Its distinguishing setup keeps the base profile and optimizer update intact while learning bounded corrections during the target pretraining run.

Learning Rate Scheduling and Hyperparameter Optimization

  • Preset schedules choose warmup and decay in advance, while hyperparameter-optimization methods search profiles or evaluate short candidate segments online.MECHANIC adapts a global multiplier; SOLAR instead learns closed-loop corrections within the live target run.Source: §2 · p. 3

Learned Learning-Rate Controllers

  • SOLAR learns bounded residual corrections within the live target run, unlike prior learned LR controllers that establish state conditioning, transfer, or layer-wise control in other settings.The paper claims SOLAR is the first learned LR scheduler shown to train online throughout the same full LLM pretraining run that it improves.Source: §2 · p. 3

Adaptive and Schedule-Free Optimizers

  • Adaptive and schedule-free optimizers alter updates internally or reduce dependence on an external trajectory, whereas SOLAR leaves the optimizer update unchanged and controls LR across parameter groups.AdamW and Muon experiments test scheduler-level control across distinct optimizer families.Source: §2 · p. 3

Learning to Optimize

  • SOLAR learns a bounded residual around an external schedule rather than replacing the base optimizer’s update rule with a learned optimizer.The paper presents this lower-dimensional learned object as making online acquisition feasible within the target pretraining run.Source: §2 · p. 4

3 PRELIMINARIES

The preliminaries frame LLM pretraining as minimizing loss over parameters partitioned into controlled groups. SOLAR supplies group-specific learning rates to a standard optimizer and treats scheduling as a sequential decision process.

3 PRELIMINARIES

  • Pretraining is formulated as minimizing a loss over parameters with stochastic gradients, partitioned into controlled groups assigned positive learning rates.SOLAR supplies these rates to a standard optimizer such as AdamW or Muon.Source: §3 · p. 4
  • Online LR scheduling is framed as a sequential decision process in which the scheduler observes state and applies an action vector to an optimizer step.The optimizer step determines the next training state and group-wise rewards.Source: §3 · p. 4
  • PPO training uses parameter-shared independent updates across groups, pairing each group’s likelihood ratio with its own advantage.The clipped terms are averaged to update the shared actor–critic.Source: §3 · p. 4

4 SOLAR: STATE-DRIVEN ONLINE LEARNING RATE SCHEDULER

SOLAR combines lightweight global and group-specific states with bounded stochastic LR corrections around a base schedule. It trains from progress and stability rewards and uses a Circuit-Breaker to recover from severe loss spikes.

4 SOLAR: STATE-DRIVEN ONLINE LEARNING RATE SCHEDULER

  • A base schedule supplies warmup and decay, while SOLAR learns bounded residual actions that re-anchor to the base at every step.A Circuit-Breaker restores the last safe state after a severe loss spike.Source: §4 · p. 4

4.1 LIGHTWEIGHT STATE REPRESENTATION

  • The lightweight state representation uses loss statistics, gradient norms, and parameter norms to construct global and local group context.It avoids high-dimensional or optimizer-internal statistics to minimize online scheduling overhead.Source: §4.1 · p. 4

4.2 ACTION SPACE: STOCHASTIC GROUP-WISE RESIDUAL MODULATION

  • SOLAR samples bounded stochastic residual actions with group-specific means and a shared learned scalar variance across controlled groups.The executed action is clipped for numerical safety.Source: §4.2 · p. 5
  • Residual actions are applied as multipliers to the current base LR, with the residual range warmed up from zero during early training.This lets the policy focus on state-dependent corrections instead of relearning warmup and decay.Source: §4.2 · p. 5

4.3 REWARD DESIGN FOR SUSTAINED OPTIMIZATION

  • The reward combines immediate loss improvement, a longer-horizon EMA trend, and group-wise stability shaping.The stability term credits post-update gradient norms below their recent trend and penalizes growth above it.Source: §4.3 · p. 5

4.4 SAFETY AND RECOVERY: CIRCUIT-BREAKER MECHANISM

  • A global emergency trigger terminates unsafe rollouts and restores model and optimizer states from the latest safe checkpoint after severe loss spikes.The truncated trajectory receives an explicit instability penalty, while the updated PPO state is retained.Source: §4.4 · pp. 5–6

4.5 IMPLEMENTATION DETAILS AND OVERHEAD

  • The implementation uses a two-layer MLP actor–critic trained with PPO.The paper separates steady-state step time from full-run wall-clock accounting in Appendix C.14.Source: §4.5 · p. 6

5 EXPERIMENTS

Across dense and MoE pretraining, SOLAR improves final validation perplexity over comparison schedules. The experiments also examine frozen-policy transfer, runtime, and how SOLAR allocates and adjusts learning rates.

5.1.1 DENSE LLAMA 2 PRETRAINING ACROSS OPTIMIZERS

  • At 1B, SOLAR-online lowers AdamW+Cosine PPL from 16.52 to 14.83 and Muon+Cosine from 14.36 to 13.79.Table 1 reports seed-52 final-checkpoint validation perplexity on C4.Source: §5.1.1 · p. 6, Table 1 · p. 7

5.1.2 MOE PRETRAINING UNDER NON-STATIONARY DYNAMICS

  • On Qwen2-MoE 1B, SOLAR improves final PPL from 9.61 to 9.34 while using a higher average LR and producing a lower, smoother gradient-norm trace than Cosine.Figure 3 shows one attention-projection group's effective LR, with a 200-step moving average and step-wise variation.Source: §5.1.2 · p. 7, Figure 3 · p. 8
  • On 1B AdamW, online SOLAR adds 1.23% to full-run wall-clock time and 1.27% to steady-state step time; frozen SOLAR adds 0.76% and 0.81%, respectively.No main-table or MoE run triggers Circuit-Breaker rollback.Source: §5.1.2 · p. 7

5.2 FROZEN RESIDUAL-POLICY TRANSFER ACROSS MODEL SCALES

  • Frozen SOLAR improves over target-tuned Cosine and WSD at every larger scale, with no target PPO updates or SOLAR-specific retuning.It also beats matched Cosine in all six settings across 0.5×, 1×, and 2× target base LRs, completing both 2× runs where Cosine diverges.Source: §5.2 · pp. 7–8

5.3 MECHANISTIC EVIDENCE

  • 27.09 final PPL falls to 23.74 with re-anchoring and action bounds, then to 22.87 with group-wise control (n = 2).The matched controls use seeds 42 and 52.Source: §5.3 · pp. 8–9
  • SOLAR maintains a higher average LR than Cosine, while a normalized static group profile reaches 24.19 PPL versus 22.79 for SOLAR-online on the same seed.AvgLR Replay ends at 25.61 PPL; Figure 4 panels (a) and (b) show average effective LR and single-group LR dynamics.Source: §5.3 · p. 9, Figure 4 · p. 9
  • Vocabulary-facing components receive larger LR multipliers, while internal attention and MLP projections are more constrained.The passage describes this stable module-level allocation as suggesting optimization capacity shifts toward modules that appear more sensitive to LR variation.Source: §5.3 · p. 9
  • Dense transformation matrices show negative LR–GradNorm correlation, while several post-attention LayerNorm parameters show positive correlation.The learned mapping differs across modules rather than applying a uniform inverse function of gradient norm.Source: §5.3 · p. 9, Figure 4 · p. 9

6 CONCLUSION AND LIMITATIONS

SOLAR retains the base schedule’s coarse profile while adapting group-wise corrections to training state; its conclusion reports improved final PPL across dense and MoE settings, with frozen-policy transfer to larger dense scales.

6 CONCLUSION AND LIMITATIONS

  • Matched-seed controls report 27.09 to 23.74 PPL with base anchoring and action bounds, and 22.87 with group-wise control.A 60M-proxy policy also transfers to larger dense scales without target PPO updates or SOLAR-specific retuning, beating matched Cosine runs across the tested fourfold base-LR range.Source: §6 · p. 10

A SOLAR IMPLEMENTATION DETAILS

  • The appendix provides implementation details covering state construction, action parameterization, PPO configurations, and the automated Circuit-Breaker.These details describe the implementation of the scheduler summarized above.Source: A SOLAR IMPLEMENTATION DETAILS · p. 14

A.1 STATE DEFINITION AND FEATURE EXTRACTION

  • The scheduler state concatenates global optimization features with local, parameter-group-specific features.Global features capture overall training dynamics; local features capture tensor-specific statistics.Source: §A.1 · p. 14

A.2 ACTION SPACE, ARCHITECTURE, AND DISTRIBUTED EXECUTION

  • SOLAR uses a lightweight actor-critic policy to produce per-group actions that apply multiplicative residuals to the base learning rate.The action scale warms up linearly for the first 10% of training and has a maximum of 1.3.Source: §A.2 · p. 15

A.3 REWARD DESIGN AND CIRCUIT-BREAKER MECHANISM

  • Each group’s reward combines shared progress signals with stability feedback based on that group’s post-update gradient trend.The progress terms reflect stepwise improvement and the longer-term loss trend; rising gradient growth reduces the stability shaping reward.Source: §A.3 · p. 17
  • A global Circuit-Breaker responds to severe loss spikes, while the outer training loop restores the latest safe model and optimizer state.The mechanism truncates the corrupted trajectory and retains the forced PPO update.Source: §A.3 · pp. 17–18

A.3.1 CONTROLLER COEFFICIENT CENSUS AND REWARD LEAVE-ONE-OUT ANALYSIS

The coefficient census finds small final-PPL variation under one-at-a-time coefficient sweeps, while reward leave-one-out tests show larger penalties when progress terms are removed.

A.3.1 CONTROLLER COEFFICIENT CENSUS AND REWARD LEAVE-ONE-OUT ANALYSIS

  • Coefficient sweeps yield final PPL between 22.80 and 22.85, versus 22.79 at the default; the largest deviation is below the 0.08 seed standard deviation.The largest deviation is 0.06 PPL.Source: §A.3.1 · p. 18
  • Removing immediate progress, trend, and stability raises final PPL by 1.94, 0.90, and 0.30, respectively.Removing stability also produces the only Circuit-Breaker activation observed under nominal hyperparameters.Source: §A.3.1 · p. 18

B.1 DENSE MODEL SETUP

The experiments use a shared training framework and tuned base schedules across dense and MoE settings. The appendix details model and optimizer configurations, scheduler baselines, reporting choices, and the positioning of SOLAR among learned controllers.

B.1 DENSE MODEL SETUP

  • The dense experiments use C4, batch size 512, sequence length 256, and Llama 2 scales from 60M to 1B.Training spans 11K, 20K, 60K, and 100K steps respectively; SOLAR inherits each scale’s selected Cosine base without a separate base search.Source: §B.1 · p. 19, Table 5 · p. 20

B.2 HYPERPARAMETER SELECTION AND FINAL-PPL REPORTING

  • Dense comparisons tune peak learning rate and weight decay on shared grids, select configurations using seed 42, and report final-update validation PPL on seed-52 runs.The common settings include weight decay values from 0.01 to 0.2 and peak learning rates from 10^-4 to 10^-2.Source: §B.2 · p. 20
  • In a 28-point 60M calibration, interim and final rankings correlate at ρ = 0.96 for AdamW and 0.93 for Muon; advancing two configurations retains both final winners.Advancing only the top interim configuration would increase selected final PPL by 0.14.Source: §B.2 · p. 20
  • The one-time SOLAR search costs 7.46 wall-clock hours, while AutoLRS costs 1.98–2.00× its matched baseline.Across both optimizers, pre-submission base searches total 616.91 wall-clock hours and 4,935.3 GPU-hours.Source: §B.2 · p. 20

B.3 DATA, SEEDS, GROUPS, AND RECOVERY STATE

  • The dense implementation controls each trainable tensor as a group, while rollback restores model, optimizer, scheduler, precision, RNG, and data-loader state.Checkpoints are kept every 1K updates; PPO state is retained, followed by a 1K-update cooldown after rollback.Source: §B.3 · p. 21

B.4 MOE MODEL SETUP

  • The Qwen2-MoE setup trains on The Pile for 100K steps, activating four of 32 experts per token, with auxiliary load-balancing and router losses.It uses sequence length 256 and batch size 512.Source: §B.4 · p. 21

B.5 OPTIMIZER CONFIGURATIONS

  • The appendix specifies optimizer configurations and compares tuned scheduler baselines under a shared training setup; experiments use RTX 4090 GPUs up to 1B and A800 GPUs for the 3B MoE run.Warmup-decay baselines generally share a minimum LR ratio of 0.1; hardware assignments are described for the dense and MoE experiments.Source: §B.6 · p. 22, §B.7 · pp. 24–25

C.1 POSITIONING AMONG LEARNED LR CONTROLLERS

  • SOLAR learns during the same full autoregressive LLM pretraining trajectory it aims to improve, unlike prior learned-controller acquisition regimes summarized in Table 8.“Same live run” means policy updates occur inside the final run being improved.Source: §C.1 · p. 25, Table 8 · p. 25

C.2 REPEATED FINAL-CHECKPOINT RESULTS

Repeated runs show SOLAR’s PPL gains across dense scales, while frozen reuse applies a source-trained policy without target PPO updates. Online and frozen modes show different strengths across target settings.

C.2 REPEATED FINAL-CHECKPOINT RESULTS

  • SOLAR is lower in every paired 350M and 1B run, with paired gaps of 1.23/1.08/1.35 PPL at 350M and 1.54/1.69 at 1B.At 130M, online/frozen mean PPLs are 22.86/22.76 under AdamW and 22.05/22.09 under Muon.Source: §C.2 · p. 26, Table 9 · p. 26

C.3 ONLINE ACQUISITION AND FROZEN REUSE

  • SOLAR-online updates the policy inside the target run, whereas SOLAR-frozen reuses a source-acquired policy without target PPO updates.The frozen policy still recomputes group-wise actions from each target state.Source: §C.3 · p. 26
  • Frozen reuse improves from 25.04 PPL at K = 0 to 22.41 at K = 5 source runs.Each source run is a complete online pretraining trajectory.Source: §C.3 · p. 26
  • Both SOLAR modes remain below Cosine in optimizer-matched rows; frozen reuse has lower PPL on matched C4 targets, while online is lower under corpus and optimizer changes.The repeated-results table reports final PPL by seed, with frozen runs reusing one source-trained policy across target seeds.Source: Table 9 · p. 26, §C.3 · p. 27

C.4 MATCHED LEARNED-CONTROLLER COMPARISONS

Matched controls probe SOLAR’s anchored, bounded residual interface, group-wise actions, and learned state-conditioned control. The comparisons also expose failure modes for compounding LR actions and sensitive local updates.

C.4 MATCHED LEARNED-CONTROLLER COMPARISONS

  • On common seeds 42 and 52, anchoring each bounded multiplier to the base improves 27.09 to 23.74 PPL; moving from one global action to 111 group actions reaches 22.87.The third SOLAR run yields a three-run mean of 22.86 ± 0.08; Table 12 reports matched control means and three-run static-baseline summaries.Source: §C.4 · p. 27, Table 12 · p. 27
  • Under frozen transfer from 60M to 130M, SOLAR-frozen reaches 22.76 mean PPL, compared with 24.95 for the adapted GANNO controller.The reported gap is 2.19 PPL between the two LLM controller adaptations.Source: §C.4 · p. 27
  • Replacing SOLAR’s residual action with a target-trained absolute-LR action changes PPL from 22.79 to 24.74, while state-blind and near-deterministic variants reach 25.21 and 25.03.AvgLR Replay reaches 25.61 at the final checkpoint.Source: §C.4 · p. 27
  • N3’s group-wise residual with a myopic hypergradient rule does not recover the same gain as the learned state-conditioned loop.N1–N3 isolate individual design axes; N4 is a complete competing controller adapted to the LLM setting.Source: §C.4 · p. 27
  • In an α = 2.6 stress run, the compounding-action controller reaches a 5.7× mean LR multiplier and 79.58 PPL without a loss spike; a one-decade hypergradient-coefficient change shifts screened PPL from 31.05 to 47.31.The stress run has n = 1.Source: §C.4 · pp. 27–28

C.5 FROZEN-POLICY ROBUSTNESS AND µP COMPATIBILITY

Frozen SOLAR policies are evaluated across target base rates, parameterizations, and corpora, alongside replay and state ablations. The results show robustness across several tested shifts and distinguish learned state-conditioned control from open-loop replay.

C.5 FROZEN-POLICY ROBUSTNESS AND µP COMPATIBILITY

  • The frozen policy beats same-base Cosine in all six target-LR settings and completes both 2× runs without target PPO updates.At 130M, its negative-action fraction rises from 0.29 to 0.73 as the base LR moves from 0.5× to 2×.Source: §C.5 · p. 28
  • SOLAR improves each paired µP run by 1.57–1.59 PPL online and 1.79–1.82 frozen, while µP+Cosine reaches 24.49 PPL versus 24.50 for the searched SP base.The frozen configuration performs neither LR/WD search nor PPO updates at the 130M target.Source: §C.5 · pp. 28–29, Table 15 · p. 28
  • On The Pile, online has a 0.22 lower mean PPL than frozen across seeds 42 and 52; both modes remain below matched Cosine baselines.On the seed-52 C4 run, frozen is 0.38 PPL lower than online.Source: §C.5 · p. 29, Table 16 · p. 29

C.6 STATIC AND OPEN-LOOP REPLAY CONTROLS

  • SOLAR-online reaches 22.79 PPL, compared with 24.19 for static group allocation, 24.63 for level-corrected scalar replay, and 25.61 for AvgLR Replay.Static group allocation improves Cosine from 24.52 to 24.19 PPL.Source: §C.6 · p. 29, Table 17 · p. 29
  • At 2.0× base LR, SOLAR-frozen completes training at 22.68 PPL, while Cosine and both open-loop replays diverge.The learned policies remain below both replay controls across the tested seed, base-LR, and corpus shifts.Source: §C.6 · p. 30, Table 18 · p. 30

C.7 IMPORTANCE OF GLOBAL AND LOCAL STATE FEATURES

  • Combining global and local state features gives 22.79 PPL, compared with 23.11 for global-only and 23.57 for local-only state.The ablation is on the 130M model.Source: §C.7 · p. 30, Table 19 · p. 30

C.8 STOCHASTIC EXPLORATION VS. NEAR-DETERMINISTIC SCHEDULING

  • Suppressing stochasticity increases final PPL from 22.79 to 25.03, consistent with stochastic sampling improving exploration under delayed feedback.The near-deterministic comparison retains the same PPO update and residual action parameterization.Source: §C.8 · p. 30, Table 20 · p. 31

C.9 RESIDUAL LR MODULATION VERSUS DIRECT ABSOLUTE LR PREDICTION

SOLAR modulates a base learning-rate schedule with bounded residual actions, while Direct-Abs predicts the absolute rate. In the matched 130M comparison, the residual-and-warmup design achieved lower final perplexity.

C.9 RESIDUAL LR MODULATION VERSUS DIRECT ABSOLUTE LR PREDICTION

  • Direct-Abs maps a bounded action to an absolute LR, leaving it responsible for both the rate’s scale and its time evolution from training’s start.The action is mapped in log-space, with ηmin set to 0.1ηmax and ηmax selected from the same seven-point LR grid as other schedulers.Source: §C.9 · p. 31
  • Early miscalibrated Direct-Abs actions can trigger aggressive updates and loss spikes that are often difficult to recover from.The passage links these disruptions to degraded final performance.Source: §C.9 · p. 31
  • SOLAR multiplies the base LR by a bounded, state-dependent adjustment and warms the residual strength from 0 to α.The warmup limits residual perturbations before the scheduler becomes reliable.Source: §C.9 · p. 31
  • 22.79 final PPL for residual modulation with warmup, compared with 24.74 for Direct-Abs under matched LR search.The comparison is reported for the 130M model.Source: Table 21 · p. 31, §C.9 · p. 31

C.10 ROBUSTNESS TO SUBOPTIMAL BASE LEARNING RATES

The supplementary evaluations test SOLAR’s response to suboptimal base LRs, unsafe exploration, and a different MoE stack, alongside policy controls and runtime accounting. The results include a lower final PPL than WSD in the single-run 3B MoE comparison.

C.10 ROBUSTNESS TO SUBOPTIMAL BASE LEARNING RATES

  • Across the 130M peak-LR sweep from 10^-4 to 10^-2, SOLAR remained trainable over a wider observed interval and used negative residual actions at high base LRs.The reported response lowers affected groups’ effective LRs while retaining the long-horizon base profile.Source: §C.10 · pp. 31–32, Figure 5 · p. 32

C.11 EFFICACY OF THE AUTOMATED CIRCUIT-BREAKER MECHANISM

  • Under α = 1.8 stress testing, an aggressive action triggered a severe loss and gradient-norm spike, but the resumed trajectory completed without another severe spike.The Circuit-Breaker rolls back to the last safe checkpoint after detecting the anomaly, then resumes training.Source: Figure 6 · p. 32, §C.11 · p. 33

C.12 ADDITIONAL VALIDATION ON THE DEEPSEEK-V2 3B MOE MODEL IN MEGATRON

  • 10.38 final PPL for SOLAR versus 10.73 for WSD on DeepSeek-V2 3B MoE in Megatron.Each method ran once, so the comparison does not estimate run-to-run variance.Source: Figure 7 · p. 33, §C.12 · p. 33

C.13.1 STATE-AGNOSTIC STOCHASTIC SCHEDULER

  • 25.21 PPL for the state-agnostic stochastic scheduler, compared with 22.79 for SOLAR.The control retains the stochastic residual family but replaces the learned state-conditioned action with state-independent actions.Source: §C.13.1 · p. 34

C.13.2 UNTRAINED FIXED-POLICY CONTROLLER

  • 25.04 PPL for the untrained fixed-policy controller, compared with 22.79 for SOLAR-online.The fixed-policy control performs no PPO updates; the control experiments use the same 130M Llama 2 AdamW setup and training budget.Source: §C.13 · p. 33, §C.13.2 · p. 34

C.14 RUNTIME ACCOUNTING

  • 1.23% full-run overhead on 1B AdamW; online overhead ranges from 0.99% to 1.48% across the listed complete runs.On 1B AdamW, measured step-time increases were 1.27% online and 0.81% frozen; full-run accounting includes evaluation, checkpointing, and SOLAR overheads.Source: Table 24 · p. 34, Table 23 · p. 34, §C.14 · p. 35
Loading 2609.34681v1…