PaperlayerView exact revision on arXiv

Source-linked AI summary

Temperon: Full-Time SAM Quality at a Third Less Wall-Clock

Stamatis Mastromichalakis

arXiv:2609.17575v1cs.LGcs.CV

TL;DR

SAM improves training outcomes but doubles the cost of every touched step, motivating evidence about where its expense should be allocated. Temperon uses a plain-SGD explorer followed by a scheduled SAM-wrapped Muon tail that owns the final anneal. It matches the best full-time-SAM accuracy on four vision datasets, reaches the hardest common target about one-third sooner on three, transfers to GPT-2 and GLUE, and is bounded by tasks where Muon buys no accuracy tier.

  • Problem

    SAM doubles the cost of every touched training step, while its benefit is concentrated near the end of training.

  • Method

    Temperon uses a plain-SGD explorer for the first 43% of the budget, then hands off once to a SAM-wrapped Muon refiner for the final cosine anneal.

  • Results

    Temperon matches the best full-time-SAM accuracy on all four vision datasets and reaches the hardest common target 35%, 34%, and 32% sooner on three.

  • Takeaways & Limitations

    The allocation law transfers to GPT-2 pretraining and GLUE fine-tuning, while the measured economics make the time-versus-accuracy exchange predictable before training.

  • Takeaways & Limitations

    Evidence is limited in scale: vision uses one 27.6M CNN, and the language-model result is a 124M single-seed pilot with a measured noise floor.

Abstract

from arXiv · show

Sharpness-aware minimization (SAM) doubles the cost of every training step, yet its benefit concentrates where training ends. We study where an expensive training mode should be spent and propose Temperon: a plain-SGD explorer for the first 43% of the epoch budget, then one scheduled hand-off that gives the entire final cosine anneal to a SAM-wrapped Muon refiner. On CIFAR-10/100, SVHN and Tiny ImageNet (five seeds, times reported as epochs-to-target times an idle-GPU-calibrated epoch cost), Temperon matches the best full-time-SAM recipe on accuracy everywhere while reaching the hardest common target 35%, 34% and 32% sooner on three of the four, and sits a tier above the published SAM+SGD recipe at level cost. Ablations make the attribution exact: the Muon refiner is worth +0.85pp with everything else fixed; the explorer's shape and its restarts are worth nothing, and we withdraw them as contributions. Re-running the closest rival, late-phase SAM, at matched budget shows the frontier: it is fastest to every mid-level target, but the tier the Muon refiner buys (0.83 on CIFAR-100, 0.97 on CIFAR-10) is reached by no SGD-refined method in any seed, and on Tiny ImageNet, where Muon buys no tier, the rival simply wins -- the measured boundary of the method. The allocation law transfers to GPT-2 pretraining (full-SAM quality at -29% wall-clock) and GLUE fine-tuning (never worse than full-time SAM at a third of its SAM cost). Two constants organize the economics: skipping SAM early buys a fixed credit, and a Muon epoch costs 1.50x a SAM+SGD epoch on all four datasets. Finally, the hand-off cannot be timed from the trajectory: under cosine schedules the accuracy curve is plateau-then-surge, so the information lives in the schedule, making the scheduled switch principled rather than convenient. Code and a pip-installable implementation are released.

1 Introduction

Temperon treats expensive training as an allocation problem: use the expensive mode only in the tail and let it own the final anneal. The recipe combines a plain-SGD explorer, one scheduled hand-off, and a task-specific refiner, with measured cost and attribution evidence.

  • Motivation: SAM doubles touched-step cost and Muon costs 2.4× an SGD epoch, while full-time SAM+Muon can plateau during the middle of training.This motivates concentrating the surcharge near the end, where the remaining progress arrives with the anneal.
  • Contribution: Temperon uses a scheduled hand-off from cheap exploration to an expensive tail that owns the entire final learning-rate anneal.The switch occurs where the last learning-rate descent begins, never inside it.
  • Attribution: The Muon refiner raises accuracy by +0.85pp, whereas the explorer’s allocation shape and warm restarts do not contribute.The paper explicitly withdraws the schedule shape and restart pattern as contributions.
  • Method: The reference recipe explores with plain Nesterov SGD for 43% of the budget, then gives the remaining 57% to SAM-wrapped Muon under a fresh cosine anneal.The incoming optimizer inherits momentum and uses warmup and perturbation-radius ramps to avoid a shock at hand-off.
  • Economics: Three datasets pay a 1.50× Muon-tail premium from the same early-SAM credit and reach −2% net cost, while Tiny ImageNet keeps the credit with an SGD tail.The exchange is measured using idle-GPU-calibrated epoch costs.

3 Experimental setup

The evaluation re-runs vision baselines in one five-seed pipeline with calibrated timing, and extends the allocation study to language-model pretraining and transformer fine-tuning. Headline vision timing is reported as calibrated time to the hardest common target.

  • Vision setup: Vision experiments use a 27.6M-parameter wide ResNet-110 for 100 epochs, with identical five seeds and re-run baselines inside one pipeline.The datasets are CIFAR-10/100, SVHN, and Tiny ImageNet.
  • Timing: Calibrated time equals epochs-to-target multiplied by optimizer- and dataset-specific seconds per epoch measured on an idle GPU.This avoids workstation load affecting method comparisons while retaining raw wall-clock in released logs.
  • Transfer experiments: The language-model evaluation uses GPT-2 124M on WikiText-103 with a 400M-token budget and one seed, while GLUE fine-tuning uses RoBERTa-base on four tasks with five seeds.The GPT-2 run includes a measured noise floor; GLUE uses best dev score fixed before results were seen.
  • Headline measurements: Table 1 reports five-seed mean ± standard deviation for final test accuracy and calibrated time to the hardest target any method reaches, including the fraction of seeds reaching it.Bold marks the best accuracy and time per row, and one row highlights the dataset where the rival wins outright.
  • Headline measurements: Figure 2 compares four methods across four datasets using calibrated target times, with annotations for seconds and seed reach fractions.Its stated comparison is that the rival is fastest to every mid target, while later analysis identifies what it cannot reach.

4 Vision results

Temperon matches the best full-time-SAM accuracy on all four vision datasets and reaches the hardest common target about one-third sooner on three. SVHN is the saturated exception where full-time SAM+Muon crosses the target during its plateau.

  • Main result: Temperon ties the best full-time-SAM recipe on final accuracy across all four datasets and reaches the hardest common target 35%, 34%, and 32% sooner on three.It does not win on time on SVHN, where the target lies below full-time SAM+Muon’s mid-run plateau.
  • Baseline comparison: Against published SAM+SGD, Temperon is ahead on accuracy on three datasets by +0.63pp, +0.27pp, and +0.09pp at level cost, while staying within 3% on all four.The comparison is attributed to the measured exchange between early SAM credit and the tail premium.
  • Allocation comparison: Uniform periodic SAM reaches only 0.8183 ± 0.0016 on CIFAR-100, 1.12pp below the hand-off despite using an equal-or-greater budget.The reported difference is statistically significant at 𝑝=0.001.

5 What earns the accuracy: attribution

The Muon refiner, not the explorer’s schedule or restarts, determines the accuracy tier. Controlled ablations attribute the gain to the refiner and withdraw the earlier shape claim.

  • Attribution: Changing the explorer’s shape does not improve accuracy and is 0.25pp worse than a plain mid-schedule switch under an SGD refiner.The comparison requires seven additional epochs to reach 0.82.
  • Attribution: Removing cyclic restarts leaves final accuracy indistinguishable from a single cosine schedule.The reported test gives p=0.61, so restarts are not an accuracy contribution.
  • Attribution: The paper withdraws earlier claims that the allocation shape contributed to accuracy.The attribution is limited to the Muon refiner; the schedule shape and restarts are not claimed as contributions.
  • Attribution: +0.85pp separates SGD- and Muon-refined CIFAR-100 tiers when only the refiner changes.The SGD tier is 0.8210–0.8234, while the Muon tier is 0.8285–0.8295.

6 The closest rival, measured

Late-phase SAM is the fastest route to every mid-level target, while Temperon’s Muon refiner reaches higher tiers that SGD-refined methods do not. On Tiny ImageNet, where Muon adds no tier, the rival wins outright.

  • Rival comparison: The rival independently reproduces or exceeds full-time SAM+SGD across all four datasets at a matched 57-epoch SAM budget.It is within noise on CIFAR-100, Tiny ImageNet, and SVHN, and is +0.20pp on CIFAR-10.
  • Rival comparison: Late-phase SAM reaches every mid target fastest: 34% sooner on CIFAR-100, 36% on CIFAR-10, 10% on Tiny ImageNet, and slightly faster on SVHN.If the mid target is sufficient, the paper recommends late-phase SAM rather than Temperon.
  • Rival comparison: No late-phase SAM seed reaches 0.83 on CIFAR-100 or 0.97 on CIFAR-10, whereas Temperon reaches those Muon-enabled tiers.Temperon reaches 0.83 in 3/5 CIFAR-100 seeds and 0.97 in 2/5 CIFAR-10 seeds.
  • Rival comparison: On Tiny ImageNet, late-phase SAM matches Temperon’s final accuracy and reaches 0.68, 0.69, and 0.70 13%, 10%, and 7% sooner.The rival’s final accuracy is 0.7020 versus Temperon’s 0.7003, with p=0.42.
  • Boundary: The Tiny ImageNet boundary occurs because Muon provides no tier there, leaving Temperon’s tail equivalent to plain SAM+SGD.Full SAM+Muon reaches 0.6838, 1.8pp below both methods; Temperon retains only the explorer’s early head start.

7 Transfer: GPT-2 pretraining

The allocation law transfers beyond vision: a late SAM tail under Muon matches full-time SAM quality in GPT-2 pretraining at lower wall-clock and remains no worse than full-time SAM in GLUE fine-tuning. The transfer has important scope limits because SAM’s benefit depends on radius and data regime.

  • GPT-2 pretraining: GPT-2 pretraining matches full-time SAM within the noise floor at 29% lower wall-clock and beats it by 0.063 nats at equal wall-clock.The transfer uses Muon with SAM switched on for the final 30% of steps under a WSD schedule.
  • Scope: The GPT-2 result is not evidence that SAM itself helps every regime: full-time SAM gains only 0.008 nats over plain Muon for 73% more wall-clock and worsens validation loss under data repetition.With 20 passes over a 20M-token slice, SAM gives 5.196 versus 5.014 validation loss for no-SAM.
  • GLUE fine-tuning: At ρ=0.05, the measured GLUE cost is 1.28× no-SAM for the tail versus 1.97× for full-time SAM, a 36% wall-clock reduction.These ratios are close to the designed 1.30× and 2.00× costs.

8 Transfer: GLUE fine-tuning

GLUE fine-tuning supports a narrower transfer claim: late SAM allocation is never worse than full-time SAM at one-third of its SAM cost, while apparent gains depend on perturbation radius.

  • GLUE fine-tuning: Full-time SAM at ρ=0.05 is harmful on all four tasks, so the tail’s apparent advantage partly reflects avoiding that harm rather than improving performance.Smaller radii stop the damage but do not turn it into a gain.
  • GLUE fine-tuning: At a corrected perturbation radius, the late-phase tail is never worse than full-time SAM on any GLUE task while paying SAM on 30% of steps.The stronger edge at ρ=0.05 shrinks at ρ=0.02 and disappears on the noisiest task.
  • GLUE fine-tuning: The robust conclusion is equivalence with full-time SAM at a third of the SAM cost, not a universal accuracy improvement.At the corrected radius, an accuracy edge survives only on MRPC.

9 Why the switch is scheduled

The hand-off is scheduled because cosine-trained accuracy can plateau before a later surge that trajectory-based rules cannot anticipate. Offline replay therefore favors an anneal-aligned boundary, while refiner selection requires information beyond a single run.

  • Why the switch is scheduled: Under cosine schedules, backward-looking stopping rules mistake the mid-run plateau for exhaustion and switch before the unseen accuracy surge.Vision rules fired at epochs 16–35 instead of the known-good 43, with cross-seed spreads up to ±33 epochs.
  • Why the switch is scheduled: The stopping rule works only on WSD, where the stable phase genuinely has diminishing returns, firing at 0.61–0.70 versus the manual 0.70.This contrasts with cosine schedules, whose plateau does not reliably signal the coming surge.
  • Why the switch is scheduled: Muon’s tier cannot be predicted reliably within a run; a practical refiner choice instead uses a baseline run plus a 43%-budget probe.The hand-off plateau ratio separates Tiny ImageNet, but its denominator is unavailable to a single-run online decision.
  • Why the switch is scheduled: The switch’s useful information lives in the schedule’s future rather than the trajectory’s past, making the anneal-aligned hand-off the method itself.Switching inside the decay loses, whereas the schedule-aligned hand-off wins.

10 Honest scope

The method’s scope is bounded by cheap-method ceilings, task saturation, refiner usefulness, data regime, and limited experimental scale. These boundaries identify when Temperon’s allocation does not provide an advantage.

  • Honest scope: Temperon does not accelerate targets already reachable by cheaper methods: plain SGD reaches 0.80 on CIFAR-100 in 1567s versus Temperon’s 4416s.The method is intended for targets above the cheap optimizer’s ceiling.
  • Honest scope: Late-phase SAM reaches mid targets sooner on every dataset, while Temperon’s advantage begins above SGD-refined ceilings such as 0.8234 on CIFAR-100 and 0.9688 on CIFAR-10.No arm without a Muon refiner crosses those ceilings.
  • Honest scope: On Tiny ImageNet, where Muon buys no tier, the simpler rival wins outright; on saturated SVHN, the target lies below the expensive method’s plateau and needs no anneal.These are measured boundary cases for the allocation.
  • Honest scope: Twenty passes over a 20M-token slice made SAM worse rather than better, showing that data scarcity is not equivalent to task difficulty.
  • Honest scope: The evidence spans regimes but not scale: vision uses one 27.6M CNN, while language-model evidence is a 124M single-seed pilot with a measured noise floor.The transfer evidence is diversity of regime, not large-scale validation.

11 Related work

Prior work motivates late SAM allocation, optimizer hand-offs, and SAM–Muon pairing, while Temperon distinguishes itself by measuring when the expensive mode should be paid rather than changing its geometry or switching trigger.

  • Related work: Zhou et al. establish that SAM’s generalization benefit concentrates late, providing the closest precedent for Temperon’s tail allocation.Temperon re-runs their late-phase method at matched budget and extends the question to Muon and other regimes.
  • Related work: SWATS supplies a hand-off skeleton and trajectory-based re-entry machinery, but it switches Adam to SGD rather than allocating an expensive training mode.Temperon’s scheduled boundary addresses a different allocation problem.
  • Related work: Concurrent SAM–Muon work studies perturbation geometry, whereas Temperon studies when to pay for the expensive mode; the questions compose because the allocation law is geometry-agnostic.The concurrent result independently supports the strength of the pairing beyond the scales measured here.

12 Conclusion

Temperon places expensive training modes in a tail owning the final anneal, achieving full-time-SAM quality more cheaply across multiple regimes while exposing clear economic and empirical boundaries. The paper attributes gains to the Muon refiner, identifies settings where the rival wins, and leaves scale and generality open.

  • Temperon reaches full-time-SAM quality at roughly a third less wall-clock across vision pretraining, GPT-2 pretraining, and GLUE fine-tuning.Reported examples include GPT-2 at −29% wall-clock and GLUE at −36%.
  • Skipping SAM early buys a fixed credit, while a Muon tail costs a flat 1.50× premium against a SAM+SGD epoch.The practitioner can convert the credit into time or accuracy.
  • The Muon refiner provides the accuracy contribution, while the allocation shape and restarts do not.The ablations support withdrawing the allocation shape and restarts as contributions.
  • Late-phase SAM is faster to every mid-level target and wins Tiny ImageNet, so Temperon benefits only when the expensive refiner buys an unreachable accuracy tier.These boundary measurements define when practitioners should not use the recipe.
  • The evidence is limited to a 27.6M CNN, a 124M language model, and RoBERTa-base, leaving larger-scale generality untested.ImageNet-class vision and larger language models are identified as obvious next measurements.
Loading 2609.17575v1…