Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers

• Stochastic-gradient noise can drive Brownian-like weight expansion, which weight decay counteracts by producing an equilibrium whose weight norm scales with learning rate and weight decay.Source: §1 · pp. 1–2
• Learnable multipliers reparameterize matrix weights so their scale can adapt to training data instead of being fixed by the noise-WD equilibrium.The scalar example replaces W with sW, where s is learnable.Source: §1 · p. 2
• The paper identifies architecture symmetries, width scaling, and interactions with gradient clipping as practical issues for stable and effective multiplier use.Source: §1 · p. 2
• Learnable multipliers maintain an increasing performance gap over the baseline throughout a longer end-to-end pretraining run, supporting richer model representations.The paper also reports that tuned learning-rate multipliers remain important, while tuned forward and weight-decay multipliers are not needed for the same performance level.Source: §1 · p. 2
• Experiments apply multipliers to attention, SSM, and MLP residual blocks in a hybrid architecture, with similar performance gains and behavior patterns using Adam and Muon.Source: §1 · p. 2

• A scalar multiplier lets the effective matrix norm adapt to the data even while the learnable matrix weight remains subject to the noise-WD equilibrium.Source: §2 · p. 3
• Row and column multipliers separately learn the scales of individual matrix rows and columns, extending scale freedom beyond a single whole-matrix multiplier.Source: §2 · p. 3
• Scalar multipliers aggregate gradients across the whole effective-matrix gradient, while row and column multipliers aggregate across respective columns or rows, reducing multiplier gradient noise.The passage describes this averaging as an intuitive explanation for why multipliers do not experience the same noise-driven Brownian expansion requiring weight decay.Source: §2 · p. 3
• RMSNorm weights in Pre-LN architectures already act like column multipliers for a block's first linear layer, while multipliers in remaining parts can further improve performance.Source: §2 · p. 3
• Log-scale multiplier parameterization can offer at most a slight performance advantage but poses stability issues.Source: §2 · p. 3
• Using both row and column multipliers indiscriminately can be redundant, and redundant multipliers may create parameter symmetries that lead to NaNs during training.Source: §2 · p. 4
• Multipliers can be merged into effective matrices for inference without memory or latency overhead; training can use explicit reparameterization or manually handled multiplier dynamics.The explicit approach is expected to reduce throughput by at most a couple of percent, while optimizer-level handling can further reduce that drop.Source: §2 · p. 4
• Scalar learnable multipliers are a learnable version of muP multipliers that reduce the need to tune forward and weight-decay multipliers, but require separate analysis of model-size scaling.The source gives an example of prior tuning involving 12 forward, 16 learning-rate, and 7 weight-decay multipliers.Source: §2 · p. 4

3. What is learned by the multipliers?

• With a frozen projector norm, performance drops at extreme equilibrium scales; scalar and vector multipliers keep logit norms stable while compensating for projector-norm changes.Across configurations, projector weights follow noise–WD equilibrium scaling, except when optimization pressure pulls them away at small equilibrium scales.Source: §3 · p. 6
• Projector-weight trajectories match Adam Brownian Motion under strong weight decay and nearly match under weak decay, whereas multiplier trajectories do not resemble their Brownian-motion counterparts.The same pattern holds for final norm values.Source: §3 · p. 6
• Without learnable multipliers, MLP loss degrades at larger matrix norm scales; the authors attribute this to a magnitude mismatch with attention/SSM blocks, with similar behavior under Adam and Muon.This suggests the noise–WD equilibrium trap is not specific to Adam.Source: §3 · p. 6

3.1 Features scale diversity

• Residual-block output norms trend upward toward later layers, while attention scales in the second half and SSM dt-projection scales vary across layers.The authors suggest later layers may benefit from larger residual contributions and that dt-projection variation reflects specialization in temporal scales.Source: §3.1 · pp. 6–7, Figure 2 · p. 7
• Without vector multipliers, row norms of attention/SSM input and MLP gate projections have a narrow distribution; adding them broadens the within-layer norm distribution.The authors suggest greater diversity of internal feature scales benefits the model.Source: §3.1 · p. 7, Figure 3 · p. 7

4.1 Symmetries

• Architecture symmetries can let parameter scales drift without changing model output, causing large activations and eventual divergence; small multiplier weight decay is a simple effective countermeasure.The paper identifies multiplicative and normalization symmetries and uses λ_lrm = 2 × 10^-3 to handle them.Source: §4.1 · pp. 8–9

4.2 Scaling with model width

• Matrix norms stay nearly constant across widths, so the width-independent equilibrium breaks LR-only μP scaling by causing output activations to grow with width.The paper connects this breakdown to omitted weight-decay scaling, and describes moving the scaling to a learnable multiplier as an alternative.Source: §4.2 · pp. 9–10
• Projector, attention QK, and SSM dt activation norms remain stable across widths, while learned multipliers scale slightly slower than μP predictions as input-weight alignment decays.The paper notes that the alignment decay contradicts the standard μP limit and leaves further investigation beyond its scope.Source: §4.2 · pp. 10–11, Figure 5 · p. 10

4.3 Gradient clipping

• Including multiplier gradients in global clipping can shrink all parameter updates excessively; excluding them mitigates the issue and improves performance.Multiplier gradients initially push the global norm above the clipping threshold even when the norm excluding them is below it.Source: §4.3 · p. 11, Figure 6 · p. 11

4.4 Learning the projector scale

• Adding per-logit projector row multipliers degrades performance, consistent with a shortcut that fits marginal token frequencies without updating internal features.Removing projector column multipliers and sweeping a fixed scalar also failed to yield the expected improvement.Source: §4.4 · p. 11, Figure 7 · p. 12

5.1 Multiplier tuning ablation

• Learnable multipliers improve every parent configuration, with the strongest gains when learning-rate multipliers are tuned; tuned forward and weight-decay multipliers are unnecessary for their performance.The authors speculate that forward and weight-decay tuning transfers poorly because the tuning and training data mixes differ.Source: §5.1 · p. 12

5.2 Long training validation

• 1.21% for Adam and 1.10% for Muon in average benchmark score: learnable multipliers provide similar gains over each optimizer’s baseline.Muon itself scores 31.88% versus Adam’s 30.80%.Source: Table 2 · p. 13, §5.2 · p. 13

5.1 Multiplier tuning ablation

• The loss gap from learnable multipliers grows during the constant-learning-rate stage and shrinks slightly during learning-rate decay.The authors suggest the shrinking may reflect reduced noise and a less restrictive noise–weight-decay equilibrium.Source: §5.1 · p. 13

5.2 Long training validation

• Improvements span capabilities, with modest gains on knowledge benchmarks and larger gains on reasoning benchmarks, though the authors say this pattern needs comprehensive investigation.The cited benchmarks include ARC-C and MMLU for knowledge, and BBH, MATH lvl5, and GSM8K for reasoning.Source: §5.2 · p. 13

Conclusion

• The noise–weight-decay equilibrium reduces the scale diversity of internal representations, and learnable multipliers are added to suitable model locations to address it.The multipliers adjust matrix-layer scales to the training data and support richer representation scales; the paper reports sizable improvement in long pretraining without added inference speed or memory cost.Source: §6 · pp. 13–14

Open questions

• The approach relies on matrix dynamics being noise-dominated and scalar/vector dynamics being signal-dominated, but the boundary between those regimes remains unexplained.The authors identify a continuous spectrum of gradient signal-to-noise ratios and call for a measurable noise indicator or a model of both regimes.Source: §6 · p. 14
• A complete scaling theory for learnable multipliers remains open, including whether their learning rates and weight decay should scale with model size.The authors also ask whether multipliers automatically ensure maximal feature learning at infinite width without manual muP scaling rules.Source: §6 · p. 14
• How multiplier benefits vary across capabilities and how their stability and performance scale with model size require further study.The authors raise as a hypothesis whether learned multipliers enhance only certain model circuits.Source: §6 · p. 14
• The authors frame unlearned matrix scale as one training flaw and leave open whether other unlearned matrix structure also prevents reaching population-loss optima.They ask whether standard training has other correctable flaws, including matrix properties beyond row and column norms.Source: §6 · p. 14

A. Experiment settings

• Falcon-H1-0.5B was chosen for its 35 extensively tuned muP multipliers, moderate compute cost, and available hybrid attention–SSM infrastructure.The tuned multipliers provide a strong baseline for learnable multipliers.Source: A. Experiment settings · p. 19
• Most experiments run for 30GT, while final validation runs for 240GT to examine multiplier behavior over longer training.The 30GT schedule comprises 25GT at constant learning rate and 5GT of exponential decay with an eightfold learning-rate reduction; the authors describe it as sufficient for equilibrium stabilization and three times Chinchilla compute-optimal duration.Source: A. Experiment settings · p. 19
• Training follows prior hyperparameter settings, including global weight decay 0.1 and learning-rate sweeps for each multiplier configuration.Runs use 0.1GT warmup, square-root learning-rate batch-size rampup, and tuned weight-decay multipliers; z-loss is disabled for the projector experiment to avoid confounding interpretation of logit norms.Source: A. Experiment settings · p. 19
• The width-scaling experiment uses a 12-layer model and frozen RMSNorms to support comparisons across a wider range of widths.Freezing RMSNorms maintains unit-norm block inputs.Source: A. Experiment settings · p. 19

B. MLP experiment

• Learnable MLP multipliers remove performance degradation as MLP matrix scale varies, while their compensation keeps residual-block outputs constant.Figure 8 includes the performance comparison and norm behavior across the tested configurations.Source: B. MLP experiment · pp. 19, 21, Figure 8 · p. 20
• Figure 9 shows strong layer-specific patterns, while its layer-averaged measures in Figure 8 are described as adequately capturing norm behavior with MLP scale.Figure 8 varies MLP norm scale; its subplots share y-axis scales within rows, and its bottom-right plot shows performance across four configurations.Source: Figure 8 · p. 20
• The MLP matrix norms follow the equilibrium scale as the experiment varies learning rate and weight decay while keeping their product constant.RMSNorm weights are fixed to restrict scale adaptation, and learnable MLP multipliers are added in the multiplier configuration.Source: B. MLP experiment · pp. 19, 21
• Without multipliers, the MLP scale change creates an expected scale imbalance and loss gap, but optimization pressure also realigns residual-block scales.The MLP matrix norms deviate slightly from noise–weight-decay scaling, consistent with pressure to reduce the scale gap.Source: B. MLP experiment · p. 21
• Attention outputs grow with MLP scale alongside attention matrix norms, with alignment and escape from noise–weight-decay equilibrium proposed as contributing mechanisms.The authors suspect both mechanisms contribute; they note that attention-output growth lags SSM growth and may reflect a quality cost.Source: B. MLP experiment · pp. 21–22
• SSM outputs grow despite stable SSM matrix norms, with vector-shaped conv1d and skip-connection D parameters acting as learnable multipliers.The observed growth of these parameters accounts for output growth at fixed matrix-layer norm.Source: B. MLP experiment · p. 22
• Embedding matrix norm remains constant while its output norm rises as frequent-token vectors partially escape equilibrium and low-frequency vectors remain trapped.The embedding output norm averages token-vector norms weighted by token frequency.Source: B. MLP experiment · p. 22
• Adam and Muon show nearly identical behavior across most examined aspects, with moderate differences in attention, conv1d, and D growth.The authors suggest the noise–weight-decay equilibrium may apply across a wide range of optimizer update rules.Source: B. MLP experiment · p. 22

C. Multipliers placement

• The proposed symmetry-aware placement removes multiplier redundancy without reducing scale-adaptation expressivity, with principles the authors believe can generalize beyond their hybrid architecture.The section examines Falcon-H1, and Figure 9 complements Figure 8 by showing layer-specific residual-block output norms.Source: C. Multipliers placement · pp. 22, 24, Figure 9 · p. 23
• Symmetry-aware placement performed slightly worse than using all multipliers, and residual-normalization symmetry had no reliable fix.The experiments therefore use both row and column multipliers and weight decay to handle symmetry; the authors still consider symmetry-aware placement useful for understanding multiplier effects.Source: C. Multipliers placement · p. 24

C.1 Embedding and projector

• Removing embedding multipliers results in suboptimal performance, possibly because backbone blocks dominate the residual and may ignore the fixed embedding scale.The proposed embedding configuration therefore uses both column and vector multipliers.Source: §C.1 · p. 24
• The projector is left unchanged because the RMSNorm immediately before it already contains column multipliers for the projector matrix.Source: §C.1 · p. 24
• The recommended placement combines vector and column multipliers on the embedding while adding none to the projector.Source: §C.1 · p. 24

C.2 Residual blocks

• The blocks follow a pre-LN design, and RMSNorm weights are kept as column multipliers for the matrix layer that follows.The section’s notation includes RMSNorm weights inside x to focus on the additional multipliers.Source: §C.2 · p. 24
• The gated MLP configuration uses row and column multipliers for W down, row-only multipliers for W gate, and none for W up.These choices assume RMSNorm weights supply column multipliers; removing them and adding column multipliers to W gate and W up is a more expressive option that remains untested.Source: §C.2 · p. 25
• Attention uses row and column multipliers for W out, row-only multipliers for W Q, and none for W V or W K.With Group Query Attention, query multipliers allow per-head scales, whereas key multipliers would give per-group scales; W out similarly enables per-head output scales.Source: §C.2 · p. 25
• For attention, multiplier placement avoids redundant row scaling on both key and query matrices and favors query-side per-head scales under Group Query Attention.Source: §C.2 · p. 25
• Mamba2 uses row multipliers for W Z and W dt, none for W X, W B, or W C, and row-only W out multipliers unless internal RMSNorm is skipped.The convolution is more expressive than row rescaling because it also mixes nearby time steps; W Z and W dt feed nonlinearities without native scale parameters.Source: §C.2 · p. 26

D. Additional plots

• MLP and attention output norms grow in later training, with different width-dependent offsets attributed to residual-normalization symmetry drift.Embedding output norms also grow to match residual growth.Source: Figure 10 · p. 26
• Projector output norms grow and then plateau at a similar level, with model width affecting them only during intermediate training.The passage suggests the plateau corresponds to a reasonable logits scale.Source: Figure 10 · p. 26
• Figure 11 plots checkpoint evaluation scores and 20-checkpoint running averages, with frequent evaluations beginning at 100GT due to compute constraints.Training includes a ×32 exponential decay from 160GT to 200GT, followed by 40 gigatokens at minimal learning rate.Source: Figure 11 · p. 27