Emergent One-Third Scaling Law as Attention Tries to Concentrate

1 INTRODUCTION

• Neural scaling laws describe loss decreasing as a power law with training dataset size and model size, while their origins remain debated.The introduction connects these laws to longer training and more parameters leading to better performance.Source: §1 · p. 1
• Prior work proposed that a single softmax learning peaked distributions can generate power-law dynamics, but multiple softmaxes and layers in LLMs remained insufficiently studied.The paper frames this as a question about effects on the loss landscape and training dynamics.Source: §1 · p. 1
• LLM experiments find peaked distributions in attention and the LM head, but power-law logit growth close to exponent 1/3 in some attention heads rather than the LM head.LLM losses also decay with exponents close to 1/3.Source: §1 · p. 2
• When any softmax learns a peaked distribution, it can become a bottleneck and produce overall loss scaling with exponent 1/3, regardless of its position.The paper also reports that the scaling is not specific to loss definitions.Source: §1 · p. 2

2 TOY MODEL

• The gate-head model has a gate layer that uses a key matrix and softmax to weight value matrices, followed by a head layer that projects hidden states to distributions.The student is trainable and is matched to a fixed teacher with the same architecture.Source: §2 · pp. 2–3
• Logit magnitude is used to characterize softmax nonlinearity: small logits permit a Taylor expansion and make softmax approximately linear.The model controls logit scale through initialization and normalization.Source: §2 · p. 3
• The experiments sweep target gate and head coldness from 1 to 1000 while fixing m = 32, d = 8, and n = 128, and plot loss against effective gradient-flow time.That time is defined from the accumulated learning rate over training steps.Source: §2 · p. 3
• The gate-head model exhibits 1/3 time scaling when target gate coldness, target head coldness, or both are large.When both operate at high temperature, loss can converge substantially faster.Source: Figure 2 · p. 4, §2 · p. 4
• With large gate coldness and low head coldness, gate logit magnitude grows with exponent 1/3 after the head saturates, sustaining 1/3 loss scaling.Different head coldness values yield different crossover times.Source: Figure 2 · p. 4
• Any softmax, whether in the gate or head, becomes a training bottleneck when required to produce a sufficiently peaked distribution, preventing faster-than-1/3 loss decay.The result applies when at least one such bottleneck is present.Source: §2 · p. 4

• With a large target gate coldness and a small target head coldness, the head converges while gate coldness continues growing and the loss crosses into 1/3 scaling.The head coldness saturates over the loss crossover, whereas gate coldness continues to grow approximately with positive 1/3 time scaling.Source: §3 · p. 4
• In the gate-only regime, loss and gate coldness enter 1/3 scaling as value-matrix displacement saturates, leaving continued key-matrix growth to govern late dynamics.Different learning-rate curves collapse when plotted against τ.Source: §3 · pp. 5–6, Figure 3 · p. 5
• The power law is not specific to KL divergence: probability MSE and higher even-power losses retain the scaling exponent, changing only the integral coefficient.Source: §3 · p. 6
• For aligned students with nonzero zero-gap density, any reasonable loss scales as 1/β at large β and β* ≫ β, yielding the same 1/3 time exponent.The loss must be non-negative, vanish only when distributions match, and have bounded differentiable gradients even when the target is one-hot.Source: §3 · p. 6
• In the full GH model, late loss decomposes into contributions from imperfect K and W, each scaling as τ^-1/3 when the other has converged.Cross terms are higher order at late times.Source: §3 · p. 6
• One or more non-convergent softmax modules that must learn peaked distributions can bottleneck late training and yield overall τ^-1/3 loss scaling, regardless of position.Source: §3 · p. 6

4 LLM EXPERIMENTS

• The study evaluates dense models with softmax attention, including open-source Pythia and OLMo checkpoints.The stated focus is comparison with the original neural scaling laws.Source: §4 · p. 7
• Later-layer attention heads and the LM head lie in the low-temperature regime, with some attention heads reaching extremely low entropies.Other evaluated models show similar results.Source: §4 · p. 7
• For both attention and the LM head, energy-gap densities approach non-zero values as the gap tends to zero, satisfying another condition for predicted 1/3 time scaling.The attention example is a representative later-layer Pythia-12B head; similar behavior occurs across other low-temperature heads.Source: §4 · p. 7
• Training-dependent loss fits have exponents close to 1/3 across Pythia model sizes, whose curves collapse against dynamic time τ; OLMo models show the same behavior.The growing logits occur in attention heads rather than the LM head, suggesting attention is the late-training bottleneck.Source: §4 · p. 8
• With MSE between final logits, attention-head coldness continues approximately positive 1/3 growth after LM-head coldness saturates, and the training-dependent MSE has a 1/3 scaling tail.This experiment bypasses the output softmax and rules out the LM head as the sole bottleneck.Source: §4 · p. 8, Figure 5 · p. 8

5 RELATED WORKS

• Under online one-epoch training with a fixed learning-rate schedule shape, dynamic time is proportional to dataset size, making 1/3 time scaling interpretable as 1/3 dataset-size scaling.The paper reports this as consistent with Chinchilla-data exponents of 0.28 ∼0.37.Source: §5 · p. 9
• Earlier sample-limited theories attribute scaling exponents to data properties, while many training-dynamics accounts require power-law data structures to yield power-law loss.The paper notes that such high-level theories can be difficult to test directly in LLMs and that some analyses ignore irreducible loss.Source: §5 · p. 9
• The paper generalizes the single-softmax analysis of Liu et al. (2026b) to multiple softmax functions and identifies training attention as the bottleneck for 1/3 loss scaling in LLMs.K¨uhn et al. (2026) analyze the single-softmax case more rigorously but do not test the theory against LLMs.Source: §5 · p. 9
• Prior observations of attention sinks and massive activations suggest attention concentration, but the authors say this work first proposes a connection to neural scaling laws.The cited observations include Xiao et al. (2024), Sun et al. (2024), Zucchet et al. (2025), and Queipo-de Llano et al. (2026).Source: §5 · p. 9

6 DISCUSSION

• Attention concentrating on specific tokens is presented as the likely bottleneck underlying the 1/3 power-law loss scaling in LLMs.The paper frames this as a possible connection between a key Transformer feature and neural scaling laws.Source: §6 · p. 9
• The analysis focuses on training-time scaling and gradient-flow dynamics, leaving model-size scaling, gradient noise, optimizer-specific effects, and hyperparameter scaling outside its main treatment.It also focuses on original neural scaling laws; the authors expect MoE to retain the 1/3 training-time exponent, while noting that newer architectures may modify scaling-law form.Source: §6 · p. 9
• The authors suggest structural priors might accelerate learning by reducing how sharply attention must focus, while also raising that selecting a few relevant elements may be fundamental.They state that distinguishing these possibilities could enable more efficient scaling or reveal principles of language and intelligence.Source: §6 · pp. 9–10

AI USE STATEMENT

• The authors used generative AI for brainstorming, code drafting, appendix drafting, and readability, but not to develop theoretical models, prove claims, or perform data analysis.They report reviewing AI-assisted work and taking responsibility for the final content.Source: AI USE STATEMENT · p. 10

REPRODUCIBILITY STATEMENT

• The paper says code reproducing all results is available, with appendix explanations of experiment logic and theoretical assumptions and proofs.The reproducibility statement references corresponding code files.Source: REPRODUCIBILITY STATEMENT · p. 10

A TOY MODEL EXPERIMENTS

• The appendix identifies GH-I as the type-I GH model and says GH-II and GH-III vary its gate architecture, supporting generality beyond that architecture.The appendix says GH-I analysis suggests a saturating nonlinear origin for emergent 1/3 scaling.Source: A TOY MODEL EXPERIMENTS · p. 12

A.1.1 EXPERIMENT METHODS

• The GH-I appendix describes experiments using Gaussian inputs, KL-divergence training, SGD, and sweeps over β*G, β*H, and learning rate η.The setup specifies m = 32, d = 8, n = 128, 10,000 steps, and 192 runs; the main-text architecture is referenced rather than restated.Source: §A.1.1 · p. 13

A.1.2 DATA ANALYSIS

• GH-I scans plot test loss and the βH and βG proxies across parameter combinations, finding qualitatively consistent behavior in the large-β* regime.The plotted curves are representative of their regimes rather than particular parameter choices.Source: §A.1.2 · p. 13
• GH-I plots compare loss and coldness trajectories against −1/3 and +1/3 reference slopes, with vertical bands marking the transition to the 1/3 regime.For β*H ≈ 7.2, the crossover is smoother and longer, so a single vertical line marks only its onset.Source: §A.1.2 · p. 13

A.2 G-I MODEL

• The G-I model isolates the GH-I gate layer, omitting the head layer while retaining the gate architecture and GH-I hyperparameters.It trains the gate against teacher hidden states using MSE, with 64 runs across β*G and learning-rate sweeps.Source: §A.2 · p. 13, §A.2.1 · pp. 13–14

A.2.2 DATA ANALYSIS

• Across the large-β∗G scan, curves from different learning rates approximately collapsed against τ, and behaviors were qualitatively consistent.The scan overlaid all eight learning rates per panel; selected curves were described as representative.Source: §A.2.2 · p. 14
• At τ = 1500, the reported onset of 1/3 scaling aligned across loss, βG, and value-tensor displacement panels.The displacement panel showed saturation rather than a power-law trend.Source: §A.2.2 · p. 14

A.3 ADAM OPTIMIZER

• Adam produced qualitatively similar approximate 1/3 power laws in loss and β in both GH-I and G-I, despite its harder-to-analyze adaptive dynamics.The authors report that effective β dynamics appear similar under Adam and SGD, with parameter-update differences acting as higher-order corrections.Source: §A.3 · p. 14

A.3.1 GH-I EXTRA EXPERIMENTS

• The GH-I Adam sweep comprised 192 runs with learning rates {0.01, 0.03, 0.1}, and dynamic time was defined as τ = ηt.Adam replaced SGD while the architecture, hyperparameters, and sweep design otherwise matched Appendix A.1.1.Source: §A.3.1 · p. 14
• With large gate and small head coldness, βH saturated before βG grew approximately as τ^1/3, alongside continued loss decay.The sequence mirrors the SGD behavior, though alignment with the reference slopes was less tight.Source: §A.3.1 · pp. 14–15, Figure 7 · p. 15
• In the Adam coldness sweep, approximate 1/3 loss scaling appeared in the three settings with large gate coldness, head coldness, or both.When both coldnesses were small, loss initially fell rapidly but Adam became unstable at late times.Source: §A.3.1 · p. 15, Figure 6 · p. 15
• Under Adam, gate-only training with MSE showed approximate 1/3 scaling in both loss and gate logit magnitude.The caption describes this as mirroring the SGD results.Source: Figure 8 · p. 16

A.3.2 G-I EXTRA EXPERIMENTS

• Under Adam, G-I reproduced approximate 1/3 loss scaling and βG growth, confirming that the observed phenomena are not specific to SGD.The reported sweep used eight learning rates and 64 runs; curves approximately collapsed against τ.Source: §A.3.2 · p. 16

A.4 GH-II MODEL

• Both sigmoid and tanh GH-II gates reproduced τ^-1/3 loss and τ^1/3 gate-coldness scaling with large gate and small head coldness.The sigmoid components correspond to binary softmaxes, while tanh is a rescaling of sigmoid; the paper connects both to the softmax argument.Source: §A.4 · p. 18
• A low-temperature GLU gate may add a training bottleneck, and MoE routing may retain 1/3 scaling if its distribution becomes peaked.The paper suggests that GLU or MoE can lower the achievable loss floor through greater expressivity or capacity, while convergence to that floor may still follow the same power law.Source: §A.4 · p. 18, §A.5 · p. 19

A.5 GH-III MODEL

• In GH-III, replacing the gate with real single-head attention preserved τ^-1/3 loss scaling and τ^1/3 gate-coldness growth.The explanation carries over when the student Q, K, and V matrices are aligned with the teacher; under that assumption, the loss reduces to a function of βG alone.Source: §A.5 · p. 19, Figure 10 · p. 19

B.1 GRADIENT FLOW

• The gate-dynamics analysis studies the student key matrix under gradient flow and simplifies late-time behavior by assuming alignment with the teacher.In the small-norm regime, the student matrix rapidly aligns; the late-time dynamics are then dominated by norm growth.Source: §B.1 · p. 20, §B.2 · p. 20

B.3 LOW TEMPERATURE EXPANSION

• For complex, diverse data, the low-temperature expansion yields loss scaling at large βG when the energy-gap density at zero is nonzero.In the toy setup with i.i.d. standard-normal energies, the gap distribution is approximated using extreme-order statistics; βG much larger than √(2 ln d) marks a rough low-temperature boundary.Source: §B.3 · pp. 21–22
• For large βG, the low-temperature regime has an intermediate 1/3 power-law phase; loss deviates from that law and converges exponentially as βG approaches β∗G.When both β∗G and βG are small, the model is effectively linear and, in the stated isotropic setup, loss converges exponentially.Source: §B.3 · p. 22

B.4 LOSS UNIVERSALITY

• The 1/3 scaling applies to final and intermediate softmaxes even though the final loss depends on their outputs differently.The final softmax uses KL divergence, whereas an intermediate softmax’s loss is more like MSE after later layers converge.Source: §B.4 · p. 22
• Under the theorem’s assumptions, any reasonable loss satisfies L = Θ(β^-1) and ∇βL = Θ(β^-2) when β is sufficiently large and β∗≫β.The setup assumes an aligned student and complex, diverse data.Source: §B.4 · p. 23
• If β follows effective gradient-flow dynamics, the theorem’s loss and gradient scalings imply β = Θ(τ^1/3) and L = Θ(τ^-1/3).The paper says multiple low-temperature softmaxes can each contribute a τ^-1/3 term, with higher-order interaction when each is already close to optimal.Source: §B.4 · pp. 22–23

C.1.1 EVALUATION METHODS

• For Pythia and OLMo-2 checkpoints, the evaluation measures cross-entropy and LM-head logit coldness, plus per-head attention-logit coldness and entropy.Pythia uses Pile and FineWeb text, while OLMo-2 uses FineWeb; checkpoints are approximately logarithmically spaced across training.Source: §C.1.1 · p. 25
• The evaluation also histograms energy gaps between the largest and second-largest logits to check whether their density near zero is nonzero, as required by the theory.For attention, gaps use valid causal key positions; for the LM head, they use the full vocabulary.Source: §C.1.1 · p. 25

C.1.2 ANALYSIS METHODS

• Dynamic time τ is the cumulative learning-rate sum, placing model sizes with different peak learning rates on a common axis.The schedule uses linear warmup followed by cosine decay for both Pythia and OLMo-2.Source: §C.1.2 · pp. 25–26
• The entropy thresholds approximate the high- versus low-temperature boundary; attention heads below their threshold are classified as low-temperature.The attention threshold averages simulated softmax entropies across valid key counts, while the LM-head threshold uses vocabulary size; the LM-head target entropy is compared with its threshold.Source: §C.1.2 · p. 26
• The loss fit includes a free irreducible-loss floor, which is subtracted to isolate the training-dependent component.Fits use nonlinear least squares on log loss, excluding early checkpoints that remain in rapid alignment.Source: §C.1.2 · p. 26
• Attention-logit coldness is plotted against τ without fitting, while energy-gap histograms are normalized into probability densities.A +1/3 reference line supports visual comparison of coldness growth; non-zero energy-gap density near zero tests the theoretical condition.Source: §C.1.2 · p. 26

C.1.3 ADDITIONAL RESULTS

• Pythia-160M, 410M, and 12B all contain low-temperature attention heads across network depth, so the pattern is not specific to one model scale.Later layers tend to have lower entropy, but concentrated heads also appear in early layers.Source: §C.1.3 · p. 27, Figure 11 · p. 27
• For most Pythia-12B attention heads across all 36 layers, energy-gap density is non-zero as ∆ϵ approaches zero.Some early-layer heads instead have density approaching zero at small energy gaps, consistent with their higher entropies.Source: §C.1.3 · p. 27, Figure 12 · p. 28
• Across Pythia-12B’s 36 layers, low-temperature attention heads broadly grow in logit coldness at approximately +1/3 late in training.This growth is not limited to the representative head and is consistent with the predicted behavior of a bottleneck softmax.Source: §C.1.3 · p. 27, Figure 13 · p. 28
• In Pythia-160M and 410M, representative attention heads have non-zero energy-gap density near zero and coldness growing approximately as τ^1/3, while LM-head coldness saturates.These results reproduce the reported behaviors across model sizes beyond Pythia-12B.Source: §C.1.3 · p. 27, Figure 14 · p. 29
• Eight Pythia model sizes show a training-dependent loss component that collapses onto a power law close to 1/3 after subtracting the fitted loss floor.The raw loss approaches a positive floor that decreases with model size, so including the floor in the fit separates model-size offsets from training-dependent decay.Source: §C.1.3 · p. 30, Figure 15 · p. 30
• OLMo-2-1B, 7B, and 13B have fitted loss exponents of 0.35, 0.30, and 0.36, respectively, while a highlighted early-layer head shows the predicted low-temperature behavior.That head has non-zero energy-gap density near zero and logit coldness growing as τ^1/3; the deepest low-temperature heads appear in early layers.Source: §C.1.3 · p. 30

C.2.1 TRAINING METHODS

• A randomly initialized Pythia-160M student is trained against a frozen, pretrained Pythia-160M teacher using MSE between their final logits.Training runs for 79,000 steps on the deduplicated Pile with a warmup-and-cosine schedule.Source: §C.2.1 · pp. 30, 32
• OLMo-2’s fitted loss exponents are 0.35, 0.30, and 0.36 for its 1B, 7B, and 13B models, respectively.The replication also finds an early-layer attention head whose coldness grows as τ^1/3 while LM-head coldness saturates.Source: Figure 16 · p. 31
• Mean-centering the student and teacher logits makes the MSE invariant to global logit shifts that do not affect their softmax distributions.The loss averages squared differences between mean-centered vocabulary-logit vectors over tokens.Source: §C.2.1 · p. 32

C.2.2 ANALYSIS METHODS

• Dynamic time τ is computed by the Appendix C.1.2 cumulative-sum procedure using the Pythia learning-rate schedule, with peak learning rate 6 × 10^-4 and tmax 143,000.LM-head logit coldness is the standard deviation of pre-softmax LM logits over the vocabulary, evaluated at each checkpoint.Source: §C.2.2 · p. 32
• The MSE loss is fit after LM-head coldness saturates, using a power-law-plus-constant form and subtracting the fitted floor.The late-training fit uses nonlinear least squares on log MSE, and the resulting training-dependent component is compared with a slope −1/3 reference.Source: §C.2.2 · p. 32