Scaling Latent Reasoning via Looped Language Models

1 Introduction

• LoopLM recursively reuses shared parameters to adapt computation: simple inputs can exit early, while complex inputs receive more iterations without extending the output sequence.This separates compute depth from parameter count and is presented as an architectural alternative to scaling model size, training data, or inference-time CoT generation.Source: §1 · p. 2

Our Contribution

• Training to 7.7T tokens, the study characterizes LoopLM’s scaling trajectory and saturation behavior and uses controlled ablations to isolate the sources of its gains.The study asks whether recurrence substitutes for unshared depth and whether gains vary with loop count.Source: §1 · p. 2, Our Contribution · p. 2
• 1.4B and 2.6B LoopLMs match 4B and 8B standard transformers on most benchmarks, yielding 2-3× parameter efficiency; R4 variants are competitive with 4B and match or exceed 8B models on reasoning datasets.The R4 comparisons are across multiple math and science datasets.Source: Our Contribution · p. 2, Figure 2 · p. 3
• Controlled experiments find recurrence leaves raw knowledge storage at approximately 2 bits per parameter but improves knowledge manipulation on fact-composition and multi-hop reasoning tasks.Source: Our Contribution · p. 3
• LoopLM reduces harmfulness on HEx-PHI as recurrent steps increase, while its latent-update reasoning traces align better with final outputs than CoT traces.The paper presents this alignment as evidence of greater causal faithfulness rather than post-hoc rationalization.Source: Our Contribution · p. 3
• The study frames loop depth as a third scaling axis beyond model size and data and releases the 1.4B and 2.6B Ouro model family.Source: Our Contribution · p. 3

• Recent work explores recurrent depth for latent test-time computation [17], looped transformers for reasoning [7], recursive LoRA adapters [16], continuous pondering [18], adaptive inner thinking [21], and token-routed recursion [22].Source: §2 · p. 3
• Looped architectures can be viewed as deep Transformers with tied layer weights and as latent reasoning, with hidden states progressively refining representations to solve tasks.The two perspectives connect internal computation reuse with reasoning improvement without increasing parameter count.Source: §2 · p. 4

Perspective 1: Parameter Sharing for Model Efficiency

• Parameter sharing reuses Transformer blocks or submodules across depth to reduce parameters; examples include ALBERT [23], machine-translation studies [24] [25], and MoE expert reuse in Megrez2 [26].Takase et al. [25] studied sharing strategies to balance compression and accuracy; Megrez2 [26] presents expert reuse as a path toward edge deployment with limited memory.Source: §2 · p. 4

Perspective 2: Latent Reasoning and Iterative Refinement

• More recurrent steps improve complex-reasoning performance [7] [17]; Coconut [27] and CoTFormer feed latent states back into inputs, unlike LoopLM’s implicit hidden-state evolution.PHD-Transformer [29] is another example of work exploring shared-parameter iteration for efficient sequence-length expansion.Source: §2 · p. 4

3.1 LoopLM Architecture

• The model reuses a depth-L transformer stack for t recurrent steps, producing a language-model output at each step; t = 1 recovers the non-looped model.The adaptive objective weights token-level losses by the gate’s exit-step distribution.Source: §3.1 · p. 5

3.2 Adaptive Computation via Gating Mechanism

• A parallel gate learns per-step exit probabilities, and inference exits at the first step whose cumulative probability crosses threshold q, trading compute for depth.Smaller q favors earlier exits; larger q allows deeper computation, and the gate is fine-tuned with the LM frozen in Stage II.Source: §3.2 · pp. 5–6

3.3 Stage I: Learning an Entropy-Regularized Objective

• Naive training can self-reinforce probability mass on later steps until the gate collapses onto Tmax; entropy regularization maintains a spread reflecting input difficulty.The uniform prior is depth-unbiased, decoupling exit decisions driven by input difficulty from a global compute preference.Source: §3.3 · pp. 6–7

3.4 Stage II: Focused Adaptive Gate Training

• With LM parameters frozen, Stage II trains the gate to balance marginal loss improvement from another loop against its additional compute.A greedy continuation target derived from realized performance gains tells the gate whether to continue or exit.Source: §3.4 · pp. 7–8

4.1 Transformer Architecture

• Ouro uses recurrently applied decoder-only Transformer blocks with MHA, RoPE, SwiGLU, and sandwich RMSNorm, sharing a 49,152-token SmolLM2 vocabulary.The architecture prioritizes a clean implementation of looped computation; the tokenizer is optimized for code and Latin-alphabet language.Source: §4.1 · p. 9

4.2 Data

• The corpus spans web text, mathematics, code, and long-context documents across multiple stages; Stage 1 uses 6.3T-token Nemotron-CC, while CT annealing extends sequences to 16K tokens and LongCT uses ProLong’s 64K subset (20B tokens).Source: §4.2 · pp. 9–10
• Stage 4 mid-training samples 90B tokens from 20+ open-source SFT datasets, replays 30B Stage 1 and 180B Stage 2 tokens, and yields an effective 300B-token volume.The data include both question-answer and question-CoT-answer samples, are decontaminated against mainstream evaluation benchmarks, and are converted to ChatML.Source: §4.2 · p. 10

4.3 Training Stability and Adaptive Configuration

• Reducing recurrent steps from 8 to 4 addressed Stage 1 training instabilities, while larger batches and a lower KL coefficient were also used to support stable optimization.The authors report that recurrent architectures require smaller learning rates than parameter-matched Transformers, but did not run exhaustive learning-rate sweeps.Source: §4.3 · pp. 10–11

4.3.1 Stage-wise Training Details

• After Stage 1 instability, Ouro-1.4B retained 24 pre-trained layers and Ouro-2.6B duplicated them to 48, both using 4 recurrent steps; shared weights made upcycling smooth.Stage 2 annealed the learning rate to 3 × 10−5, Stage 3 used β = 0.05 with 64K sequences, and Stage 4 used a 1 × 10−5 cosine schedule.Source: §4.3.1 · p. 11

4.4 Supervised Fine-Tuning

• SFT uses approximately 8.3M examples, emphasizing mathematical reasoning (3.5M) and code generation (3.2M), with scientific reasoning (808K) and conversation (767K).Training runs for 2 epochs with a maximum sequence length of 32K tokens and a cosine-decayed Adam learning rate of 2 × 10−5.Source: §4.4 · pp. 11–12

4.5 Reinforcement Learning Attempts

• Exploratory RLVR with DAPO and GRPO did not improve on the final SFT checkpoint, and LoopLM’s variable-depth computation broke fixed-path rollouts.Neither off-policy rollouts nor fixed four-round RL surpassed the checkpoint; the authors note that limited post-SFT headroom may explain the latter result, while inference still used fewer rounds when beneficial.Source: §4.5 · p. 12

5.1 Base Model Evaluation

• The 1.4B Ouro matches or exceeds 4B Qwen3-Base on BBH (71.02 vs 70.95), GSM8K (78.92 vs 72.86), and MATH500 (82.40 vs 59.60).It achieves comparable performance across most benchmarks with four recurrent steps.Source: §5.1 · p. 13, Table 7 · p. 13
• The 2.6B Ouro surpasses 8B Qwen3-Base on MMLU-Pro (55.73 vs 53.72), BBH (80.46 vs 77.65), and MATH500 (90.85 vs 62.30).The paper reports that the model outperforms dense models up to 8B on reasoning-intensive benchmarks.Source: §5.1 · p. 13, Table 7 · p. 13

5.2 Reasoning Model Evaluation

• On OlympiadBench and BeyondAIME, Ouro-1.4B-Thinking scores 71.55 and 34.0, while Ouro-2.6B-Thinking scores 76.44 and 39.0.The corresponding Qwen3-4B scores are 73.18 and 31.0; Qwen3-8B scores are 75.25 and 38.0.Source: Table 9 · p. 14, §5.2 · p. 15

5.3 Performance by Recurrent Depth and Extrapolation

• Base-model benchmark performance generally improves through the trained depth T = 4, then moderately degrades when extrapolated to T = 5–8.The passage contrasts this task-performance decline with safety alignment improving beyond T = 4; it points to Section 7.1 for that result.Source: Table 10 · p. 15, Table 11 · p. 15, §5.3 · p. 16
• Ouro-Thinking performance is very low at T = 1, peaks around T = 4 or 5 for 1.4B and T = 3 or 4 for 2.6B, then degrades at T = 6–8.Neither model peaks strictly at T = 4 across all tasks.Source: §5.3 · p. 16, Table 12 · p. 16, Table 13 · p. 16

5.4.1 Early Exit Strategies

• Adaptive early exit lets LoopLM vary recurrent computation by input complexity; the specialized-trained Ponder gate achieves the best MMLU accuracy at every compute budget.At an average exit round of 2.5, it reaches 66% accuracy versus approximately 64% for the standard gate.Source: §5.4 · pp. 16–17, §5.4.1 · p. 17, Figure 5 · p. 17
• The standard Ponder gate outperforms static exit, while hidden-state-difference exit comes within 1%–2% of the specialized gate at 2–3 average rounds.Specialized training improves accuracy by approximately 2%–3% at most operating points; static exit rises from 40% at 1 round to 67.35% at 4, with diminishing returns.Source: §5.4.1 · p. 18

5.4.2 KV Cache Sharing for Inference Efficiency

• During decoding, last-step and averaged KV-cache reuse stay within 0.3 GSM8K points of the full-cache baseline while reducing memory by 4×.First-step reuse collapses to 18.73 on GSM8K, down from 78.92; reuse during prefilling degrades GSM8K by more than 10 points.Source: §5.4.2 · pp. 18–19, Table 14 · p. 18

6.1 LoopLMs does not increase knowledge capacity

• At equal parameter counts, looping neither increased factual knowledge capacity nor improved its scaling; looped and baseline models reached about 2 bits/parameter.Capacity was estimated from how much information models memorized from synthetic biographies.Source: §6.1 · p. 19, Figure 6 · p. 20

6.2 LoopLMs prevails in knowledge manipulation

• On synthetic Mano [69], looped models beat iso-parameter baselines at every tested difficulty and often beat iso-FLOP models; on multi-hop QA [70], they learn with fewer examples.The authors conclude that LoopLMs have a better inductive bias for flexible knowledge manipulation, rather than increased knowledge capacity.Source: §6.1 · p. 19, Figure 6 · p. 20, §6.2 · pp. 20–21

6.3 Discussion: towards understanding why LoopLM helps knowledge manipulation

• The authors attribute improved manipulation to recurrence: shared blocks can reuse stored knowledge, retrieve needed facts, and apply structured procedures across loops.They describe this as efficient reuse of knowledge and algorithms in parameter space, including when information was not used in earlier calculations.Source: §6.3 · pp. 21–22
• For graph reachability, a one-layer width-2n looped transformer answers queries in O(1 + log2 max{1, D}) loops under the theorem’s assumptions.The comparison concerns sequential steps: the table gives O(n^2) for discrete CoT, O(1 + D) for continuous CoT, and O(1 + log max{1, D}) for Universal Transformer. It is not a matched-compute result; the construction uses full attention and thresholding, and its extension to causal attention and standard normalization is outside the theorem.Source: §6.3 · p. 22
• They hypothesize that parameter sharing reduces sample complexity by restricting deep models to a smaller realizable hypothesis class for tasks requiring repeated knowledge manipulation.The proposed explanation is that untied deep models may need to explore a larger function class when learning repeated procedures across layers.Source: §6.3 · p. 22

7.1 Safety

• At 4 recurrent steps, Ouro 1.4B Thinking and 2.6B Thinking have harmful rates of 0.009 and 0.003, respectively, while Qwen3-4B-Thinking has 0.009.The figure reports improving HEx-PHI safety alignment as recurrence increases, including evaluation at steps 5–8 beyond the 4-step training configuration.Source: §7.1 · pp. 22–23, Figure 8 · p. 23

7.2 Faithfulness

• Within recurrent steps 2–4, Ouro’s current-step representations predict that step’s answer, while the preceding-step representation does not reliably predict it.By contrast, a linear probe on Qwen3-4B-Thinking’s final-token logits reaches 0.99 ROC AUC for predicting its eventual answer; the figure plots probe ROC AUC across layers.Source: Figure 9 · p. 24, §7.2 · p. 25
• Across 1,000 Quora Question Pairs, only 361 step-2 answers match step 4 and 551 match step 3; agreement between adjacent steps approaches 1,000 from step 4 onward.The authors interpret disagreement through step 4 as decision updating, with later agreement reflecting convergence and the model’s 4-loop training limit.Source: Figure 9 · p. 24, §7.2 · p. 25
• LoopLM’s latent-state trajectory is the reasoning substrate, while decoded intermediate text is an instrumented readout of states directly supervised by the language-model loss.The paper defines faithful thinking as procedurally correct and causally coupled to the final answer, unlike chain-of-thought that may rationalize a pre-formed decision.Source: §7.2 · p. 24

7.3 More Discussion

• Intermediate and final language-model heads form a native speculative-decoding proposal–verification pair, sharing parameters and cached activations without an external draft model.Draft tokens from an earlier recurrent step are verified at the maximum deployment step using standard acceptance tests.Source: §7.3 · p. 25
• Safety checks can screen draft distributions or candidates before streaming, halting or rerouting violations; the Q-exit threshold adjusts compute, consistency, and safety strictness.The final-step verifier retains final quality while the aligned step-s prediction serves as a proxy for early screening.Source: §7.3 · pp. 25–26
• LoopLM exposes a predictive distribution at every recurrent step, enabling intermediate-depth inference under latency constraints, but deeper computation is not guaranteed to reduce next-token loss.Stage I jointly trains predictions, Stage II trains the exit gate with the language model frozen, and speculative draft tokens should be verified before commitment.Source: §7.3 · p. 26

8 Conclusion

• Ouro’s 1.4B and 2.6B models consistently match or exceed 4B and 8B standard transformers, showcasing a 2–3× efficiency gain.The models integrate iterative computation and adaptive depth into pre-training on 7.7T tokens.Source: §8 · p. 26
• Synthetic experiments and theoretical analysis attribute Ouro’s advantage to superior knowledge manipulation rather than increased knowledge storage.Source: §8 · p. 26
• Entropy regularization with a uniform prior learns adaptive depth, while validated KV cache sharing strategies support practical deployment.Source: §8 · p. 26
• Iterative refinement provides causally faithful reasoning traces that mitigate the post-hoc rationalization seen in standard CoT.Source: §8 · p. 26
• Safety alignment improves with increased recurrent steps, even when extrapolating, and iterative latent computation is proposed as a third scaling axis beyond parameters and data.Future work should improve performance extrapolation at greater depths and explore more complex recurrent mechanisms.Source: §8 · p. 26

Contributions

• Rui-Jie Zhu proposed the project and led Ouro’s pre-training, while Ge Zhang co-led and supervised the project and Kai Hua curated the pre-training data mixtures.Zhu also optimized pre-training and inference infrastructure; Zhang provided insights during pre- and post-training, and Hua contributed insights during pre-training.Source: Contributions · p. 27
• Zixuan Wang led analysis and experiments on LoopLM’s superiority, and contributed to adaptive early-exit strategies, training, and safety analysis.Zixin Wen contributed to theoretical analysis and adaptive exit strategies, as well as Physics of LLMs experiments, paper writing, and RLVR.Source: Contributions · p. 27
• Tianyu Zhang led consistency, safety, and faithfulness analysis, designed the faithfulness-evaluation pipeline, and contributed to post-training, probing, and efficient KV-cache design.Boyi Wei contributed to safety analysis, including HEx-PHI evaluations and PCA of model representations.Source: Contributions · p. 27
• Ziniu Li led post-training by developing supervised fine-tuning and contributing to RLVR exploration; Rui-Jie Zhu also explored RLVR.Ge Zhang provided insights during post-training.Source: Contributions · p. 27
• Haoran Que led LoopLM scaling-law analysis, investigating relationships between performance, model size, and recurrent depth.Source: Contributions · p. 27
• Fan Yin optimized Ouro’s vLLM and SGLang implementations for inference efficiency, while He Xing contributed to vLLM infrastructure development and optimization.Rui-Jie Zhu also optimized pre-training and inference infrastructure and developed the initial vLLM implementation.Source: Contributions · p. 27

A Empirical Validation of Prior Choice

• Uniform priors achieve lower training loss and cleaner convergence, while geometric priors plateau higher and oscillate more as their bias toward early exit increases.The uniform prior maintains exploration across depths; stronger geometric priors concentrate probability on shallow steps and weaken supervision for deeper iterations.Source: Figure 10 · p. 34, A Empirical Validation of Prior Choice · p. 34
• The prior-choice experiments use a 776M-parameter LoopLM with 4 recurrent steps, trained for 20B FineWeb-Edu tokens while varying only the KL-term prior.Each condition uses multiple random seeds; geometric-prior parameter λ ranges from 0.1 to 0.9, alongside a uniform prior.Source: A Empirical Validation of Prior Choice · p. 34
• At a fixed average step budget, uniform-prior models achieve a strictly better accuracy–compute Pareto frontier than geometric-prior models.The authors attribute this trade-off to unbiased depth exploration during pre-training.Source: A Empirical Validation of Prior Choice · p. 35

B Physics of LoopLMs

• The appendix identifies its contents as experimental settings for knowledge capacity, knowledge manipulation, synthetic QA, and theoretical results.It locates those topics in Sections B.1, B.2, B.3, and B.5, respectively.Source: B Physics of LoopLMs · p. 35

B.1 Capo: knowledge capacity

• The Capo task measures factual knowledge memorization as bits per parameter, including partial correctness when evaluating recalled biography attributes.Its synthetic biographies contain randomly generated people and attributes; the evaluation normalizes stored information by model parameter count.Source: §B.1 · pp. 35–36

B.2 Mano: knowledge manipulation

• The Mano task tests manipulation of stored knowledge without intermediate thoughts [69] using modular-arithmetic expression trees over F23 and exact-match accuracy on the hardest length.Expressions use prefix notation and addition, subtraction, and multiplication.Source: §B.2 · p. 36

B.3 Multi-hop question answering on synthetic relations

• With 100,000 and 140,000 unique QA pairs, models with more loops learn faster and achieve better performance on synthetic multi-hop QA [70].The task evaluates three-hop questions over hierarchical relations; models are compared with 1, 2, or 4 loops.Source: §B.3 · pp. 36–37, Figure 11 · p. 37

B.3.1 Additional experimental results

• The appendix notes that the iso-flop baseline at N = 1.2 × 10^5 did not significantly outperform the shallower main-paper version, possibly due to randomness or insufficient tuning.The authors say further experiments are needed to validate this appendix conclusion.Source: §B.4 · p. 37
• With 100,000 and 120,000 unique QA pairs, models with more loops learned faster and performed better than non-looped baselines, including an iso-flop transformer.The loop-2 model was weaker than the iso-flop transformer at 100,000 pairs but surpassed the baseline with more data.Source: §B.3.1 · p. 37, Figure 12 · p. 38

B.4 Case study: improvements across different categories in MMLU

• The MMLU analysis measured relative improvement by comparing accuracy at the single recurrent step, Loop 1, with accuracy at fully trained depth, Loop 4.The category-level results cover all 57 MMLU categories.Source: §B.4 · p. 38
• LoopLM’s MMLU gains indicate stronger knowledge manipulation rather than simply greater accessible knowledge or raw storage.The passage contrasts gains in logic and math with nearly flat improvement in global_facts.Source: §B.4 · p. 38

B.5 Theory: latent thought with LoopLM

• The simplified looped transformer can solve graph reachability with part of the graph stored in its parameters in O(1 + log n) loops.The construction uses full attention over graph tokens and an explicit elementwise threshold, unlike the causal attention and normalization of the language models.Source: §B.5 · p. 38
• The theoretical task combines a fixed knowledge graph encoded in the model parameters with a context graph, and asks whether a directed path connects a source to a target in their union.Paths of length zero are allowed; the maximum finite shortest-path distance among reachable pairs is at most n − 1.Source: §B.5 · p. 38

B.4 Case study: improvements across different categories in MMLU

• Reasoning-heavy MMLU categories improved most from Loop 1 to Loop 4, while fact- and world-knowledge-heavy categories showed more modest gains.The category-level results are reported for all 57 MMLU categories in Table 15.Source: §B.4 · p. 38, Table 15 · p. 39

Transformer architecture

• The theorem assumes an adjacency-column input encoding with an identity matrix in R0, and reads query (s, t) from (RL)t,s after L loops.The identity represents zero-length paths and ensures every query has a selected key, including at isolated vertices.Source: Transformer architecture · p. 40
• With one layer, one attention head, and width 2n, the transformer solves all reachability queries in 1 + ⌈log2 max{1, n−1}⌉ loops and remains correct thereafter.Its parameters are independent of the context graph and query; the loop bound does not require knowing the graph diameter.Source: Transformer architecture · pp. 40–41
• Each loop shares a single-head attention layer and two-layer ReLU MLP; attention composes represented paths while the MLP inserts hidden-graph edges via Boolean repeated squaring.The construction uses full attention and a Boolean threshold, not a causal mask or standard LayerNorm or RMSNorm.Source: Transformer architecture · pp. 40–41
• A positive threshold is essential: it prevents softmax’s positive weights on unselected keys from creating false positives in the Boolean path recurrence.The recurrence preserves the diagonal and yields the exact Boolean-matrix update.Source: Transformer architecture · p. 41
• Each loop uses O(n3) arithmetic operations and O(n2) state entries, so logarithmically many loops do not imply logarithmic total work.The construction uses Θ(n) coordinates per token, but does not establish that this width is necessary for every representation.Source: Transformer architecture · p. 41
• Converting an edge-list input, as considered in [71], to the theorem’s adjacency-column format requires an additional construction whose cost and masking requirements are not bounded here.That conversion is excluded from the stated loop bound.Source: Transformer architecture · p. 41

C.1 Evaluation Settings

• Base-model evaluation settings and benchmark sources are documented in Table 16.Source: §C.1 · p. 41, Table 16 · p. 42
• Ouro-Thinking reasoning benchmarks use an in-house evaluation harness and an LLM-as-judge protocol with a fixed rubric; settings and protocol are documented in Table 17.Source: §C.1 · p. 42, Table 17 · p. 42

D Scaling Law for LoopLMs

• The scaling study investigates predictability and scalability through three questions about the standard-model versus LoopLM performance gap, recurrent-step effects on loss, and the connection between total and step-wise loss.Source: D Scaling Law for LoopLMs · p. 42

D.1 RQ1: What is the performance gap between standard models and LoopLM?

• The RQ1 experiments use 53M, 134M, 374M, 778M, and 1.36B models, recurrent depths 1, 2, 4, and 8, and six benchmarks; all train on 20B FineWeb-Edu tokens.Standard-model recurrent steps effectively multiply the model's layer count; performance is reported from the final step, using LoopLM's maximum recurrent step for evaluation.Source: §D.1 · pp. 42–43
• For both LoopLM and Standard models, benchmark performance rises with model size at every recurrent step and generally rises with recurrent step, except for LoopLM at 778M and 1.364B.The study reports average performance over six benchmarks.Source: §D.1 · p. 43, Figure 13 · p. 43, Figure 14 · p. 44
• The Standard model outperforms LoopLM in every experiment under the same conditions, with a positive benchmark-score gap.The gap is defined as Standard model score minus LoopLM score.Source: §D.1 · p. 43, Figure 13 · p. 43, Table 18 · p. 44
• The benchmark gap generally grows with recurrent steps and shrinks as model size increases, particularly consistently at recurrent step 4.The reported size trend applies when the maximum recurrent step is relatively low.Source: §D.1 · pp. 43–44, Table 18 · p. 44

D.2 RQ2: How do recurrent step impact the total loss and step-wise loss in the context of LoopLM?

• R2=0.9596 for the Total Loss Scaling Law, which relates total loss to model size, training data size, and maximum recurrent step.The actual and predicted loss curves are described as highly consistent.Source: §D.2 · pp. 44–45, Figure 15 · p. 45
• R2 values for the Step-wise Loss Scaling Law were 0.8898, 0.8146, and 0.795 at maximum recurrent steps 2, 4, and 8, respectively.The law captures the overall trends, although step-wise loss is less stable than total loss; its positive γ indicates that step-wise loss decreases as recurrent step increases.Source: §D.2 · p. 46

D.3 RQ3: What is the inherent connection between total loss and step-wise loss?

• When model size is insufficient, shallow step-wise loss can rise as training data grows, as gating favors deeper steps to reduce expected task loss while entropy regularization limits deviation from the prior.The text attributes the shallow-loss increase to limited information capacity: raising shallow step-wise loss lets the gating weights favor deeper recurrent steps, and suggests a larger model may better preserve the normal trend.Source: §D.3 · pp. 46, 49
• With maximum recurrent step fixed, the two scaling-law forms are consistent; for maximum step 4, an estimated step-wise-to-total-loss fit achieved R2=0.961.The estimate used averaged gating weights and ignored the entropy term after training stabilized; the resulting predicted curve closely matched the actual curve.Source: §D.3 · pp. 49–50, Figure 20 · p. 50

E.1 Generalizability for the Total Loss Scaling Law

• R2 values were 0.9385, 0.9609, and 0.962 when the first 25%, 50%, and 75% of training-data points were used for fitting, respectively.Consistency improved as more data points were included; the text says predictions at larger training sizes improve when fitting data points are closer to those larger sizes.Source: §E.1 · p. 50
• Average R2 was 0.9542 across 10 model-size fits using three of five sizes, demonstrating total-loss scaling-law generalizability across model size.The five sizes were 53M, 134M, 374M, 778M, and 1.364B; Figure 21 shows a fit using the three largest sizes predicting the two smallest.Source: §E.1 · p. 50, Figure 21 · p. 51
• Average R2 was 0.9581 when fitting on two of the three maximum recurrent steps and validating on all data points, demonstrating generalizability across maximum recurrent step.The tested maximum recurrent steps were 2, 4, and 8.Source: §E.1 · p. 52

Model Size Generalizability

• Average R2 is 0.8815 versus full-data 0.8898 at max recurrent step 2, and 0.797 versus 0.8146 at step 4, supporting model-size generalizability.Fits used three of five model sizes; Figure 22 illustrates predictions for unseen smaller and larger sizes.Source: §E.2 · p. 52, Figure 22 · p. 53

Training Data Generalizability

• As fitting data points increase, fitting accuracy improves: reported R2 values are 0.8686, 0.8882, and 0.8896, while max recurrent step 4 gives R2 values 0.793, 0.813, and 0.8142.These results are similar to full-data values of 0.8898 and 0.8146, demonstrating generalizability of the Step-wise Loss Scaling Law with respect to training data.Source: §E.2 · p. 52

Recurrent Step Generalizability

• Average R2 is 0.8118 when fitting three of four recurrent steps at max step 4, close to the full-data value of 0.8146, supporting recurrent-step generalizability.Figure 23 shows a fitted curve closely matching actual results at unseen recurrent step 4.Source: §E.2 · p. 52, Figure 23 · p. 54