Predictable Scaling: Efficient Model Exploration via Scaling-Law Prediction

1 Introduction

• PS formalizes reduced-scale evaluation as prediction over relative performance gaps and introduces six composable scaling functions for reducing compute along distinct resource axes.The functions are designed to create proxies for full-scale evaluation.Source: §1 · p. 1
• The simplest predictor achieves error within the A/A noise floor, while the Gaussian Process provides calibrated uncertainty for reliability screening.The paper compares three prediction strategies of increasing expressiveness.Source: §1 · p. 1
• Across two ranking models, PS correctly ranks actionable hypotheses with 79–88% compute reduction.The evaluations cover a task-specific model and a foundation model.Source: §1 · p. 1

Scaling laws in deep learning

• Classical language-model scaling laws describe power-law performance, but their homogeneous-Transformer setting contrasts with recommendation systems that couple sparse embeddings, dense layers, and sequential components.Gadre et al. (2024) report that scaling laws hold under over-training, which is relevant to PS’s under-trained proxy regime.Source: §2 · pp. 1–2

Efficient candidate evaluation

• Hyperband and BOHB adaptively terminate configurations within a fixed architecture, while µP transfers hyperparameters zero-shot; PS instead predicts structural changes requiring distinct training runs.Those changes include architecture, features, and data.Source: §2 · p. 2

Small-to-large prediction; Recommendation system scaling

• Small-to-large prediction work on LLMs and recommendation systems targets absolute performance, whereas PS targets relative gaps between hypotheses and their baseline.Prior recommendation scaling studies address capacity planning, including quality changes with parameter or data scaling.Source: §2 · p. 2

3.1 Setting

• A relative performance gap compares each hypothesis’s normalized-entropy metric with the baseline metric after convergent training.A negative gap indicates improvement.Source: §3.1 · p. 2

3.2 Scaling Functions

• Six scaling-function families are parameterized by a reduction ratio, and a scaling mechanism binds one family to a ratio in (0, 1].Curated scaling profiles compose mechanisms to produce proxies at a target compute reduction.Source: §3.2 · p. 2, Table 1 · p. 2

3.3 Gap Extrapolation

• A predictor maps proxy observations to predicted full-scale gaps; linear regression is suited to approximately affine gap transfer.The raw proxy is also considered as a baseline predictor.Source: §3.3 · pp. 2–3
• The Empirical Gaussian Process uses historical training curves to extrapolate proxy learning-curve convergence and provides calibrated uncertainty for rejecting unreliable predictions.Predictions exceeding a coefficient-of-variation threshold are rejected, signaling that full-scale training is needed.Source: §3.3 · p. 3

3.4 Optimization Objective and Success Criteria

• The evaluation objective minimizes prediction error while constraining the number of mechanisms and their compute to a budget; actionable-hypothesis ranking uses Spearman ρ above the A/A noise floor.The secondary ranking metric is applied to hypotheses with |∆| > τAA.Source: §3.4 · p. 3
• PS assumes convergence stability, transferable gaps, monotonic signal preservation, and a low-dimensional predictive relationship between proxy and full-scale gaps.These are the framework’s stated regularity conditions.Source: §3.4 · p. 3

3.5 Compute Budget-Optimal Proxy Selection

• Proxy selection enumerates feasible configurations, evaluates them on a benchmark corpus with known full-scale gaps, and chooses the scaling mechanisms and predictor with the lowest cross-validated error.The search is kept tractable with a small function set, coarse ratio grid, depth at most three, and compute-tier filtering.Source: §3.5 · p. 3

Evaluation setup

• The evaluation uses two industrial ranking models, and a public Criteo evaluation with four public models is provided separately.Normalized entropy (NE) is the evaluation metric; NE < 1 indicates improvement over the base rate.Source: §4 · p. 3

TS-1: Predictor ablation

• 15% relative MAE improvement: on TS-1 CTR, EGP achieves 0.738 τAA MAE versus 0.865 τAA for the raw proxy, while all predictors rank actionable hypotheses with ρ = 1.0.The comparison covers 14 held-out hypotheses; Table 3 reports the prediction accuracy.Source: Table 3 · p. 4, §4 · p. 4
• Figure 1 plots TS-1 CTR predictions by hypothesis, with the diagonal marking perfect prediction and the shaded band marking ±τAA.Filled points are held-out hypotheses and open points are training hypotheses.Source: Figure 1 · p. 4

FM-1: GP on a foundation model

• On FM-1 across seven held-out hypotheses, GP reduces raw-proxy CTR-head MAE from 2.15 τAA and 1.75 τAA to 0.70–0.85 τAA, yielding a 57–67% improvement.Source: §4 · p. 4

• TS-1 and FM-1 reduce trainers by 86%/88%, and samples by 80%/50%.Source: Compute reduction · p. 4

5 Discussion and Conclusion

• PS works best with fast NE convergence, low AA variance, and effects exceeding τAA; sign reversals above 15–20% rates should be rejected.WAS reduces the NE gap by 38% relative to ES at α=0.5; newer architectures with unverified regularity may call for PS screening plus selective full-scale validation.Source: §5 · pp. 4–5
• Unlike classical scaling laws that predict absolute performance for homogeneous architectures, PS predicts relative gaps across coupled axes in heterogeneous models.The discussion extends PS to LLMs and estimates a ∼5–8× reduction in per-hypothesis compute and carbon footprint.Source: §5 · p. 5
• 79–88% compute reduction, noise-floor prediction accuracy, and perfect rank preservation (ρ = 1.0) among actionable hypotheses are achieved across both models.Source: §5 · p. 5
• For newer architectures with unverified regularity conditions, hybrid PS screening with selective full-scale validation offers the best quality balance.Source: §5 · p. 5

A.1 Hypothesis Categories

• Each hypothesis changes one model aspect; architecture, feature, and data changes are the three categories, and full-scale evaluation is the bottleneck.Source: §A.1 · p. 8

A.2 Scaling Configurations

• TS-1 uses DD, HSR, and ES to halve trainer count while preserving data-distribution fidelity; FM-1 scales five axes for 79% compute reduction under a 50% data-reduction ceiling.Table 5 summarizes the selected scaling-function compositions for each model.Source: §A.2 · p. 8, Table 5 · p. 8

A.3 Extended Per-Head and Predictor Analysis

• On TS-1, all three predictors fall within τAA; EGP improves mean MAE by 0.13 τAA and SE by 0.12 τAA over the raw proxy.EGP raises Spearman ρ from 0.755 to 0.8 and provides calibrated uncertainty for rejecting unreliable predictions, unlike the raw proxy and linear predictor.Source: §A.3 · p. 8
• On FM-1, the GP reduces CTR-1 and CTR-2 MAE by 67% and 57%, respectively; its prior corrects greater distortion on CTR heads than CVR-1.The prior is learned from 9 training hypotheses; all three GP-predicted heads rank the same 7 held-out hypotheses in the same order (ρ = 0.714).Source: §A.3 · pp. 8–9
• Confirming the predictor patterns and extending comparison to a larger foundation-model hypothesis pool remain future directions.Source: §A.3 · p. 9

A.4 Compute reduction

• Compute reduction models training compute from trainer count and sample volume, adjusted for per-device throughput.PS proxy and full-scale runs use the same hardware type, so trainer-count changes reflect infrastructure reduction without throughput differences between hardware types.Source: §A.4 · p. 9

B.1 Rank Preservation

• Rank preservation correlates predicted and true gaps only for non-neutral hypotheses, excluding gaps indistinguishable from noise; perfect preservation is ρrank = 1.Preserved rankings do not guarantee accurate gap magnitudes.Source: §B.1 · p. 9

B.2 Normalized Entropy (NE)

• NE normalizes predictive quality against base-rate entropy, enabling comparisons across tasks, periods, and data splits; multi-task models report it per head.A model matching the base rate has NE = 1; lower values indicate better predictive performance.Source: §B.2 · pp. 9–10

C.1 Quantized Scaling Operator

• The quantized scaling operator keeps scaled integer parameters positive and hardware-aligned while leaving values below a minimum threshold unchanged.The operator rounds to an alignment granularity such as 8 and enforces a minimum value such as 1.Source: §C.1 · p. 10

C.2 Scaling Function Descriptions

• Scaling functions reduce different resource axes: DD preserves data distribution, HSR reduces embedding-table memory, ES narrows embeddings, FS prunes features, WAS scales dimensions uniformly, and DS truncates layers.DD uses uniformly sampled data; HSR leaves hash sizes below 5,000 unchanged; feature pruning respects shared-table and dependency constraints; DS removes deepest layers first.Source: §C.2 · pp. 10–11

D Assumption Verification

• All proxy and full-scale runs decrease monotonically in NE and plateau within the final 10% of training; repeated-baseline terminal variation defines τAA.The passage reports τAA as small relative to the hypothesis gaps of interest.Source: D Assumption Verification · p. 11
• Proxy gaps cluster around the identity line, and gap-transferability MAE is below 1.03 τAA across models and heads.Source: D Assumption Verification · p. 11
• On TS-1, DD + HSR + ES has a sign-reversal rate below 10%, concentrated among near-neutral hypotheses; high actionable-hypothesis reversals flag a mechanism for exclusion.Source: D Assumption Verification · p. 11

E Comparison of Scaling Functions

• ES scales embedding-related dimensions only, while WAS scales all dimension parameters uniformly.For tightly coupled FM-1, WAS removes width asymmetry; the paper adopts WAS for FM-1 and ES for simpler, embedding-dominated TS-1.Source: E Comparison of Scaling Functions · pp. 11–12
• 38% lower NE gap with WAS: the gap changes from −0.0090 to −0.0056, with 25% higher QPS and 42.9% lower FLOPs per sample than ES.The comparison uses scaling ratio 0.5 on a foundation model approximately 100× TS-1 FLOPs, trained on 1,024 NVIDIA B200 trainers.Source: E Comparison of Scaling Functions · p. 12, Table 8 · p. 12

F Compute Budget-Optimal Proxy Selection: Full Procedure

• The procedure searches scaling-function compositions and prediction models under a compute budget, then selects the combination with the lowest cross-validated prediction error.It uses a benchmark corpus of hypotheses with known full-scale gaps; the search limits composition depth to three and incurs its main cost in one-time proxy training.Source: F Compute Budget-Optimal Proxy… · pp. 12–13

G.1 Reference Curve Selection

• Reference curves are filtered by the range of their relative NE-gap trend, then ranked by correlation; references must reach ρ ≥ 0.9999 or prediction is rejected.The procedure uses a single reference because larger bases can let OLS overfit short-window alignment noise.Source: §G.1 · pp. 13–14

G.2 Empirical Gaussian Process Details

• The EGP prior uses the pointwise mean and sample covariance of historical training curves, capturing their correlation patterns without manual kernel design.A coefficient-of-variation threshold rejects predictions deemed unreliable.Source: §G.2 · p. 14

H Predictable Scaling on Public Data

• The public evaluation tests whether PS generalizes to four recommendation architectures trained on Criteo Terabyte, using raw proxy, linear predictor, and EGP across six scaling configurations.The models are DLRM, DeepFM, xDeepFM, and DCN-v2.Source: H Predictable Scaling on Public Data · p. 14

H.1 Hypothesis Design

• The 27-hypothesis pool spans embedding size, DNN width and depth, joint embedding–architecture changes, and training hyperparameters, with 17 training and 10 stratified held-out hypotheses.The effect-size tiers include neutral, weak, and strong changes.Source: §H.1 · pp. 14–15

H.2 Scaling Configurations

• Six public configurations scale different resource axes at a uniform 0.5 factor, enabling comparison of single-axis and multi-axis proxy fidelity.The design also tests whether combining axes introduces interaction effects.Source: §H.2 · p. 15, Table 9 · p. 15

H.3.1 GP Predictor Value

• 37–80%: the GP reduces MAE when raw-proxy error exceeds 1 τAA across the four public models.When raw-proxy error is already low, the GP adds negligible or modest benefit.Source: §H.3.1 · p. 16

H.3 Cross-Model Results and Analysis

• Across architectures, the GP improves MAE most where the raw proxy is degraded, while linear prediction adds little beyond the raw proxy on xDeepFM and DCN-v2.Linear prediction improves DLRM MAE on five of six configurations but is identical to the raw proxy on five of six configurations for each of xDeepFM and DCN-v2.Source: §H.3.1 · p. 16

H.3.2 Scaling Function Effectiveness and Architecture Dependence

• The best scaling configuration differs across all four architectures, so applying one configuration universally would be suboptimal for at least three.The reported best choices are data_ds, hash_only, hash_data, and hash_emb.Source: §H.3.2 · p. 16

H.3.3 Regularity Condition Violations and Failure Modes

• Embedding-only scaling has poor rank preservation on all four public models, and GP fitting does not repair the structural signal loss.This contrasts with its reported effectiveness on the full-scale models; configurations with poor benchmark-corpus ranking can be excluded during proxy selection.Source: §H.3.3 · p. 17
• Scaling compositions can introduce interaction effects that violate rank-preservation conditions more severely than either component alone.The selection procedure evaluates compositions on a benchmark corpus and rejects those that degrade proxy fidelity.Source: §H.3.3 · p. 17
• DeepFM hash_emb ranks hypotheses worse (ρ = −0.491) than either hash-only (ρ = 0.939) or embedding-only (ρ = 0.115) scaling alone.The passage attributes the distortion to jointly reducing embedding-table rows and vector widths in the factorization-machine layer.Source: §H.3.3 · p. 17
• DCN-v2 is the hardest tested architecture: its best GP rank correlation is ρ = 0.515, and all_scaled has GP MAE of 1.063 τAA.The text attributes the difficulty to tightly coupled cross-network interactions and suggests that uniform α = 0.5 scaling may be too aggressive.Source: §H.3.3 · p. 17

H.3.4 Cross-Architecture Synthesis

• The GP reduces MAE on all 24 model-configuration pairs, with improvements from 3% to 80% over the raw proxy.The best scaling configuration differs across all four architectures, and universal use would be suboptimal for at least three.Source: §H.3.4 · pp. 17–18
• Configurations with negative ρ are identifiable in the benchmark corpus and would be excluded by the selection procedure before use on new hypotheses.Source: §H.3.4 · p. 18
• Tightly coupled feature interactions make DCN-v2 and xDeepFM harder to scale than DLRM, and the GP cannot fully compensate when scaling destroys the signal.Source: §H.3.4 · p. 18
• Embedding scaling has poor rank preservation across the public architectures, unlike its reported effectiveness on full-scale models.Source: §H.3.4 · p. 18
• Data downsampling’s robustness on larger datasets does not fully transfer to smaller Criteo, where 50% downsampling may remove too much signal for complex-interaction architectures.Source: §H.3.4 · p. 18
• The best configurations achieve GP MAE of 0.285–0.541 τAA on three models; DCN-v2 reaches 0.430 τAA MAE but has limited rank preservation (ρ = 0.515).Source: §H.3.4 · p. 18
• The figures plot predicted against actual accuracy gaps for six configurations per model; tighter clustering around the diagonal indicates greater proxy fidelity.Source: Figure 2 · p. 23, Figure 3 · p. 24, Figure 4 · p. 25