PaperlayerView exact revision on arXiv

Source-linked AI summary

Small-Scale Experiments: Are We There Yet?

Nicholas Lourie, Kyunghyun Cho, Karen Ullrich, Sanae Lotfi

arXiv:2608.11859v1cs.LG

TL;DR

Small-scale experiments have seemed unreliable because scaling laws require extensive hyperparameter tuning. The paper develops a holistic, model-centric methodology and shows that well-tuned small models reveal scaling laws, while extrapolation remains statistically limited.

  • Problem

    Scaling laws have not reliably transferred to small models, leaving unclear whether cost-effective small-scale experiments can deliver their promised value.

  • Method

    The paper combines extensive hyperparameter exploration, loss-surface analysis, and diagnostic checks into a holistic methodology for model-centric research.

  • Results

    Scaling laws extend to models as small as 4M parameters, but require rigorous tuning because small models are highly hyperparameter-sensitive.

  • Takeaways & Limitations

    Thoroughly explore hyperparameters at small scale, then use simple rules to scale model-centric conclusions upward.

  • Takeaways & Limitations

    The approach is restricted to model-centric research because changing data breaks pretraining loss as a proxy for downstream capabilities.

Abstract

from arXiv · show

Scaling laws promised cost-effective experiments; six years later, they have yet to fully deliver. Instead, researchers have found them unreliable at small scales (starting at 4M parameters) and concluded that sizable models cannot be avoided. We show this is not the case: the confounding factor is hyperparameters. Small models are highly sensitive, but hyperparameter sensitivity fades with scale. This small-scale sensitivity makes scaling laws easy to miss because they only emerge on the fully tuned frontier, and reaching that frontier requires an extensive search far beyond what most ever run. By ablating the basic scaling law recipe, we show well-tuned hyperparameters matter more than any other ingredient. Further, we reveal why those hyperparameters become easier to find: as scale increases, the hyperparameter loss surface becomes lower dimensional. Nevertheless while scaling laws exist in small models, extrapolation hits statistical limitations. A holistic approach is required. Synthesizing our insights with the recent literature, we develop a new methodology for model-centric research and demonstrate it on a question that once took the field years to settle: where to place normalization layers in the transformer architecture. From small-scale experiments, we recover the large scale result: pre-normalization works better as models grow in size. With the right tools and a better understanding, small-scale experiments can deliver on scaling laws' long-awaited promise.

1 Introduction

Small-scale experiments have struggled to reveal transferable scaling laws because hyperparameters disproportionately affect small models and require extensive tuning. The paper argues that rigorous small-scale exploration, diagnostics, and simple scaling rules can reduce costs while improving research rigor.

  • Rigorous hyperparameter tuning is the key ingredient determining final quality, while scaling laws remain robust to choices such as parameter counting and learning-rate decay.Poorly tuned hyperparameters can be disastrous, especially for small models.
  • Scaling laws emerge down to models with 4M parameters, but predictable scaling appears only on the fully tuned frontier because smaller models are harder to tune.Revealing these laws may require searching hundreds of configurations, although 4M-parameter models train in under an hour on a single GPU.
  • Small-scale models are difficult to tune because good configurations occupy less of hyperparameter space, while increasing parameters and data lowers the loss surface’s intrinsic dimension.As scale grows, hyperparameters become easier to find and simple rules can carry them upward.
  • The proposed methodology combines noisy quadratic limits, scaling laws, and perplexity–capability correspondence to turn assumptions about scaling into diagnostics for qualitative understanding.Small-scale experiments provide exploration that can test the assumptions underlying larger-scale conclusions.
  • The resulting model-centric regimen is to thoroughly understand hyperparameters at small scale, then use simple rules to scale up, reducing costs while increasing rigor.The approach focuses on improving the model while holding data fixed; data-centric research may require different techniques.

2 Background

This section introduces scaling laws, perplexity–capability correspondence, and the noisy quadratic limit for analyzing hyperparameter tuning, then describes experiments that test extrapolation from small to larger models.

  • Hyperparameter loss surfaces: Near optima, hyperparameter loss surfaces are approximately quadratic, low-rank, and normally distributed around their mean with additive noise.The noisy quadratic limit uses these properties to characterize tuning difficulty and attainable performance.
  • Hyperparameter loss surfaces: The noisy quadratic limit makes tuning difficulty practical to estimate by locating the asymptotic regime with a scalar threshold instead of estimating the full Hessian.Its parameters describe attainable performance, score concentration, effective hyperparameter dimension, and random-seed noise.
  • Scaling laws: Scaling laws describe the best attainable pretraining loss by balancing model parameters and data under a fixed compute budget.Their classical formulation omits hyperparameters and therefore describes the fully tuned frontier.
  • Experimental design: Experiments probe hyperparameter loss surfaces across scales using random search, with training on 4M–34M models and extrapolation evaluated on held-out 67M–268M models.The study uses effective parameter counts and discards the two smallest token budgets because shorter runs fit scaling laws poorly.

3 Estimating (Small-Scale) Scaling Laws

Small-scale scaling laws are robust to parameter-count definitions and methodological refinements, but rigorous hyperparameter tuning is essential: insufficient search can hide the law entirely, while broader search makes it accurate.

  • 3.1 Methodology: Parameter-count definitions have limited impact on scaling laws, so the study uses effective parameters—counting attention and unembedding but excluding embedding—to keep c = 6pd exact.Six parameter-count variants each yield a reasonable scaling law.
  • 3.1 Methodology: Methodological refinements improve precision without enabling the law: per-budget tuning cuts test MSE by 50%, learning-rate decay by 98%, while tying exponents has little effect.A recognizable law appears even with the basic approach, and tying exponents slightly improves test performance but hurts validation.
  • 3.2 Hyperparameter tuning: 256 configurations are required for an accurate scaling law: with 4 it is absent, with 16 incomplete, and with 64 visible but weakly extrapolative.Suboptimal configurations inject noise that obscures the law; only the largest sweep yields an accurate estimate.
  • 3.2 Hyperparameter tuning: Rigorous hyperparameter tuning is the decisive ingredient because small-scale laws require searches far more extensive than most researchers run.This search requirement explains why scaling laws are easy to miss in small-scale experiments.
  • 3.2 Hyperparameter tuning: As models scale, hyperparameter sensitivity decreases because near-optimal configurations occupy more of the search space and random configurations increasingly approach the optimum.Scaling parameters and data together collapses the gap between minimum and upper-percentile validation losses, whereas scaling only one factor can reduce sensitivity less.

4 Understanding the Hyperparameter Loss Surface Across Scales

As models scale, hyperparameter sensitivity decreases because the loss surface becomes lower dimensional, with parameter count driving this change more than data. Consequently, small models demand extensive hyperparameter search, whereas large models are easier to tune and adapt.

  • Search-space structure: As scale increases, near-optimal configurations occupy more of the search space, concentrating random-search scores near the best achievable loss.This makes larger models trained longer easier to tune because good configurations become more common.
  • Drivers of sensitivity: Sensitivity decreases through both parameters and data, but neither factor alone explains the effect, while parameters have the stronger influence on effective dimension.The sensitivity gap between near-optimal and minimum validation losses measures how difficult the search is.
  • Loss-surface geometry: Hyperparameter sensitivity decreases with scale because the hyperparameter loss surface becomes lower dimensional, with parameters driving this change more than data.The effective dimension remains similar across training checkpoints, while increasing parameters clearly reduces it.
  • Loss-surface geometry: The effective hyperparameter count γ, defined as the loss surface’s intrinsic dimension near the optimum, drops toward 1 as models scale.γ is estimated by fitting a noisy quadratic to near-optimal random-search results.
  • Experimental implications: Small-scale experiments require extensive hyperparameter search, whereas large-scale experiments are gentler, lower-dimensional, and easier to adapt with standard techniques.Simple quantitative models capture coarse surface behavior but fail to characterize finer details, limiting extrapolation to larger scales.

5 Designing Small-Scale Experiments

The section proposes a holistic methodology: tune and measure at small scales, understand pretraining-loss scaling, diagnose failures, and scale up the winner. Applied to normalization placement, small-scale experiments recover pre-norm as the better transformer choice while highlighting extrapolation’s statistical limits.

  • §5.1 Methodology: The proposed methodology tunes and measures small models, tracks how pretraining loss changes, then carries the winner to larger scales because capabilities correspond to perplexity and hyperparameter sensitivity decreases with scale.It uses fixed pretraining data, rigorous tuning, small-scale scaling laws, and the expectation that final scale-up is easier.
  • §5.1 Methodology: Diagnostics test whether tuning, loss-capability correspondence, and scaling laws behave as expected, turning failures into evidence of hyperparameter, model, or implementation problems.The noisy quadratic limit diagnoses whether random search approached the optimum, while common loss–accuracy trends support equal-loss equal-capability comparisons.
  • §5.2 Statistical limits: Scaling laws can exist at small scales, but extrapolation eventually compares unreliable irreducible-error estimates because sampling noise is magnified away from the observed data.The law is more reliable near the data, and models cannot simply be ranked by their lowest extrapolated final loss.
  • §5.3 Normalization case study: In the normalization case study, post-norm is harder to tune: its noisy-quadratic asymptote emerges deeper in the search tail, indicating greater hyperparameter sensitivity.Pre-norm’s limit emerges readily, whereas post-norm maintains a large suboptimal peak and reaches its asymptotic regime later.
  • §5.3 Normalization case study: Pre-norm scales better over the explored sizes and remains more efficient when both architectures are assumed to reach the same final loss.With separate irreducible errors, post-norm appears to win beyond tested scales; sharing the irreducible error preserves pre-norm’s advantage, which is most reliable near the data.

6 Related Work

Related work established scaling laws as a tool for extrapolating large-scale training from smaller experiments, but found small-scale estimates difficult to reproduce and downstream capabilities irregular. This paper revisits the small-model regime, arguing that rigorous tuning restores scaling laws and situates the approach among loss-based, capability-based, and hyperparameter-scaling methods.

  • Scaling laws: Scaling laws evolved from modeling data effects to using parameters and compute to manage the rising cost of experiments (Kaplan et al., 2020).Subsequent work found discrepancies across major studies and unreliable estimates from small models, whereas this paper reports scaling laws at a few million parameters with rigorous tuning (Kaplan et al., 2020).
  • Capability prediction: Downstream scaling laws have had mixed success because matching parameters, data, and compute does not ensure matching pretraining quality (Ivgi et al., 2022; Mahmood et al., 2022).Researchers therefore modeled downstream performance from pretraining loss, but extrapolation remains task-dependent: some tasks are predictable while others exhibit breakthrough or inverse-scaling behavior (Xia et al., 2023; Huang et al., 2024; Du et al., 2024; Gadre et al., 2025; Chen et al., 2025; Lourie et al., 2025b; Wei et al., 2022; Srivastava et al., 2023; McKenzie et al., 2023; Wilcox et al.).
  • Capability prediction: Equal pretraining loss often corresponds to equal capabilities across changes in scale, hyperparameters, architecture, and tokenizer, offering a more general regularity than direct capability extrapolation.Carefully chosen metrics can mitigate, but not eliminate, irregular capability behavior (Schaeffer et al., 2023, 2025; Du et al., 2024; Zhao et al., 2026).
  • Hyperparameter scaling: Hyperparameter transfer uses parametrizations with scale-stable optima or scaling laws for optimal values, while related work also models performance from complete hyperparameter configurations across scales (Yang et al., 2021, 2024; Dey et al., 2025; Mlodozeniec et al., 2026; DeepSeek-AI et al., 2024; Li et al., 2025b; Bergsma et al., 2025; Zhang et al., 2026).The latter approach differs in aim by fitting across publicly available training runs rather than extrapolating from small scales.

7 Conclusion · Appendix

Small-scale experiments can reveal scaling laws, but only on a fully tuned frontier reached through extensive hyperparameter searches. They offer affordable, robust methodology while requiring careful diagnostics and remaining limited for data-centric prediction.

  • 7 Conclusion: Scaling-law conclusions drawn at small scales do not reliably transfer when regularity is assumed to emerge only at large scales.This belief reflects the unfulfilled promise that scaling laws would keep experiments cheap.
  • 7 Conclusion: Small models are hyperparameter-sensitive, so scaling laws emerge only on the fully tuned frontier that researchers rarely reach.Scaling laws extend to very small sizes, but finding the frontier requires extensive search.
  • 7 Conclusion: Pretraining loss can proxy capability only when data is held fixed, because changing the data breaks the perplexity–capability correspondence.Scaling laws predict pretraining loss, and better loss does not always imply better capability outside the fixed-data condition.
  • 7 Conclusion: Small-scale experiments make rigorous testing affordable by enabling hundreds of model trainings and statistical diagnostics such as the noisy quadratic limit.Their value extends beyond compute savings because they can make conclusions more robust.
  • 7 Conclusion: Scaling laws can exist at small scales, but extrapolation faces statistical challenges and requires judgment across multiple diagnostics.Researchers should assess tuning difficulty, whether tuning becomes easier, and how scaling compares where the law is most trusted.
  • 7 Conclusion: Without pretraining loss as a proxy or reliable downstream scaling laws, small-scale experiments have limited ability to predict large-scale outcomes for data-centric research.The authors identify unlocking small-scale experiments for data-centric research as an important direction for future work.

A Experimental Design

This section presents the experimental details and explains the rationales behind the experiments.

  • The section documents the experiments’ methodological details.
  • It explains the rationales motivating the experimental design.
  • Together, the section provides the documented basis for understanding how the experiments were designed.

A.1 Modeling

The study uses a Llama-based modeling recipe implemented in Meta Lingua, with controlled data, architecture, optimization, scale, and evaluation choices. Experiments span effective parameter scales from approximately 4M to 268M, training each scale for 32 times its target parameter count in tokens.

  • Architecture: The recipe uses a decoder-only Llama model from Meta Lingua, with RMSNorm, SwiGLU, RoPE, and pre-normalization by default.Post-normalization is used only when explicitly specified and follows Xiong et al. (2020).
  • Data: Training uses the 100B-token FineWeb-Edu subset, with preprocessing and data shuffling configured for the single-GPU scale range studied.The experiments consider models from 4M to 134M parameters in one data chunk, with a separate eight-way split described for larger-scale training.
  • Optimization: Models are trained with AdamW and a warmup-stable-decay schedule whose linear cooldown reaches 1e-6 times the peak learning rate.Effective parameter counts range from 2^22 ≈4M to 2^28 ≈268M, and training uses 32 times the target count in tokens.
  • Evaluation: Pretraining is evaluated with bits-per-character on validation data, while capabilities use Lingua’s EleutherAI lm-evaluation-harness integration.BPC adjusts for tokenization differences, and validation evaluation is limited to 1,000 steps.

A.2 Search Distributions

The experiments use search distributions over hyperparameters across model scales, with effective parameters defining scale and shared non-architectural settings isolating architectural effects. The study compares a hand-crafted model ladder with prenorm and postnorm searches, including expanded postnorm tuning because it is harder to tune.

  • Search setup: Three experiments are defined by distinct hyperparameter search distributions, each randomly sampling configurations across scales measured by effective parameters proportional to FLOPs per token.Compute is related by c = 6pd, where c is compute, p effective parameters, and d tokens.
  • Search setup: Configurations sharing an index use the same non-architectural hyperparameters across designs, so experimental differences arise from architecture alone.The shared-hyperparameter search distribution is shown in Figure 16, and appropriate sampling scales such as logarithmic scales are important for efficient search.
  • Model Ladder (ladder): The model-ladder experiment searches scales from approximately 4M to 268M effective parameters and uses hand-crafted architectural settings subject to implementation constraints.After excluding failed runs, it searches 867, 826, 867, 934, 128, 128, and 64 configurations at 4M, 8M, 17M, 34M, 67M, 134M, and 268M, respectively.
  • Pre-Normalization (prenorm) vs. Post-Normalization (postnorm): Prenorm and postnorm are evaluated at approximately 4M, 34M, and 134M effective parameters, reusing random-experiment configurations while rerunning them under post-normalization.Postnorm receives expanded searches because its architecture increases tuning difficulty, yielding 511, 512, and 128 configurations at the three scales after excluding one failed run.

A.3 Analysis

The analysis evaluates scaling laws by fitting on small models and testing extrapolation to larger ones, while modeling hyperparameter loss surfaces using asymptotic-regime configurations and transformed features.

  • Noisy Quadratic Limit: Noisy quadratic limits are fitted with opda using LD Highest Density confidence bands and thresholds set at varying score-distribution percentiles.Percentiles differ across parameter–data budgets, with the asymptotic regime generally increasing.
  • Scaling Laws: Scaling laws are fit on 4M–34M models, used for decisions on 67M–134M models, and finally tested by extrapolation to 268M models.Training fits the laws, validation guides research decisions, and testing provides the final evaluation.
  • Scaling Laws: The scaling-law fits use Equation 3 on selected validation losses for each parameter–data budget pair, excluding the first two checkpoints to improve fit quality.Depending on the analysis, the selected loss is either the best validation loss or the loss from the run achieving minimum final loss at each scale.
  • Interpretable Models of the Hyperparameter Loss Surface: Hyperparameter loss-surface models use all configurations within the asymptotic regime rather than only the single best configuration, after scale-appropriate feature transformations.Transformations include logarithmic scales for most parameters and a log-plus-one scale for warmup.
  • Interpretable Models of the Hyperparameter Loss Surface: The loss-surface pipeline expands features quadratically, applies RobustScaler preprocessing, and predicts with a tuned multilayer perceptron fitted to percent excess loss after the scaling law.The MLP uses one hidden layer of 128 units and grid-searches alpha and initial learning rate.

A.4 Implementation

The implementation adapts training across model scales through gradient accumulation, scale-specific batch settings, and hardware allocation. It also modifies Meta Lingua and uses a fixed A100-based cluster setup for training.

  • Software: The models are trained with Meta Lingua, modified for the 100B-token FineWeb-Edu subset, linear-decay WSD scheduling, and PyTorch SDPA instead of Flex Attention.Flex Attention was replaced because it could not generate kernels for some randomly sampled transformer shapes.
  • Hardware: Training uses 80GB NVIDIA A100 GPUs on a SLURM cluster, with 1 GPU for 4M–134M models and 8 GPUs for 268M models.

B Interpretable Models of the Hyperparameter Loss Surface

The section develops interpretable models of the hyperparameter loss surface by combining scaling laws for optimal loss with noisy quadratic models of hyperparameter effects. Low-rank and scale-dependent variants capture some extrapolation behavior, but static quadratics remain surprisingly strong and dynamic variants are inconsistent.

  • Model construction: The proposed model combines scaling-law predictions of optimal loss with a quadratic that increases reducible loss as hyperparameters move away from the optimum.The quadratic minimum is set using the scaling law, while the reducible component reflects distance from optimal hyperparameters.
  • Model comparison: Table 2 shows that static quadratics already extrapolate strongly, while varying the optimum helps and varying curvature hurts; the nonlinear model performs worst among hyperparameter-based models.Models are fit on small scales and evaluated on larger held-out scales using MSE.
  • Low-rank structure: Diagonalizing the Hessian projects hyperparameters into a lower-dimensional subspace, enabling a low-rank quadratic model of the loss surface.The projection retains the non-zero Hessian eigendirections and represents the optimum in that latent space.
  • Model variants: The model variants include static, dynamic-optima, dynamic-sensitivity, dynamic, and polynomial forms, with the Hessian rank restricted to at most γ = 3.Dynamic optima vary latent optimal hyperparameters with log scale; dynamic sensitivity varies Hessian eigenvalues as scale power laws.
Loading 2608.11859v1…