PaperlayerView exact revision on arXiv

Source-linked AI summary

Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models

Houyi Li, Wenzhen Zheng, Qiufeng Wang, Zhenyu Ding, Haoying Wang, Zili Wang, Shijie Xuyang, Ning Ding, Shuigeng Zhou, Xiangyu Zhang, Daxin Jiang

arXiv:2506.10972v3cs.LGcs.AI

TL;DR

Expensive LLM training creates a scaling gap because small-scale findings may not transfer reliably to production scales. Farseer models the loss surface L(N, D) with N-dependent data scaling and differential, iterative fitting, achieving better empirical fit and extrapolation than Chinchilla while offering compute-allocation guidance. Its scope is limited to Llama-style decoder-only Transformers and two pre-training data distributions.

  • Problem

    Small-scale LLM experiments often fail to transfer to expensive production scales, hindering efficient evaluation of new training methods.

    Source: §1 · p. 3

  • Method

    Farseer combines an N-dependent loss formulation with Differential Piecewise and Multi-round Iterative fitting to model L(N, D).

    Source: §1 · p. 3, §3.2 · p. 5

  • Results

    Farseer provides a significantly better empirical fit and more robust extrapolation than Chinchilla, with lower average relative error and variance across fitting ranges.

    Source: §1 · p. 3, §5.2 · p. 9

  • Takeaways & Limitations

    Farseer supports reliable large-scale performance prediction from tractable small-scale experiments and provides data-driven guidance for compute allocation.

    Source: §5.3 · p. 9, §7 · p. 10

    The evaluation is restricted to Llama-style decoder-only Transformers and two pre-training data distributions.

    Source: J Discussion and Future Work · p. 32

Abstract

from arXiv · show

Training Large Language Models (LLMs) is prohibitively expensive, creating a critical scaling gap where insights from small-scale experiments often fail to transfer to resource-intensive production systems, thereby hindering efficient innovation. To bridge this, we introduce Farseer, a novel and refined scaling law offering enhanced predictive accuracy across scales. By systematically constructing a model loss surface $L(N,D)$, Farseer achieves a significantly better fit to empirical data than prior laws (e.g., Chinchilla's law). Our methodology yields accurate, robust, and highly generalizable predictions, demonstrating excellent extrapolation capabilities, improving upon Chinchilla's law by reducing extrapolation error by 433\%. This allows for the reliable evaluation of competing training strategies across all $(N,D)$ settings, enabling conclusions from small-scale ablation studies to be confidently extrapolated to predict large-scale performance. Furthermore, Farseer provides new insights into optimal compute allocation, better reflecting the nuanced demands of modern LLM training. To validate our approach, we trained an extensive suite of approximately 1,000 LLMs across diverse scales and configurations, consuming roughly 3 million NVIDIA H100 GPU hours. We are comprehensively open-sourcing all models, data, results, and logs at https://github.com/Farseer-Scaling-Law/Farseer to foster further research.

1 Introduction

Scaling laws aim to predict LLM performance as model size and training data increase, but Chinchilla’s fixed data-scaling parameters limit accuracy across model sizes. Farseer addresses this scaling gap with an N-dependent refined law and reports more accurate fitting and extrapolation.

1 Introduction

  • Chinchilla’s fixed data-scaling parameters capture average behavior, reducing accuracy at model sizes far from its calibration range.The limitation is especially relevant for predicting performance across arbitrary (N, D) surfaces.Source: §1 · p. 2
  • Figure 2 shows Farseer predictions lying almost exactly on the ground-truth BPC curve, while Chinchilla systematically under- and overestimates at small and large D.The comparison indicates a better empirical fit across the plotted data range.Source: Figure 2 · p. 2
  • Farseer is introduced as a refined scaling law and methodology developed from training over 1,000 LLMs.The approach is intended to bridge the gap between tractable small-scale experiments and expensive production-scale training.Source: §1 · p. 3
  • Farseer explicitly models data-scaling effects as dependent on model size N, improving the fit to empirical LLM data.This targets Chinchilla’s assumption that data-improvement rates are uniform across model sizes.Source: §1 · pp. 2–3
  • Farseer also provides compute-allocation guidance and releases models, data, logs, and code for further research.The contribution list links the refined law to optimal D/N allocation and comprehensive open-sourcing.Source: §1 · p. 3

2 Preliminaries

The preliminaries formulate loss as model-size, data-size, and interaction components, then describe the experimental setup used to study this surface. The study trains roughly 1,000 Llama-style models and evaluates them with BPC on an unseen, high-quality validation set.

2.1 General Loss Formulation

  • Farseer decomposes loss into constant, model-size, data-size, and N–D interaction components while fitting the functional forms from small-scale data.The interaction term H(N, D) is explicitly included alongside U(N) and V(D).Source: §2.1 · p. 3

2.2 Basic Settings

  • Approximately 1,000 models are trained with web, mathematical, and code data using a 65,536-token BPE vocabulary.The models use a standard language-modeling objective and two distinct data mixtures.Source: §2.2 · p. 3
  • Model sizes N and data sizes D follow a geometric progression with common ratio 2 in the Llama-style training setup.The architecture uses AdamW, warmup followed by cosine decay, and a fixed sequence length of 2,048 tokens.Source: §2.2 · p. 3
  • Evaluation uses BPC on a separate 30-million-token validation set containing filtered web pages, books, and academic papers.The validation samples are unseen and the metric measures compression efficiency rather than validation loss.Source: §2.2 · pp. 3–4
  • Figure 3 examines differential BPC terms as functions of D and N, with the strongest relationship being ΔDL versus D at fixed N (R2 = 0.9807).This relationship motivates adopting a power-law form in D for the main analysis.Source: Figure 3 · p. 4

3 Methodology

Farseer uses differential analysis to establish model–data interaction, fits its data-dependent and model-dependent loss components, and combines them into a scaling law for L(N, D).

3.1 Differential Analysis for the Existence of Higher-Order Interaction Term H(N, D)

  • Differential losses depend systematically on both N and D, showing that the interaction term H(N, D) cannot be reduced to a single variable or additively separated.This motivates a fitting procedure that explicitly captures coupled model-size and data-size effects.Source: §3.1 · p. 4

3.2 Functional Form for Data-Dependent Term V (D) + H(N, D)

  • The data-dependent component is modeled as B(N)D^-A(N), with model-size-dependent exponent A(N), coefficient B(N), and residual error εR(N, D).The form follows the observed power-law relationship between ΔDL(N, D) and D.Source: §3.2 · p. 5
  • A two-stage procedure estimates discrete A_N and B_N values, then selects and iteratively refines continuous stretched-exponential functions fA and fB.Transformation pairs are chosen by residual minimization, and exponent refinement typically converges in 1–2 iterations.Source: §3.2 · p. 5

3.3 Functional Form for Model-Dependent Residual E + U(N)

  • After subtracting the fitted data-dependent term, averaging residuals over D yields G(N) as a reliable estimate of E + U(N).The residual behaves nearly like Gaussian white noise, with amplitude shrinking from 2×10^-3 to 4×10^-4 as N increases.Source: §3.3 · p. 6
  • The model-dependent residual is fitted through transformation selection and grid search, producing a stretched-exponential form with relative error of only 0.09%.Combining the fitted components yields the fully specified scaling law in Eq. 3.Source: §3.3 · p. 6

4 Farseer’s Properties: Robustness, Generalization, and Extrapolation

Farseer is evaluated for robustness to fitting-data volume, generalization across data recipes, extrapolation beyond the fitting range, and monotonicity in model and data scale.

4.1 Robustness to Fitting Data

  • Relative error on excluded 6.4 B models falls from 6.01 × 10^-3 to 5.87 × 10^-4 as the fitting range expands from 1.9 B to 5.4 B parameters.The error decreases nearly monotonically as additional model-size points are incorporated.Source: §4.1 · p. 7, Figure 6 · p. 7

4.2 Data Distribution Generalization

  • On the English-Chinese recipe, error on excluded 3.2 B models steadily converges to 7.60×10^-4, supporting generalization across substantially different data mixtures.The result indicates reliable predictions under an altered bilingual training-data composition.Source: §4.2 · pp. 7–8

4.3 Extrapolation Capabilities

  • Farseer is tested beyond its fitted (N, D) grid using larger models, larger datasets, and off-grid combinations, with relative errors annotated for each validation point.The extrapolation evaluation includes a 25.1 B model.Source: Figure 7 · p. 8, §4.3 · p. 8

4.4 Formula and Monotonicity

  • Farseer’s loss L(N, D) decreases monotonically as either model size N or data size D increases.This property follows from the fitted functional form and parameter constraints imposed during fitting.Source: §4.4 · p. 8

5 Comparative Analysis of Farseer and Chinchilla

Farseer is compared with Chinchilla on predictive fit, robustness, extrapolation, and compute allocation. Across these evaluations, Farseer models size-dependent data-scaling effects more accurately and provides scaling guidance that grows with compute.

5 Comparative Analysis of Farseer and Chinchilla

  • The comparison evaluates Farseer against Chinchilla for predictive accuracy, extrapolation, and practical guidance on allocating model size and data.These criteria address both formula quality and the usefulness of scaling-law predictions for training decisions.Source: §5 · p. 8, §5.2 · p. 9, §5.3 · p. 9

5.1 Formula Comparison: Predictive Power

  • Farseer’s predicted BPC closely matches empirical values across N and D, while Chinchilla underestimates at low D and overestimates at high D.Farseer’s A(N) and B(N) capture decay rates specific to each model size rather than a single average trend.Source: §5.1 · pp. 8–9

5.2 Property Comparison: Robustness and Extrapolation

  • Farseer achieves lower average relative error and smaller error variance within the fitting range, while retaining low error when extrapolating to a 25.1 B model beyond that range.The comparison varies the fitted N upper bound and finds Chinchilla’s error persistently high in the far-extrapolation setting.Source: §5.2 · p. 9, Figure 8 · p. 9

5.3 Application Comparison: Optimal Computing Resource Allocation

  • Farseer predicts an optimal D/N ratio that steadily increases with compute budget, unlike Chinchilla’s roughly constant ratio around 20.The increasing trend aligns with configurations reported for recent large language models, including Qwen and Llama.Source: §5.3 · p. 9

6 Related Works

Prior scaling-law work established power-law relationships for model and data scale, refined compute allocation, and extended scaling analysis beyond pre-training. The literature also identifies sensitivity to data quality, reproducibility, and broader efficiency and generalization questions.

Related scaling laws

  • OpenAI’s foundational scaling work [22] related cross-entropy loss to model and data scale, while Chinchilla [20] proposed proportional model-data scaling for improved efficiency.Chinchilla used a smaller model with more training data than Gopher under the same compute budget and reported downstream gains.Source: §6 · pp. 9–10
  • Recent evidence questions the reproducibility and universality of Chinchilla-style ratios, because optimal token-to-parameter ratios depend on data quality.These laws remain useful heuristics but require calibration for particular settings.Source: §6 · p. 10

Broader scaling-law research

  • Scaling-law research now includes observational laws, data curation, theoretical mechanisms, inference dynamics, parameter and communication efficiency, and downstream tasks.These directions seek broader explanations of performance patterns and more efficient training or deployment.Source: §6 · p. 10

7 Conclusions

The paper concludes that Farseer characterizes L(N, D) for controlled model families and data recipes, enabling more reliable scaling comparisons. Its framework depends on standardized training and architecture choices so that changes in N and D have interpretable effects.

7 Conclusions

  • Farseer models the loss surface L(N, D) and offers better empirical fit and extrapolation than prior laws such as Chinchilla’s.The paper presents this as a basis for predicting large-scale performance from tractable experiments and informing compute allocation.Source: §7 · p. 10

A General Loss Function Metric Formulas

  • For a fixed model family and data recipe, L(N, D) maps model size and training-data size to expected performance under a fixed training strategy.The formulation treats performance as primarily dependent on N and D after other training factors are controlled or standardized.Source: A General Loss Function Metric Formulas · p. 15
  • The framework assumes other hyperparameters are appropriate enough to prevent severe underperformance, making L primarily a function of N and D.This assumption limits the interpretation of the fitted surface to controlled training conditions rather than arbitrary optimization settings.Source: A General Loss Function Metric Formulas · p. 15

A.1 Standardizing Hyperparameter Settings

  • Appropriate rather than strictly optimal learning rates and batch sizes are sufficient when the performance landscape is relatively flat near competent settings.The study therefore uses scale-aware or standardized hyperparameter choices to reduce fluctuations without requiring exact optima.Source: §A.1 · p. 16

A.2 Defining an Architecturally Standardized LLM Family

  • The paper defines model families broadly enough to compare how architecture and data distributions influence distinct scaling surfaces.A family can share training methods, data pipelines, or architectural principles, but its scaling behavior must be determined empirically.Source: §A.2 · p. 16
  • An LLM family is defined so that a parameter budget N uniquely determines architectural details, including aspect ratio, FFN ratio, attention type, normalization, activation, and MoE use.This standardization lets models within a family share a scaling-law surface while different families or data recipes produce distinct surfaces.Source: §A.2 · p. 16
  • Controlled experiments found broad, flat optima for aspect ratio and FFN ratio, so the paper fixes these shape variables across model sizes.The chosen values need not be globally optimal, but they must remain constant as N varies.Source: §A.2 · pp. 16–17, Figure 9 · p. 17

B Distribution of the Training Dataset

The appendix describes the training-data recipes and the differential piecewise fitting workflow used to construct Farseer. The procedure estimates size-dependent terms, refines their parameterizations, and fits the remaining model-dependent residual.

B Distribution of the Training Dataset

  • The experiments use baseline and English-Chinese data recipes, with dataset mixture weights specified for distinct capability targets.The accompanying table distinguishes baseline performance, coding and mathematical reasoning, and bilingual proficiency strategies.Source: B Distribution of the Training Dataset · p. 17, Table 1 · p. 17

C.1 Algorithm Workflow: Differential Piecewise Fitting

  • Differential Piecewise Fitting estimates A_N and B_N for each model size from finite differences, then fits continuous functions to their discrete estimates.The method uses transformations such as logarithms or power laws to linearize relationships before regression.Source: §C.1 · pp. 18–19
  • The fitting procedure iteratively refines power-law exponents, fits the residual E + U(N), and returns the final scaling-law parameters after convergence.The algorithm uses stretched-exponential forms for the size-dependent functions and minimizes a global residual during refinement.Source: §C.1 · pp. 18–19

C.2 Core Insight: Power-Law Scaling with Data Size (D)

Farseer selects a data-difference perspective whose power-law behavior is stable across model sizes, then fits the resulting coefficient, exponent, and residual functions with transformed functional forms. The evaluation setup uses a filtered 30-million-token validation set and BPC-based compression metrics.

C.2 Core Insight: Power-Law Scaling with Data Size (D)

  • 0.9807 average R2 supports a power-law fit of the loss finite difference against data size across studied model sizes.The log-log fits show small, randomly distributed residuals across the model-size ranges.Source: §C.2 · pp. 19–20
  • The selected differential view models the data-dependent loss as ∆DL(N, D) ≈ B(N)D−A(N), with model-size-dependent coefficient and exponent functions.This formulation follows cancellation of E and U(N) terms and underlies subsequent fitting stages.Source: §C.2 · p. 20
  • B(N) scales tightly with N and A(N) varies smoothly, more regularly than analogous functions from the other differential perspectives.These trends, together with high fit quality, motivate prioritizing the ∆DL-versus-D analysis.Source: §C.2 · p. 20, Figure 10 · p. 20, Figure 11 · p. 21

C.3 Functional Form Selection for A(N) and B(N)

  • Logarithmic transformations of A(N) and B(N), combined with power-law transformations of N, minimize the joint projection residual and yield stretched-exponential forms.The transformation search evaluates candidates from identity, logarithmic, and power functions using least-squares projections.Source: §C.3 · p. 21, Table 2 · p. 21

C.4 Functional Form Selection for Model-Dependent Residual E + U(N)

  • The model-dependent residual E + U(N) is estimated after removing the fitted data-dependent term, averaged over D, and fit with a logarithmic transform of G(N) and a power-law transform of N.The exponent γ is selected by grid search, producing another stretched-exponential form.Source: §C.4 · p. 25

D.1 Validation Set

  • The validation set contains 30 million tokens across web pages, books, and academic papers spanning 55 domains, while BPC measures compression efficiency and requires R1·R2 for cross-family comparisons.Within one model family, comparisons can focus on R1 because vocabulary differences are controlled.Source: §D.1 · p. 25, §D.2 · pp. 25–26

E Point, Line, and Surface Comparison

Farseer uses fitted loss surfaces to compare training recipes continuously across model size and data axes rather than relying only on isolated points or lines. In the English-mixture example, these surface comparisons reveal that small-scale conclusions can reverse at larger scales.

E Point, Line, and Surface Comparison

  • Surface comparisons map relative behavior continuously over model size N and data size D, extending point and fixed-axis line comparisons.The method fits separate Farseer curves from modest experiments and compares the resulting predicted loss surfaces.Source: E Point, Line, and Surface Comparison · p. 26
  • In the 85%- versus 50%-English experiment, regions above the zero plane favor the 50% mixture, whereas regions below it favor the 85% mixture.The zero plane denotes parity in normalized relative BPC difference.Source: E Point, Line, and Surface Comparison · p. 26, Figure 15 · p. 26
  • Small-scale point and line comparisons can suggest one mixture is better, but the surface comparison shows that such conclusions can reverse at larger scales.This illustrates why evaluating the full (N, D) surface matters for extrapolating recipe performance.Source: Figure 15 · p. 26, E Point, Line, and Surface Comparison · p. 27

F Model Parameter Configuration Table

  • The baseline study spans 21 systematically scaled transformer configurations, with non-embedding parameter counts from 2.01 × 10^8 to 6.37 × 10^9.Training data ranges from roughly 1.00 × 10^9 to 5.12 × 10^11 tokens across the configurations.Source: F Model Parameter Configuration Table · p. 27, Table 4 · p. 27
  • The configurations systematically vary transformer depth, width, attention, and data coverage to support analysis across model scales.The models follow Llama architectural principles and are trained with substantial, diverse data intended to avoid under-fitting.Source: F Model Parameter Configuration Table · pp. 27–28
  • An additional EN-ZH configuration uses nine systematically scaled transformer models to test data-generalization capabilities.These settings are analogous to the baseline configurations while changing the training-data recipe.Source: F Model Parameter Configuration Table · p. 28, Table 5 · p. 28

G Consideration of Parameters with Embedding Layers

The paper tests whether embedding parameters should count toward model size and compares fitting procedures for Farseer and Chinchilla. Excluding embeddings improves large-scale extrapolation, while Differential Piecewise Fitting is substantially better for Farseer than standard nonlinear regression.

G Consideration of Parameters with Embedding Layers

  • At Nval = 25.1B, including embeddings gives prediction error 0.029, whereas excluding them yields an error more than four times smaller.Both definitions fit comparably near the training points, but their extrapolations diverge at the larger validation scale.Source: G Consideration of Parameters with… · p. 28, Figure 16 · p. 29
  • The embedding ablation supports excluding embedding-layer parameters from N when seeking robust and generalizable scaling laws.The authors suggest embeddings may not scale with capacity in the same way as transformer compute parameters.Source: G Consideration of Parameters with… · p. 28

H Non-linear End-to-End Fitting

  • Chinchilla fits slightly better with standard nonlinear regression than with Differential Piecewise Fitting.The comparison applies both fitting paradigms to both scaling-law formulations.Source: Figure 17 · p. 29, H Non-linear End-to-End Fitting · p. 30
  • Farseer’s standard nonlinear-regression fit has errors approximately 10 times higher than its Differential Piecewise Fitting result.The nonlinear procedure used over 30,000 initializations but struggled to reach an optimal global solution for Farseer.Source: H Non-linear End-to-End Fitting · pp. 29–30, Figure 17 · p. 29
  • The main comparison uses Chinchilla with standard nonlinear regression and Farseer with Differential Piecewise Fitting.Each model is represented by the fitting method that performed best in the empirical evaluation.Source: H Non-linear End-to-End Fitting · p. 30

I Analysis of Formula Properties

The fitted Farseer loss decreases with both data size and model size across its analyzed domain. The appendix supports this behavior analytically for data and numerically for model size.

I Analysis of Formula Properties

I.1 Monotonicity

  • U(N), B(N), and A(N) are strictly positive for all admissible N and D in the fitted formulation.These positivity conditions support the sign analysis of the partial derivatives.Source: §I.1 · p. 30
  • The derivative with respect to D is strictly negative because A(N), B(N), and D−A(N)−1 are positive.Therefore, increasing data size monotonically lowers L(N, D).Source: §I.1 · p. 30
  • Figure 19 visualizes the negative ∂L/∂D surface across a broad range of log N and log D.The visualization matches the analytical sign conclusion for the data derivative.Source: §I.1 · p. 30, Figure 19 · p. 31
  • The analytical form of ∂L/∂N is prohibitively complex, so its sign is verified numerically across the model-size range.The numerical derivative remains negative throughout the evaluated domain.Source: §I.1 · p. 30
  • Figure 20 shows that ∂L/∂N remains negative, confirming that L decreases monotonically as N increases.This provides numerical rather than closed-form verification for the model-size direction.Source: §I.1 · p. 30, Figure 20 · p. 32

J Discussion and Future Work

The discussion identifies important scope limitations while presenting Farseer as a framework for comparing training settings across arbitrary model and data scales. It also outlines open research directions spanning architectures, data, theory, validation, and training dynamics.

Limitations

  • Farseer has been evaluated only on Llama-style decoder-only Transformers and two pre-training distributions because of computational constraints.The paper reports ongoing investigation of MoE and code/math-dominant data settings.Source: J Discussion and Future Work · p. 32
  • Validation reaches 25 B parameters but not larger models, because larger-scale checks require additional distributed-training implementations that can introduce performance noise and demand more resources.The authors regard 25 B as sufficient for their extrapolation validation, while acknowledging the absence of larger experiments.Source: J Discussion and Future Work · pp. 32–33

Evaluation limitations

  • BPC on a carefully constructed validation set is used instead of downstream tasks, but whether low BPC necessarily implies strong post-training model quality remains open.The validation set is designed around unseen integrity, bias-free composition, diversity, and quality.Source: J Discussion and Future Work · p. 33

Theoretical limitations

  • Farseer characterizes scaling laws from extensive observational data without providing a theoretical derivation from fundamental machine-learning principles.The paper presents this as a broader unresolved challenge for scaling-law research.Source: J Discussion and Future Work · p. 33

Methodological contribution

  • Farseer fits loss surfaces for competing settings, enabling precise comparisons across arbitrary model sizes and dataset sizes rather than only at one tested configuration.The methodology can identify conditions under which one setting outperforms another across the N,D plane.Source: J Discussion and Future Work · p. 33

Future work

  • Future work includes extending scaling-law analysis to code and math data, sparse and linear-attention models, new architectures, data quality and repetition, synthetic data, and broader model classes.The authors also release extensive experimental resources and call for further study of fitting, loss-surface applications, and training dynamics.Source: J Discussion and Future Work · pp. 33–34

Related papers

Loading 2506.10972v3…