ScAn-Bench: Evaluating Scaling Analysis Methodology

1 Introduction

• Existing evaluations focus overwhelmingly on LLMs, often confound methods with inconsistent training choices, and rarely test interactive data-acquisition strategies.Prior static scaling datasets support extrapolation studies but not interactive, reproducible evaluation of acquisition strategies.Source: §1 · p. 1
• ScAn-Bench-LLM and ScAn-Bench-VLM contain 4,524 and 8,024 checkpoints, respectively, and their surrogate predictor approximates training runs in CPU-seconds rather than GPU-hours.The benchmarks span 9 scaling and hyperparameter settings over 1,000 training runs each.Source: §1 · p. 2
• The benchmarks enable reproducible, apples-to-apples comparisons of scaling methods across multiple data modalities and foundation-model families.The authors describe this as the first such comparison.Source: §1 · p. 2
• The paper contributes a systematic framework, two surrogate benchmarks, and a controlled evaluation of sequential data acquisition and extrapolation strategies.The benchmarks target research on scaling analysis methodology.Source: §1 · p. 2

2 A systematic view on scaling analysis

• Scaling analysis approximates the configuration that minimizes pretraining loss under a target compute constraint through data acquisition followed by extrapolation.The compute constraint depends on compute-scaling hyperparameters.Source: §2 · p. 2
• Data acquisition samples and evaluates configurations within a search budget that is usually insufficient for adequate evaluations at the target compute scale.Fair allocation comparisons require standardizing how baselines handle non-scaling hyperparameters, which may otherwise rely on tuned prior knowledge.Source: §2 · p. 2
• The runtime comparison covers 100 evaluations, measuring surrogate queries in CPU-hours and model training in GPU-hours.The caption specifies an 8-core AMD EPYC 9655 CPU and A100 GPUs.Source: Table 1 · p. 3
• Loss extrapolation predicts the minimum achievable loss at a target compute budget along the empirically observed Pareto-optimal frontier.Source: §2 · p. 3
• Coarse extrapolation predicts optimal model size and token count, whereas full extrapolation directly predicts the joint optimal configuration of scaling and optimization hyperparameters.Full extrapolation avoids reliance on heuristic priors.Source: §2 · p. 3

3 ScAn-Bench-LLM and ScAn-Bench-VLM

• The surrogate benchmarks approximate configuration-to-performance mappings with fast CPU queries, enabling evaluations that would otherwise require hours of GPU training.Unlike static tabular benchmarks, they can be queried outside the collected configurations and support continuous design spaces.Source: §3 · p. 3

3.1 ScAn spaces and data collection

• The VLM pipeline trains CLIP models on 60M LAION-400M samples, while the LLM pipeline trains decoder-only transformers on SlimPajama.Source: §3.1 · p. 3
• The bounded ScAn spaces vary model-family-specific scale variables alongside optimization hyperparameters; the pipelines record 8,024 VLM and 4,524 LLM checkpoints.VLM scale variables include encoder widths and sample count; LLM variables include layers, attention heads, embedding dimension, and training tokens.Source: §3.1 · pp. 3–4

3.2 Creating the surrogate benchmarks

• TabPFN provides the strongest upstream-performance prediction across benchmarks and is selected as the default surrogate.The comparison uses Spearman rank correlation and RMSE; the final predictors model each metric independently, with an additional VLM divergence predictor.Source: §3.2 · p. 4

4.1 Evaluated ScAn Methodology

• The evaluation compares Iso-Parameter Profiling, Iso-FLOP Profiling, CARBS, and Random Search, alongside loss-, coarse-parameter-, and full-parameter extrapolation approaches.Full extrapolation uses independent linear regressions for each hyperparameter dimension.Source: §4.1 · p. 5

4.2 Experimental setup

• Target compute is set to 20×Cmax, with acquisition restricted below Cmax; results average 10 seeds and report standard error.Cmax is 2 × 10^16 FLOPs for OpenCLIP and 1.0 × 10^19 FLOPs for LLM.Source: §4.2 · pp. 5–6

4.3 Results

• CARBS achieves the lowest PER and better early incumbent loss, Iso-Parameter yields the highest Hypervolume, and Random Search later surpasses CARBS on best loss.The figure tracks PER, Hypervolume, and Incumbent Loss against cumulative search budget.Source: §4.3 · p. 6, Figure 1 · p. 6
• No single acquisition and loss-extrapolation combination is optimal across benchmarks: Chinchilla 3 is closer under Iso-Parameter, while under CARBS Chinchilla favors OpenCLIP and Kaplan favors LLM.Under Iso-Parameter, Kaplan predictions improve more steadily as compute increases.Source: §4.3 · p. 7, Figure 3 · p. 7
• With standard C ≈ 6ND compute estimation, Chinchilla fails to predict OpenCLIP's optimal model size, while empirical Power-Law fits remain robust across both domains.Power-Law predictions are accurate for all tested acquisition strategies except CARBS, whose localized search yields poor predictions.Source: §4.3 · pp. 7–8, Figure 5 · p. 8, Figure 6 · p. 8
• Random Search outperforms CARBS across tested budgets, and its predictions improve with more acquired data, but independent hyperparameter predictions miss coupled OpenCLIP tower interactions.The comparison evaluates actual loss at predicted configurations against the best empirical loss at the target compute budget.Source: §4.3 · p. 9, Table 4 · p. 9

• Many benchmarks center on smaller MLP/CNNs, fixed training budgets, or spaces without suitable scaling parameters; transformer benchmarks omit independent runs or hyperparameter variation.Source: §5 · p. 9
• LLM studies report optimal-scale prediction discrepancies from inconsistent parameter and compute counting, and high variance and pitfalls when extrapolating from intermediate checkpoints.Source: §5 · p. 9
• Data-constrained training with repeated corpora shows similar power-law dynamics with diminishing returns [49].Source: §5 · p. 10
• Architecture-shape effects over the compute horizon have been studied, but evaluation remains restricted to a highly constrained pool of model sizes.Source: §5 · p. 10

• ScAn-Bench-VLM and ScAn-Bench-LLM provide surrogate benchmarks for systematic, computationally feasible study of scaling methodology and evaluation of existing methods.Source: §6 · p. 10
• The benchmarks operate at compute scales substantially smaller than current state-of-the-art models, so observed trends may not transfer to significantly larger regimes.Source: §6 · p. 10
• Findings do not always align across VLM and LLM settings, architectures, or training recipes, cautioning against broad conclusions from a single pipeline.Source: §6 · p. 10
• The analysis covers upstream scaling behavior, while the surrogate benchmarks also support extending evaluation to downstream tasks.Source: §6 · p. 10

• The checklist reports open access to data and code, with justification referring readers to the methodology, benchmark construction, evaluation, and results sections.Source: NeurIPS Paper Checklist · pp. 16, 18
• The checklist directs readers to Section 6 for the paper’s discussion of limitations.Source: NeurIPS Paper Checklist · p. 16
• The checklist states that the paper does not include theoretical results.Source: NeurIPS Paper Checklist · p. 17
• Figures in Sections 4.3 and Appendix E show the standard error of the mean.Source: NeurIPS Paper Checklist · p. 18

10. Broader impacts

• The released repository provides documentation and implementation details for the benchmark assets, while existing assets and their applicable licenses and terms are documented in released repositories.The appendix also includes licensing guidance for existing and repackaged assets.Source: §10 · pp. 20–21

A.1.1 LLM

• The LLM search space combines five optimization hyperparameters with four scale parameters, including architecture dimensions and token budget.Its parameter ranges are informed by scaling-law literature; batch size is derived from token budget using the Step Law relationship.Source: §A.1.1 · p. 23

A.1.2 VLM

• The VLM search space varies vision and text widths, training samples, and six optimization hyperparameters, with additional model-design choices fixed across configurations.Attention head dimension is generally 64, reduced to 32 for the smallest models; head counts are based on tower width.Source: §A.1.2 · pp. 23–24

A.2 Sampling strategies

• Most parameters use random or log-uniform sampling, while LLM configurations specify architecture, token budget, and optimization settings.The VLM sample count uses a warped log-uniform distribution between 6 × 10^5 and 1.2 × 10^8, with α = 0.5, to relax equal-per-decade allocation.Source: §A.2 · p. 24, §A.2.1 · p. 25, §A.2.2 · p. 25

A.2.2 VLM

• VLM vision and text widths are selected from nine candidate values, with the weighting scheme favoring smaller widths.The candidate widths range from 32 to 512, and the weighting uses α = 0.5.Source: §A.2.2 · p. 25

A.3 Multi-fidelity modeling and checkpointing

• Intermediate checkpoints give each configuration multiple observations at different normalized training-progress levels, supporting multi-fidelity surrogate modeling.Progress runs from initialization at 0 to the fully trained model at 1.Source: §A.3 · p. 25

A.3.1 LLM

• LLM checkpoint counts scale with training steps: the longest run receives 10 checkpoints, while other runs receive 2 to 10 in proportion to their steps.Training steps are determined from each model's token budget and global batch size.Source: §A.3.1 · pp. 25–26

A.3.2 VLM

• VLM checkpoint frequency is set through the epoch count, which depends on planned training samples and yields 2 to 10 checkpoints per configuration.The maximum samples per epoch is set to 6 million.Source: §A.3.2 · p. 26

A.4.1 LLM

• LLMs are trained from scratch on SlimPajama with a decoder-only transformer, AdamW, and a warmup-stable-decay schedule, then evaluated on language-model losses and six downstream benchmarks.Training steps follow the token budget and global batch size; downstream benchmarks use their respective few-shot settings.Source: §A.4.1 · p. 26, Table 10 · p. 26

A.4.2 VLM

• VLMs use AdamW with a heuristic global batch size and are evaluated on 100,000-sample validation and test splits plus the zero-shot 40-dataset DataComp suite.The DataComp suite spans classification, retrieval, robustness, and fairness tasks.Source: §A.4.2 · pp. 26–27, Table 11 · p. 27

B.1 Surrogate candidates

• Surrogate candidates adapt the HW-GPT-Bench setup and add TabPFN v2.5, whose in-context learning permits reuse across downstream targets without separate model training for each metric.The setup also excludes some candidates or configurations for performance, stability, or implementation reasons.Source: §B.1 · p. 27

B.2 Data splitting

• All checkpoints for a configuration are assigned together to either the training or test split to prevent information leakage.Source: §B.2 · p. 27
• VLM configurations use an 80/20 split stratified within four training-compute groups, preserving representation from each compute range.Source: §B.2 · p. 27
• LLM configurations use a 90/10 random split without additional balancing, while VLM divergence prediction uses a separate random 80/20 configuration split.The divergence dataset labels configurations as diverged or not and does not treat checkpoints as separate examples.Source: §B.2 · p. 27

B.3 Evaluation metrics

• Surrogate predictive performance is assessed using error, relative-deviation, determination, and correlation metrics, with calibration metrics omitted.The reported table covers upstream performance across model families on their respective train-test splits.Source: §B.3 · pp. 27–28, Table 12 · p. 28

B.4 Divergence prediction (VLMs)

• VLM divergence prediction labels failed configurations and uses an ensemble of gradient-boosted XGBoost classifiers; predicted failures receive no performance prediction at query time.The passage identifies extreme hyperparameter settings, particularly high learning rates, as the primary empirical cause of divergence.Source: §B.4 · p. 28

C Surrogate API

• The surrogate API accepts a configuration and training-progress value and returns predicted upstream and downstream metrics, uncertainty estimates, and configuration-derived metadata.For VLM configurations predicted to diverge, performance outputs are withheld.Source: C Surrogate API · p. 28, Table 13 · p. 29

C.1 Predictive uncertainty

• TabPFN uncertainty is represented by the difference between the 90th and 10th percentiles of its predictive distribution.Source: §C.1 · p. 28

C.2 Parameter count

• The appendix specifies parameter-count calculations for LLMs and VLMs, including separate VLM vision- and text-encoder counts and text non-embedding parameters.Source: §C.2 · pp. 28–29

C.3 FLOPs estimation

• LLM training FLOPs are estimated using a factor of six per parameter-token contribution, while VLM compute multiplies profiled per-sample forward-pass FLOPs by training samples.The LLM estimate accounts for forward and backward passes and depends on total tokens and sequence length.Source: §C.3 · p. 29

E.3 Loss Extrapolation

• Chinchilla 3 models the OpenCLIP optimal trajectory better than Kaplan across acquisition methods, while LLM regret ranges are comparable and favor Chinchilla 3 overall in modeling optimal loss.Source: §E.3 · p. 30

E.4 Coarse Extrapolation

• Chinchilla 3 has superior model-size prediction regret on the LLM benchmark, with the passage attributing this to compute-cost estimation in its parametric loss function.Source: §E.4 · p. 30

E.5 Robustness of Extrapolation at 10× Target Scale

• CARBS achieves the lowest final parameter-prediction regret across both OpenCLIP and LLM benchmarks.Figure 9 measures absolute logarithmic regret between predicted and empirical optimal model sizes over cumulative profiling FLOPs.Source: Figure 9 · p. 32
• At 10× target scale, extrapolation-method rankings remain preserved when the prediction shift is synchronous with the empirical optimal-loss shift.The power-law scaling form makes increasing the target compute budget a scalar change that shifts the trajectory in log-log space.Source: §E.5 · pp. 30, 32
• The section adds plots of loss and optimal-parameter prediction trajectories at 10× extrapolation to make large-scale predictions more transparent.The cited text says the 10× predictions are consistent with the 20× setting.Source: §E.5 · pp. 30, 32

E.6 Robustness and Generalizability of the Surrogate Model

• XGBoost yields comparable performance trends and interaction effects, while training TabPFN on disjoint data subsets preserves relative rankings and scaling behaviors.Together, these ablations support generalization across surrogate architecture substitutions and data variations.Source: §E.6 · p. 33, Figure 16 · p. 35, Figure 17 · p. 35
• At 10× scale, the empirical Power-Law fit remains robust across architectural domains, while CARBS degrades in ranking with its localized exploitation bias.The caption says acquisition-method rankings remain consistent with baseline scale results.Source: Figure 12 · p. 34
• At 10× scale, Chinchilla Approach 3 retains structural prediction failures on OpenCLIP multi-modal architectures under its standard compute-estimation heuristics.The figure describes these as parametric limitations of the approach.Source: Figure 13 · p. 34
• The interaction plots compare acquisition strategies with loss and model-size prediction methods across OpenCLIP and LLM benchmarks, including relative rankings.The primary-surrogate plots are Figures 14 and 15; the XGBoost and disjoint-subset TabPFN plots repeat these comparisons.Source: Figure 14 · p. 34, Figure 15 · p. 35