SLDBench v2: Active Scaling Law Discovery

1 INTRODUCTION

• Agents must choose evidence as well as a law because inexpensive measurements can leave candidate laws indistinguishable, while other scales or parameter combinations may help distinguish them.They must judge what evidence can establish, what to acquire next, and whether its expected value justifies its cost.Source: §1 · pp. 1–2
• The benchmark collects 127 scenarios and retains 30 for which a reference power-law fit scores R2 ≤0.8 in both settings, spanning 14 sources and 339,092 observations.The source-reported training compute of only two sources already totals 1.09 × 10^23 FLOPs.Source: §1 · p. 2
• Active SLD gives agents priced candidate configurations, reveals selected outcomes over rounds, and requires them to revise their law and acquisition choices before stopping.Historical replay charges recorded experiment prices while preserving the passive setting and shared hidden extrapolation targets.Source: §1 · p. 2
• Across 8 systems, median R2 spans 0.723–0.753 in Passive and 0.235–0.570 in Active; Active losses arise from what agents buy, and cheap purchases can leave conditions needed for extrapolation unobserved.Runs that buy the full pool match their Passive counterparts; Claude Opus 5.5 leads in Active at 4.01% median cost.Source: §1 · p. 2
• SLDBench v2 tests how agents spend compute to develop future models by bringing experimental acquisition into scaling-law discovery.The benchmark reports predictive quality and experimental cost separately.Source: §1 · pp. 2–3

2 SLDBENCH V2

• An SLD task comprises experimental choices, a group or control context, and a measured target; a shared-form symbolic law with group-specific parameters predicts hidden larger-scale experiments.The law’s form is not supplied, and Passive SLD provides the complete candidate dataset upfront.Source: §2 · p. 3
• Active SLD exposes a priced catalogue and revealed history, then requires an agent to choose batches, revise its law, and decide when to stop.The agent jointly selects evidence and a law because purchased points can fit competing laws that diverge beyond the candidate scale.Source: §2 · p. 3
• Task T22 has 173 catalogue runs across 22 groups and 58 hidden test runs, with a 15,006-fold price range and no candidate exceeding the test runs’ 0.64B-sample lower bound.The example illustrates a breadth-versus-depth trade-off: cheap points across every group may not identify per-group exponents, while full series in a few groups rely on transferable exponents.Source: §2 · p. 3
• The benchmark collects 127 scenarios, applies active-eligibility and reference-fit difficulty screens, and retains 30 tasks from 14 sources with 339,092 records.The retained tasks have 40 to 118,480 candidates; median test prices exceed median candidate prices by 1.9 to 218 times, with a median ratio of 7.4.Source: §2 · pp. 3–4
• Tasks run in paired containers: agents request outcomes through a root-only oracle that logs charges, then submit a law for separate hidden-test verification.The score is unclipped R2, or zero for non-finite predictions; spending is reported separately rather than traded against accuracy, and replay does not launch training jobs.Source: §2 · pp. 4–5

3.1 EXPERIMENTAL SETUP

• The experiment evaluates eight agent systems with 720 runs per setting, reporting predictive quality separately from acquisition cost.Quality includes median R2 and threshold shares; cost is the fraction of full-pool cost.Source: §3.1 · p. 5

3.2 MAIN RESULTS

• Passive median R2 spans 0.723–0.753, but Active median R2 spans 0.235–0.570, and the Passive ranking only weakly predicts Active ranking (Spearman rs = 0.21, n = 8).Every system loses quality, with source-aggregated Active/Passive error multipliers from 1.49 to 4.71; all 95% intervals exceed one.Source: §3.2 · pp. 5–6, Table 1 · p. 6
• The pooled share of runs with R2 ≤0 or no score rises from 11.4% Passive to 31.7% Active, while the share above 0.5 falls from 72.2% to 46.4%.Runs buying the full candidate pool have near-zero median Active–Passive difference, indicating that the loss stems from what agents buy rather than sequential access.Source: §3.2 · p. 6, Figure 3 · p. 6

3.3 WHAT AGENTS BUY

• Agents’ median share of candidates bought exceeds their median share of cost, and their first batches are cheaper within groups than their last batches.The typical episode is cheap exploration followed by costlier confirmation; Hy4-Preview buys 40.4% of candidates for 3.44% of the cost.Source: §3.3 · p. 7, Figure 4 · p. 7
• Although 89.3% of Active runs observe every group, 16.7% do so with reduced independent parameter variation and 10.7% omit at least one group.A Hy4-Preview OpenWebText run bought 51 points but missed two test-set batch-size and learning-rate combinations, scoring R2 = −83.3.Source: §3.3 · p. 7

3.4 WHAT THE PURCHASED DATA ARE WORTH

• The fixed reference law’s median R2 falls from 0.455 with the full pool to at most 0.085 at ρ = 10%, and is negative under every rule at ρ ≤2%.The four strongest agents’ Active medians are 0.514–0.570 at median costs of 4–35%, above every acquisition-rule curve and the full-pool reference, but this comparison does not establish efficient data selection.Source: §3.4 · p. 8, Figure 5 · p. 8
• With a fixed reference law, price-spread selection beats cheapest-first on 63% to 77% of tasks at budgets ≤5%, while cheapest-per-group beats it on at most 46.7%.At ρ = 2%, price-spread raises median R2 from −1.456 for cheapest-per-group to −0.005; spanning within-group price ranges matters more than early group coverage.Source: §3.4 · p. 8, Figure 5 · p. 8
• In a DeepSeek V4 Pro T22 run, 14 sparsely sampled groups contribute 96.4% of test squared error after the agent transfers a shared exponent to them.The run bought 78 points for 4.407% of pool cost, yet the failure reflects an untested transfer assumption rather than too few points.Source: §3.4 · p. 8, Figure 5 · p. 8

AI for AI

• SLDBench v2 targets the computation needed to predict future-model behavior, building on AI systems’ roles in model development and research workflows.Examples include searching learning algorithms and update rules, optimizing matrix multiplication and sorting, and automating experimentation.Source: §4 · pp. 8–9

Research-agent benchmarks

• SLDBench v2 makes experimental evidence part of an agent’s decision and connects extrapolation quality to expenditure and purchase coverage.Its paired settings hold the prediction target fixed while varying access to observations.Source: §4 · p. 9

Scaling laws and symbolic discovery

• Earlier approaches fix either the data or the law family, whereas SLDBench v2 asks agents to construct the law and choose measurements.Related work also frames these choices against scaling-law dependence on observed scale range, selected data, and fitting procedure.Source: §4 · p. 9

Limitations

• The benchmark replays recorded experiments, so agents cannot propose new configurations, and its catalogue bounds what they can learn.Prices mix reported FLOPs, estimates, and proxies; cost is reported separately from quality, rather than used to rank systems on one axis.Source: §5 · p. 9
• The difficulty screen uses one reference law and therefore favors tasks on which that law fails.Evaluation covers 8 systems with three repeats on tasks from 14 sources, making per-system estimates coarse.Source: §5 · p. 9

Conclusion

• Agents that must buy experiments separate widely despite near-indistinguishable passive performance, and cheap purchases can leave extrapolation-relevant conditions unobserved.Acquisition cost alone does not determine prediction quality; under a fixed law, within-group scale variation matters more than reaching every group, and is also costlier.Source: §5 · p. 9
• Agents that select experiments to discriminate between candidate laws are a natural next step.The paper concludes that experimental planning remains a key challenge for agents assisting model development.Source: §5 · p. 9

Reproducibility

• Scripts reproduce corpus statistics, compute accounting, and non-agent baselines from the task files.They also regenerate every experiment figure and table from recorded evaluation results.Source: REPRODUCIBILITY STATEMENT · pp. 9–10
• Rerunning the agents additionally requires the released Harbor tasks and the models and agent harnesses listed in Table 1.Source: REPRODUCIBILITY STATEMENT · p. 10
• The scripts regenerate the experiment figures and tables from recorded evaluation results.Source: REPRODUCIBILITY STATEMENT · p. 10

AI USE STATEMENT

• An LLM agent assisted with writing, figures, and analysis scripts, and the authors verified the manuscript’s text and figures.Source: AI USE STATEMENT · p. 10

A.1 TASK INVENTORY AND CONSTRUCTION

• Records sharing all input features and a group are merged using their median target, and source-specific metadata defines scale ordering and test-set construction.The appendix lists upstream sources in Table 3.Source: §A.1 · p. 12, Table 3 · p. 13

A.2 REFERENCE FITTING AND THE DIFFICULTY SCREEN

• Structural curation yields 127 scenarios, 60 meet eligibility conditions for a genuine information-cost trade-off, and the difficulty screen retains 30 relative to a fixed reference law.The selected tasks are not a representative sample of the corpus.Source: §A.2 · pp. 12, 14
• The reference is a shared additive power-law fit using bounded multi-start nonlinear least squares with a soft-L1 penalty on normalized residuals.Amplitudes and exponents may have either sign, accommodating decreasing losses and increasing scores; fits use candidates alone and fall back to linearized or pooled fits for groups with too few points.Source: §A.2 · pp. 13–14

A.3 EVALUATION PROTOCOL AND ACQUISITION ACCOUNTING

• Claude Opus 5.5 runs with Claude Code 2.1.280, a 1,000,000-token context window, and high reasoning effort.Agent and verifier time limits are 3,600 and 600 seconds, respectively.Source: §A.3 · p. 14
• Both settings share the same features, groups, and targets, while Passive files additionally expose task-specific columns.Agents submit law.py, explain.md, and any required parameter files; only the verifier accesses test targets.Source: §A.3 · p. 14
• Acquisition cost sums prices for distinct purchased candidates and is normalized within each task.Repeated queries incur no additional charge; Portuguese MTEB is priced by parameter count because FLOPs are unavailable.Source: §A.3 · p. 14

A.4 SOURCE-UNIT COMPUTE COVERAGE

• Costs are classified as C3 source-reported FLOPs, C2 FLOPs derived under documented assumptions, C1 non-FLOP proxies, or C0 unavailable costs.Portuguese MTEB parameter-count pricing is C1, while OpenWebText training compute derived from 6ND is C2.Source: §A.4 · p. 14
• Compute is aggregated over training runs or configurations, and shared pool-and-test data are counted once.For tokenization, the largest reported budget per configuration gives a lower bound without assuming each budget is an independent run.Source: §A.4 · p. 14
• The two C3 sources alone establish a lower bound of 1.09 × 10^23 FLOPs, excluding C2 estimates, C1 proxies, and sources without verified cost mappings.This is neither a corpus total nor agent expenditure.Source: §A.4 · p. 14, Table 4 · p. 15

B.1 SCORE INCLUSION AND AGGREGATION

• Medians and error ratios use finite scores, and paired Active–Passive effects are summarized by source with 95% bootstrap intervals from 10,000 source resamples.The comparison pairs runs by system, task, and repeat index, takes the median effect within each of 14 sources, then the median across sources.Source: §B.1 · p. 15, Table 5 · p. 15

B.2 BEST-OF-THREE PREDICTION QUALITY

• Best@3 takes each task’s highest held-out R2 across three repeats, then reports the median of the 30 task-wise maxima separately by setting.This complements run-level medians by summarizing the best result across three attempts per task.Source: §B.2 · p. 15, Table 6 · p. 15

B.3 NON-AGENT ACQUISITION BASELINES

• Non-agent baselines fit the fixed reference law on outcome-blind purchase sets at budgets from 0.5% to 100% of full-pool cost.Cheapest-first, random, cheapest-per-group, and price-spread acquisition rules are compared; Table 7 reports median R2 and the share of tasks beating cheapest-first.Source: §B.3 · p. 16, Table 7 · p. 16

C PURCHASE BEHAVIOR AND SYSTEM-SPECIFIC PATTERNS

• Purchase analyses report full-pool use, point and cost shares, price preference, spending spread, and repeat stability.Purchase statistics include runs with at least one purchase; quality-related measures also require a finite score.Source: C PURCHASE BEHAVIOR AND… · p. 16, Table 8 · p. 16, Table 9 · p. 17

D COVERAGE AND EXTRAPOLATION CASE STUDIES

• Coverage distinguishes the fraction of candidates purchased, the fraction of groups observed, and task-specific checks of parameter values, combinations, and scale ranges.Independent parameter variation is assessed through design-matrix rank within each group under a fixed input representation; unobserved groups count as losses.Source: D COVERAGE AND EXTRAPOLATION CASE… · p. 17, Table 10 · p. 17
• Hy4-Preview acquired 51 OpenWebText observations covering 13 of 15 batch-size and learning-rate combinations, but omitted learning rate 10^-4 with batch sizes 64 and 128.Both omitted combinations occur in the catalogue and test set.Source: D COVERAGE AND EXTRAPOLATION CASE… · p. 17, Table 11 · p. 17
• In T22, DeepSeek V4 Pro shared the exponent across groups and bought all eight candidates in four inexpensive groups, with two candidates in each of 14 groups.It bought three to six candidates in each of the remaining four groups; Table 12 sorts all 22 groups by share of squared test error.Source: D COVERAGE AND EXTRAPOLATION CASE… · p. 18, Table 12 · p. 18

• SLDBench v2 jointly evaluates acquisition and law construction under a priced catalogue.Source: E COMPARISON WITH RESEARCH BENCHMARKS · p. 18
• SLDBench evaluates law construction from supplied observations.Source: E COMPARISON WITH RESEARCH BENCHMARKS · p. 18
• SL2 evaluates acquisition under specified law families.Source: E COMPARISON WITH RESEARCH BENCHMARKS · p. 18