PaperlayerView exact revision on arXiv

Source-linked AI summary

Test-Time Scaling for Scientific Equation Discovery

Haowei Lin, Hubert Lim, Xiangyu Wang, Letian Huang, Di He

arXiv:2608.28660v1cs.CLcs.AIcs.LG

TL;DR

The paper asks how test-time compute should be allocated for open-ended equation discovery, where candidate equations are evaluated iteratively rather than matched to a fixed answer. It formulates discovery as a unified iterative search and compares minimal controllers under fixed budgets. On LLM-SRBench, search width dominates allocation, with wider searches favored at larger budgets and appropriate width improving wall-clock efficiency.

  • Problem

    Prior TTS evidence mainly concerns closed-ended mathematics and coding, leaving compute allocation for open-ended equation discovery with external verification less directly studied.

  • Method

    The paper models equation discovery as iterative sample–generate–evaluate–prune search and compares minimal parallel controllers using population size, branching factor, and grouping under fixed budgets.

  • Results

    Search width is the dominant allocation parameter: larger budgets generally favor wider searches, while population–branching splits and controller choice have smaller effects.

  • Takeaways & Limitations

    Given an informative verifier, compute allocation is a central design choice for equation-discovery search, and appropriate width can improve both accuracy and wall-clock efficiency.

  • Takeaways & Limitations

    The conclusions are specific to LLM-SRBench equation discovery, especially its Biology and Material Science splits, and should not be treated as universal laws for scientific-discovery workflows.

Abstract

from arXiv · show

Test-time scaling (TTS) improves language model reasoning by allocating additional test-time compute, but prior work mainly studies closed-ended tasks such as math and coding. We study TTS for automated equation discovery, an open-ended setting where models search over candidate equations and rely on observed datapoints for feedback. We formulate LLM-driven equation discovery as an iterative search process that unifies Best-of-N, sequential refinement, tree search, and evolution-style methods under a common compute-allocation view. To isolate allocation effects from prompt engineering and other heuristics, we compare minimal parallel controllers under fixed budgets. On LLM-SRBench equation-discovery tasks, we find that search width is the dominant allocation parameter: the best width in our sweep generally increases with the compute budget, while the population--branching split and controller choice matter less. Appropriate width selection also improves wall-clock efficiency by increasing parallelism. These results suggest that, given an informative verifier, controlling exploration and exploitation is central to scaling LLM-based equation discovery.

1 Introduction

The paper studies test-time scaling for open-ended equation discovery, where candidate equations are iteratively generated and evaluated rather than matched to a single target answer. It unifies these searches as compute allocation under fixed budgets and finds that search width is the dominant allocation choice.

  • Motivation: Equation discovery provides an open-ended TTS setting in which models generate candidate equations or programs and improve them using an external verifier.Unlike mathematics or coding benchmarks, progress is measured by improvement under a task-specific evaluator rather than recovery of one known target string.
  • Motivation: Existing discovery systems combine prompts, mutation rules, ensembles, and pruning heuristics, making control-flow effects difficult to isolate.The paper therefore treats control flow and compute allocation as objects of study rather than attributing gains to entire engineered workflows.
  • Unified formulation: The unified view subsumes Best-of-N, sequential refinement, tree search, and evolution-style systems as different allocations across exploration and exploitation.Best-of-N emphasizes one-shot breadth, sequential refinement emphasizes narrow multi-step exploitation, and other methods combine branching, pruning, and persistent populations.
  • Unified formulation: The framework represents equation discovery as iterative sample, generate, evaluate, and prune operations governed by a shared test-time compute budget.It abstracts away domain-specific prompt and mutation choices while retaining selection policies, branching rules, verification, and pruning.
  • Experimental design: The study uses minimal parallel controllers and a large controlled empirical evaluation on LLM-SRBench to separate allocation effects from additional engineering.The reported study spans roughly 4,000 H100 GPU hours and compares allocation choices under shared budgets.
  • Results: Search width is the dominant control variable: the best-performing width is intermediate, increases with larger budgets, and improves wall-clock efficiency through parallelism.Population size–branching-factor splits and the choice between the two simple controllers have smaller effects after width is selected.

2 Method

The paper frames equation discovery as fixed-budget test-time search balancing exploration and exploitation, then studies simple parallel controllers parameterized by population, branching, and grouping.

  • Unified formulation: The unified abstraction evaluates control flows by how they balance exploration and exploitation under a fixed compute budget.It covers classical baselines, tree search, and evolutionary systems.
  • Unified formulation: Evolutionary systems maintain multiple isolated populations but add complex ensembles, crossover rules, and prompting heuristics whose necessity remains unclear.The paper therefore tests whether compute allocation explains much of their benefit.
  • Controller design: The reduced controller design uses population size n, branching factor k, and group count g, with global width defined as w = nk.Grouping changes search isolation and synchronization granularity.
  • Controller design: The study isolates selection and pruning effects with two clean parallel controllers while leaving richer selection, adaptive branching, diversity-aware pruning, and iteration-aware allocation for future work.This keeps the tunable allocation space small and highly parallel.
  • Controller design: PBeam expands each group’s current frontier and retains top-scoring candidates, while PIE expands from the full historical pool to preserve exploration.Both controllers share allocation variables and use parallel execution.

3 Experiments

Experiments on LLM-SRBench evaluate parallel equation-discovery search across budgets, widths, groupings, and controller choices. Search width dominates the performance–runtime trade-off, while grouping and other decompositions have smaller effects.

  • Experimental setup: The benchmark covers 24 Bio tasks and 25 Material tasks, each providing descriptions, variable definitions, training data, and held-out test data.Models propose Python programs specifying equations and fitting optimizers.
  • Experimental setup: The sweep uses budgets N ∈ {1, 2, 4, 8, 16, 32, 64, 128} and varies population size n, branching factor k, group count g, and iterations T.The budget satisfies N = w×T, with w = nk.
  • Result analysis: Search width is the most important allocation variable across domains and models, with the best configurations generally lying between very narrow and one-shot wide searches.Narrow searches overemphasize sequential refinement, whereas very wide searches underuse iterative feedback.
  • Result analysis: Small budgets favor narrow searches, while larger budgets tend to benefit from moderately wider synchronized expansion, although this remains an empirical trend rather than a universal scaling law.The sweep has few discrete budget levels and nearby widths often perform similarly.
  • Result analysis: Width 32 reduced Bio runtime from 7622 s to 1586 s while increasing train Acc0.1 from 0.965 to 0.984 versus width 1.The runtime reduction was nearly 80%, reflecting greater inference parallelism and fewer sequential rounds.
  • Result analysis: Increasing groups from g = 1 to g = 8 yielded up to a 1.075× speedup, but grouping had weak and inconsistent effects on search quality.Over-grouping can reduce per-group width and budget, so g is treated primarily as a throughput parameter.
  • Result analysis: Once global width is fixed, the n–k decomposition and controller choice have minor effects; PBeam and PIE train Acc0.1 usually differ by at most ±2%.Width determines the operating regime, while these choices mainly fine-tune performance.
  • Result analysis: The recommended workflow maximizes feasible budget, pilots widths using powers of two, then sets remaining hyperparameters moderately and treats controller choice as refinement.Suggested settings include k ∈ [2, 8] and mild grouping.

4 Related work

The paper situates LLM-based equation discovery within scientific modeling and recent language-model-driven symbolic regression. These systems generate hypotheses and iteratively improve them through test-time workflows and external tools.

  • Scientific discovery couples hypothesis generation with empirical verification across the research pipeline.
  • Equation discovery, or symbolic regression, seeks equations balancing predictive accuracy and simplicity for a dataset.
  • Classical symbolic-regression approaches include search-based optimization, Monte Carlo search, reinforcement learning, and pretrained Transformer models.
  • LLM-based symbolic regression generates equation hypotheses and iteratively improves them through test-time-scaling workflows while leveraging model knowledge.

5 Extended Discussion

The discussion argues that lightweight, width-matched controllers can compete with engineered evolutionary systems because global compute allocation matters more on this benchmark. It also emphasizes that these findings depend on verifier quality and differ from closed-ended reasoning settings.

  • PBeam and PIE remain competitive with OpenEvolve when width and depth are matched, and slightly outperform it under the paper’s empirical parameter recipe.OpenEvolve’s inability to use k > 1 or mild grouping with g < w makes it less expressive in these comparisons.
  • Additional engineering appears secondary to obtaining the right global compute allocation on this benchmark.
  • Width-matched minimal baselines are needed to avoid confounding engineering gains with operating in a better width–depth regime.
  • Unlike closed-ended math reasoning, open-ended discovery with an external verifier requires deciding how to allocate a fixed budget across exploration and exploitation.
  • The conclusions are conditional on an informative verifier, because search can overfit measured proxies rather than the true scientific target.The benchmark verifier uses training-set performance as an imperfect proxy.

6 Conclusion

The paper studies equation-discovery test-time scaling as compute allocation. On LLM-SRBench, search width dominates the sweep, with appropriate width improving accuracy and wall-clock efficiency.

  • Search width is the dominant factor in the LLM-SRBench sweep, and larger budgets generally favor wider searches.
  • Appropriate width selection improves both equation-discovery accuracy and wall-clock efficiency.
  • The findings motivate studying compute allocation in broader open-ended scientific-search settings with informative verifiers.

Limitations

The study’s conclusions are bounded by its benchmark setting, verifier assumptions, limited model and controller coverage, compute proxy, and lack of scientific validation.

  • Scope: The empirical conclusions apply specifically to LLM-SRBench equation discovery, not to the full complexity of scientific discovery.The benchmark omits experimental design, noisy measurements, causal interpretation, safety constraints, and long-horizon laboratory validation.
  • Verifier: The analysis assumes an informative verifier and optimizes training Acc0.1, which can permit overfitting to the training split or proxy objective.Improved verifier fidelity remains a bottleneck for real scientific applications.
  • Coverage: The study covers limited models, domains, and controller families, so optimal width and controller behavior may change elsewhere.The experiments primarily use gpt-oss-20b, with Qwen3-30B-A3B as an auxiliary check, and instantiate PBeam and PIE.
  • Compute accounting: The compute budget counts generated proposals N, but practical costs also vary with token lengths, serving latency, verifier runtime, hardware utilization, failures, and synchronization.A fuller systems-level accounting is needed before direct transfer to heterogeneous production or laboratory settings.
  • Validation: Strong benchmark performance does not guarantee scientifically valid discoveries or replace domain expertise, independent validation, and experimental confirmation.The generated equations are candidate models optimized against benchmark feedback.

Ethical Considerations

The paper frames its contribution as benchmark-level compute-allocation analysis rather than unsupervised automation of high-stakes scientific decisions.

  • Scope: The work studies inference-time compute allocation in a benchmark setting, not high-stakes scientific decisionmaking without oversight.Its stated goal is to understand more effective compute allocation.
  • Risks: The systems can generate plausible but incorrect equations, hypotheses, or programs when the verifier is imperfect or measures only a proxy objective.Search is optimized against benchmark feedback on the training split.
  • Oversight: Real-world use would require domain-expert review and additional experimental validation.Strong search performance is not equivalent to validated scientific discovery.
  • Efficiency: Large-scale test-time search can consume substantial compute resources, motivating allocation strategies that improve performance while reducing wall-clock time and synchronization overhead.Efficiency is treated as a practical consideration alongside performance.

A Compute budget N and FLOPs

The paper defines compute budget N as the total number of candidate proposals generated across the search and argues that, under its controlled setting, N is equivalent to FLOPs up to a task-dependent constant.

  • Budget definition: N counts the total number of candidate proposals produced across all search steps.This provides the paper’s operational measure of test-time compute.
  • Per-rollout cost: A single generation pass produces one proposal, with inference cost determined by model size and input and output lengths.The paper introduces Lin and Lout for one autoregressive transformer rollout.
  • Search accounting: All generated proposals are indexed by iteration, selected parent candidate, and newly generated proposal within a branching step.The indexing makes the total search cost additive across rollouts.
  • Equivalence assumptions: With a fixed base model and comparable bounded contexts and outputs, each rollout costs between task-dependent constants.The controlled setting keeps model size fixed and rollout lengths within a bounded range.
  • FLOPs relation: Summing rollout costs yields total FLOPs proportional to N, so N and FLOPs induce the same budget ordering in these experiments.The equivalence holds up to a task-dependent constant factor.
  • Excluded costs: The budget captures LLM generation cost; selection, pruning, and verifier evaluation are treated as negligible or fixed across methods.This assumption supports the stated N–FLOPs equivalence.

B Benchmark and implementation details

The experiments use a controlled LLM-SRBench implementation spanning 49 Biology and Material Science tasks, with minimal prompts, a shared weak initialization, and parallel grouped search.

  • Benchmark: The benchmark includes 24 Biology tasks and 25 Material Science tasks, for 49 LSR-Synth tasks overall.Each task contains a specification, executable scaffold, evaluator, and local dataset.
  • Data: Each synthetic task has 5,000 samples, with 4,000 used for training and the remainder reserved for held-out testing.Biology and Material tasks use shared input-output mappings with task-specific hidden symbolic laws and numeric ranges.
  • Prompting: Prompts combine the task instruction, fixed scaffold boundaries, and at most one in-context inspiration from the current best valid program.Only the code inside the evolve block is decoded and inserted between the unchanged prefix and suffix.
  • Evaluation interface: The reported evaluation fields include acc01 and combined_score, while the implementation uses a shared executable scaffold with restricted packages.Available packages are numpy, scipy, and scikit-learn.
  • Initialization: All tasks share a deliberately weak affine least-squares initialization as a domain-agnostic starting point.The same initialization scaffold is used across all 49 tasks, runs, and groups.
  • Search and execution: The grouped tree search uses population size n, branching factor k, and groups g, with synchronized width w = nk.The implementation also evaluates candidates asynchronously under fixed concurrency caps and uses three random seeds.
  • Evaluation protocol: Held-out test performance is measured after each synchronized iteration but is never fed back into the search loop.The main experiments use gpt-oss-20b on Biology and Material Science, with Qwen3-30B-A3B as a cross-backbone check.
  • Reproducibility: The paper reproduces benchmark ground-truth equations for all 49 tasks considered.Task identifiers follow the LLM-SRBench Table 4 naming convention.

C.1 Empirical budget–width trend

The empirical width frontier suggests that preferred search width generally rises with compute budget, but the evidence supports a qualitative allocation guide rather than a universal scaling law.

  • Width frontier: The width frontier compares the best score at each (N, w) after optimizing decompositions and controllers.The reported algorithm is whichever method attains the envelope at that budget.
  • Width frontier: The best width tends to increase with budget, although noisy measurements can make nearby widths similarly competitive.The frontier uses eight discrete budget levels, so this is a qualitative trend.
  • Limitations: A universal width law is unsupported because the sweep has only eight power-of-two budgets bounded by N ≤128.The selected argmax can also change with seed fluctuations, verifier quality, model family, task difficulty, and hardware.
  • Practical guide: A practical deployment strategy is to pilot candidate widths, preferably powers of two, then choose the smallest width near the observed frontier.This aims to preserve most quality gains while avoiding over-interpreting sparse fitted points.
  • Performance: Tuned-width training accuracy improves with additional compute but shows diminishing returns at larger budgets.The authors report this descriptively because the number of budget levels is small and high-budget accuracy approaches saturation.

C.2 The study of second-order effect allocation choices

After width is selected, decomposition, controller, grouping, and backbone choices generally have smaller effects, though grouping can trade quality for throughput and exact optima remain context-dependent.

  • Decomposition: The exact population–branching split is second-order after choosing width, with most alternatives within about one percentage point of their regime optimum.A poorly chosen width can cost several percentage points, reaching about 5%, even after optimizing the split and controller.
  • Controller choice: PBeam and PIE usually differ by no more than ±2 percentage points, far less than the 10 to 20 point penalty associated with the wrong width.This supports treating the two controllers as largely interchangeable when comparing width effects.
  • Controller choice: PBeam and PIE generally outperform OpenEvolve at matched (N, w) when using mild grouping g = 2 and balanced decomposition.OpenEvolve is represented as the fully partitioned case (n, k, g) = (w, 1, w).
  • Grouping: Grouping improves inference efficiency through reduced synchronization overhead but should be used cautiously when g becomes large.The empirical analysis does not generally support grouping as a mechanism that improves reasoning through enforced diversity.
  • Backbone comparison: The width–budget trend is qualitatively similar across Qwen3-30B-A3B and gpt-oss-20b, while exact preferred widths remain model- and budget-dependent.The trajectories overlap through N ≤8 but diverge at N = 16, where Qwen favors w⋆ = 2 and GPT favors w⋆ = 8.
  • Grouping: Moderate grouping can improve throughput with modest quality loss, whereas excessively large groups collapse per-group width and hurt performance.At Material width 16, g = 2 gives a 1.043× speedup and 0.295-point accuracy gain; Bio width 16 with g = 8 loses 1.54 points despite a 1.053× speedup.
Loading 2608.28660v1…