PaperlayerView exact revision on arXiv

Source-linked AI summary

SymbolicArena: A Unified Infrastructure for Benchmark Distillation and Dynamic Evaluation in Symbolic Regression

Ziwen Zhang, Xiju Wu, Yuheng Jing, Runxiang Wang, Boxiao Wang, Yifan Zang, Yifan Zhang, Yang Wang, Kai Li, Yifan Zhang, Huilin Xu, Jian Cheng

arXiv:2609.35113v1cs.LG

TL;DR

Symbolic-regression benchmarks trade evaluation cost against evidence that compact sets preserve benchmark validity. SymbolicArena distills a standardized task pool into Core50 and evaluates methods with a shared protocol; across evaluated methods, no single method dominates all dimensions, and numerical accuracy does not consistently correspond to symbolic recovery.

  • Problem

    Large SR benchmark collections are costly to evaluate, while compact collections have limited evidence that they preserve task coverage and algorithm discrimination.

    Source: §1 · p. 1

  • Method

    SymbolicArena standardizes 664 SR tasks, distills Core50 using task information and algorithm responses, and applies a unified execution protocol with numerical, symbolic, and search metrics.

    Source: §1 · p. 2

  • Results

    Across 15 representative methods, no method dominates all dimensions, and strong numerical accuracy does not consistently correspond to faithful symbolic recovery.

    Source: §6 · p. 9

  • Takeaways & Limitations

    Core50 reduces evaluation workload by 92.5% while providing a reproducible setting for studying differences across SR performance dimensions.

    Source: §6 · p. 9

    Core50 fidelity is validated only for the current task distribution and evaluated algorithmic paradigms; absent scientific domains and operator families remain outside its validated scope.

    Source: §5 · p. 8

Abstract

from arXiv · show

Symbolic regression (SR) seeks concise and interpretable mathematical expressions from data for scientific equation discovery. Existing SR benchmarks face a tradeoff between evaluation cost and benchmark validity. Repeated evaluation of large task pools is expensive, and compact benchmarks lack systematic evidence of preserved task diversity and algorithm discriminability. SymbolicArena provides a unified infrastructure for benchmark distillation and dynamic evaluation. The framework standardizes 664 heterogeneous tasks with executable ground truth expressions and distills the Full Task Set into Core50, a validated benchmark of 50 tasks. The distillation process preserves task coverage and algorithm discrimination under explicit balance constraints. SymbolicArena applies a unified execution protocol to heterogeneous SR algorithms and produces comparable outputs and search trajectories. Multi Axis Evaluation characterizes numerical quality, symbolic quality, and search behavior. Core50 reduces evaluation workload by 92.5% and maintains agreement with Full Task Set evaluations. Experiments show that SymbolicArena achieves 72.6% to 86.7% lower approximation error than alternative selectors, further supporting its fidelity to the Full Task Set. Evaluation reveals a substantial gap between numerical fitting and symbolic recovery across current SR methods, suggesting that reliable equation recovery remains an open challenge.

1 INTRODUCTION

SymbolicArena addresses the cost and validity challenges of evaluating symbolic regression by distilling a validated compact benchmark and standardizing evaluation across methods. Its reported results support benchmark fidelity and show that numerical, symbolic, and search measures capture distinct dimensions of performance.

1 INTRODUCTION

  • Existing benchmarks trade broad task coverage at substantial evaluation cost against cheaper compact collections with limited evidence of preserved validity.The introduction identifies the lack of a systematic strategy for compact SR benchmarks that preserve task coverage and algorithm discrimination.Source: §1 · p. 1
  • SymbolicArena combines benchmark distillation with a unified evaluation protocol that records search trajectories and organizes metrics across numerical, symbolic, and search dimensions.The framework uses task information and algorithm responses from 664 standardized tasks to construct Core50, a validated 50-task benchmark.Source: §1 · p. 2
  • 72.6% to 86.7% lower approximation error than alternative selectors supports Core50’s fidelity to the Full Task Set, with strong ranking agreement across nine algorithms on ID and OOD.The OOD ordering of FePySR, SymbolFit, and JAXSR is also preserved.Source: §1 · p. 2
  • Core50 is formulated as a distillation problem and derived as a validated 50-task subset from 664 executable tasks.The selection is intended to preserve task coverage and algorithm discrimination at a reduced evaluation budget.Source: §1 · p. 2
  • SymbolicArena standardizes heterogeneous SR algorithms under a shared execution protocol for reproducible comparison.Source: §1 · p. 2
  • Multi Axis Evaluation assesses numerical quality, symbolic quality, and search behavior to reveal tradeoffs beyond final numerical error.Source: §1 · p. 2

2 RELATED WORK

Prior symbolic-regression benchmarks span controlled equation-recovery settings, broad regression suites, and specialized equation-discovery tasks. Work on compact benchmark selection has focused mainly on NLP and general machine learning, while compact SR benchmark fidelity remains less explored.

Symbolic Regression Benchmarks

  • Controlled equation-recovery and extrapolation settings include Nguyen, Keijzer, Korns, and Vladislavleva.Source: §2 · p. 2
  • Broader and specialized suites include SRBench (La Cava et al., 2021), SRBench 2025 (Imai Aldeia et al., 2025), SRSD, and LLM SRBench (Shojaee et al., 2025b).Their stated emphases range from synthetic and real-world regression to predictive accuracy, expression complexity, scientific equation rediscovery, and LLM-based discovery.Source: §2 · p. 2

Efficient Benchmarking

  • Compact subset selection methods such as tinyBenchmarks, Anchor Points, and SubLIME mainly target NLP and general machine learning.The supplied passage identifies compact SR evaluation as less explored because of heterogeneous tasks and diverse algorithm responses.Source: §2 · pp. 2–3

3 SYMBOLICARENA

SymbolicArena constructs Core50 from a standardized task pool, then applies a shared execution protocol and multi-axis metrics to evaluate SR methods. Distillation combines task coverage, probe informativeness, and balance constraints.

3.1 BENCHMARK DISTILLATION

  • The task schema retains feature ordering, targets, fixed splits, executable ground truth, and structural metadata; validation leaves 664 valid tasks from a candidate pool exceeding 800.Source: §3.1 · p. 3
  • Calibration tasks are selected using disagreement and single-probe successes from PySR and LLM-SR, while four probes are chosen for usability across at least three paradigms.The probes run on calibration tasks; the selected set is DSO, PyOperon, iMCTS, and uDSR.Source: §3.1 · p. 4
  • Core50 selection uses probe responses across 664 tasks and three seeds, combining task informativeness with structural and response coverage and balance constraints.The best observed subset is selected through greedy initialization and feasible single-task swaps before fidelity validation.Source: §3.1 · p. 4

3.2 UNIFIED EVALUATION PROTOCOL

  • The unified protocol evaluates 15 algorithms on Core50 with shared inputs and implementation, while recording standardized outputs and expression trajectories selected by each algorithm’s native rule.Source: §3.2 · p. 4

3.3 MULTI AXIS EVALUATION

  • Multi Axis Evaluation groups ID and OOD as numerical quality, SYM and MIN as symbolic quality, and EFF and STAB as search behavior.ID and OOD use fixed test splits; EFF tracks approach to best within-run numerical quality, while STAB combines numerical, validity, and structural consistency.Source: §3.3 · pp. 4–5
  • Symbolic and minimality metrics assess expression recovery and size, while metric aggregation uses a common [0, 1] scale with empirical averaging conventions.SYM prioritizes symbolic-equivalence judgments, and MIN measures expression minimality relative to the reference.Source: §3.3 · pp. 4–5

4 EXPERIMENTS

Experiments evaluate Core50’s fidelity to the Full Task Set and compare the diagnostic profiles of symbolic regression methods. Core50 closely reproduces aggregate and algorithm-level behavior, while the evaluated methods show distinct strengths across numerical, symbolic, and search measures.

4.1 EXPERIMENTAL SETUP

  • Fidelity validation uses a separate one-hour evaluation of nine algorithms; formal Core50 evaluation covers 15 algorithms with a three-hour run budget.Construction and fidelity validation use separate records from formal evaluation.Source: §4.1 · p. 6

4.2 FIDELITY TO THE FULL TASK SET

  • Core50 achieves the lowest aggregate score MAE, 0.1388, among the evaluated 50-task selectors.Its representation of the task population and its algorithm responses jointly support fidelity; optimizing task information or source proportions alone is insufficient among the evaluated selectors.Source: Figure 2 · p. 6, §4.2 · p. 6
  • Across nine algorithms, Full Task Set and Core50 scores reach Spearman correlations of ρ = 0.9833 for ID and ρ = 0.9667 for OOD.The three validation algorithms preserve their OOD ordering, while one ID pair changes order.Source: §4.2 · p. 6, Figure 3 · p. 7

4.3 DIAGNOSTIC ANALYSIS OF SR METHODS

  • On Core50, iMCTS leads ID and OOD, FePySR leads SYM and MIN, and JAXSR leads EFF and STAB.PySR ranks second on ID, OOD, SYM, and MIN; the six-axis profiles show distinct capability specialization.Source: Table 2 · p. 7, §4.3 · p. 7, Figure 4 · p. 8
  • Strong numerical performance does not consistently correspond to symbolic fidelity: iMCTS leads both numerical axes yet has a SYM score of 42.15.A clean Nguyen-9 example shows that near-zero numerical error can coexist with failure of strict symbolic equivalence, so final numerical error alone is insufficient to characterize equation recovery.Source: §4.3 · pp. 7–8, Table 2 · p. 7
  • Search behavior and attained quality are distinct: JAXSR has near-maximal EFF and STAB with moderate ID and OOD, while iMCTS leads the numerical axes with a STAB score of 23.31.EFF is normalized to each run’s observed maximum, and high STAB indicates repeatability rather than attained quality.Source: Table 2 · p. 7, §4.3 · p. 8

5 LIMITATIONS

Core50’s validated fidelity is bounded by the task distribution and algorithmic paradigms represented in its evaluation, and by the configurations used to assess algorithms.

5 LIMITATIONS

  • Core50 fidelity is validated only for the current task distribution and evaluated algorithmic paradigms; absent scientific domains and operator families remain outside its validated scope.New task families or substantially different search paradigms may require renewed fidelity validation; Core50 also inherits the Full Task Set’s coverage boundary.Source: §5 · p. 8
  • Evaluation uses default or recommended configurations without algorithm-specific hyperparameter optimization, so absolute scores and relative rankings may change with dedicated tuning.Reported results characterize methods in a standardized benchmark configuration.Source: §5 · p. 8

6 CONCLUSION

SymbolicArena distills a large symbolic-regression task population into a compact benchmark and provides a unified evaluation framework. The conclusion reports fidelity to the full set while showing that numerical accuracy, symbolic recovery, and search behavior do not consistently align.

6 CONCLUSION

  • Core50 reduces evaluation workload by 92.5% while preserving strong ranking agreement with the Full Task Set.It is distilled from 664 executable tasks, and validation on algorithms excluded from construction supports its fidelity.Source: §6 · p. 9
  • Across 15 methods, numerical accuracy, symbolic recovery, and search behavior differ, and no method dominates every dimension.Strong numerical accuracy does not consistently imply faithful symbolic recovery, while search progress or repeatability does not guarantee high attained solution quality.Source: §6 · p. 9

A RELATED WORK

  • Existing benchmarks offer broad task collections, but construction, execution, and diagnostic coverage remain heterogeneous.The related-work discussion identifies a need for a unified framework that preserves benchmark fidelity and supports cross-paradigm evaluation.Source: A RELATED WORK · p. 15

B Full Task Set CONSTRUCTION AND STANDARDIZATION

  • The Full Task Set contains 664 tasks collected from multiple benchmark sources, with duplicate control and composition auditing.Tasks with ambiguous or unresolvable definitions are excluded; admitted tasks require executable ground truth, consistent variables and operators, data splits, and metadata.Source: B Full Task Set CONSTRUCTION AND… · p. 16, §B.1 · p. 16

B.2 UNIFIED TASK REPRESENTATION

SymbolicArena standardizes tasks and uses calibration responses to select probes and construct Core50. The subset balances task coverage, information, and distributional constraints.

B.2 UNIFIED TASK REPRESENTATION

  • The Full Task Set schema standardizes task representations and preserves raw formulas, normalized expressions, and structural representations for comparison and recovery analysis.Fixed ID and OOD test splits are retained or generated deterministically; all algorithms share the materialized splits.Source: §B.2 · pp. 16–17

B.3 Full Task Set COMPOSITION

  • Task complexity is grouped by operator counts in the stored source implementation, with boundaries at 5 and 8.Function calls receive no additional weight, and auxiliary implementation logic can affect the count.Source: §B.3 · p. 17

C.1 Calibration Set MINING

  • Calibration Set uses 200 tasks chosen to expose algorithm differences, including high-disagreement, one-sided-evaluability, and moderate-gap cases.Its source-family composition includes 70 SRSD, 70 LLM-SRBench, and 33 SRBench1.0 tasks.Source: §C.1 · p. 18

C.2 SELECTION OF FOUR PROBES

  • The four probes are DSO, PyOperon, iMCTS, and uDSR, selected for calibration usability while covering three primary paradigms.The selection objective favors reliable responses; calibration usability combines finite-result availability and resistance to numerical explosion.Source: §C.2 · pp. 18, 20

C.3 Core50 CONSTRUCTION

  • Core50 is selected from the 664-task Full Task Set using task descriptors and probe responses, subject to coverage and balance constraints.The construction combines coverage, mean task information, and distributional balance, with feasibility rules limiting duplicate, family, difficulty, and failure-mode representation.Source: §C.3 · pp. 20–21
  • Task information combines probe discrimination and construction stability, so high information requires both probe separation and seed consistency.Greedy initialization and feasible single-task swaps are used to refine the objective before the highest observed feasible subset is frozen.Source: §C.3 · pp. 20–21

C.4 SELECTION BASELINES AND FIDELITY VALIDATION

Core50 is evaluated against alternative 50-task selectors for aggregate-score fidelity, with accompanying ranking validation. The section also reports the benchmark manifest and the shared execution contract.

C.4 SELECTION BASELINES AND FIDELITY VALIDATION

  • The six alternative selectors use random, family-stratified, metadata-diverse, response-space, top-information, or difficulty-balanced selection strategies.Selectors use the same construction records and scoring rule; random selectors are evaluated over 100 sampled subsets.Source: §C.4 · pp. 21–22
  • Core50 achieves aggregate score MAE 0.1388 and NMAE 2.33% against the Full Task Set.Two of 100 uniform random draws obtain lower MAE than fixed Core50, while no sampled family-stratified draw reaches its value.Source: §C.4 · p. 22, Table 6 · p. 23

C.5 Core50 MANIFEST

  • Core50 includes 16 LLM-SRBench tasks, 15 SRSD tasks, and 9 SRBench1.0 tasks, with smaller counts from five other source families.The manifest provides ground-truth formulas grouped by source family.Source: §C.5 · p. 23, Table 7 · p. 23

D UNIFIED EXECUTION PROTOCOL

  • Each unified-protocol run specifies an algorithm, task, seed, and time budget, while fixed test splits and ground-truth expressions remain reserved for evaluation.Algorithms receive training data and permitted task information; configurations are reported separately.Source: D UNIFIED EXECUTION PROTOCOL · p. 25

D.1 WRAPPER AND EXECUTION ARCHITECTURE

SymbolicArena standardizes SR execution across isolated algorithm wrappers and records comparable outputs and trajectories. Its evaluation framework separates numerical, symbolic, and search-quality measures, with supplementary checks for uncertainty and training noise.

D.1 WRAPPER AND EXECUTION ARCHITECTURE

  • Wrappers expose shared fitting, prediction, and expression-retrieval operations while preserving native search procedures and model-selection rules.Algorithms run in isolated environments; outputs are converted to a shared symbolic representation while raw outputs are retained for auditing.Source: §D.1 · p. 25

D.2 UNIFIED RUN PROCEDURE

  • Each run records auditable checkpoint incumbents, evaluates them on fixed ID and OOD splits, and retains explicit unavailable-output statuses.Training-label noise modifies only the training split, and test scores remain external to search and candidate selection.Source: §D.2 · p. 26

D.3 RECORDED RESULTS AND SEARCH TRAJECTORIES

  • Final artifacts and checkpoint trajectories preserve selected expressions, evaluation results, run status, and provenance under a defined reproducibility boundary.Formal evaluation uses 180 minutes with one checkpoint per minute; the latest auditable expression is carried forward between updates and after early termination.Source: §D.3 · p. 26, §D.4 · p. 26

E MULTI AXIS EVALUATION DETAILS

  • Six fixed axes separate ID/OOD numerical quality, SYM/MIN symbolic quality, and EFF/STAB search behavior across 50 tasks and three seeds.All axes except STAB are averaged across 150 task-seed runs; STAB is averaged across 50 tasks.Source: E MULTI AXIS EVALUATION DETAILS · p. 27

E.1 NUMERICAL QUALITY MAPPING; E.3 OPUS ASSISTED SYMBOLIC POSTPROCESSING

  • ID and OOD transform NMSE into bounded quality scores, while symbolic postprocessing simplifies and validates expressions before frozen judgments are aggregated.The postprocessing prompt excludes coefficient refitting, variable renaming, and additional domain assumptions; positive model judgments are evaluation evidence, not formal proof.Source: §E.1 · p. 27, §E.3 · pp. 27–28

E.4 SYMBOLIC QUALITY; E.5 SEARCH BEHAVIOR

  • SYM assesses agreement with the reference, MIN scores expression complexity relative to it, and EFF measures numerical search progress while STAB combines consistency and output validity across seeds.A larger MIN score indicates lower complexity relative to the reference; STAB combines numerical consistency, output validity, and structural consistency.Source: §E.4 · pp. 28–29, §E.5 · pp. 29–30

E.6 BOOTSTRAP CONFIDENCE INTERVALS

  • Bootstrap intervals quantify task-resampling variability within the empirical Core50 sample, not significance, while evaluation outputs and symbolic judgments remain fixed.Intervals use 1,000 task bootstrap replicates and the 2.5th and 97.5th percentiles.Source: §E.6 · p. 30

E.7 SUPPLEMENTARY TRAINING NOISE ROBUSTNESS

  • Supplementary training-noise evaluation perturbs training labels at 1% and 5% of target standard deviation while keeping ID and OOD test sets clean.Noise robustness is reported separately from the six formal evaluation axes.Source: §E.7 · p. 30

F.1 PROBE4 SELECTION STABILITY AND REPLACEMENT STRUCTURE

Probe4 remains dominant or near-dominant under calibration resampling and selection-rule changes, while Core50 selection is robust to moderate weight variation. Across joint weight samples, the four-probe ranking is preserved.

F.1 PROBE4 SELECTION STABILITY AND REPLACEMENT STRUCTURE

  • Probe4 is the unique optimum in 63.4% of calibration bootstrap replicates and belongs to the optimal set in 73.4%.Its tie-averaged selection mass is 68.3%, with median competition rank 1 and interquartile range 1–2.Source: §F.1 · p. 31, Table 8 · p. 31
  • Bootstrap inclusion concentrates on uDSR and PyOperon, with DSO, iMCTS, and QLattice also frequent; the complete 12-algorithm procedure imposes no prior filtering.Table 8 summarizes calibration usability and bootstrap inclusion frequencies for the 12 eligible candidates.Source: §F.1 · p. 31, Table 8 · p. 31
  • Across 45 usability-rule settings, Probe4 is uniquely optimal in 31, belongs to the optimal set in 33, and ranks in the top three throughout.The analysis varies α from 0.30 to 0.70 and τ across 10, 30, 100, 300, and 1000, with m = 3 fixed.Source: §F.1 · p. 31
  • Removing a Probe4 member yields structured replacements that retain three paradigms, with maximum usability loss 0.007500.QLattice replaces DSO, iMCTS, or uDSR, while gplearn replaces PyOperon.Source: §F.1 · p. 31, Table 9 · p. 32
  • Requiring three paradigms resolves the tie at two without reducing maximum usability; requiring four makes the default panel infeasible.At four paradigms, the best feasible panel is {PyOperon, iMCTS, TPSR, uDSR}, with usability 0.944375.Source: §F.1 · p. 32

F.2 Core50 CONSTRUCTION

  • Under single-weight perturbations, the median Jaccard similarity to default Core50 is approximately 0.710, or 41–42 overlapping tasks.The median within-setting similarity across restarts is 0.667, indicating roughly 40 consistently selected high-quality tasks and interchangeable marginal positions.Source: §F.2 · p. 32, Table 10 · p. 32
  • All 5,000 joint weight samples preserve the four-probe Full Task Set ranking, and selected task sets average 0.850 Jaccard similarity to the default.The default-weight optimal set accounts for 40.24% of selections, and 37 tasks are selected in all 5,000 runs.Source: §F.2 · p. 33, Table 12 · p. 33
  • Across joint weight variations, optimal sets substantially overlap Core50 and the resulting algorithm ranking remains unchanged.The default-weight optimal set accounts for 40.24% of the 5,000 sampled weight vectors.Source: §F.2 · p. 33, Table 11 · p. 33

G.1 CORRELATION AMONG EVALUATION AXES

The six evaluation axes capture distinct association patterns, and task-sampling uncertainty varies by metric and algorithm. Noise robustness differs from clean numerical performance, while near-exact fitting can still miss symbolic equivalence.

G.1 CORRELATION AMONG EVALUATION AXES

  • ID and OOD rankings are strongly correlated (ρ = 0.98), as are SYM and MIN (ρ = 0.82), while numerical quality correlates less with symbolic quality.Numerical quality correlates moderately with SYM (ρ = 0.57 to 0.64) and weakly with MIN (ρ = 0.25 to 0.29); EFF has little association with ID or OOD.Source: §G.1 · p. 34, Figure 8 · p. 34

G.2 UNCERTAINTY IN SIX AXIS SCORES

  • Bootstrap interval widths differ across algorithms and metrics, so small point-estimate gaps should be interpreted alongside task-sampling intervals.The intervals quantify task-resampling uncertainty for all six axes.Source: §G.2 · p. 34, Figure 9 · p. 35

G.3 ROBUSTNESS TO TRAINING NOISE

  • Noise robustness varies across algorithms: iMCTS retains the highest OOD score across evaluated conditions, while DSO has the smaller absolute drop from its clean score.High clean OOD performance does not consistently correspond to small degradation under noise.Source: §G.3 · p. 34, Table 13 · p. 35

G.4 NUMERICAL APPROXIMATION AND SYMBOLIC RECOVERY

  • On a clean Nguyen-9 PySR run, ID and OOD NMSE are 4.54×10^-15 and 8.28×10^-17, yet the prediction is not_equivalent and scores mSYM = 0.4886.The prediction differs from the reference by a small additive residual despite negligible numerical error.Source: §G.4 · pp. 36–37

H EVALUATED ALGORITHMS AND EXPERIMENTAL SETTINGS

SymbolicArena evaluates 15 algorithms spanning six search paradigms on frozen Core50. Language models support equation search in LLM-SR and DrSR, while Opus 5 performs postprocessing after search.

H EVALUATED ALGORITHMS AND EXPERIMENTAL SETTINGS

I LLM USAGE AND INFORMATION BOUNDARIES; I.1 LLAMA IN LLM-SR AND DRSR

  • Llama assists equation search for LLM-SR and DrSR, while Opus 5 simplifies final expressions and supplies symbolic judgments only after search completion.Search models cannot access ground truth expressions or ID/OOD evaluation targets; candidate selection and coefficient fitting use training objectives.Source: I LLM USAGE AND INFORMATION BOUNDARIES · p. 40, §I.1 · p. 40

I.1 LLAMA IN LLM-SR AND DRSR

  • LLM-SR iteratively requests four structures per iteration using task context and up to two candidates selected by training score; DrSR adds experience feedback and residual analysis.For each DrSR structure, BFGS optimizes coefficients using training MSE.Source: §I.1 · pp. 40–41

I.2 OPUS 5 IN SYMBOLIC EVALUATION

Opus 5 is confined to symbolic processing of completed runs, while search, evaluation targets, and score aggregation remain separated. The protocol also limits semantic information exposed to search models and isolates benchmark construction from LLM-generated responses.

I.2 OPUS 5 IN SYMBOLIC EVALUATION

  • Opus 5 processes completed-run expressions for simplification and comparison, with ground truth and predictions handled independently under fixed operator semantics and domain assumptions.Source: §I.2 · pp. 41–42
  • Opus 5 does not compute evaluation scores; deterministic code aggregates frozen expressions and judgments, with its axis contributions recorded in Table 19.Source: §I.2 · p. 42, Table 19 · p. 42

I.3 INFORMATION ISOLATION AND LEAKAGE CONTROL

  • Search models cannot access ground truth or evaluation targets; variable names are normalized, and semantic descriptions are optional inputs kept separate from formulas and targets.Tasks lacking semantic metadata use a generic prompt asking for the mathematical function skeleton from the target description and feature data.Source: §I.3 · p. 42
  • For SRSD tasks with distractors, models receive an unordered multiset of semantic roles, not their mapping to variables, preventing direct identification of role-linked numerical columns.Source: §I.3 · p. 42
  • Core50 selection uses responses from four non-language-model probes—DSO, PyOperon, iMCTS, and uDSR—so final Full Task Set responses contain no LLM-generated probe outputs.Source: §I.3 · p. 42
  • Evaluator separation confines Opus 5’s access to ground truth to completed-run symbolic processing; its outputs cannot alter candidate generation or trajectories, and judgments are frozen before aggregation.Model identifiers and prompt hashes are retained for auditing.Source: §I.3 · p. 42
Loading 2609.35113v1…