Source-linked AI summary
ToolGate: An Executable Acceptance Pipeline for Tool-Dependent Scientific Benchmark Construction
Ke Zhang, Yankang Liu, Roya Zandi, Maziar Raissi
TL;DR
Generated scientific benchmark questions still require acceptance checks because scripts can fail and models may solve questions without specialist software. ToolGate applies executable verification, randomized no-tool screening, and tool-enabled solving, yielding 128 unique FEniCSx protocol survivors from 500 attempts. The pipeline makes repeated checking and difficulty screening auditable while leaving domain design and final review to experts.
Problem
Generated scientific questions require acceptance checks because scripts may fail or disagree with proposed answers, while models may solve them without the target software.
Method
ToolGate keeps candidates only when executable verification passes, specified no-tool screens fail, and a tool-enabled agent solves the item under fixed conditions.
Results
128 unique protocol survivors remained from 500 FEniCSx generation attempts after verification, no-tool screening, tool-enabled solving, and exact deduplication.
Takeaways & Limitations
ToolGate shifts repeated software execution and difficulty screening into an auditable process tied to named models, budgets, and presentation rules.
Takeaways & Limitations
Acceptance remains operational and protocol-relative: a post-hoc audit found a recurring FEniCSx data-layout defect among pre-deduplication survivors, with unresolved cases.
Abstract
from arXiv · showhide
Scientific benchmarks are commonly built by domain experts who write tasks and cross-check one another's work, or who adapt existing material from textbooks, published papers, and online resources. These routes can produce strong evaluations, but they require substantial per-item labor. Language models can reduce this repeated work by proposing candidates quickly. The remaining problem is acceptance. We target scientific questions whose answers require computations with specialist software rather than unaided reasoning alone. A candidate is invalid if its script fails or returns a different answer, or trivial if a model answers it without the software. We present ToolGate, which treats every generated item as a proposal and keeps it only if three gates pass. First, an executable solution script must reproduce the proposed answer when run with the scientific software. Second, randomized no-tool screening rejects candidates that models can already solve from the prompt alone. Third, a tool-using agent must solve each survivor within a fixed time limit. We instantiate ToolGate in FEniCSx with 500 generation attempts. The local-verification gate retains 478 candidates. For final reporting, we rescreen this pool after generation: two randomized no-tool screens exclude 222 from the reported pool, and direct GPT-5.5 API calls at medium reasoning (the API default) exclude another 121. Of the remaining 135, a GPT-5.5 Codex CLI agent with access to FEniCSx solves 130; exact deduplication leaves 128 unique protocol survivors. ToolGate turns repeated answer checking and difficulty screening into an auditable process while leaving domain design and final review to experts.
Introduction
ToolGate addresses the acceptance problem for generated scientific benchmark questions whose answers require specialist software. It combines executable verification, no-tool difficulty screening, and tool-enabled solving, with safeguards for screening artifacts and duplicates.
- Generated scientific questions can be invalid when scripts fail or produce different answers, and trivial when models solve them without specialist software.
- ToolGate accepts a candidate only when its script reproduces the proposed answer, specified no-tool screens fail, and a tool-enabled agent succeeds.Each candidate includes a question, four options, a proposed answer, and a solution script.
- 500 FEniCSx generation attempts yielded 478 locally verified candidates; post-generation screens excluded 222 and then 121 more, leaving 135.
- 130 of the remaining 135 candidates were solved by a GPT-5.5 Codex CLI agent with FEniCSx, and exact deduplication left 128 unique protocol survivors.
- The pipeline also randomizes option order, grades by answer value, and removes exact duplicate questions to address letter-preference and template-repetition artifacts.
ToolGate
ToolGate treats each generated item as a proposal and records whether it passes three operational gates. The process separates executable answer checking from no-tool difficulty screening and tool-enabled solving under named conditions.
- Each candidate contains a question, four options, a proposed answer, and a solution script using the target scientific software.
- The generator writes and tests code, but its own execution is not accepted as evidence; the saved solution is checked independently.
- Local verification: Local verification passes only when the script finishes, prints exactly one answer, and matches the proposed answer.This gate verifies reproducibility but does not judge difficulty.
- No-tool screening: No-tool screening rejects candidates when one model answers correctly in at least two of three independently randomized presentations.A candidate must pass every specified no-tool screen.
- Tool-enabled solving: A tool-enabled agent independently implements and runs a solution in a fresh workspace, and solved candidates are accepted within the fixed time limit.The agent does not see the proposed answer or generator solution.
- ToolGate saves models, reasoning settings, option presentation, time limits, and outcomes because passing is relative to reported test conditions.
FEniCSx Experimental Setup
The study applies ToolGate to FEniCSx workflows that combine mesh construction, finite-element computation, assembly, and post-processing. An end-to-end candidate uses a deformed triangular mesh, marked boundary facets, coefficient functions, and a numerically checked answer.
- FEniCSx provides DOLFINx, UFL, and Basix for finite-element solves, variational forms, and element definition and tabulation.Its workflows produce precise outputs that can be checked by running modest programs.
- Five expert-written seeds cover variants of a nonlinear Poisson problem, while generation expands to deformed meshes, variational problems, quadrature, and Basix operations.
- ToolGate loop: Figure 1 depicts a generator proposing a complete candidate before three gates decide acceptance and append machine-readable outcomes to the run database.
- Models and execution settings: The construction loop gives the generator and tool solver FEniCSx and code execution, while the no-tool screen receives only the public question and options.
- Models and execution settings: The no-tool screen makes three calls, rejects candidates with at least two correct answers, and assigns 300-second limits to generation, submitted solutions, and tool solving.
- End-to-end example: Item 0023 asks for a DOLFINx solve on a 17×14 triangular unit-square mesh, with original boundary facets marked before coordinate deformation.
- End-to-end example: The example solves for uh with Dirichlet and Neumann conditions, uses coefficient functions in moved coordinates, and applies quadrature degree 5.
- End-to-end example: Local verification and the independent FEniCSx-enabled solver both returned option (B), while the original no-tool screen selected (C) in all three fixed-order calls.
Results
The 500-candidate construction run recorded substantial attrition through local verification, no-tool screening, and tool-enabled solving, while later audits distinguished these pilot outcomes from the final pool.
- 500 generated candidates were tested in the original construction run.
- 478 candidates passed local verification, with scripts reproducing their proposed answers.
- 218 candidates remained after the original GPT-5.4 no-tool screen, and the GPT-5.5 Codex CLI agent solved 212 with FEniCSx.
- The original construction counts are pilot outcomes rather than the final reported pool because the fixed-order no-tool screen was later audited.
- 29.4 agent-hours covered 500 generator calls, while 5.49 agent-hours covered 218 construction-loop tool-solve calls.
Protocol Audits and Corrections
Post-generation audits corrected answer-position bias, tested transfer across model families, strengthened no-tool screening, and produced a conservative deduplicated survivor pool.
- Answer presentation: The fixed-order GPT-5.4 screen was position-sensitive: its accuracy was 86.0% when C was correct versus 29.4% otherwise.
- Answer presentation: Independent option shuffling and value-based grading reduced positional accuracy to 28%–33%, and 191 of 260 fixed-order easy labels no longer remained easy.
- Answer presentation: The proposed answer was one of the two middle numeric values in 80% of verified candidates, so a middle-value heuristic succeeded on 40.0%.
- Cross-family validation: Opus solved 209 of 212 GPT-5.5 tool-agent candidates, or 98.6%, with all three failures being timeouts.
- Stronger no-tool reasoning: The corrected post-generation audit used 256 candidates after two randomized no-tool screens, then excluded 121 more with GPT-5.5 medium reasoning, leaving 135.
- Stronger no-tool reasoning: The final tool check solved 130 of 135 candidates, and exact deduplication left 128 unique protocol survivors.
- Stronger no-tool reasoning: On a fixed cohort of 115 candidates, GPT-5.5 solved 16 without reasoning, 47 at medium reasoning, and the FEniCSx-enabled Codex agent solved all 115.
Related Work
ToolGate’s contribution is an upstream acceptance pipeline for generated scientific evaluation candidates, complementing expert domain design and final review with automated execution and difficulty screening.
- Tool-use data and scientific agents: ToolGate retains candidates only when local verification, no-tool failure, and tool-enabled success provide evidence for acceptance.
- Expert authorship and automatic construction: ToolGate moves repeated execution and difficulty screening into a shared automated process while retaining experts for domain design and final review.
- Tool-use data and scientific agents: APIGen filters general function-calling data, whereas ToolGate makes no-tool failure and tool-enabled success per-item criteria for scientific evaluation.
Discussion
The discussion shows that ToolGate’s yield and difficulty measurements depend on protocol choices, model and scaffold conditions, gate ordering, and diversity controls, with operational rather than complete semantic validity.
- Results: 343 of 478 locally verified candidates are excluded by sequential no-tool screens, leaving 135; the tool-enabled agent solves 130, and deduplication leaves 128 unique survivors.
- No-tool difficulty is protocol-relative: GPT-5.4 and Opus disagree on 167 of 478 candidates, or 34.9%, showing that no-tool difficulty is defined by named models and presentation rules.
- Gate ordering controls cost: The medium-reasoning screen runs after lighter screens, reserving its larger inference budget for 256 candidates instead of all 478.
- Outcome memory does not ensure diversity: The earlier fixed-order cohort concentrates in two families, with 147 Basix interpolation and 63 DOLFINx deformed-mesh items covering 210 of 212 candidates.
- Outcome memory does not ensure diversity: Neither seed policies nor outcome memory guarantees coverage, so future runs should combine them with topic quotas and novelty penalties against accepted history.
- Limitations: The reported gap bundles tool access with iteration and scaffolding, while model, budget, threshold, and presentation changes can move candidates across gates.
- Limitations: A post-hoc audit found a recurring FEniCSx data-layout defect in 97 of 130 pre-deduplication survivors, while five cases remained unresolved.
- Availability: The released artifacts include candidates, scripts, gate records, audit flags, rejected candidates, screening logs, and pipeline code.
Conclusion
ToolGate reduces generated scientific evaluation candidates through executable verification and tool-dependence screening, leaving a smaller auditable pool tied to explicit testing conditions.
- Conclusion: 500 FEniCSx attempts yield 478 locally verified candidates, 130 protocol survivors, and 128 unique survivors after exact deduplication.
- Conclusion: ToolGate shifts repeated software execution and difficulty screening from per-item expert labor into an auditable process.
- Conclusion: Each result remains tied to named models, budgets, and presentation rules so the process can be rerun as conditions change.
Ethical Statement
The work generates synthetic scientific evaluation questions and retains expert review and status records to address the risk of releasing mistaken or misleading items.
- Ethical Statement: Synthetic scientific questions can be mistaken or misleading if released without their status records.
- Ethical Statement: The project reports all gate outcomes, retains expert review as a release step, and uses standard open-source scientific software.
- Ethical Statement: The items add no domain-specific dual-use capability.