PaperlayerView exact revision on arXiv

Source-linked AI summary

SciAgentGym: Benchmarking Multi-Step Scientific Tool-use in LLM Agents

Yujiong Shen, Yajie Yang, Zhiheng Xi, Binze Hu, Huayu Sha, Jiazheng Zhang, Qiyuan Peng, Junlin Shang, Jixuan Huang, Yutao Fan, Jingqi Tong, Shihan Dou, Ming Zhang, Lei Bai, Zhenfei Yin, Tao Gui, Xingjun Ma, Qi Zhang, Xuanjing Huang, Yu-Gang Jiang

arXiv:2602.12984v2cs.CL

TL;DR

Existing scientific benchmarks do not adequately evaluate interactive, domain-specific tool-use across long-horizon workflows. The paper introduces SciAgentGym and SciAgentBench, then proposes SciForge, whose verified dependency-graph trajectories let SciAgent-8B outperform a much larger Qwen3-VL-235B-Instruct while supporting cross-domain transfer.

  • Problem

    Existing scientific benchmarks primarily use static question answering and fail to capture interactive, tool-mediated scientific workflows.

  • Method

    The paper builds SciAgentGym with 1,780 domain-specific tools, evaluates agents with tiered SciAgentBench tasks, and synthesizes verified dependency-graph trajectories through SciForge.

  • Results

    SciAgent-8B outperforms Qwen3-VL-235B-Instruct after fine-tuning on SciForge trajectories, while current models show substantial degradation on longer interaction horizons.

  • Takeaways & Limitations

    Scientific tool-use remains a bottleneck for current models, while execution-grounded, logic-aware training improves tool-use capabilities and transfers positively across domains.

Abstract

from arXiv · show

Scientific reasoning inherently demands integrating sophisticated toolkits to navigate domain-specific knowledge. Yet, current benchmarks largely overlook agents' ability to orchestrate tools for such rigorous workflows. To bridge this gap, we introduce SciAgentGym, a scalable interactive environment featuring 1,780 domain-specific tools across four natural science disciplines, supported by a robust execution infrastructure. Complementing this, we present SciAgentBench, a tiered evaluation suite designed to stress-test agentic capabilities from elementary actions to long-horizon workflows. Our evaluation identifies a critical bottleneck: state-of-the-art models still struggle with complex scientific tool-use, and their performance degrades substantially as interaction horizons extend. To address this, we propose SciForge, a data synthesis method that models the tool action space as a dependency graph to generate logic-aware training trajectories. By fine-tuning on these trajectories, our SciAgent-8B outperforms the significantly larger Qwen3-VL-235B-Instruct while exhibiting positive cross-domain transfer of scientific tool-use capabilities. These results underscore the promising potential of next-generation autonomous scientific agents.

1. Introduction

Scientific workflows require iterative, tool-mediated reasoning, but existing benchmarks largely evaluate static question answering. The paper introduces an interactive environment, benchmark, and graph-based synthesis method to address this gap and improve long-horizon scientific tool-use.

  • Existing scientific benchmarks largely target static question answering rather than interactive, tool-mediated workflows.
  • SciAgentGym integrates 1,780 domain-specific tools across four scientific disciplines for multi-turn scientific reasoning.
  • SciAgentBench evaluates agents from elementary actions to high-fidelity long-horizon workflows across 259 tasks and 1,134 sub-questions.
  • GPT-5 achieves 41.3% overall success, with performance falling from 58.8% to 34.6% as interaction horizons increase.
  • SciForge models the tool environment as a dependency graph and synthesizes training trajectories from verified execution paths.
  • Fine-tuning on SciForge trajectories gives SciAgent-8B a +6.7% improvement and enables it to outperform Qwen3-VL-235B-Instruct.

2. Related Work

Prior work includes interactive environments for general-purpose or scientific tasks and benchmarks centered on static scientific question answering. SciAgentGym targets the missing combination of feedback-driven interaction, scientific tool-use, and long-horizon planning.

  • Interactive environments support closed-loop perception, action, and feedback, but most target digital or general-purpose tasks.
  • Scientific benchmarks such as ScienceQA, SciBench, MMMU, GPQA, and SuperGPQA traditionally evaluate scientific knowledge through static question answering.
  • Existing benchmarks generally omit iterative exploration, tool-use, and long-horizon planning.
  • SciAgentGym provides feedback-driven, tool-augmented evaluation through an interactive environment.

3. SciAgentGym

SciAgentGym is an integrated execution environment for multi-step scientific tool-use, combining typed domain tools with sandboxed state and reproducible interaction infrastructure. Its toolkit is organized for extensibility, validation, and execution-grounded feedback.

  • SciAgentGym governs agent-environment dynamics through a modular architecture and interaction protocol.
  • The environment uses type safety, reproducibility, and extensibility as its core design principles.
  • Its formal action space contains 1,780 domain-specific tools plus execute_code and query_database infrastructure primitives.
  • A sandboxed filesystem stores intermediate artifacts and execution history while preserving read-only problem assets.
  • Tools use scientific typed signatures spanning primitive, structured, and domain-specific types such as SMILES and Protein Structure.
  • The toolkit pipeline extracts computational patterns, wraps established scientific packages, organizes tools by function and granularity, and applies automated testing.

4. SciAgentBench

SciAgentBench evaluates scientific agents on verified multi-step tool-use across diverse disciplines and reasoning horizons. It emphasizes compositional, long-horizon tasks while measuring both complete-task correctness and path efficiency.

  • SciAgentBench contains 259 tasks and 1,134 sub-questions spanning four natural science domains.
  • Tasks are standardized from source benchmarks into 1,780 validated tools with typed signatures.
  • Physics contributes 109 tasks, Chemistry 81, Materials Science 37, and Life Sciences 32.
  • The benchmark stratifies reasoning complexity as L1 (≤3 steps), L2 (4–7 steps), and L3 (≥8 steps), with L2+L3 comprising 79% of tasks.
  • Success Rate measures the proportion of tasks whose sub-questions are all answered correctly.
  • Success Weighted by Path Length measures agent path efficiency relative to an expert-verified reference path.

5. SciForge: Execution-Grounded Synthesis

SciForge synthesizes execution-grounded scientific tool-use trajectories by modeling tool dependencies, enforcing executable workflow constraints, and incorporating runtime feedback. It converts verified traces into non-trivial natural-language problems while preserving solvability without exposing exact intermediate outputs.

  • Tool Dependency Graph and Program Sampling: SciForge models each domain’s tool environment as a dependency graph and samples executable program graphs under argument-binding and logical-stage constraints.Edges encode type-level compatibility; sampled inputs must bind to compatible predecessor outputs or root initializers, while stage-aware sampling favors realistic workflow order.
  • Tool Dependency Graph and Program Sampling: Stage-aware ε-greedy sampling balances logical workflow adherence with exploration of complex, non-linear tool dependencies.The method prioritizes stage-compliant predecessors with probability 1 − ε and explores alternatives with probability ε.
  • Forward Execution with Environment: Sampled program graphs are executed in topological order, producing verified traces from validated outputs or fine-grained error feedback that updates environment state.Root bindings are initialized from domain-specific priors before tool execution.
  • Forward Execution with Environment: Execution errors become recovery data: failed calls are paired with corrected inputs and successful re-executions to teach adaptive recovery from environmental feedback.The augmented trajectories explicitly interleave failed attempts with their successful corrections.
  • Trace-to-Question Generation: SciForge converts verified Golden Traces into natural-language scientific problems using domain-specific rubrics, while concealing exact intermediate outputs through semantic abstraction.Quantitative intermediates are mapped to qualitative descriptors so problems remain non-trivial and solvable without leaking exact solutions.

6. Experiments

Experiments evaluate scientific tool-use across models, interaction horizons, failure recovery, disciplines, and training designs. Results show substantial long-horizon degradation, redundant invocation loops, and improved tool-use from execution-verified scientific trajectories.

  • Experimental Setup: Models are evaluated with and without tools using ReAct-style interaction versus chain-of-thought-only reasoning.The benchmark includes recent proprietary and open-source multimodal models, with SciAgent-8B and SciAgent-4B trained on 11,074 trajectories.
  • Long-Horizon Performance: 47.4% average L1 accuracy falls to 16.4% on L3 tasks, a 65.4% relative degradation across models.GPT-5 declines from 58.8% to 34.6%, while Claude-Sonnet-4 drops from 57.4% to 20.3%.
  • Tool Augmentation: Claude-Sonnet-4 improves from 22.4% to 35.9% with tools, while SciAgent-8B gains +6.8%.Tool integration also benefits Qwen3-VL-235B-Think and Qwen3-VL-32B-Think by +3.6% and +3.5%.
  • Cross-Disciplinary Results: Life Sciences shows the largest average tool-related improvement at +8.4%, compared with +2.5% for Physics and +7.0% for Chemistry.Materials Science shows a +3.7% average improvement.
  • Failure Patterns: GPT-5 reaches 41.3% accuracy with 3.41 tool calls, whereas Qwen3-VL-8B-Inst averages 16.55 calls and achieves 23.4% accuracy.The analysis associates excessive calls with repetitive loops; fine-tuning reduces calls while improving tool-augmented accuracy by about 7%.
  • Failure Recovery: Strong models can recover after difficult trajectory phases, whereas Qwen3-VL-8B’s recovery rate declines monotonically from 29% to 10%.Claude-Sonnet-4 follows a 40%→57%→9%→63% recovery pattern across step intervals.
  • Training Ablations: Scientific tool-use training transfers across scientific disciplines, while non-scientific tool fine-tuning produces a 4.9-point overall-score drop.Training without error-recovery trajectories also underperforms across most subjects.
  • Data Scaling: Tool-augmented success improves with training-data scale, while tool-free reasoning saturates early and plateaus.Additional trajectories teach interaction patterns, verification habits, and error handling.

7. Conclusion

The paper shifts scientific evaluation from static knowledge to dynamic, tool-augmented reasoning through SciAgentGym and SciAgentBench. SciForge then uses execution-grounded synthesis to produce trajectories that enable an 8B model to outperform 200B+ baselines.

  • Conclusion: SciAgentGym and SciAgentBench shift scientific evaluation from static knowledge to dynamic, tool-augmented reasoning.The framework evaluates scientific tool use across reproducible infrastructure and diverse domains.
  • Conclusion: SciForge generates logic-aware training data from real, execution-grounded interactions.The method is presented as a synthesis approach for verified trajectories.
  • Conclusion: Training on verified trajectories enables the 8B model to outperform 200B+ baselines.The conclusion frames this result as a foundation for future scientific agents.
  • Evaluation Scope: SciAgentBench aggregates scientific tasks while accounting for effective evaluated sets after unresolved non-response cases.The benchmark also reports per-discipline results with and without tools.
  • Benchmark Construction: The benchmark aggregates source datasets under licensing agreements and reports the included sources and licenses.Its construction is described as respecting the intellectual property rights of original contributors.
  • Benchmark Construction: Candidate tasks are filtered by frontier-model accuracy, stratified to preserve domain distribution, and verified executable in SciAgentGym.The pipeline retains tasks whose ensemble mean accuracy is below 50%.

B.4. Tool Signature and Serialization Specifications

SciAgentGym specifies typed scientific tool interfaces and serializes their inputs and outputs for executable LLM function calling. Its organization combines scientific tool categories with discipline coverage and standardized schemas.

  • Tool Signatures: Each tool v∈V_d specifies a mapping between scientific types.The signature formalizes type-level relationships for tools in discipline d.
  • Tool Coverage: The toolkit covers numerical computation, data processing, visualization, and database queries across six listed scientific disciplines.Physics, Astronomy, and Statistics are grouped as the Physics domain in the main text.
  • Serialization: Inputs and outputs use JSON-compatible serialization, with complex scientific objects reconstructed from identifiers such as SMILES, POSCAR, file paths, and database IDs.Outputs follow a unified dictionary schema with scientific metadata including units and status.
  • Tool Organization: Tools are organized by function—query, computation, analysis, and visualization—and by granularity from atomic primitives to composite operations.The taxonomy supports discoverability and composability in multi-step tool chains.
  • Interface Requirements: The executable interface specification covers input serialization, output encapsulation, and traceability constraints for LLM function-calling compatibility.These requirements are summarized in Table 7.

C.1. Benchmark Construction

The benchmark construction reuses an existing source-benchmark pool while adding structured trajectories, validated tools, quality checks, and diagnostics for feedback utilization. Its efficiency metric is reproducible but measures alignment with one expert strategy rather than absolute planning optimality.

  • Dataset pipeline: The pipeline reuses the source-benchmark pool and curation process, with alternative models replacing the original difficulty filter.The replacement models are GPT-5, Claude-Sonnet-4.5, Gemini-2.5-Pro, and DeepSeek-R1.
  • Dataset structure: The benchmark includes unimodal and multimodal questions with reference solutions, intermediate decompositions, and expert-verified tool-use trajectories.Canonical tool inputs and outputs are included when applicable for reproducible execution and fine-grained supervision.
  • Evaluation design: SPL weights successful solutions by their tool-use length relative to an expert reference path.Because multiple valid solution paths exist, SPL evaluates alignment with the reference strategy rather than absolute planning optimality.
  • Tool organization: The tool taxonomy organizes tools by workflow function and operational granularity.The listed functions are query, computation, analysis, and visualization, while granularity distinguishes atomic primitives from composite operations.
  • Feedback diagnostics: Adaptation, tuning, switching, and loop escape form a diagnostic pipeline for how models use execution feedback.The metrics distinguish changing an action, correcting inputs with the same tool, switching strategies, and avoiding repeated failed actions.
  • Quality control: Every sample receives at least one end-to-end human review, with complex cases double-reviewed.Review checks cover clarity, reasoning and tool-call ordering, recomputed tool outputs, final answers, and template compliance.

E.1. Evaluation Details

Evaluation uses OpenAI-compatible chat-completion models under ReAct-style tool interaction, with explicit planning and execution controls. Outputs are scored through hierarchical matching, semantic verification for text fields, and a backward construction procedure for executable programs.

  • Interaction protocol: With-tools evaluation exposes only task-relevant tools through the OpenAI tool schema and enforces a Planning → Execution procedure.The protocol uses explicit Thought–Action–Observation loops; reverse evaluation applies a convergence prompt for strict JSON output.
  • Inference configuration: Tool-enabled runs use automatic tool choice, disable parallel calls, allow 50 interaction rounds, and impose a 300-second request timeout.All models otherwise use temperature 0.7, followed by final-answer normalization for strict formats.
  • Scoring: Strict hierarchical accuracy recursively matches predicted JSON against ground truth with a numeric tolerance of 0.05.An LLM-based verifier using gpt-4.1 handles failures limited to textual fields.
  • Program construction: Backward program construction takes a target tool, dependency graph, maximum depth, and exploration rate to produce an executable program graph.The algorithm filters candidate tools for stage compliance.

E.2. Training details

Training generates normalized tool-use traces and fine-tunes a Qwen3-VL-8B-Instruct backbone while preserving its vision components. Evaluation examples contrast expert workflows with failure-prone or incomplete model traces.

  • Trace generation: Training traces are generated by a function-calling agent with automatic tool choice, capped at 50 rounds, then normalized into task-specific JSON.Trace generation uses temperature 0.3, while normalization uses temperature 0.1.
  • Supervised fine-tuning: Full-parameter SFT fine-tunes Qwen3-VL-8B-Instruct for 3 epochs while freezing the vision backbone and projector modules.The setup uses DeepSpeed ZeRO-3, gradient checkpointing, FlashAttention, and a learning rate of 1 × 10^-6.
  • Prompt settings: Forward tool mode supplies ReAct instructions in the system prompt and appends an answer-only constraint to the user message.The reverse setting instead appends a template-compliant JSON prompt after tool interaction, while without-tools mode omits the ReAct system prompt.
  • Output verification: Normalization converts final responses into strict task-specific JSON, and verifier prompts enforce rubric-based scoring.Secondary judge prompt templates provide additional verification.
  • Qualitative examples: The thin-film example contrasts a partial-failure interaction trace with an expert workflow integrating analytic derivation and numerical or visual verification.The truss comparison shows the tool-tuned variant returning both force-direction results, unlike the generic baseline’s incomplete final response.
  • Qualitative examples: The plasmid-replacement example requires database queries for plasmid properties followed by difficulty and strategy computation before protocol planning.The trace illustrates precise tool execution in a Life Sciences task.

F.4. Failure Modes

The failure cases show two recurring problems: repetitive tool-use loops and formula-level errors that persist despite plausible reasoning. Examples span mechanics, mass spectrometry, thin-film interference, truss loading, and plasmid replacement.

  • Repetitive tool-use loops: Degenerate tool-use loops repeatedly invoke failed or unproductive actions until the round budget is exhausted.Case 67 repeats a shear-stress subroutine without updating torque or applying allowable-stress constraints; Case 53 shows repetitive peak extraction and invalid outputs.
  • Analytical near-misses: Formula-level errors can produce structurally incorrect closed forms even when the surrounding reasoning appears plausible.Case 68 contrasts an unnecessary functional transformation with a comparatively simple correct expression.
  • Thin-film interference: Thin-film interference traces include partial tool-call failures, whereas the expert trace combines analytical derivation with numerical and visual verification.The paired figures provide a controlled contrast between failure-prone interaction and reference workflow.
  • Truss permissible load: On the truss permissible-load task, the stronger variant evaluates both force directions and returns a complete two-part answer, while the weaker variant omits required output parts.The comparison concerns two Qwen3-VL-8B variants on the same case.
  • Plasmid replacement: The plasmid-replacement trace combines database-backed retrieval with procedural planning.The task requires querying plasmid properties and computing downstream difficulty or strategy before composing a protocol-level plan.
  • Termination failure: A mechanics example records 50 rounds and termination without a boxed answer being extracted.This is a concrete instance of failure under the tool-round limit.
  • Classical statics: The classical statics case uses phase-structured traces and tools for contact geometry, friction forces, equilibrium, verification, plotting, and sensitivity analysis.The task asks for the maximum equilibrium angle of a rod on a horizontal surface with friction.
Loading 2602.12984v2…