Source-linked AI summary
SciMaster: Towards General-Purpose Scientific AI Agents, Part I. X-Master as Foundation: Can We Lead on Humanity's Last Exam?
Jingyi Chai, Shuo Tang, Rui Ye, Yuwen Du, Xinyu Zhu, Mengcheng Zhou, Yanfeng Wang, Weinan E, Yuzhi Zhang, Linfeng Zhang, Siheng Chen
TL;DR
The paper addresses the challenge of building scientific AI agents that can engage with frontier human knowledge, using HLE as a demanding evaluation. It introduces X-Master and the X-Masters inference-time workflow; X-Masters scores 32.1% on HLE, a new record above 30%.
Problem
Scientific AI agents need to demonstrate a deep grasp of frontier human knowledge, and HLE provides an exceptionally challenging test of that capability.
Source: §1 · p. 2Method
X-Master uses code to interact flexibly with external tools, and X-Masters scales it through a scattered-and-stacked multi-agent workflow.
Source: §1 · p. 2Results
32.1% on HLE set a new record, crossing 30% and surpassing the previous OpenAI and Google Deep Research records of 26.6% and 26.9%.
Source: §6 · p. 10Takeaways & Limitations
The paper shares a practical roadmap for open-source models to attain or surpass state-of-the-art performance on challenging benchmarks without extensive retraining.
Source: §6 · p. 10
Abstract
from arXiv · showhide
The rapid advancements of AI agents have ignited the long-held ambition of leveraging them to accelerate scientific discovery. Achieving this goal requires a deep understanding of the frontiers of human knowledge. As such, Humanity's Last Exam (HLE) provides an exceptionally challenging touchstone for evaluating scientific AI agents. In this work, we aim to construct the foundational architecture for general-purpose agents and validate the capabilities through leading performance on HLE. To achieve this, we introduce X-Master, a tool-augmented reasoning agent designed to emulate human researchers by interacting flexibly with external tools during its reasoning process. This agent, guided by the conceptualization of code as an interaction language, can flexibly leverage built-in Python libraries and our customized tools to augment the reasoning. We further scale its capabilities through X-Masters, a scattered-and-stacked agentic workflow that systematically enhances breadth and depth of reasoning. Our open-source solution, X-Masters, sets a new state-of-the-art record on HLE with a score of 32.1%, surpassing OpenAI's and Google's Deep Research (26.6% and 26.9%) and becoming the first to exceed the 30% threshold. This work allows us to gain a deeper understanding of complex task-solving and accumulates valuable experience that can inform future advancements, guiding subsequent model training.
1 Introduction
The paper presents X-Master as a foundation for general-purpose scientific agents and evaluates its capabilities on Humanity’s Last Exam (HLE). X-Masters combines tool-augmented reasoning with a multi-agent workflow, achieving a new HLE record while sharing practical know-how with the research community.
1 Introduction
- The paper shares its methods and insights to encourage participation and inform future advances, including subsequent model training.It presents inference-time computation as a practical route that bypasses extensive model training.Source: §1 · pp. 2–3
- X-Master uses code as an interaction language to invoke external tools, then incorporates their outputs into the agent’s continuing reasoning.The architecture is intended to emulate researchers’ shifts between internal reasoning and tool use.Source: §1 · p. 2
- X-Masters scales inference-time reasoning by coordinating multiple X-Master instances as Solvers, Critics, Rewriters, and a Selector.The workflow is designed to broaden exploration and deepen refinement.Source: §1 · p. 2
- X-Masters set a new record on Humanity’s Last Exam at 32.1%, exceeding previous OpenAI and Google Deep Research records by 5.5 and 5.2 points, respectively, and becoming the first to surpass 30%.Source: §1 · pp. 2–3
2 Tool-Augmented Reasoning Models as Agents
This section describes X-Master’s tool-augmented reasoning architecture and its interaction with external environments. It explains how code-based interaction, initial reasoning guidance, and search and parsing tools support that process.
2.1 Overview
- X-Master extends open-source models’ reasoning by enabling them to access external environments, with execution results returned to context for continued reasoning.The agent can repeat interactions to acquire information or validate conclusions.Source: §2.1 · p. 3
2.2 Code as Interaction Language
- Python code serves as the interaction language: code is extracted and executed with available libraries and tools, and results are appended to the model’s context.The paper identifies universality, accuracy, and compatibility with existing Python libraries as advantages of this design.Source: §2.2 · p. 4
2.3 Initial Reasoning Guidance
- Initial Reasoning Guidance inserts first-person instructions after the opening think token to prompt non-agentic reasoning models to use tools.The authors say this can elicit code execution and environmental interaction without explicit agentic fine-tuning.Source: §2.3 · p. 4
2.4 Tools
- Web search supplies entity facts, page previews, and related queries, while web parse extracts relevant content from general webpages or scientific papers.The tools support moving from finding relevant pages to examining selected pages more deeply.Source: §2.4 · pp. 4–5
3 Scattered-and-Stacked Agentic Workflows for Inference-Time Computation
X-Masters scales X-Master at inference time through a scattered-and-stacked workflow that explores multiple solutions and then refines and selects among them. The section details its stages and relates the workflow to exploration and exploitation in reinforcement-learning rollouts.
3 Scattered-and-Stacked Agentic Workflows for Inference-Time Computation
- X-Masters coordinates Solver, Critic, Rewriter, and Selector roles to broaden solution exploration and iteratively improve answers at inference time.Its four stages generate initial solutions, refine them, rewrite them, and select a final answer.Source: §3 · p. 5, §3.1 · p. 5
3.2 Scattering for Broad Exploration
- Five Solver agents generate solutions concurrently, Critics assess and improve them, and a second scattering generates five rewritten versions.The workflow uses stochastic decoding to explore diverse initial paths and parallel rewriting to explore multiple refinements.Source: §3.2 · p. 6
3.3 Stacking for Sequential Refinement
- Rewriters synthesize the scattered solutions into a coherent draft, then five Rewriters produce further candidates for a Selector to compare.The Selector chooses among those candidates based on logical consistency and factual accuracy.Source: §3.3 · p. 6
3.4 Discussion: An Analogy to Rollouts in Reinforcement Learning
- The authors compare scattering to rollout exploration and stacking to aggregation and exploitation, with diverse paths followed by synthesis of their strongest elements.They describe the workflow as operationalizing exploration and exploitation at inference time.Source: §3.4 · p. 6
4 Experiments
X-Masters leads HLE with 32.1%, while experiments also report strong biology-benchmark results and improvements across workflow stages. The evaluation uses the text-only HLE subset and compares against research agents and advanced models.
4.1 Experimental Setups
- Evaluation uses 2,518 text-only HLE samples, averages three workflow runs, and uses o3-mini as judge; baselines include research agents and advanced models.DeepSeek-R1-0528 drives the agent with a 64k maximum completion and temperature 0.6.Source: §4.1 · pp. 6–7
4.2 Main Results on Humanity’s Last Exam
- 32.1% on HLE is the highest reported score, surpassing Google Deep Research at 26.9% and OpenAI Deep Research at 26.6%.The authors report this as the first result above 30% on HLE.Source: §4.2 · p. 7
4.4 Analysis
- Workflow stages progressively raise HLE accuracy, with Selector reaching 32.1%.Tool support lifts pass@1 from 17.7% to 21.1%; Critic raises average accuracy from 21.1% to 25.0%; Rewriter adds 5.6 points to reach 30.6%. Figure 7 reports more frequent higher correctness counts after rewriting, especially five correct answers.Source: §4.2 · p. 7, Table 1 · p. 8, §4.4 · p. 8
4.3 Comparisons with Scientific Agents on Biology
- 27.6% accuracy on HLE Biology/Medicine exceeds Biomni’s 17.3% and STELLA’s approximately 26%, but X-Masters covers all 222 text-only questions.Biomni and STELLA evaluated on selected questions from the category.Source: §4.3 · p. 7
- 67.4% on TRQA-lit (choice) gives X-Masters state-of-the-art performance; standalone X-Master scores 62.1% and uses only two web tools against OriGene’s 500-plus expert tools.TRQA-lit comprises 172 multiple-choice questions on biological research.Source: §4.3 · pp. 7–8, Figure 6 · p. 8
4.4 Analysis
- Without scattering, accuracy falls to 25.5%; without stacking, it falls to 25.0%, versus 32.1% with both.The authors associate scattering with diverse solutions and stacking with distilling them into a higher-quality answer.Source: §4.4 · p. 9, Table 2 · p. 9
5 Related Works
The related work describes limitations in existing tool-use strategies and agentic workflows. X-Master uses Python code for flexible tool access, while X-Masters supports iterative multi-turn reasoning and tool use.
Tool-augmented LLMs
- Existing tool-use approaches include structured-output prompting, special-token invocation, and code generation, with limitations in adaptability, tool extensibility, or scope.X-Master uses Python code to access libraries and custom tools and can build tools during inference.Source: §5 · p. 9
Agentic workflows
- Prior agentic workflows commonly use fixed single-turn actions, whereas X-Masters agents can iteratively alternate internal reasoning and external tool use.The passage cites AI co-scientist, ChatDev, MetaGPT, and MAS-GPT as examples of agentic workflows.Source: §5 · p. 9
6 Conclusions and Future Works
The conclusion presents X-Master as the foundational agent for SciMaster and frames the paper’s contribution as openly shared practical know-how for open-source capability gains.
6 Conclusions and Future Works
- X-Master is a tool-augmented reasoning agent, while X-Masters scales its reasoning breadth and depth through a scattered-and-stacked workflow.Source: §6 · pp. 9–10
- 32.1% on HLE set a new record, exceeding OpenAI’s 26.6% and Google DeepMind’s 26.9% and crossing the 30% threshold.Source: §6 · p. 10
- The paper emphasizes openly sharing practical know-how for open-source models to reach or exceed state-of-the-art benchmark performance without extensive retraining.Source: §6 · p. 10
- Future work includes specialized scientific agents and tools for literature analysis, scientific computing, and experimental workflows, plus end-to-end trained agents.Source: §6 · p. 10
7 Cases
The cases illustrate X-Master’s responses to unexpected tool behavior, including trying alternative strategies and checking its answers.
7 Cases
- When tools do not work as expected, X-Master adjusts their use and cross-checks its answer.Source: Figure 8 · p. 13
- When tools return unexpected content, X-Master tries different strategies.Source: Figure 9 · p. 14
- When computations produce unmatched results, X-Master adjusts its strategy and verifies the final answer with Python code.Source: Figure 10 · p. 15