Source-linked AI summary
ASI-Bench: At the Dawn of Artificial Superintelligence
Junwei Zhou, Zhen Sun, Binyu Li, Jiangyu Zhou, Yuexi Pan, Hengyu Wang, Honghe Ren, Xiaohan Jia, Xueyang Zhou, Xiaoyu Cao, Yongchao Chen, Yuanning Feng, Junhao Wu, Cheng Zhang, Sijia Chen, Haoyu Xue, Chengsong You, Huan Wang, Koutian Wu, Peigan Gao, Jiakun Wu, Wenzhe Li, Ergan Shang, Qingyuan Zheng, Jingjing Zhou, Ruixuan Jia, Yan Xu, Hongrui Zhang, Xiao-Han Ma, Zhengxiang Cheng, Yuexing Hao, Liting Mai, Xianglin Ji, Wenjun Zhang, Zhuofan Chen, Yixiao Huang, Chi Wang, Wenyue Hua, Yilun Hao, Yuantao Zhai, Ziyan Zhao, Jingyan Xie
TL;DR
Existing benchmarks provide limited evidence that AI can autonomously conduct scientific research when both problems and methods are not specified by humans. ASI-Bench evaluates this capability by progressively withdrawing methodological guidance, finding average scores fall from 50.91 to 29.10 and then 26.62.
Problem
Existing evaluations provide limited evidence about whether AI can autonomously conduct scientific research when problems and methods are not substantially specified by humans.
Method
ASI-Bench evaluates 60 project-level research tasks across 11 scientific domains under matched conditions with progressively reduced methodological guidance and validated task construction.
Results
Average performance drops from 50.91 with full guidance to 29.10 when only the method is specified and 26.62 when agents determine the method themselves.
Takeaways & Limitations
ASI-Bench provides a common reference point for measuring progress toward general intelligence, innovation, and autonomous execution as methodological guidance is withdrawn.
Takeaways & Limitations
No fixed benchmark or single research team can fully represent the breadth of difficult, meaningful, and verifiable problems future AI systems must confront.
Abstract
from arXiv · showhide
Artificial superintelligence (ASI) requires AI to move beyond mastering existing knowledge toward exploring the unknown, creating new knowledge, and turning new ideas into verifiable results. However, the capabilities of today's AI systems are still largely built on learning, compressing, and applying existing human knowledge. Accordingly, existing benchmarks primarily test whether AI can produce correct answers based on learned knowledge, or whether it can complete tasks under extensive human guidance. We therefore introduce ASI-Bench, the first benchmark to jointly evaluate AI systems' capabilities of innovative exploration and autonomous scientific execution across general research domains, and the first to progressively withdraw human methodological guidance within the same research project to test how far AI can proceed on its own. Built by over 40 experts with the cost of 31,000+ human hours, ASI-Bench contains 60 project-level research tasks across 11 scientific domains and progressively reduces methodological guidance to test whether AI can independently select methods, conduct research, and produce verifiable results. All tasks undergo expert review, AI-assisted auditing, sandbox execution, and scorer validation. Across 18 state-of-the-art agent--model configurations, the average score drops from 50.91 with full methodological guidance to 29.10 with only the method specified and 26.62 when agents must determine the method themselves. This sharp decline shows that current systems remain heavily dependent on human guidance and are still far from autonomously conducting end-to-end, project-level scientific research. ASI-Bench is open to the world. We invite researchers and builders everywhere to contribute new tasks, challenge the limits of today's AI, and help accelerate humanity's collective path toward artificial superintelligence at https://asibench.apexin.ai/submit.
1 Introduction
ASI-Bench evaluates whether AI can move beyond applying existing human knowledge to explore unfamiliar problems, create new solutions, and produce verifiable results. Across 18 Agent×Model configurations, performance drops sharply as methodological guidance is withdrawn, exposing current systems’ dependence on human direction.
- Motivation and contribution: ASI-Bench targets AI’s ability to explore unfamiliar problems, develop new solutions, and turn them into verifiable results beyond applying existing human knowledge.The benchmark addresses capabilities needed for general intelligence, innovation, and autonomous execution.
- Benchmark construction: Tasks are collected from traceable scientific sources, engineered by domain experts, and validated through AI-assisted auditing and four rounds of human cross-review.Reviews examine scientific logic, task specifications, reference artifacts, scoring code, and information leakage.
- Empirical findings: 50.91 with full methodological guidance falls to 29.10 when only the method is specified and 26.62 when agents must determine the method themselves.This substantial decline reveals a gap between scientific execution and autonomous discovery.
- Contribution: ASI-Bench provides a common reference point for tracking the emergence of general intelligence, innovation, and autonomous execution as methodological guidance is withdrawn.Its purpose is to measure the transition from systems that learn, compress, and apply human knowledge toward autonomous scientific capability.
2 Benchmark Design
ASI-Bench jointly evaluates cross-domain project-level research, independent method selection, and end-to-end execution while progressively withdrawing human methodological guidance. It contains 60 tasks across 11 scientific domains, uses matched guidance conditions, and is rigorously constructed and validated.
- Benchmark Design: ASI-Bench jointly measures cross-domain generality, methodological autonomy, and end-to-end scientific execution through project-level research and a B1–B3 guidance gradient.The gradient tests whether capabilities persist as human methodological guidance is progressively withdrawn.
- Autonomous Research with Reduced Human Guidance: Matched guidance conditions keep the task, data, required outputs, and scoring criteria fixed while methodological guidance is progressively reduced to isolate scientific autonomy.The same research objective and evaluation conditions are retained as agents move from executing prescribed procedures toward selecting methods themselves.
- Autonomous Research with Reduced Human Guidance: Each task requires a long-horizon process spanning problem understanding, method selection, implementation, experimentation, failure diagnosis, refinement, and validation, producing verifiable scientific artifacts.Across the 60 tasks, these processes involve more than 2,600 interaction turns, 2,400 execution steps, and over 35 hours of agent execution.
- Broad Scientific Coverage: The benchmark contains 60 project-level research tasks spanning 11 scientific domains, requiring one Agent×Model system to generalize across data types, methods, and validation criteria.Covered domains include mathematics, physics, chemistry, biology, astronomy, materials science, earth science, medicine and biostatistics, computer science, robotics, and electrical engineering.
- Rigorous Construction and Validation: The benchmark was built iteratively from more than 1,300 candidate research ideas through five review rounds, over 1,100 review assignments, and more than 2,000 task revisions.This construction process is presented as large-scale and iterative rather than a single-pass collection.
- To measure frontier progress: 26.62 is the average B3 score across 18 state-of-the-art Agent×Model configurations, indicating substantial room for future systems to improve under minimal methodological guidance.The benchmark is described as far from saturated and intended to distinguish progress toward more autonomous, higher-order intelligence.
3 Experiments
Experiments on 18 Agent×Model configurations across 60 ASI-Bench tasks show that autonomous scientific research remains limited, with performance strongly dependent on detailed methodological guidance and harness design. Complete guidance also reduces computational cost, while higher spending does not consistently yield better performance.
- Scientific Autonomy Remains Limited: B3 rises from 40.86 to 51.60 when GPT-5.6 Sol inference increases from xhigh to ultra, a gain of 10.74.Stronger inference-time reasoning improves independent method selection and workflow construction, but the resulting capability remains limited overall.
- Dependence on Detailed Methodological Guidance: Average performance falls 21.82 points from 50.91 in B1 to 29.10 in B2, while removing the method in B3 causes only a further 2.48-point drop to 26.62.These results identify converting a scientific method into a complete research procedure as the main bottleneck rather than method selection itself.
- Harnesses Shape Model’s Capability: MiMo V2.5 Pro improves from 16.17 with MiMo Code to 23.25 with Claude Code, while Kimi K2.7 rises from 19.72 to 27.34.The harness effect is not uniform: Kimi K3 differs only slightly between Kimi Code and Claude Code, at 36.22 versus 37.09.
- Complete Guidance Reduces Compute Cost: B1 requires 4.35M tokens and 37.8 minutes per task on average; removing procedural guidance raises consumption by 25% in B3 and 30% in B4.B1 specifies the method, implementation steps, and parameter choices, making complete guidance the least expensive setting.
- Higher Cost Does Not Guarantee Better Performance: Codex with GPT-5.6 xhigh scores 40.86% at approximately $684 per run, closely matching Claude Opus 5 with Claude Code at 40.70% despite its $2,728 cost.Higher spending can improve absolute performance, but comparable scientific performance can occur at substantially different computational budgets.
4 Conclusion · A Related Work · B Authors
ASI-Bench is positioned as open, evolving infrastructure for evaluating scientific AI, extending prior benchmarks toward executable project-level autonomy and guidance dependence. Its construction combines broad participation, expert review, 60 tasks, and 11 scientific domains.
- 4 Conclusion: ASI-Bench aims to become shared research infrastructure, enabling controlled model–agent comparisons while exposing where human methodological guidance remains necessary.Its B1–B4 structure and domain-level analyses support comparisons beyond a single leaderboard.
- 4 Conclusion: The benchmark is intended to evolve through community contributions of research problems, executable tasks, evaluation methods, reviews, and verified results.The authors invite scientists, engineers, model teams, agent researchers, and benchmark developers to test and challenge it as capabilities advance.
- 4 Conclusion: An open, rigorous, continually evolving community standard is presented as a foundation for identifying genuine advances in intelligence and accelerating progress toward artificial superintelligence.The conclusion emphasizes collective development rather than a fixed benchmark or single research team representing all future problems.
- A Related Work: AI scientist systems combine language models with planning, retrieval, code execution, and scientific tools to automate increasingly complete research workflows [16].Coscientist demonstrated tool-augmented chemistry planning and experimentation, while The AI Scientist integrated generation, implementation, experimentation, visualization, writing, and review [16].
- A Related Work: Scientific-agent benchmarks range from advanced knowledge tests to executable programming, data-driven discovery, and hypothesis-generation tasks, but cover only selected capabilities [21] [13] [12] [22] [23] [24].GPQA and Humanity’s Last Exam test knowledge; SciCode tests scientific programming; DiscoveryBench, BLADE, DSBench, and HypoBench target discovery or hypothesis generation.
- A Related Work: Prior autonomy benchmarks vary available information, but ScienceAgentBench does not separate method selection from procedural guidance, while ProjectionBench focuses on hypothesis generation rather than executable project workflows [8] [33].These distinctions motivate evaluating progressively reduced guidance in project-level research.
- B Authors: ASI-Bench was built through broad participation and independent expert review, retaining 60 project-level tasks across 11 scientific domains from 21 researchers.Construction included five review rounds, more than 1,100 task-review assignments, and repeated revisions.
B.1 Task Contributors · B.2 Human Reviewers · B.3 Affiliations
The appendix identifies the researchers who contributed retained ASI-Bench tasks and participated in human review, and maps affiliation numbers to institutions. Human review comprised more than 1,100 assignments across five rounds.
- B.1 Task Contributors: The final ASI-Bench task contributors are listed by name in the appendix.The contributor list spans two passages.
- B.1 Task Contributors: The contributor roster includes Yuexi Pan, Hengyu Wang, Hengyu Wang, and other researchers identified in the appendix.The supplied roster also includes researchers such as Honghe Ren, Peigan Gao, Jiangyu Zhou, Sijia Chen, Junhao Wu, and Huan Wang.
- B.2 Human Reviewers: More than 1,100 task-review assignments were completed across five rounds of human review.The appendix introduces the researchers who participated in this review process.
- B.2 Human Reviewers: The human-reviewer roster includes Jiangyu Zhou, Yuexi Pan, Hengyu Wang, Honghe Ren, and other named participants.The supplied list also includes Xiaohan Jia, Xueyang Zhou, Cheng Zhang, Yuanning Feng, and Xianglin Ji.
- B.3 Affiliations: Affiliation numbers 1–6 correspond to Tsinghua University, MIT, Harvard University, Carnegie Mellon University, the University of Michigan, and the University of Illinois Urbana–Champaign.The listed locations span Beijing, Cambridge, Pittsburgh, Ann Arbor, and Urbana–Champaign.
- B.3 Affiliations: Affiliation numbers 7–13 correspond to Boston University, the University of Queensland, USTC, the Flatiron Institute, Microsoft Research, AG2 AI, and independent researchers.The appendix provides locations for the universities and New York for the Flatiron Institute.
C Contributing to ASI-Bench
ASI-Bench is designed as an evolving, community-driven benchmark whose initial 60 tasks across 11 scientific domains are only a starting point. Contributors can add genuine, reproducible scientific problems through a standardized B1–B4 task format, local testing, and scientific and technical review.
- Growing ASI-Bench with the Research Community: Each contributed task must define a genuine, non-trivial scientific objective and produce artifacts that can be evaluated using explicit, reproducible criteria.Contributors retain the scientific substance of their problems while expressing them through the standardized ASI-Bench format.
- What Constitutes a Complete Task?: A complete contribution includes the scientific objective, four prompt variants, explicit input/output specifications, evaluation configuration, runtime dependencies, and local-testing evidence.The package is intended to support reproducible execution and evaluation rather than merely describe a scientific question.
- Constructing the B1–B4 Information Gradient: The B1–B4 gradient controls methodological information from complete procedural guidance in B1 to objective-and-constraint specifications in B3, with B4 adding only non-essential information.B2 provides the intended methodological approach while leaving implementation decisions to the agent; B3 and B4 preserve the same task objective and required artifacts.
- Test, Submit, and Revise: Before submission, contributors test B1–B4 and report scores alongside execution, environment, sandbox, and scorer-replay evidence to assess difficulty, leakage, reproducibility, and evaluation stability.The process uses a public task template, an online submission portal, or a CLI launcher, followed by review and revision.
- An Open Benchmark for the Path Ahead: The benchmark invites researchers to contribute new scientific challenges and developers to refine evaluations, identify failure cases, and validate tasks as AI capabilities evolve.The current 60 tasks are explicitly treated as a starting point rather than a closed test set.
D Case Study … B1 Prompt — Full Equation and Solver Guide
The case study evaluates whether an agent can analyze observations of a two-dimensional nonlinear dynamical system, reconstruct its dynamics, forecast a future field, and produce verifiable scientific artifacts. In the fully guided B1 condition, the prompt specifies the anisotropic Kuramoto–Sivashinsky-type equation, Fourier pseudospectral discretization, dealiasing, and ETDRK4 integration.
- Task: 2D Anisotropic Stiff Dynamics: The task supplies observations of a two-dimensional nonlinear dynamical system on a periodic spatial domain and asks the agent to characterize its dynamics.
- Task: 2D Anisotropic Stiff Dynamics: The agent must construct a model, predict the field at a future target time, extract physical diagnostics, and provide executable scientific outputs.Required artifacts include a predicted field, spatial spectrum, physical diagnostics, visualizations, data-analysis results, and complete simulation code.
- Task: 2D Anisotropic Stiff Dynamics: The scientific objective, input data, artifacts, and evaluation criteria remain fixed across B1–B4; only the prompt information changes.
- B1 Prompt — Full Equation and Solver Guide: B1 gives the full conserved anisotropic Kuramoto–Sivashinsky-type equation on a doubly periodic domain and provides all domain and equation parameters.
- B1 Prompt — Full Equation and Solver Guide: The prescribed spatial method is a two-dimensional Fourier pseudospectral scheme with conserved-form nonlinear evaluation and 2/3-rule dealiasing.
- B1 Prompt — Full Equation and Solver Guide: The Fourier-space linear operator combines second- and fourth-order anisotropic derivative terms with the supplied coefficients.
- B1 Prompt — Full Equation and Solver Guide: Because fourth-order terms make the system stiff, B1 instructs the agent to use ETDRK4 with precomputed exponentials and Kassam–Trefethen contour-integral coefficients.The prompt also specifies the intermediate ETDRK4 states and final Fourier-state update using E, Q, f1, f2, and f3.
- B1 Prompt — Full Equation and Solver Guide: The B1 assignment is to analyze the observed data, reconstruct the numerical solver, predict the target-time field, and produce the required scientific artifacts.
B2 Prompt — Method Background
The task concerns a stiff, fourth-order nonlinear PDE in two spatial dimensions with directionally uneven, complex dynamics. It requires modeling the observed system, forecasting its future field, and extracting interpretable physical diagnostics.
- Task: The objective is to analyze observed data, build a numerical model reproducing the dynamics from the supplied initial condition, predict the future field, and recover physical diagnostics.The two spatial directions should be analyzed separately where appropriate before selecting a solver.
- Physical and Numerical Background: The system combines destabilizing second-order growth, stabilizing fourth-order dissipation with cross-derivative coupling, nonlinear energy transfer, and direction-dependent behavior.These ingredients produce irregular pattern dynamics rather than simple steady diffusion.
- Physical and Numerical Background: Severe linear stiffness from fourth-order terms motivates spectral discretization with stiffness-aware time stepping, including ETDRK4 and semi-implicit spectral integrators.ETD-style spectral solvers are competitive but are not the only reasonable option.
B3 Prompt — Research Objective and Data Only · B4 Prompt — Research Objective with Distracting Information
The prompts define a project involving exploration of observed 2D nonlinear dynamics, model identification, field prediction, and extraction of physical quantities. B3 withholds the governing equation and evolution strategy, while B4 adds unrelated scientific suggestions and background chatter that must be ignored.
- B3 Prompt — Research Objective and Data Only: The task uses observed spatio-temporal data from a 2D nonlinear system on a periodic domain whose field develops structured, complex patterns over time.
- B3 Prompt — Research Objective and Data Only: The supplied files contain system parameters and timing metadata, observed 2D field snapshots, and the initial field for prediction.
- B3 Prompt — Research Objective and Data Only: The research objective is to characterize spatial and temporal structure, construct a model reproducing the dynamics, predict the target-time field, and extract physical quantities.
- B3 Prompt — Research Objective and Data Only: B3 leaves the governing equation, numerical representation, and effective evolution strategy unspecified, making approach identification part of the task.
- B4 Prompt — Research Objective with Distracting Information: B4 preserves the same four objectives but removes the requirement to use the initial condition and system_info.json explicitly stated in B3.
- B4 Prompt — Research Objective with Distracting Information: B4 surrounds the task with unrelated PDE families, speculative preliminary notes, and off-topic background chatter, while instructing the agent to focus on the supplied data and ignore irrelevant information.