ASI-Bench: At the Dawn of Artificial Superintelligence

• Existing evaluations mainly test problems with known answers or tasks whose methods and procedures are substantially specified by humans, leaving open-ended autonomous research weakly assessed.The introduction frames the central challenge as moving beyond applying existing human knowledge to exploring unfamiliar problems, developing solutions, and producing verifiable results.Source: §1 · p. 2
• Each ASI-Bench task is sourced, engineered, audited, cross-reviewed, and executed end-to-end in a sandbox to verify its reference solution and scoring behavior.Four rounds of human cross-review cover scientific logic, specifications, reference artifacts, scoring code, information leakage, and agent trajectories; unreliable or shortcut-prone tasks are revised or excluded.Source: §1 · p. 2
• Average performance across 18 Agent×Model configurations drops from 50.91 with full methodological guidance to 29.10 when only the method is specified and 26.62 when agents must determine the method themselves.The decline reveals a substantial gap between scientific execution and autonomous discovery.Source: §1 · p. 3
• ASI-Bench provides a common reference point for tracking progress toward AI systems capable of general intelligence, innovation, and autonomous execution as methodological guidance is withdrawn.The benchmark is intended to track capabilities that may define the dawn of artificial superintelligence.Source: §1 · p. 3

2.1 Why Do We Need ASI-Bench?

• ASI-Bench jointly evaluates cross-domain project-level research, independent method selection, and end-to-end completion as human methodological guidance is withdrawn.Existing benchmarks test important dimensions of knowledge and execution, but typically use predefined objectives and evaluation criteria; ASI-Bench’s B1–B3 guidance gradient tests whether capabilities persist as guidance recedes.Source: §2.1 · p. 3

2.2 How Is ASI-Bench Designed?

• ASI-Bench comprises 60 project-level research tasks across 11 scientific domains, with matched guidance conditions that keep each task’s objective, data, outputs, and scoring criteria fixed.The tasks require long-horizon work spanning problem understanding, method selection, implementation, experimentation, failure diagnosis, refinement, and validation; together they involve more than 2,600 interaction turns, 2,400 execution steps, and 35 hours of agent execution. The guidance gradient ranges from supplied procedures to independent method selection, while B4 adds task-irrelevant information to test resilience to distraction.Source: §2.2 · pp. 3–4
• The benchmark retains 60 tasks after iterative expert review and sandbox validation, supported by more than 31,000 human-hours.Construction included five review rounds, more than 1,100 review assignments, over 2,000 task revisions, and more than 1,500 sandbox runs to check execution, reproducibility, artifact generation, and scoring consistency.Source: §2.2 · p. 4

2.3 When Should ASI-Bench Be Used?

• ASI-Bench can test autonomous research, compare backbones and agent designs, and help researchers locate failures across guidance levels, domains, and research stages.The authors invite researchers, engineers, model teams, agent developers, and benchmark researchers to contribute new problems, tasks, evaluation methods, and verified results.Source: §2.3 · p. 5
• 26.62 was the average B3 score across 18 state-of-the-art Agent×Model configurations, indicating substantial room to distinguish future systems.In B3, agents receive the research objective, observed data, and required outputs, but no methodological guidance; they must determine how to conduct the research.Source: §2.2 · p. 4, §2.3 · p. 5

3.1 Main Results

• 51.60 is the highest B3 score, achieved only by Codex with GPT-5.6 Sol (ultra); moving from xhigh to ultra raises its score from 40.86 to 51.60.The gain of 10.74 points still leaves systems far from reliable autonomous scientific discovery.Source: §3.1 · p. 5
• 50.91 in B1 falls to 29.10 in B2 (−21.82), then 26.62 in B3 (−2.48); B4 is nearly unchanged at 26.99.This asymmetry identifies turning a specified method into a complete research procedure as the primary bottleneck, rather than selecting a method or handling irrelevant context.Source: §3.1 · p. 5, Table 2 · p. 6
• MiMo V2.5 Pro rises from 16.17 with MiMo Code to 23.25 with Claude Code, while Kimi K3 changes only from 36.22 to 37.09.Harness effects vary by model, so scientific capability cannot be attributed to the backbone alone.Source: §3.1 · p. 6

3.2 Computational Cost

• B1 costs 4.35M tokens and 37.8 minutes per task, while B2 costs 6.91M tokens and 49.7 minutes, making B2 the most expensive guidance setting.Compared with B1, B3 uses 25% more tokens and 22% more time, while B4 uses 30% more tokens and 18% more time; incomplete guidance can require extra exploration and iteration.Source: Figure 4 · p. 7, §3.2 · p. 7
• 40.86% at approximately $684 per run for Codex with GPT-5.6 xhigh closely matches 40.70% at $2,728 for Claude Opus 5 with Claude Code.Codex with GPT-5.6 Ultra reaches the highest B3 score, 51.60%, at approximately $1,550 per run; xhigh offers the strongest cost–performance balance when efficiency matters.Source: Figure 4 · p. 7, §3.2 · p. 7

4 Conclusion

• ASI-Bench’s B1–B4 structure and domain analyses enable controlled model-agent comparisons and reveal where human methodological guidance remains necessary, but no fixed benchmark can represent every meaningful problem.The benchmark’s continued value therefore depends on participation across disciplines and the frontier of model development.Source: §4 · p. 7
• ASI-Bench invites global researchers and builders to contribute tasks, methods, evaluation approaches, reviews, and verified results, test systems, and challenge the benchmark as it evolves.The authors frame it as an open, rigorous community standard for identifying advances in intelligence and accelerating progress toward artificial superintelligence.Source: §4 · pp. 7–8

A Related Work

• AI scientist systems span tool-augmented chemistry, integrated machine-learning research [16], template-light experimentation [17], and collaborative long-horizon investigation [18] [19], but human-specified objectives or methods limit evidence of autonomy [16].The paper distinguishes autonomous workflow construction from reliable execution of provided methodology [8] [9].Source: A Related Work · p. 10
• Scientific benchmarks have expanded from knowledge and programming tests [21] [13] to discovery [12] [22] [23] [24] and executable, replication, and end-to-end workflows [8] [10] [11] [9] [25] [26] [27] [28] [29] [30] [31] [32], but most fix task specifications.Consequently, their scores show completion of assigned workflows, not whether agents constructed them independently or derived them from supplied methods.Source: A Related Work · p. 10
• ASI-Bench matches project instances while withdrawing methodological guidance across four levels, testing execution, method identification, autonomous workflow construction, and robustness to distractors.The instances keep the research objective, data, required outputs, and evaluation criteria fixed.Source: A Related Work · p. 10

B Authors

• The final benchmark contains 60 project-level tasks across 11 domains, retaining contributions from 21 researchers after five review rounds and more than 1,100 task-review assignments.Task contributors are ordered primarily by the number of tasks retained in the final benchmark.Source: B Authors · p. 10

B.1 Task Contributors

• Task contributors included Yuexi Pan1, Hengyu Wang1, Honghe Ren1, Peigan Gao9, Jiangyu Zhou1, Sijia Chen10, Junhao Wu1, and Huan Wang1.Source: §B.1 · p. 10
• Task contributors also included Koutian Wu13, Cheng Zhang13, Yuanning Feng1, Qingyuan Zheng1, Wenzhe Li1, Jiakun Wu1, Ruixuan Jia1, and Junwei Zhou5.Source: §B.1 · p. 10
• Other retained-task contributors were Ergan Shang4, Jingjing Zhou1, Yan Xu2, Hongrui Zhang7, and Liting Mai6.Source: §B.1 · p. 10

B.2 Human Reviewers

• More than 1,100 task-review assignments were completed across five rounds of human review.Source: §B.2 · p. 11
• Human reviewers included Jiangyu Zhou1, Yuexi Pan1, Hengyu Wang1, Honghe Ren1, Xiaohan Jia1, Xueyang Zhou1, Cheng Zhang13, and Yuanning Feng1.Source: §B.2 · p. 11
• The cited passage lists Sijia Chen10, Junhao Wu1, Huan Wang1, Koutian Wu13, Qingyuan Zheng1, Peigan Gao9, Wenzhe Li1, and Jiakun Wu1 among the names shown.Source: §B.2 · p. 11
• The remaining listed reviewers were Jingjing Zhou1, Ruixuan Jia1, Yuexing Hao2, Yan Xu2, Hongrui Zhang7, Zhengxiang Cheng1, and Xianglin Ji2.Source: §B.2 · p. 11

Growing ASI-Bench with the Research Community

• The benchmark’s 60 tasks are a starting point; the authors invite researchers and developers to contribute new scientific challenges and refine evaluations as AI capabilities evolve.Contributors whose tasks pass scientific and technical review and enter an official release are included in that release’s contributor list.Source: C Contributing to ASI-Bench · pp. 11–13

What Constitutes a Complete Task?

• A complete contribution must pair a non-trivial, reproducible scientific objective with prompt variants, specified inputs and outputs, reference generation, evaluation criteria, dependencies, and evidence from all four conditions.Contributors retain the scientific substance of their problem while expressing it in ASI-Bench’s standardized task format.Source: C Contributing to ASI-Bench · p. 11

Constructing the B1–B4 Information Gradient

• B1 supplies the method and procedures, B2 leaves implementation decisions to the agent, B3 requires independent method selection, and B4 adds non-essential context without revealing the method.B3 and B4 must still specify complete inputs and outputs.Source: C Contributing to ASI-Bench · pp. 11–12

How to Contribute a Task

• Contributors formulate a deterministic scientific question, build B1–B4 prompts and scoring, then test all conditions, submit evidence, and revise the task in response to review.Local testing records scores, execution time, environment, sandbox provenance, and scorer-replay evidence to assess difficulty, leakage, reproducibility, and evaluation stability.Source: C Contributing to ASI-Bench · pp. 11–12

Task: 2D Anisotropic Stiff Dynamics

• The agent receives observations of a two-dimensional nonlinear dynamical system on a periodic spatial domain, plus system metadata, field snapshots, and an initial condition.Source: Task: 2D Anisotropic Stiff Dynamics · p. 13
• The task requires characterizing observed dynamics, reproducing them with a model, predicting a future field, and extracting physical diagnostics.Required outputs include the predicted field, spatial spectrum, physical diagnostics, scientific visualizations, data-analysis results, and executable simulation code.Source: Task: 2D Anisotropic Stiff Dynamics · p. 13
• Across B1–B4, the scientific objective, input data, required artifacts, and evaluation criteria remain fixed; only the prompt information changes.Source: Task: 2D Anisotropic Stiff Dynamics · p. 13

B1 Prompt — Full Equation and Solver Guide

• B1 specifies a conserved anisotropic Kuramoto–Sivashinsky-type equation on a doubly periodic domain and provides its domain lengths and coefficients.The listed parameters include alpha, nu, mu, gamma, delta, lambda_xx, lambda_xy, and lambda_yy.Source: B1 Prompt — Full Equation and Solver… · p. 13
• Use a 2D Fourier pseudospectral method for spatial discretization; compute the nonlinear term pseudospectrally in conserved form and apply 2/3-rule dealiasing to that term.Source: B1 Prompt — Full Equation and Solver… · pp. 13–14
• Because the fourth-order terms make the system stiff, B1 prescribes ETDRK4 with precomputed exponential factors and Kassam–Trefethen contour-integral coefficients.The prompt gives the ETDRK4 intermediate-stage and update formulas, then asks the agent to reconstruct the solver, predict the target-time field, and produce the required artifacts.Source: B1 Prompt — Full Equation and Solver… · p. 14

• The goal is to build a model that reproduces the observed dynamics from the supplied initial condition, predicts the future field, and extracts physical diagnostics.Source: B2 Prompt — Method Background · p. 14
• The system is a fourth-order nonlinear 2D PDE combining destabilizing second-order growth, stabilizing fourth-order dissipation with cross-derivative coupling, conservation-type nonlinear transfer, and directionally different behavior.Source: B2 Prompt — Method Background · p. 14
• Because fourth-order terms can cause severe linear stiffness, use spectral discretization with stiffness-aware time stepping, and inspect the data before selecting a solver.Analyze the two spatial directions separately where appropriate and aim to recover interpretable physical structure, not only forecast the field.Source: B2 Prompt — Method Background · p. 15

B3 Prompt — Research Objective and Data Only

• B3 supplies system metadata, observed 2D field snapshots, and an initial condition for a nonlinear system on a periodic domain.The field develops structured patterns and evolves over time.Source: B3 Prompt — Research Objective and… · p. 15
• B3 asks agents to characterize the data, construct a model, predict the field at a target time, and extract physically meaningful quantities.The target time is specified in system_info.json; quantities are to be extracted from observed and predicted fields.Source: B3 Prompt — Research Objective and… · p. 15
• B3 leaves the governing equation, numerical representation, and effective evolution strategy unspecified, making method selection part of the task.Outputs must be finite and agents are instructed to use only the provided data files.Source: B3 Prompt — Research Objective and… · p. 15

B4 Prompt — Research Objective with Distracting Information

• B4 retains the core workflow of characterizing observed data, modeling its dynamics, predicting a target-time field, and extracting physical quantities.Unlike B3's model instruction, B4 does not specify that modeling begins from the supplied initial condition.Source: B4 Prompt — Research Objective with… · p. 15
• B4 adds unrelated PDE families, numerical solvers, and spatial or temporal analysis techniques as possible context, rather than identifying the correct approach.The suggestions range from reaction–diffusion and phase-field models to neural operators, Fourier analysis, and recurrence analysis.Source: B4 Prompt — Research Objective with… · p. 15
• B4 includes speculative preliminary model and shortcut suggestions plus unrelated chatter, while instructing agents to focus on the data and ignore irrelevant information.The notes mention possible phase-field or reaction–diffusion models, a neural surrogate, and comparing observed and target frames.Source: B4 Prompt — Research Objective with… · pp. 15–16