PaperlayerView exact revision on arXiv

Source-linked AI summary

ATLAS: A High-Difficulty, Multidisciplinary Benchmark for Frontier Scientific Reasoning

Hongwei Liu, Junnan Liu, Shudong Liu, Haodong Duan, Yuqiang Li, Mao Su, Xiaohong Liu, Guangtao Zhai, Xinyu Fang, Qianhong Ma, Taolin Zhang, Zihan Ma, Yufeng Zhao, Peiheng Zhou, Linchen Xiao, Wenlong Zhang, Shijie Zhou, Xingjian Ma, Siqi Sun, Jiaye Ge, Meng Li, Yuhong Liu, Jianxin Dong, Jiaying Li, Hui Wu, Hanwen Liang, Jintai Lin, Yanting Wang, Jie Dong, Tong Zhu, Tianfan Fu, Conghui He, Qi Zhang, Songyang Zhang, Lei Bai, Kai Chen

arXiv:2511.14366v2cs.CL

TL;DR

Established benchmarks are increasingly saturated, and existing high-difficulty tests leave gaps in multidisciplinary scientific reasoning, realistic answer formats, and contamination resistance. ATLAS combines a scientific benchmark with expert creation, adversarial filtering, and multi-stage review. The authors report that current models perform poorly on it, with analyses providing diagnostic information about their reasoning deficiencies.

  • Problem

    Benchmark saturation and gaps in disciplinary breadth, realistic answer formats, and contamination resistance limit existing evaluations of frontier scientific reasoning.

    Source: §1.1 · p. 2, §1.3 · p. 3

  • Method

    ATLAS combines expert-generated scientific problems with originality and difficulty filtering, multi-stage review, and answer refinement across seven core scientific disciplines.

    Source: §3.3 · p. 8, §4.1 · p. 9

  • Results

    Systematic evaluation finds that all tested models perform poorly on ATLAS, and performance and error analyses provide diagnostic information about their scientific reasoning.

    Source: §7 · p. 20

  • Takeaways & Limitations

    The authors present ATLAS as a touchstone for frontier AI-for-Science capabilities and report that its fine-grained evaluation results may benefit reinforcement learning with verifiable rewards.

    Source: §5.2 · p. 12, §7 · p. 20

    ATLAS has about 800 problems—fewer than some larger benchmarks—and its current version is predominantly Chinese and limited to seven core scientific subjects.

    Source: §6.2 · p. 19

Abstract

from arXiv · show

The rapid advancement of Large Language Models (LLMs) has led to performance saturation on many established benchmarks, questioning their ability to distinguish frontier models. Concurrently, existing high-difficulty benchmarks often suffer from narrow disciplinary focus, oversimplified answer formats, and vulnerability to data contamination, creating a fidelity gap with real-world scientific inquiry. To address these challenges, we introduce ATLAS (AGI-Oriented Testbed for Logical Application in Science), a large-scale, high-difficulty, and cross-disciplinary evaluation suite composed of approximately 800 original problems. Developed by domain experts (PhD-level and above), ATLAS spans seven core scientific fields: mathematics, physics, chemistry, biology, computer science, earth science, and materials science. Its key features include: (1) High Originality and Contamination Resistance, with all questions newly created or substantially adapted to prevent test data leakage; (2) Cross-Disciplinary Focus, designed to assess models' ability to integrate knowledge and reason across scientific domains; (3) High-Fidelity Answers, prioritizing complex, open-ended answers involving multi-step reasoning and LaTeX-formatted expressions over simple multiple-choice questions; and (4) Rigorous Quality Control, employing a multi-stage process of expert peer review and adversarial testing to ensure question difficulty, scientific value, and correctness. We also propose a robust evaluation paradigm using a panel of LLM judges for automated, nuanced assessment of complex answers. Preliminary results on leading models demonstrate ATLAS's effectiveness in differentiating their advanced scientific reasoning capabilities. We plan to develop ATLAS into a long-term, open, community-driven platform to provide a reliable "ruler" for progress toward Artificial General Intelligence.

1. Introduction

Established benchmarks have become less able to distinguish frontier models, while existing high-difficulty tests leave gaps in integrated scientific reasoning and realistic answer formats. ATLAS addresses these gaps with original multidisciplinary problems and a rigorous evaluation pipeline.

1.1. Benchmark Saturation Phenomenon

  • MMLU accuracies above 90% and MATH scores rising from below 10% to above 90% in three years illustrate benchmark saturation’s reduced ability to distinguish frontier models.The introduction argues that evaluation needs more challenging tools to assess frontier capabilities.Source: §1.1 · p. 2

1.2. Evaluation Needs for Frontier Scientific Reasoning

  • AI for Science requires models to combine scientific knowledge with advanced, multi-step, interdisciplinary reasoning, motivating benchmarks tailored to those capabilities.ATLAS aims to serve as a touchstone for assessing models’ scientific reasoning abilities.Source: §1.2 · pp. 2–3

1.3. Limitations of Existing High-Difficulty Benchmarks

  • Existing high-difficulty benchmarks can be narrow in disciplinary scope, untailored to integrated AI-for-Science reasoning, vulnerable to contamination, or reliant on simplified answer formats.ATLAS seeks to address these gaps with original problems and complex natural and symbolic expressions.Source: §1.3 · p. 3

1.4. Our Contributions

  • ATLAS contributes an expert-created benchmark, a human–AI construction pipeline, a scalable LRM-as-Judge workflow, and a planned community-driven platform for continued evaluation.The platform is intended to support long-term tracking of progress toward AGI.Source: §1.4 · pp. 3–4

2. Related Work

The related work traces shifts from broad and human-centric tests to frontier-difficulty and specialized STEM benchmarks. ATLAS combines aspects of these lines for AI for Science, alongside discussion of LLM-as-a-Judge.

2. Related Work

  • The literature review situates ATLAS amid shifts toward human-centric, frontier-difficulty, and specialized STEM evaluations, alongside work on LLM-as-a-Judge.It frames benchmark evolution as a response to rapid LLM progress and saturation.Source: §2 · p. 4

2.1. From Broad Coverage to Human-Centric Benchmarks

  • Human-centric benchmarks use standardized-test questions, but their source material limits difficulty and public questions create contamination risk.The cited examples include AGIEval and C-Eval; ATLAS instead uses expert-authored original problems.Source: §2.1 · p. 4

2.2. The Rise of Frontier-Difficulty Reasoning Benchmarks

  • Frontier-difficulty benchmarks use expert creation and adversarial filtering, while specialized STEM sets emphasize deep reasoning in fields such as mathematics and physics.ATLAS adopts methodological principles from GPQA and HLE and draws on complex reasoning in specialized STEM benchmarks.Source: §2.2 · p. 4

2.3. Synthesis and Positioning of ATLAS

  • ATLAS applies originality-focused methods and deep-reasoning goals to a broad but coherent suite of AI-for-Science subjects.Table 1 compares prominent high-difficulty benchmarks by their goals and subject scope.Source: §2.3 · p. 5, Table 1 · p. 5

2.4. Additional Discussion of LLM-as-Judge

  • LLM-as-a-Judge offers a scalable, economical proxy for human assessment of open-ended outputs, but bias, inconsistency, and prompt sensitivity remain concerns.The cited discussion identifies these issues as targets for continued progress.Source: §2.4 · p. 5

3. ATLAS Construction Pipeline

ATLAS construction combines expert problem creation with machine-based filtering and human review. Its questions favor realistic, complex responses and are refined into canonical answers for evaluation.

3. ATLAS Construction Pipeline

  • The construction process uses a multi-stage approach to ensure problem quality, difficulty, and originality.The paper presents methodological rigor as central to benchmark credibility.Source: §3 · p. 6

3.1. Question Design for Real-World Fidelity

  • More than half of ATLAS questions have multiple sub-parts, and answers can require equations, chemical products, or simplified proofs rather than simple selections.Questions are available in both English and Chinese.Source: §3.1 · p. 6

3.2. Core Design Principles

  • Problems are newly authored or substantially re-engineered by domain experts for graduate-level-or-higher, multi-step reasoning rather than information retrieval.The design pairs a single objectively verifiable answer with complex answer formats and expert review alongside adversarial filtering.Source: §3.2 · pp. 6–7

3.3. Data Generation and Quality Assurance Workflow

  • Experts from more than 25 institutions submit original problems with canonical solutions, which undergo formatting normalization and similarity screening against an offline database.This pre-screening is intended to filter duplicates and support novelty.Source: §3.3 · p. 7
  • Submissions are screened with RAG-based originality scoring, then tested by an ensemble of reasoning models; problems exceeding 40% accuracy over ten attempts are returned for possible refinement or discard.Only submissions meeting a high originality threshold advance; the difficulty criterion accepts problems at 40% accuracy or less.Source: §3.3 · pp. 7–8
  • Three anonymous same-domain reviewers assess correctness, clarity, and difficulty, with a senior meta reviewer resolving discrepancies before a final online-disclosure check.Only problems that clear the validation stages are admitted to the database.Source: §3.3 · p. 8
  • A language-model agent extracts answer components, checks scientific reasoning, and reformats solutions into clear, stepwise canonical answers.The described example corrects an initial hydrolysis hypothesis to solvent extraction.Source: §3.3 · p. 8
  • The authors characterize the combination of adversarial model filtering and multi-stage manual review as a dual-filter definition of high-quality difficulty.They distinguish machine challenge from difficulty grounded in scientific depth rather than flaws, ambiguity, or obscure knowledge.Source: §3.3 · p. 8

4. Dataset Analysis: The ATLAS Corpus

The first ATLAS corpus contains approximately 800 scientific reasoning problems across seven disciplines, with text-only and multimodal problems. Representative examples illustrate its demands for expert knowledge, multi-step reasoning, and precise answer generation.

4.1. Quantitative Overview

  • Approximately 800 problems span seven scientific disciplines, with a balanced distribution of text-only and multimodal problems.The corpus is presented as the first phase of the ATLAS project.Source: §4 · p. 8, §4.1 · p. 9

4.2. Qualitative Examples

  • The mathematics example requires counting vectors satisfying a bilinear-form condition, using claims that yield a recurrence relation.The passage says solving the recurrence gives the final result as a function of p, m, and N.Source: §4.2 · p. 9
  • The MAVS example links its central disordered region to 3’UTR binding and asks how RNase affects the signalosome and transcription-factor phosphorylation.The refined answers identify the MAVS signalosome complex and the phosphorylation level as affected.Source: §4.2 · p. 10
  • ATLAS examples require expert domain knowledge and multi-step reasoning, including symbolic derivation and mechanistic integration of molecular components and effects.The mathematics example uses recurrence relations over a finite ring; the biology example requires connecting components, binding targets, and functional consequences.Source: §4.2 · p. 10

4.3. Language and Structural Characteristics

  • Problem statements average about 65 words, sometimes exceed 200, and require generated short answers, precise values, or LaTeX expressions.The paper says this generative format can reduce guessing relative to multiple-choice questions.Source: §4.3 · p. 11

5. Evaluation and Performance

ATLAS is evaluated through a structured workflow in which LLM judges assess generated answers, and the validation results distinguish models overall and by subject. Further analyses examine answer errors, output budgets, and judge-model differences.

5.1. Setup

  • The evaluation covers closed- and open-source reasoning models, uses OpenAI o4-mini and GPT-OSS-120B as judges, and reports average accuracy across multiple inferences and G-Pass@k to assess performance stability.Closed-source models use official APIs; open-source models use serving frameworks. The maximum generation length is 32,768 tokens, the sampling temperature is 0.6, and four predictions are generated per question.Source: §5.1 · p. 11

5.2. Evaluation Workflow

  • The evaluation workflow generates JSON predictions, parses answers, has judges assess subanswers against ground truth, and parses judgments into metrics.The authors say its fine-grained judgments may benefit applications in Reinforcement Learning with Verifiable Rewards.Source: §5.2 · p. 12

5.3. Quantitative Results

  • 42.9% accuracy made OpenAI GPT-5-High the validation leader, with mG-Pass@2 34.7% and mG-Pass@4 32.1%; OpenAI o3-High followed at 35.3%.Table 3 reports average accuracy and mG-Pass scores across four predictions, with higher mG-Pass indicating greater stability.Source: Table 3 · p. 13, §5.3 · p. 13
  • Across subjects, GPT-5 leads most fields, Grok-4 tops computer science, and Qwen3-235B-A22B-2507 is competitive in mathematics.The reported mG-Pass@{2, 4} trends generally align with accuracy, and GPT-5 scores highly across domains.Source: §5.3 · pp. 13–14, Figure 7 · p. 14

5.4. Further Analysis

  • Most selected models performed better with a 64k rather than 32k output budget, but the longer limit incurred significant inference overhead.The authors identify inference efficiency as an important area for improvement.Source: Table 5 · p. 15, §5.4 · p. 16
  • Numerical discrepancies were the most frequent sampled error category at 27.0%, followed by mathematical errors at 16.5%, missing components at 13.0%, and structural mismatches at 11.0%.Almost all evaluated models had a 0.00% JSON parse-error rate, while truncation ranged from 0.00% for OpenAI o4-mini to 10.38% for Grok-4.Source: §5.4 · pp. 15–16, Table 4 · p. 15
  • Across nearly all models and metrics, Qwen3-235B-A22B typically allocates lower scores than GPT-OSS-120B; it is more susceptible to errors in knowledge and semantic understanding, while assessing judgment-model effectiveness requires further investigation.Source: §5.4 · p. 17

6. Discussion and Future Work

ATLAS results are presented as a measure of the remaining gap between current models and human experts in demanding scientific reasoning. The authors identify limits in dataset scale and scope, and propose continuous development of the platform.

6.1. Implications of the Results

  • ATLAS results indicate a large gap between current models and human experts in scientific domains requiring deep, rigorous, comprehensive reasoning.The authors connect this gap to the distance from reliable scientific discovery and progress toward AGI.Source: §6.1 · p. 19

6.2. Limitations of ATLAS

  • The initial dataset contains about 800 problems, fewer than some larger-scale benchmarks, and is predominantly Chinese across seven core scientific subjects.The authors identify these as limitations and starting points for future work.Source: §6.2 · p. 19

6.3. The ATLAS Platform: A Future Roadmap

  • ATLAS is planned as an open, collaborative platform for continuous construction rather than a one-time data release.The roadmap proposes regular problem-pack updates, broader fields, languages and task formats, and expert collaboration.Source: §6.3 · p. 20
  • Planned problem-pack updates are intended to keep pace with model development and prevent benchmark overfitting.The roadmap also proposes inviting domain experts worldwide to create and review problems.Source: §6.3 · p. 20

7. Conclusion

ATLAS combines a rigorous construction process with evaluation results showing poor performance by leading models. The appendices point readers to the benchmark’s subject coverage, question-type distribution, and contributor institutions.

7. Conclusion

  • ATLAS combines expert-original problem creation, adversarial model filtering, and multi-stage blind review to construct a high-difficulty, multidisciplinary benchmark.The initial dataset focuses on core AI for Science areas and is presented in Chinese.Source: §7 · p. 20
  • Systematic evaluation found that all tested advanced models perform poorly on ATLAS, revealing deficiencies in deep scientific reasoning.The authors say their analysis of performance and error patterns provides diagnostic information and research directions for future model improvements.Source: §7 · p. 20
  • The authors frame ATLAS and its continued development as a tool for measuring and guiding AI progress in scientific discovery.They state that it may contribute to progress toward AGI.Source: §7 · p. 20

A.1. ATLAS Subjects

  • Table 8 presents the subjects and sub-fields represented in ATLAS.The table is described as giving question counts per sub-field.Source: §A.1 · p. 27

A.2. ATLAS Question Type Distribution

  • The paper analyzes ATLAS question types in Figure 8, which shows their hierarchical distribution, and provides the question-type definition in Table 9.Source: §A.2 · p. 27, Figure 8 · p. 27

B. Question Contributors

  • ATLAS contributors include scholars from over 25 universities or research organizations, with institution statistics shown in Figure 9.Source: B. Question Contributors · p. 27

C.1. Peer Review Stage

ATLAS uses expert peer review followed by a meta-review stage for questions that pass peer review. The appendix also shows contributor-institution statistics.

C.1. Peer Review Stage

  • Two to three domain experts independently score each problem on content and format, scientific value, and difficulty; advancement requires an average score of at least 3.0.The scoring rubric is exemplified for mathematics in Table 10.Source: §C.1 · p. 27
  • Figure 9 presents statistics on the institutions contributing questions to ATLAS.Source: Figure 9 · p. 28

C.2. Meta Review Stage

  • The meta-review stage invites experts to review every question that passed peer review and summarizes reasons for 50 sampled failures in Table 11.Source: §C.2 · p. 28

E. Prompts for Evaluation

The appendix points to ATLAS dataset and question-type tables, a peer-review rubric and rejection analysis, and prompts for prediction and answer judgment.

E. Prompts for Evaluation

  • Prompt E.1 and Prompt E.2 provide the detailed instructions used for evaluation.Source: E. Prompts for Evaluation · p. 28
  • Table 8 reports the number of ATLAS questions in each sub-field.Source: Table 8 · p. 29
  • Table 9 categorizes question forms by formal structure and expected answer type, not by subject matter.Source: Table 9 · p. 30
  • Table 10 gives the expert peer-review scoring rubric for the mathematics domain as a template.Source: Table 10 · p. 31
  • Table 11 analyzes reasons for rejection during meta question review.Source: Table 11 · p. 31

Instructions:

  • The prediction prompt asks for step-by-step solutions but requires final answers only in a JSON list, one item per sub-question.A single-question response should also be a one-item list.Source: E. Prompts for Evaluation · p. 32, Instructions · p. 32
  • The judgment prompt instructs an expert grader to compare the candidate’s final answer with the standard answer without generating or modifying answers.Source: Instructions · p. 32

Evaluation Guidelines 1. Reference Standard

The evaluation instructions treat the standard answer and question as authoritative, then compare only the candidate’s final answer using question-appropriate equivalence rules. They first reject invalid responses, apply an all-parts standard to multipart answers, and produce a structured judgment.

Evaluation Guidelines 1. Reference Standard

  • The standard answer is always accepted as correct, and evaluators must not critique or reinterpret the question.They also must not regenerate, fix, or complete the candidate’s answer.Source: Evaluation Guidelines 1. Reference… · p. 32

2. Comparison Strategy

  • Comparison rules depend on the question and answer format; assess only the final answer and accept equivalent formatting or mathematical expressions.The guidance also gives a relative-error tolerance of ±0.1 for decimal or fraction comparisons.Source: §2 · pp. 32–33

3. Multi-part Answers

  • For multipart answers, compare every component individually; any incorrect part makes the entire answer incorrect.Source: §3 · p. 33

4. Validity Check

  • Incomplete, repetitive, or refusal answers are rejected as invalid and assigned label C.Source: §4 · p. 33

Grading Scale

  • The grading scale labels exact or semantically equivalent matches A, deviations or partial matches B, and invalid answers C.Source: Grading Scale · p. 33

Evaluation Procedure & Output Format

  • The procedure checks validity first, compares content if valid, then returns a structured judgment for each sub-question.Content comparison accounts for question-specific strictness, allowed tolerances, equivalence, and formatting flexibility.Source: Evaluation Procedure & Output Format · p. 34

Task Input

The evaluation prompt asks for step-by-step analysis of a candidate answer followed by a final structured judgment. The test-set evaluation reports accuracy and stability across multiple predictions.

Task Input

  • The evaluation prompt asks the evaluator to analyze the candidate’s answer step by step before giving a final judgment.Source: Task Input · p. 35
  • The test-set results are judged by GPT-OSS-120B using four predictions per model, with average accuracy and mG-Pass@2 and mG-Pass@4 reported.Source: Table 13 · p. 36
  • Higher mG-Pass scores indicate greater stability across multiple predictions.Source: Table 13 · p. 36

F. Performance on the Test Set of ATLAS

On ATLAS’s test set, GPT-5-High leads overall accuracy and stability scores, while subject-level results show model strengths that vary by domain. The section also describes how contributors supported the project and dataset.

F. Performance on the Test Set of ATLAS

G. Contributors

  • Contributors supported data collection strategy, quality screening, evaluation design, result analysis, and manuscript preparation; domain experts supplied realistic, challenging questions.Source: G. Contributors · p. 38
Loading 2511.14366v2…