PaperlayerView exact revision on arXiv

Source-linked AI summary

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao, Yixin Chen, Hengjian Gao, Yiheng Wang, Qi Li, Kun Li, Sheng Xu, Shengdu Chai, Fangchen Yu, Xiangyu Zhao, Zhangrui Zhao, Weijie Ma, Zijie Guo, Koutian Wu, Haoyu Zhou, Haoxiang Yin, Lixue Cheng, Chaofan Hu, Haoxuan Li, Lu Mi, Xuxuan Xie, Yifan Zhou, Ruizhe Chen, Zhiwang Zhou, Xingjian Guo, Yuhao Zhou, Xuming He, Shengyuan Xu, Xinyu Gu, Jiamin Wu, Mianxin Liu, Chunfeng Song, Fenghua Ling, Dongzhan Zhou, Shixiang Tang, Yuqiang Li, Mao Su, Peng Ye, Siqi Sun, Bin Wang, Xue Yang, Zhenfei Yin, Tianfan Fu, Guangtao Zhai, Wanli Ouyang, Bo Zhang, Lei Bai, Wenlong Zhang

arXiv:2606.07591v5cs.LGcs.AIcs.CL

TL;DR

Autonomous scientific research claims lack a principled end-to-end evaluation. ResearchClawBench evaluates this capability across real-paper-derived tasks, finding that current agents remain far from reliable scientific re-discovery.

  • Problem

    Existing benchmarks assess scientific subskills but do not determine whether AI systems can complete an independent, discovery-level scientific process.

  • Method

    ResearchClawBench uses 40 real-paper-derived tasks across 10 domains, hidden target papers, raw data, related literature, and expert-weighted rubrics to evaluate scientific outputs.

  • Results

    Current agents and harnessed LLMs remain far from reliable scientific re-discovery; the strongest autonomous agent, Claude Code, averages 21.5.

  • Takeaways & Limitations

    ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward end-to-end autonomous scientific research.

  • Takeaways & Limitations

    The benchmark primarily evaluates dry-lab research and cannot assess wet-lab work requiring experimental platforms, sample preparation, or instruments.

Abstract

from arXiv · show

AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.

1. Introduction

ResearchClawBench addresses the lack of principled, end-to-end evaluation for autonomous scientific research by grounding 40 real discovery tasks in published papers and expert-annotated rubrics across 10 domains. Unified evaluations show that both autonomous agents and native LLMs remain far below stable target-paper-level re-discovery.

  • Benchmark gap: Existing benchmarks cover scientific question answering, interactive environments, and paper reproduction, but not complete research from raw data with verifiable evaluation anchors.This gap makes objective assessment of open-ended scientific outputs difficult.
  • Benchmark design: ResearchClawBench converts expert-selected published papers with clear questions, accessible raw data, and practical value into executable tasks evaluated against hidden target papers.Expert-curated rubrics decompose expected outputs into verifiable criteria.
  • Autonomous-agent results: 21.5 is Claude Code’s average across seven autonomous research agents, while the best-agent-per-task frontier mean reaches only 24.6 against a 50-point target-paper match anchor.Scores above 50 indicate discoveries, whereas 50 means the output matches the target paper.
  • Native-LLM results: 20.7 is Claude-Opus-4.7’s average among seventeen native LLM baselines, while the LLM frontier mean is 26.5, showing difficulty with stable end-to-end re-discovery.ResearchHarness provides a unified lightweight tool-use evaluation setup for models without a full agent scaffold.
  • Benchmark scope: 40 real scientific discovery tasks span 10 domains and diverse scenarios, with expert-annotated rubrics enabling fine-grained evaluation.The benchmark is designed to assess end-to-end scientific discovery using real tasks and verifiable outputs.

2. Related Work

Prior scientific benchmarks largely measure local question answering, reasoning, coding, or data tasks, while newer agent evaluations examine dynamic research workflows. However, existing approaches still leave gaps in realistic, end-to-end scientific discovery across literature review, raw-data processing, experimentation, analysis, and reporting.

  • Static scientific benchmarks: Existing evaluations mainly use question-answering, exam-style, or expert-level problems to measure scientific knowledge, factual understanding, and static reasoning.Examples include SciQ, GPQA, MMLU, Humanity’s Last Exam, and SciBench.
  • Static scientific benchmarks: These benchmarks center on local tasks and usually do not require literature review, raw-data processing, experiment design and execution, figure generation, and research reporting around one question.The limitation applies even when tasks are grounded in scientific contexts.
  • Dynamic research-like evaluations: Dynamic evaluations extend coverage to scientific coding, data preparation, paper reproduction, autonomous discovery, and open-ended research workflows.Examples include SciCode, SciDataCopilot, MLAgentBench, SGI-Bench, AIRS-Bench, and MLR-Bench.
  • Remaining gaps: Important gaps remain because some systems use simulated worlds, others target local capabilities, and several concentrate on machine-learning settings with limited scientific domains and evidence types.The cited limitations concern ScienceWorld, DiscoveryWorld, SciCode, ScienceAgentBench, SciDataCopilot, MLE-bench, MLGym, and MLAgentBench.

3. ResearchClawBench

ResearchClawBench packages real scientific research tasks with related literature and raw data while hiding target papers, then evaluates agents through expert-built multimodal rubrics. Its Reference-Anchored Discovery Score balances verifiable re-discovery against scientifically meaningful advancement beyond the reference study.

  • Evaluation setup: Research outputs include experimental code and execution, intermediate results and figures, and a final report evaluated for reaching or surpassing the hidden target paper.The benchmark represents these outputs as experimental process, intermediate artifacts, and final research reporting.
  • Benchmark design: 40 tasks across 10 scientific domains are derived from high-quality published papers, with target papers hidden during evaluation for independent re-discovery.Systems receive task descriptions, related literature, and raw data rather than the target paper.
  • Task construction: Experts select scientifically valuable papers, extract executable research questions, organize literature and raw data, and validate standardized task packages.Selection emphasizes clear questions, accessible data, and academic, economic, ecological, medical, or other research value.
  • Reference-Anchored Discovery Score: RADS treats each target paper as a human reference study rather than a fixed answer key, comparing evidence, quantitative results, mechanistic analysis, and experimental reasoning.This design addresses the tension between closed-space reproduction and unconstrained open-ended evaluation.
  • Reference-Anchored Discovery Score: 50 denotes reference-level scientific evidence, while scores above 50 indicate reference-surpassing evidence and potential for new discovery.Scores below 50 reflect insufficient discovery potential from problems such as incorrect analysis, shallow experiments, missing evidence, or incomplete reporting.

4. Experiments

Experiments across 40 tasks and 10 scientific domains show that current agents and LLMs remain far from reliable end-to-end re-discovery. Failures chiefly involve protocol mismatch, evidence mismatch, and missing scientific core, while polished presentation and greater resource use do not reliably resolve substantive gaps.

  • Overall performance: 21.5 was Claude Code’s average, while the autonomous-agent frontier mean was 25.8; Claude-Opus-4.7 reached 20.7, with an LLM frontier mean of 26.5.These results indicate that both autonomous agents and ResearchHarness LLMs remain far from reliable end-to-end re-discovery.
  • Overall performance: Claude Code won 12 of 40 tasks, while task difficulty remained highly consistent across autonomous agents, with median pairwise correlation 0.77 and range 0.64–0.86.ResearchHarness LLMs showed a similar pattern, with different models leading in different domains.
  • Supplemental quality dimensions: Professionalism often exceeded 70, whereas substantive dimensions were lower and weakly correlated with rubric scores, indicating that presentation quality outpaced research-content quality.Different systems led in Professionalism, Depth, and Instruction Following.
  • Resource-score relationships: Qwen3.7-Max was the efficient knee in cost and OpenClaw in runtime, but rubric score had only a weak positive relationship with resource investment.Claude Code elevated this relationship by combining a high score with high cost and long runtime.
  • Error analysis: Failures across 280 runs concentrated in Experiment Design Mismatch, Evidence Mismatch, and Scientific Core Missing rather than execution or reporting failures.Agents commonly departed from target papers through incorrect data processing, baselines, validation settings, experimental protocols, key evidence, or mechanistic interpretation.
  • Case study: Physics 002: OpenClaw scored 27.45 on Physics 002, recovering counts-weighted linear XEB and depth-dependent fidelity decline but missing several rubric-critical components.It received 47/50 for counts-weighted linear XEB and 40/50 for N=40 depth scaling, while lacking log-XEB, MB regression, mirror-circuit inference, and gate-counting analysis.

5. Conclusion

ResearchClawBench evaluates end-to-end autonomous research across 10 scientific domains and 40 real-paper-derived tasks using expert-built rubrics. Results indicate that current agents and harnessed LLMs remain far from reliable scientific re-discovery, motivating broader task coverage and longer-horizon studies.

  • Benchmark scope: ResearchClawBench covers 10 scientific domains and 40 real-paper-derived tasks for evaluating end-to-end autonomous research.The benchmark provides a task description, related literature, raw data, and an executable environment.
  • Evaluation protocol: Systems must design experiments, execute analyses, and produce research reports judged by expert-built rubrics.
  • Results: Current agents and harnessed LLMs remain far from reliable scientific re-discovery.Many systems produce complete reports but deviate from the target, as indicated by the supplied conclusion passage.
  • Future work: Future work will expand task coverage and study longer-horizon research processes under real evidence constraints.

6. Limitations … Main Affiliations

ResearchClawBench currently evaluates primarily dry-lab research and final reports, while lacking coverage of wet-lab experimentation, fine-grained research processes, and reliable assessment of genuinely new scientific conclusions. The paper also lists its authors, scientific directors, and institutional affiliations.

  • 6. Limitations: The benchmark primarily evaluates dry-lab research using existing data, code, and literature, not wet-lab work requiring experimental platforms, sample preparation, or instruments.This limits assessment of research involving physical experimentation and instrument operation.
  • 6. Limitations: Current scoring mainly targets the final report rather than fine-grained research steps.The limitation concerns process-level evaluation granularity.
  • 6. Limitations: Evaluating truly new scientific conclusions requires more reliable methods than rubric-based evaluation.The supplied passage identifies this as a limitation of assessing novel discovery.
  • Core Authors: The core authors listed are Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao, Yixin Chen, Hengjian Gao, Yiheng Wang, Qi Li, and Kun Li.
  • Scientific Directors: The scientific directors listed are Xinyu Gu, Jiamin Wu, Mianxin Liu, Chunfeng Song, Fenghua Ling, Dongzhan Zhou, Shixiang Tang, and Yuqiang Li.The list continues with Mao Su, Peng Ye, Siqi Sun, Bin Wang, Xue Yang, Zhenfei Yin, Tianfan Fu, Guangtao Zhai, Wanli Ouyang, and Bo Zhang.
  • Main Affiliations: The main affiliations include Shanghai Artificial Intelligence Laboratory, Shanghai Jiao Tong University, and The Chinese University of Hong Kong.
  • Main Affiliations: Additional main affiliations include Hong Kong Polytechnic University, Xi’an Jiaotong-Liverpool University, Hong Kong University of Science and Technology, and Beijing Normal University.

B. Task Information

The benchmark tasks span scientific objectives from molecular and biomolecular modeling to climate, energy, materials, quantum physics, cybersecurity, and multimodal AI. Across tasks, inputs range from structured scientific data and sequences to images and literature, while outputs include predictions, simulations, designs, forecasts, and extracted calculations.

  • Molecular and biomolecular modeling: Molecular and biomolecular tasks predict molecular properties, generate biomolecular complex structures, model interactions, and produce ranked three-dimensional docking ensembles.KA-GNNs use atom- and bond-level graph features with Fourier-based modules, while other tasks use diffusion-based prediction or PDB coordinates with experimental restraints.
  • Climate and Earth systems: Climate and Earth-system tasks assess glacial mass change, cloud-seeding records, mangrove risks from cyclones and sea-level rise, and 15-day global weather forecasting.The weather task uses upper-air and surface variables from two consecutive 6-hour steps and outputs forecasts at 6-hour resolution.
  • Energy and infrastructure: Energy and infrastructure tasks model future GB power-system pathways, estimate African green-hydrogen delivery costs, and provide a multi-energy benchmark dataset for forecasting and optimization.The energy dataset covers generation, greenhouse-gas emissions, and seven weather attributes from 2014 to 2022.
  • AI and scientific computing: AI and scientific-computing tasks unify multimodal understanding and generation, improve fine-grained visual perception through task-guided cropping, and automate Hartree-Fock derivations and paper-information extraction.The Hartree-Fock task evaluates structured calculations from 15 quantum many-body physics papers and produces derived Hamiltonians, step scores, and extraction results.
  • Security, materials, and design: Security, materials, and molecular-design tasks address unknown and few-shot attacks, discover altermagnetic candidates, and generate recyclable vitrimer chemistries targeting specified glass transition temperatures.The altermagnet search uses graph-based crystal structures, including 148 labeled positive samples, while vitrimer design combines simulations, Gaussian-process calibration, and a graph variational autoencoder.
  • Life-science infrastructure: Additional life-science tasks align large protein-complex structure databases and develop Uncalled4 for fast nanopore signal alignment and improved DNA/RNA modification detection.The structure-search output includes chain correspondence, superimposition vectors, and TM scores, while Uncalled4 targets speed, file-format, compatibility, and sequencing-chemistry limitations.

C. Per-Task Results

Table 8 presents per-task total rubric scores for two system groups: autonomous agents and ResearchHarness LLMs. The supplied passages identify the systems and their abbreviations but do not include task-level score values.

  • Panels: Table 8 divides per-task total rubric scores into panel (a) for autonomous agents and panel (b) for ResearchHarness LLMs.The passages provide panel labels but no numerical task scores.
  • Autonomous agents: The autonomous-agent panel includes Claude Code, Codex CLI, ARIS Codex, OpenClaw, Nanobot, EvoScientist v0.0.4, EvoScientist v0.1.1, and ResearchClaw.The table abbreviations are C.Code, Codex, ARIS, Open, Nano, Evo0, Evo1, and RClaw, respectively.
  • ResearchHarness LLMs: The ResearchHarness LLM panel includes Claude-Opus-4.6/4.7, DeepSeek-V4-Pro, GLM-5.1, GPT-5.4/5.5, Gemini-3.1-Pro/3.5-Flash, Grok-4.1/4.3, and Kimi-K2.5/2.6.The passages also list M2P, M2.5, Q3.5, Q3.6, and Q3.7, without expanding those abbreviations.

D. Detailed Demonstrations … 2. Data and reproducible workflow

The section demonstrates representative system behaviors through high-scoring and task-winning runs, then examines Floquet-Bloch signatures in pumped graphene using provided tr-ARPES data and a reproducible workflow. The analysis targets replica bands, polarization dependence, and limitations imposed by the available time-resolved data.

  • D. Detailed Demonstrations: Figure 8 presents four demonstrations: two high-scoring runs and two task-winning runs whose absolute scores remain low.The latter runs achieve the highest scores within their respective tasks.
  • D. Detailed Demonstrations: 49 total score, 264 seconds duration, and $0.99 cost characterize the ResearchHarness/GPT-5.5 demonstration.The reported system/model is ResearchHarness / GPT-5.5.
  • Energy- and Momentum-Resolved Floquet-Bloch Signatures in Pumped Monolayer Graphene: The graphene task seeks energy- and momentum-resolved Floquet-Bloch states from monolayer epitaxial graphene under a 5 microm mid-infrared pump.The scientific goal is experimental confirmation of Floquet-Bloch states and elucidation of the underlying scattering mechanism.
  • D. Detailed Demonstrations: The evidence rubric combines an energy-momentum map, polarization-dependent intensity data with theoretical fit, and textual interpretation of Volkov final-state scattering.The rubric expects a main Dirac cone, a shifted replica band, periodic intensity variation, and distinction from simpler band replication.
  • 1. Scientific objective and context: Evaluation checks replica bands displaced by one pump photon, visibility in pump-on/pump-off spectra, pi-periodic polarization dependence, and limits on time-domain and Volkov inference.These checks connect the processed features, raw spectra, polarization pathway, and mechanism interpretation.
  • 2.1 Input files: The analysis uses raw_trARPES_data.h5, processed_band_data.json, and polarization_dependence_data.csv without modifying data/.The files contain spectra and axes, extracted Dirac-cone and replica features, and replica intensity versus pump polarization angle.
  • 2.1 Input files: A reproducible script, code/analyze_floquet_trarpes.py, regenerates numeric outputs in outputs/ and PNG figures in report/images/.This provides a saved implementation for reproducing the reported analysis artifacts.
  • 2.2 Data overview: The raw file contains 200 energy points, 150 kx points, seven polarization angles, and 2D pump-off/pump-on spectra, but no delay-indexed 4D intensity dataset.The energy axis spans -0.5 to 0.5 eV and the kx axis spans -0.3 to 0.3 Angstrom-1.

3. Methods

The analysis tests Floquet-Bloch replicas through photon-energy consistency, raw pump-induced signals, and polarization-dependent intensity modeling. The processed and raw data support photon-spaced replicas and structured but weak anisotropy.

  • Replica identification: Replica entries were tested for energy consistency using the stored pump energy of 0.248 eV.The resulting per-feature and order-averaged tables were saved as outputs/band_summary.csv and outputs/band_order_summary.csv.
  • Replica identification: Four processed replica-band entries, two at each order, showed first-order sidebands exactly one processed pump photon energy from the inferred parent.The positive-order mean intensity was slightly larger than the negative-order mean intensity.
  • Raw-spectra verification: Raw pump-on minus pump-off spectra gave positive target-window values at all polarization angles, independently supporting a pump-induced feature near the processed target.The target window was centered at target_energy = 0.248744 eV and target_kx = 0.042282 Angstrom-1.
  • Raw-spectra verification: The target-window enhancement was strongest at 0 deg, 90 deg, and 180 deg, matching the angles with high processed intensity.This agreement links the processed polarization dependence to the raw maps.
  • Polarization analysis: The pi-periodic fit produced anisotropic amplitude 0.001305, modulation contrast 0.00261, bootstrap 95% interval [0.000682, 0.036761], and R2 = 0.047.The weak absolute modulation and seven-angle dataset limit the fit, although both raw and processed data show the same high-low grouping.

5. Validation and traceability

Validation supports the Floquet-Bloch interpretation through photon-spaced replicas and positive pump-induced target-window enhancement across polarization angles. However, the available data lack delay-indexed 4D intensities and ky-resolved measurements, limiting dynamical and two-dimensional momentum analysis.

  • Validation: Positive pump-on minus pump-off enhancement appears in the target window at every measured polarization angle.The corresponding raw signal-by-angle results are reported in outputs/raw_replica_window_signal_by_angle.csv.
  • Traceability: The processed replicas are photon-spaced from their inferred parent energy by 0.248 eV for both first-order sidebands.This result is documented in outputs/band_summary.csv and outputs/band_order_summary.csv.
  • Limitations: The inspected HDF5 file lacks a delay-indexed 4D intensity dataset, preventing extraction of rise/decay constants and time-delay-dependent Floquet formation dynamics.It contains energy, kx, and time_delays axes plus 2D pump-on/off spectra by polarization angle.
  • Limitations: The analysis is effectively one-dimensional in momentum because no ky axis or ky-resolved dataset was present in the inspected HDF5 file.This limits the available momentum-resolved analysis to kx.
  • Validation: 0.248 eV separates both first-order replica sets from a common inferred parent energy near -0.042714 eV, matching the expected energy-domain signature of Floquet-Bloch states.The processed replica table provides the most direct workspace evidence for periodically driven band structure.

7. Conclusion · Local Analysis of Catalog-Scale Numerical Accuracy in Synthetic SXS Binary Black Hole Data · 1. Context and Goal

The merged sections report reproducible local accuracy diagnostics for synthetic SXS binary black hole data, alongside a conclusion showing partial recovery of a separate graphene replica-band signature. The catalog exhibits a median waveform difference of 4x10^-4, increasing errors with mode order while remaining dominated by the ell=2 mode.

  • 7. Conclusion: The analysis confirms energy- and momentum-resolved first-order graphene replica bands separated from the parent feature by the 5 microm pump photon energy.Pump-induced maps show enhanced spectral weight near the processed replica region.
  • 7. Conclusion: The generated energy-momentum maps show the main Dirac cone and additional intensity near the expected replica-energy region, but the replica features are weaker than in the paper.The maps use energy (eV) and momentum (Angstrom-1) axes.
  • 7. Conclusion: The polarization plot shows replica intensity versus thetap with clear periodic cos 2theta behavior and a theoretical fit curve.This matches the required qualitative anisotropy trend.
  • Local Analysis of Catalog-Scale Numerical Accuracy in Synthetic SXS Binary Black Hole Data: 4x10^-4 is the median waveform difference between the two highest numerical resolutions, with differences following an approximately lognormal distribution.The synthetic feature dataset contains 1500 entries representing simulations in the catalog.
  • Local Analysis of Catalog-Scale Numerical Accuracy in Synthetic SXS Binary Black Hole Data: 5 is the approximate factor by which the median difference grows from ell=2 to ell=8, while the 16th-84th percentile interval widens with ell.Higher-ell modes are described as more unstable and more affected by numerical noise.
  • Local Analysis of Catalog-Scale Numerical Accuracy in Synthetic SXS Binary Black Hole Data: 2x10^-5 and 5x10^-5 are the median differences for N=2 versus N=3 and N=2 versus N=4 extrapolation-order comparisons, respectively.Both comparisons are described as more than an order of magnitude below the resolution difference.
  • Local Analysis of Catalog-Scale Numerical Accuracy in Synthetic SXS Binary Black Hole Data: The local benchmark analyzes catalog-wide resolution error, mode-index dependence, extrapolation stability, and a quality index for combined numerical difficulty.Its main findings characterize the catalog as predominantly high-accuracy, according to the supplied abstract passage.
  • 1. Context and Goal: Numerical-relativity catalogs provide strain, curvature signals, remnant properties, and metadata for gravitational-wave inference, waveform calibration, and strong-field tests of gravity.The related literature emphasizes explicit error diagnostics and the scientific value of higher-order or subdominant modes.

2. Data and Methodology … 2. Background

The analysis builds a reproducible synthetic-catalog benchmark with distributional, mode-dependent, extrapolation, and composite-quality diagnostics, while limiting claims to supported comparative evidence. The supplied material also introduces AlphaGeometry’s formal benchmark, symbolic-neural method, research questions, and synthetic training data.

  • 2. Data and Methodology: Three CSV datasets cover 1500 waveform-disagreement simulations, mode-wise differences for l=2–8, and 1200 extrapolation comparisons.The files are fig6_data.csv, fig7_data.csv, and fig8_data.csv.
  • 2. Data and Methodology: The analysis script computes robust summaries, generates resolution, modal-accuracy, and extrapolation figures, and constructs a composite quality index.Generated artifacts are saved under benchmark-native outputs/ and report/ images/ paths.
  • 5. Claim Discipline and Limits: The composite quality index is an internal ranking device, not a physical observable or externally validated catalog statistic, and the study excludes physical-parameter claims.The strongest justified claims are distributional and comparative: most simulations are accurate, higher modes are less accurate, and demanding extrapolation comparisons are more sensitive.
  • 4. Interpretation: 4.25 x 10^-4 is the median waveform difference; 77.7% of simulations fall below 10^-3, while the maximum reaches 4.07 x 10^-2.The distribution is sharply right-skewed, with a narrow difficult-simulation tail.
  • 4. Interpretation: 2.27 x 10^-3 is the median waveform difference at l=8 versus 3.00 x 10^-4 at l=2, a 7.57 ratio indicating systematic modal degradation.A log-linear fit gives a slope of 0.144 dex per unit increase in l, and interquartile ranges broaden at larger l.
  • 4. Interpretation: 72.2% of simulations show larger N=2 vs N=4 disagreement, with median ratio 2.67 and weak correlation r = 0.036 against N=2 vs N=3.The evidence supports nonuniform sensitivity to the higher-order extraction choice.
  • 6. Reproducibility: The workflow is reproducible through code/analyze_catalog_accuracy.py, with summary metrics, mode statistics, quality summaries, and figures stored in specified output paths.The documented command is python code/analyze_catalog_accuracy.py.

3. Dataset Analysis

IMO-AG-30 covers 30 problems across 21 competition years and seven proof-goal types, with congruence most prevalent. Its complexity remains broadly stable over time, while problems typically require about 8–9 constructions and scores range from 13.0 to 37.9.

  • Dataset scope: 30 problems span 21 competition years from 2000–2022, with 8 missing years reflecting IMO problem-selection cycles.Figure 1 characterizes the dataset by goal type, complexity over time, and construction counts.
  • Goal types: 40% of problems target congruence, followed by 23% collinearity and 17% concyclicity.The benchmark includes seven distinct proof-goal types: congruence 12/30, collinearity 7, concyclicity 5, perpendicularity and equal angles 2 each, and equal ratios and parallelism once each.
  • Complexity: Problem complexity scores show no clear upward trend over time, suggesting IMO problem difficulty has remained broadly constant.Construction counts are roughly bell-shaped, centered around 8–9 constructions per problem.
  • Complexity: 13.0–37.9 is the range of the composite complexity score, which combines point count, constraint clauses, and weighted high-complexity constructions.The score correlates with proof difficulty based on whether a problem requires LM-assisted auxiliary constructions; weights include circles +3.0, incenter +2.5, orthocenter +2.0, reflections +2.0, and angle bisectors +1.5.
  • Construction primitives: on_line is the most common construction primitive, while collinearity emphasizes line intersections and orthocenter constructions and congruence relies on circles and midpoints.Other frequent primitives include on_circle, midpoint, foot, and reflect.

4. Methods … 3. Methodology

The paper combines symbolic and neural methods for geometry proving with synthetic-data training, then evaluates AlphaGeometry against classical provers and human performance. It also introduces an ECAT battery-modeling methodology using Latin hypercube sampling, a neural surrogate, and multi-objective genetic optimization.

  • 4. Methods: AlphaGeometry alternates a constructive language model with a formally verified DD+AR engine that performs all logical deduction.The language model proposes auxiliary points but does not infer; DD+AR provides machine-verifiable proofs and soundness.
  • 4. Methods: Synthetic generation produces approximately 100M proof traces and 90M derived theorems, avoiding reliance on fewer than 10,000 formalized human proofs.Random configurations are exhaustively processed by DD+AR, and retrograde analysis extracts theorem-proof pairs for training.
  • 4. Methods: 25/30 (83%) AlphaGeometry solve rate matches the human gold-medalist threshold, improving on DD+AR alone at 14/30 (47%).Classical provers solve 0 of 30 problems, while AlphaGeometry leaves 5 unsolved; congruence goals reach 83%.
  • 4. Methods: 77% accuracy for DD+AR and 67% for AlphaGeometry show that complexity predicts solvability imperfectly, with problems requiring 4+ auxiliary constructions generally unsolved.The benchmark groups problems by complexity and auxiliary-construction requirements.
  • 4. Methods: AlphaGeometry proofs average 41.7 steps, with LM-assisted proofs averaging approximately 44 steps versus approximately 30 for DD+AR-only proofs.Proof length correlates positively with complexity at r approx. 0.72, and most LM-assisted proofs require 1–2 auxiliary constructions.
  • 7. Related Work: The paper positions AlphaGeometry against classical automated geometry provers and neural theorem proving, emphasizing its hybrid construction-and-verification design.Classical systems use coordinate or rule-based methods, while prior neural theorem proving applied transformers to Metamath formal proofs.
  • 8. Conclusion: The conclusion reports that synthetic data and LM-guided auxiliary constructions raise AlphaGeometry to 25/30 (83%), matching human gold-medalist performance.The broader roadmap combines formal language design, synthetic data generation, and hybrid neural-symbolic architectures.
  • 3. Methodology: The ECAT methodology uses an 11-parameter single-particle model, 500 Latin-hypercube samples, an ANN voltage-curve surrogate, and MMGA optimization with approximately 100x speedup.The ANN maps parameters to 200-point discharge curves, while MMGA uses 100 individuals for 200 generations and a fitness weighted 70% RMSE and 30% MAE.

4. Experimental Data · 7. Conclusion

The study evaluates its battery-parameter-identification framework across complementary NASA, CALCE, and Oxford datasets, then concludes that MMGA combines voltage-prediction accuracy, computational efficiency, physical plausibility, and cross-dataset transferability. It also identifies model, surrogate, identifiability, and chemistry limitations and proposes broader models and data for future work.

  • 4. Experimental Data: NASA, CALCE, and Oxford datasets represent controlled room-temperature cycling, NCM aging stages, and highly transient urban-drive conditions, respectively.NASA uses four 18650 cells; CALCE provides an 83-point reference segment; Oxford uses 3,145 points from a dynamic drive cycle.
  • 5.2 ANN Surrogate Model Performance: 0.284 V validation RMSE, with 0.096 V median error, indicates accurate ANN surrogate predictions for most hold-out samples.The higher mean than median reflects a minority of edge-case samples contributing larger errors.
  • 5.3 MMGA Optimization Convergence: 0.165 NASA and 0.197 CS2 final fitness values follow rapid convergence within the first 50 of 200 generations.The narrowing gap between best and average fitness indicates population convergence toward the optimum.
  • 5.4 Voltage Prediction Accuracy: 0.176 V NASA and 0.212 V CS2 RMSEs show that MMGA captures discharge-curve shape and slope, with lower error on NASA.The corresponding MAEs are 0.138 V for NASA and 0.162 V for CS2.
  • 5.5 Identified Parameters and 5.7 Sensitivity Analysis: Identified parameters differ across datasets: NASA favors smaller particles and moderate reaction rates, whereas CS2 favors larger particles and higher positive-electrode reaction rates.Both optimizations reach the lower heat-transfer-coefficient bound of 5 W/m2K; sensitivity analysis finds maximum concentrations strongly correlated with voltage, reaction rates moderately correlated, particle radii slope-influential, and heat transfer minimally correlated.
  • 6.1 Computational Efficiency and 6.4 Practical Implications: 100x speedup reduces 20,000 MMGA fitness evaluations from approximately 200 seconds with direct simulation to 2 seconds with the ANN surrogate.Each SPM simulation takes approximately 0.01 seconds, while an ANN forward pass takes approximately 0.0001 seconds.
  • 6.3 Comparison with Literature and 7. Conclusion: 0.176 V NASA and 0.212 V CS2 voltage RMSEs, approximately 100x efficiency, physically plausible parameters, and cross-dataset transferability summarize the framework’s reported achievements.Future work targets full P2D models, multimodal experimental data, and adaptive sampling to improve surrogate accuracy in regions of interest.
Loading 2606.07591v5…