PaperlayerView exact revision on arXiv

Source-linked AI summary

ResearchClawBench: A Benchmark for End-to-End Autonomous Scientific Research

Wanghan Xu, Shuo Li, Tianlin Ye, Qinglong Cao, Yixin Chen, Hengjian Gao, Yiheng Wang, Qi Li, Kun Li, Sheng Xu, Shengdu Chai, Fangchen Yu, Xiangyu Zhao, Zhangrui Zhao, Weijie Ma, Zijie Guo, Koutian Wu, Haoyu Zhou, Haoxiang Yin, Lixue Cheng, Chaofan Hu, Haoxuan Li, Lu Mi, Xuxuan Xie, Yifan Zhou, Ruizhe Chen, Zhiwang Zhou, Xingjian Guo, Yuhao Zhou, Xuming He, Shengyuan Xu, Xinyu Gu, Jiamin Wu, Mianxin Liu, Chunfeng Song, Fenghua Ling, Dongzhan Zhou, Shixiang Tang, Yuqiang Li, Mao Su, Peng Ye, Siqi Sun, Bin Wang, Xue Yang, Zhenfei Yin, Tianfan Fu, Guangtao Zhai, Wanli Ouyang, Bo Zhang, Lei Bai, Wenlong Zhang

arXiv:2606.07591v5cs.LGcs.AIcs.CL

TL;DR

It remains difficult to verify whether AI systems can complete end-to-end scientific research, while existing benchmarks largely assess local tasks and research subskills. ResearchClawBench evaluates systems on hidden-paper tasks grounded in real papers, and current agents and harnessed LLMs remain far from reliable re-discovery, with reports often diverging in protocols, mechanisms, or evidence.

  • Problem

    It remains difficult to verify whether AI can complete end-to-end scientific research, while existing benchmarks largely assess local scientific tasks and research subskills.

    Source: §1 · p. 3, §2.1 · p. 4

  • Method

    ResearchClawBench evaluates 40 real-paper-derived tasks across 10 domains, hiding target papers and using expert-weighted rubrics to assess systems given task descriptions, related literature, and raw data.

    Source: §1 · p. 3, §3.2 · p. 7

  • Results

    Current agents and harnessed LLMs remain far from reliable re-discovery, with reports often diverging in protocols, mechanisms, or evidence; Claude Code averages 21.5.

    Source: §1 · p. 3, §5 · p. 13

  • Takeaways & Limitations

    ResearchClawBench offers an evaluation benchmark for measuring whether systems can reach target-paper-level re-discovery.

    Source: §1 · p. 3, §5 · p. 13

    Current tasks primarily assess dry-lab research using existing data, code, and literature, and cannot assess wet-lab research requiring real platforms, sample preparation, or instrument operation.

    Source: §6 · p. 14

Abstract

from arXiv · show

AI coding agents are increasingly used for scientific work, but their end-to-end autonomous research capability remains difficult to verify. We present ResearchClawBench, a benchmark for evaluating autonomous scientific research across 40 tasks from 10 scientific domains. Each task is grounded in a real published paper, provides related literature and raw data, and hides the target paper during evaluation. Expert-curated multimodal rubrics decompose the target scientific artifacts into weighted criteria, enabling evaluation of target-paper-level re-discovery while leaving room for new discovery. We evaluate seven autonomous research (auto-research) agents under a unified protocol and seventeen native LLMs through the lightweight ResearchHarness. Current systems remain far from reliable re-discovery: the strongest autonomous agent, Claude Code, averages 21.5, and the strongest ResearchHarness LLM, Claude-Opus-4.7, averages 20.7, with an LLM frontier mean of only 26.5. Error analysis shows that failures concentrate in experimental protocol mismatch, evidence mismatch, and missing scientific core. ResearchClawBench provides a reproducible evaluation frontier for measuring progress toward autonomous scientific research.

1. Introduction

ResearchClawBench evaluates end-to-end autonomous scientific research by grounding tasks in real papers and raw data, then scoring open-ended outputs against expert-curated, verifiable criteria.

1. Introduction

  • Existing benchmarks address adjacent settings but do not combine raw experimental data, complete research outputs, and verifiable evaluation anchors.The paper identifies scientific significance, open-ended assessment, and heterogeneous data, methods, and evidence standards as benchmark-design challenges [Welbl et al., 2017, Rein et al., 2023, Wang et al., 2022, Jansen et al., 2024, Lu et al., 2024, Starace et al., 2025].Source: §1 · p. 3
  • RCBench hides each target paper during evaluation and uses expert-curated rubrics that decompose expected outputs into weighted, verifiable sub-criteria.This design evaluates open-ended scientific outputs against a target without relying on exact-match tests or simple unit tests.Source: §1 · p. 3
  • Claude Code averages 21.5, while the best-per-task autonomous-agent frontier mean is 24.6, against a 50-point target-paper match anchor.Scores above 50 indicate discoveries.Source: §1 · p. 3
  • Claude-Opus-4.7 averages 20.7, and the native-LLM frontier mean is 26.5, showing that LLMs also struggle with stable end-to-end re-discovery.ResearchHarness enables comparison with models that lack a full agent scaffold [Liu et al., 2025b].Source: §1 · p. 3
  • RCBench covers 40 real scientific discovery tasks across 10 domains, including diagnostic analysis and metric optimization.Domain experts select papers with clear questions, accessible raw data, and practical research value, then convert them into executable task descriptions.Source: §1 · p. 3

2. Related Work

Prior scientific benchmarks measure knowledge and research subskills but generally do not test completion of an independent, discovery-level research process. Research-agent evaluations move toward realistic workflows, yet gaps remain; ResearchClawBench targets hidden-target rediscovery from real scientific papers.

2.1. Scientific Capability and Scientific Task Benchmarks

  • Static scientific benchmarks assess knowledge and domain reasoning, but do not establish whether systems can complete an independent research process that reaches discovery-level outcomes.They range from exam-style and expert-level reasoning to domain-specific assessment, while typically omitting literature review, raw-data processing, experimental design and execution, figure generation, and research reporting around one open question.Source: §2.1 · p. 4

2.2. Research-Agent Benchmarks and Autonomous Research Systems

  • Research-agent evaluations span scientific coding and data preparation, ML experimentation, paper reproduction, interactive discovery, and open-ended research; autonomous research systems also demonstrate potential for paper generation and discovery.Examples include SciCode [Tian et al., 2024], PaperBench [Starace et al., 2025], ScienceWorld, AIRS-Bench [Lupidi et al., 2026], and The AI Scientist [Lu et al., 2024].Source: §2.2 · pp. 4–5
  • ResearchClawBench addresses gaps in prior agent benchmarks by testing end-to-end rediscovery from real papers with hidden targets, while leaving room for future discovery-oriented studies.Prior work uses simulated worlds, focuses on local capabilities or machine-learning settings, or evaluates reproduction around exposed papers and code; broader natural-science domains, data modalities, and evidence standards remain under-covered.Source: §2.2 · p. 5

3. ResearchClawBench

ResearchClawBench packages real-paper tasks with related literature and raw data, then uses hidden target-paper rubrics to make open-ended research outputs verifiable.

3.1. Task Components

  • Each task combines a description, related literature, raw data, a hidden target paper, and evaluation artifacts; systems produce experimental code and execution, intermediate outputs, and a final report.The benchmark assesses whether those products are high-quality and reach or surpass the target paper.Source: §3.1 · p. 6

3.2. Data Construction

  • Experts curate high-value papers into executable tasks with literature, raw data, artifact-based rubrics, and cross-checks; systems independently re-discover hidden targets across 40 tasks in 10 domains.Research value can be scientific, economic, ecological, medical, or otherwise significant.Source: §3.2 · p. 7

3.3. Evaluation Harness for LLM baselines: ResearchHarness

  • ResearchHarness gives native LLMs a lightweight ReAct-style tool loop for web search, local file inspection, and local execution, with automatic context compaction at 128k tokens.Its compact scaffold is intended to keep evaluation closer to model capability and easier to extend.Source: §3.3 · p. 8

3.4. Evaluation Metric: From Re-Discovery to Discovery

  • RADS anchors open-ended discovery to target papers, comparing evidence, quantitative results, mechanisms, and reasoning with a human reference rather than treating it as fixed ground truth.A score of 50 represents reference-level evidence; scores above 50 suggest new-discovery potential but do not establish a validated discovery.Source: §3.4 · pp. 8–9, Table 5 · p. 10

4. Experiments

Across 40 tasks, the strongest autonomous agent and ResearchHarness LLM average 21.5 and 20.7, respectively; failures chiefly involve protocol mismatch, evidence mismatch, and missing scientific core. The Physics 002 case illustrates how recovering a direct trend can still leave key validation and modeling steps missing.

4.1. Experimental Setup

  • The evaluation compares seven autonomous agents and 17 native LLM baselines on 40 tasks, scoring final reports against rubrics with GPT-5.1.The LLM baselines are run through ResearchHarness.Source: §4.1 · p. 10

4.2. Main Results

  • Claude Code averages 21.5 (autonomous-agent frontier mean: 25.8), while Claude-Opus-4.7 averages 20.7 (LLM frontier mean: 26.5).The paper characterizes current systems as far from reliable end-to-end re-discovery.Source: §4.2 · p. 11

4.3. Four Supplemental Dimensions

  • Physics 002 rubric results range from partial recovery of XEB trends to absent gate-counting analysis, with several required estimator comparisons and validation steps missing.At fixed d=12, the increasing-N trend lacks log-XEB and unified comparison; at N=40, linear XEB decays from about 0.63 to 0.27–0.30 without multi-estimator comparison. The N=56 analysis lacks MB regression, mirror-circuit inference, and the inferred depth-12 point; gate-counting is absent, while counts-weighted linear XEB covers all required depths with 50 instances per depth.Source: §4.3 · p. 12
  • Reports often exceed 70 on Professionalism while other supplemental dimensions are lower, and these dimensions correlate weakly with rubric scores.The results distinguish polished presentation from substantive research content: recovering rubric-critical scientific evidence remains the central challenge.Source: §4.3 · p. 12, Table 6 · p. 12

4.4. Runtime and Cost Analysis

  • Qwen3.7-Max is the efficient knee for cost, while OpenClaw is the efficient knee for runtime; score has only a weak positive relationship with resource investment.The relationship is largely elevated by Claude Code, which combines a high score with high cost and long runtime; additional computation does not necessarily yield stable improvement.Source: Figure 5 · p. 11, §4.4 · pp. 12–13

4.5. Error Analysis

  • Across 280 autonomous-agent runs, failures concentrate in Experiment Design Mismatch, Evidence Mismatch, and Scientific Core Missing.Agents tend to depart from the target paper in protocol, key evidence, or mechanistic interpretation rather than simply failing to execute or report.Source: Figure 6 · p. 11, §4.5 · p. 13

4.6. Case Study

  • On Physics 002, OpenClaw scores 27.45, recovering depth-dependent linear XEB but missing key validation and gate-counting evidence.It receives 47/50 on counts-weighted linear XEB and 40/50 on N=40 depth scaling, but lacks log-XEB and multi-metric consistency, N=56 MB regression or mirror-circuit inference, and the gate-counting fidelity model.Source: §4.3 · p. 12, §4.6 · p. 13

5. Conclusion

ResearchClawBench evaluates end-to-end autonomous research across 10 scientific domains and 40 real-paper-derived tasks. Current agents and harnessed LLMs remain far from reliable scientific re-discovery.

5. Conclusion

  • ResearchClawBench covers 40 real-paper-derived tasks across 10 scientific domains.The benchmark evaluates end-to-end autonomous research.Source: §5 · p. 13
  • Many agents and harnessed LLMs produce complete reports but deviate from target-paper protocols, mechanism explanations, or evidence chains.The results indicate that complete reports do not ensure reliable scientific re-discovery.Source: §5 · p. 13
  • For each task, systems receive a description, related literature, raw data, and an executable environment, then design experiments, execute analyses, and produce reports judged by expert-built rubrics.Source: §5 · p. 13
  • Future work will expand task coverage and study longer-horizon research processes under real evidence constraints.Source: §5 · pp. 13–14

6. Limitations · A. Authors · Corresponding Authors

ResearchClawBench is limited to dry-lab research, primarily scores final reports rather than fine-grained research steps, and relies on target-paper-based rubrics that are insufficient for reliably evaluating genuinely new scientific conclusions.

6. Limitations

  • The benchmark cannot assess wet-lab research requiring experimental platforms, sample preparation, or instrument operation.Its current tasks primarily evaluate dry-lab research based on existing data, code, and literature.Source: §6 · p. 14
  • Current scoring focuses mainly on final reports rather than fine-grained research steps.Source: §6 · p. 14
  • Evaluating genuinely new scientific conclusions requires more reliable methods than rubrics built around existing target papers.Source: §6 · p. 14

B. Task Information

This section defines benchmark task inputs, outputs, data, modalities, and scientific objectives across domains including astronomy, chemistry, Earth science, energy, mathematics, neuroscience, and physics. It covers tasks involving datasets, model development, optimization, scientific discovery, proof generation, behavior classification, neural modeling, and experiments on materials and quantum systems.

B. Task Information

  • Table 7 maps each of the 40 tasks to its data name, modality, and description.The task information also describes the scientific goals and intended analyses for many tasks.Source: Table 7 · p. 20
  • Astronomy tasks cover a Local Distance Network targeting ~1% precision for H0 and synthetic SXS waveform-error data for assessing catalog accuracy.The H0 task combines multiple distance indicators; the waveform data include overall, modal, and extrapolation-order differences.Source: B. Task Information · pp. 21–22
  • Chemistry tasks include KA-GNN molecular-property prediction, using Fourier-based Kolmogorov-Arnold modules to replace conventional MLP transformations in graph networks.The stated aim is stronger predictive accuracy, computational efficiency, and interpretability.Source: B. Task Information · p. 23
  • Other chemistry tasks span diffusion-based biomolecular-complex structure prediction, HADDOCK3 integrative modeling, and LES recovery of atomic charges from energy and force data.The LES benchmark uses randomly charged configurations with Coulomb and repulsive Lennard-Jones interactions.Source: B. Task Information · pp. 24–25
  • The work seeks a consistent, high-confidence global glacial mass-change assessment by reconciling diverse observational methods, and analyzes 25 years of U.S. cloud-seeding records for temporal, geographic, operational, and methodological patterns.Source: B. Task Information · p. 25
  • Further Earth-science tasks assess mangrove risk from tropical-cyclone regime shifts and sea-level rise, and use a three-model cascade for 15-day global weather forecasts.The forecasting task uses ERA5 inputs and seeks skill comparable to the ECMWF ensemble mean.Source: B. Task Information · p. 26
  • Energy tasks include identifying lithium-ion battery model parameters and providing an open-source, high-resolution Great Britain power-system model for future energy-pathway analysis.Battery validation uses experimental aging data, while the power-system task models hourly dispatch and costs under scenarios.Source: B. Task Information · p. 27
  • Other energy tasks estimate African green-hydrogen delivery costs under financing and policy scenarios and provide hierarchical hourly building and community energy data for research.The HEEW dataset supports analyses including data cleaning, correlation, and hierarchical-aggregation consistency checks.Source: B. Task Information · p. 28

Task information

  • The tasks include a hierarchical energy-system dataset spanning 2014–2022, a unified autoregressive model for visual understanding and generation, and training-free cropping to improve fine-grained visual reasoning.The energy dataset covers thermal loads, PV generation, emissions, and weather attributes; the cropping framework addresses information loss from fixed-resolution vision encoders.Source: B. Task Information · pp. 28–29
  • One task tests structured Hartree-Fock calculations from 15 research papers, while another develops DIDS-MFL to improve intrusion detection for unknown and few-shot attacks.DIDS-MFL combines statistical and representational disentanglement, dynamic graph diffusion, and multi-scale representation fusion.Source: B. Task Information · p. 29
  • A hydrogel-design task maps protein-sequence features to adhesive strength to create synthetic hydrogels targeting robust underwater adhesion above 1 MPa.Source: B. Task Information · p. 30
  • A personalized neoantigen-vaccine task uses patient sequencing, HLA and mutation data, prediction scores, and a manufacturing budget to optimize vaccine composition and efficacy measures.Outputs include per-cell immune response probability, tumor-cell coverage, vaccine-composition IoU, and runtime.Source: B. Task Information · p. 31
  • Protein-complex search takes 3D structures and returns chain correspondences, superimposition vectors, and TM scores, while Uncalled4 aims to align nanopore signals rapidly and sensitively across DNA and RNA modifications.Source: B. Task Information · p. 32
  • Materials tasks include discovering altermagnets from crystal graphs, developing a general-purpose atomistic foundation model, and inverse-designing vitrimer polymers for target glass-transition temperatures.The altermagnet task combines unlabeled pretraining with fine-tuning on known examples; the vitrimer framework combines molecular dynamics, Gaussian-process calibration, and a graph variational autoencoder.Source: B. Task Information · p. 33
  • Other tasks cover AI workflows for materials prediction, generation, and optimization; dense-scene multi-object tracking; accelerated optimization methods; and collision-free multi-agent path finding.The tracking task outputs object trajectories, the optimization task seeks a global minimizer, and the path-finding task balances solution quality with computational efficiency.Source: B. Task Information · pp. 33–35

Math_003 imo_ag_30.txt

  • The geometry task evaluates machine-verifiable, human-readable Euclidean proofs on 30 post-2000 IMO problems, targeting autonomous solutions without human demonstrations.The benchmark is the curated Math_003 imo_ag_30.txt problem set.Source: B. Task Information · p. 35

Neuroscience_000 Together_1

  • The neuroscience behavior-classification task trains supervised models on pose-derived features and aligned Attack and Sniffing labels, producing evaluations, diagnostics, confusion matrices, and feature-importance tables.The objective is to verify that the SimBA-style workflow yields reproducible, transparent, and auditable classification evidence.Source: B. Task Information · p. 36

Neuroscience_001 flow

  • The Drosophila optic-flow task optimizes a connectome-constrained deep mechanistic network to predict activity in 45,669 neurons and analyze motion-detection mechanisms.The provided ensemble contains 50 pretrained models and configuration data for simulation without retraining.Source: B. Task Information · p. 36

Neuroscience_003 and single-cell feature selection

  • The connectomics task predicts whether adjacent over-segmented neuron fragments should be merged, while the single-cell task selects dynamically expressed features that preserve continuous cellular trajectories.The connectomics goal is to reduce manual proofreading; the single-cell setup supports lineage, glial-activation, and neurodegeneration-related analyses while reducing confounding variation.Source: B. Task Information · p. 37

Physics_000 Multi-component Icosahedral Reproduction Data

  • The icosahedral-aggregate dataset supports reproducing theoretical calculations, experimental verification, and dynamic growth simulations to aid rational design and predict nanocluster stability and growth.The stated design scope includes multi-component nanoclusters and nanoparticles with chiral or achiral symmetry and specified composition.Source: B. Task Information · p. 37

Physics_001 MATBG Superfluid Stiffness Core Dataset

  • The MATBG task uses simulated data for carrier-density, temperature, and current experiments to assess superfluid stiffness and test quantum-geometry, power-law, and quadratic-current conclusions.Its inputs and outputs concern a gate-tunable magic-angle twisted bilayer graphene device at approximately 20 mK, including resistance and microwave resonance measurements.Source: B. Task Information · p. 37

Physics_002 results

  • The RCS task estimates fidelity with uncertainty from measured bitstring samples and ideal probabilities, then compares experimental fidelity with classical approximability across circuit sizes and depths.The supplied data support XEB calculations and comparative curves over qubit count or circuit depth.Source: B. Task Information · p. 38

Physics_003 raw_trARPES_data.h5

  • The tr-ARPES task uses raw four-dimensional measurements and processed band and polarization data to observe Floquet-Bloch replica bands in pumped monolayer graphene.Its scientific objective is to confirm Floquet-Bloch states and investigate scattering involving photon-dressed Volkov final states.Source: B. Task Information · p. 38

C. Per-Task Results

Table 8 reports per-task total rubric scores for autonomous agents and ResearchHarness LLMs. A dash indicates a model crashed or failed during execution.

C. Per-Task Results

  • Table 8 reports per-task total rubric scores for each task and system.Its panels separate autonomous agents from ResearchHarness LLMs.Source: Table 8 · p. 38
  • The autonomous-agent panel identifies Claude Code, Codex CLI, ARIS Codex, OpenClaw, Nanobot, EvoScientist v0.0.4 and v0.1.1, and ResearchClaw.The table abbreviates these systems as C.Code, Codex, ARIS, Open, Nano, Evo0, Evo1, and RClaw, respectively.Source: Table 8 · p. 38, C. Per-Task Results · p. 39
  • The ResearchHarness panel lists 17 LLMs, including Claude-Opus-4.6/4.7, DeepSeek-V4-Pro, GLM-5.1, GPT-5.4/5.5, Gemini-3.1-Pro/3.5-Flash, Grok-4.1/4.3, Kimi-K2.5/2.6, MiMo-V2-Pro/V2.5, and Qwen3.5-397B-A17B/3.6-Plus/3.7-Max.The caption maps the table abbreviations to these model names.Source: Table 8 · p. 38, C. Per-Task Results · p. 40
  • A dash in Table 8 marks a model that directly crashed or failed during execution.Source: Table 8 · p. 38

D. Detailed Demonstrations … 2. Data and reproducible workflow

The graphene tr-ARPES analysis finds photon-spaced replica features and weak polarization dependence, while the available data limit mechanistic separation.

D. Detailed Demonstrations

  • Figure 8 selects four demonstrations: two high-scoring runs and two task-winning runs whose absolute scores remain low; GPT-5.5 scored 49 in 264 seconds at $0.99.Source: Figure 8 · p. 41, D. Detailed Demonstrations · p. 41
  • The graphene demonstration rubric calls for an energy-momentum replica-band map, polarization-dependent intensity plot, and an explicit Volkov-final-state scattering explanation.It distinguishes that mechanism from replication attributed only to the initial Floquet-Bloch state.Source: D. Detailed Demonstrations · pp. 41–42

Energy- and Momentum-Resolved Floquet-Bloch Signatures in Pumped Monolayer Graphene

  • Both first-order replicas sit 0.248 eV from the parent dispersion, and the pi-periodic polarization fit has a modulation contrast of 0.00261.The seven-angle series and lack of a delay-indexed 4D raw cube prevent stronger separation of initial-state Floquet dressing from Volkov final-state effects.Source: Energy- and Momentum-Resolved… · p. 42

1. Scientific objective and context

  • The analysis tests for photon-spaced replica bands in energy-momentum maps, polarization-dependent intensity, and evidence relevant to Volkov final-state effects.Related-work notes identify replica or sideband spectral weight as the signature and note that final states and matrix elements can shape replica intensity.Source: §1 · p. 42

2.1 Input files

  • The workflow uses the HDF5 spectra, processed band features, and polarization series, with a script that regenerates numeric outputs and PNG figures.Source: §2.1 · p. 42

2.2 Data overview

  • The raw spectra contain 200 energy points, 150 kx points, and seven pump polarization angles, but no delay-indexed 4D intensity dataset for reconstructing time dynamics.The overview records energy from -0.5 to 0.5 eV and kx from -0.3 to 0.3 Angstrom-1.Source: §2.2 · p. 43
  • Figure 1 shows pump-off, pump-on at 0 deg, and pump-induced difference maps, with a marker for the processed replica target region used in raw-window validation.Source: §2.2 · p. 43

3. Methods

The methods test replica-band energy consistency, validate the target against pump-on and pump-off raw maps, and fit polarization dependence with a pi-periodic model. The reported findings support photon-spaced replicas and a pump-induced signal, while the weak polarization fit does not uniquely establish a Volkov mechanism.

3.1 Replica-band energy test

  • For each replica entry with order n = ±1, the energy test computes an inferred parent energy using the processed pump energy of 0.248 eV.The per-feature and order-averaged tables are saved as outputs/band_summary.csv and outputs/band_order_summary.csv.Source: §3.1 · p. 43

3.2 Raw-map validation

  • Raw-map validation subtracts each pump-off map from its pump-on map and averages the difference around the CSV target point.The window is centered at target_energy = 0.248744 eV and target_kx = 0.042282 Angstrom-1, with half-widths 0.03 eV and 0.02 Angstrom-1.Source: §3.2 · p. 43

3.3 Polarization-dependence model

  • The replica intensity is fit with a minimal pi-periodic model to capture leading anisotropic dependence expected for a polarization-sensitive transition matrix element or Volkov-like final-state dressing.The fitted amplitude, phase, contrast, and bootstrap intervals are saved in outputs/polarization_fit.json, with the fitted curve in outputs/polarization_fit_curve.csv.Source: §3.3 · p. 43

4.1 Photon-spaced replica features

  • Both first-order sidebands are exactly one processed pump photon energy from the inferred parent, satisfying the photon-spacing criterion for Floquet-Bloch replicas.The two orders have comparable intensities, with the positive-order mean slightly larger; Figure 2 compares order-averaged separations with the 0.248 eV pump photon energy.Source: §4.1 · p. 44

4.2 Raw pump-induced signal near the replica region

  • Raw pump-on minus pump-off intensity is positive at all polarization angles, supporting a pump-induced feature near the processed target region.The target-window enhancement is strongest at 0 deg, 90 deg, and 180 deg, matching the high processed-intensity angle groups and providing an independent check.Source: §4.2 · p. 44, Figure 3 · p. 45

4.3 Polarization dependence and Volkov final-state interpretation

  • The fitted pi-periodic model gives modulation contrast 0.00261 and R2 = 0.047 for seven angle points.The bootstrap 95% interval for contrast is [0.000682, 0.036761]; the weak modulation and limited angle count constrain interpretation.Source: §4.3 · p. 45

5. Validation and traceability

The workspace supports photon-spaced replicas with polarization-dependent raw-signal enhancement, while the available data and weak polarization modulation constrain mechanistic and dynamical conclusions. Validation outputs record the measurements and claim-level support, and related-work extraction motivates the observable and its interpretation.

5.1 Directly verified from workspace data

  • The workspace records raw HDF5 structure, replica spacing, raw-window signals, polarization-fit intervals, and claim-level support in named output files.These include data_overview.json, band_summary.csv, band_order_summary.csv, raw_replica_window_signal_by_angle.csv, polarization_fit.json, and claim_recovery_table.csv.Source: §5.1 · p. 46

5.2 Related-work context

  • Related-work extraction supports pump-induced, photon-spaced ARPES sidebands as the central Floquet-Bloch observable and polarization-angle dependence as evidence for matrix-element or Volkov final-state contributions.Source: §5.2 · p. 46

5.3 Limitations and assumptions

  • No delay-indexed intensity or ky-resolved dataset was present, preventing temporal-dynamics extraction and limiting analysis to one-dimensional kx.Volkov interpretation is indirect and would require additional observables for decisive separation from initial-state Floquet replicas. The polarization fit has low R2 because modulation is small and only seven angles are available, so fitted contrast warrants caution.Source: §5.3 · p. 46
  • 0.248 eV separates both first-order replica sets from a common inferred parent near -0.042714 eV, while raw target-window enhancement is positive at every measured polarization angle.The replicas overlay the extracted Dirac-cone dispersion; raw enhancements are strongest at 0°, 90°, and 180°. The weak, pi-periodic anisotropy supports polarization-sensitive final-state dressing alongside Floquet-like replicas, but does not prove Volkov scattering.Source: §5.1 · p. 46, §5.3 · p. 47

7. Conclusion · Local Analysis of Catalog-Scale Numerical Accuracy in Synthetic SXS Binary Black Hole Data · 1. Context and Goal

The conclusion passage reports graphene replica-band and polarization signatures but notes the lack of delay-resolved data; the local SXS analysis finds high overall accuracy, rising modal errors, and comparatively small extrapolation differences.

7. Conclusion

  • The graphene analysis identifies first-order replica bands separated from the parent feature by the 5 microm pump photon energy, with enhanced replica-region spectral weight and weak π-periodic polarization anisotropy.The generated energy-momentum maps show the Dirac cone and expected replica-energy intensity, while the polarization plot reproduces a periodic angular trend.Source: §7 · p. 47
  • The anisotropy is compatible with photon-dressed Volkov final-state contributions but does not uniquely establish them; without a delay-indexed 4D tr-ARPES cube, time dynamics and causal separation remain unresolved.The generated replica features are also less distinct and more diffuse than the target image.Source: §7 · p. 47
  • Inputs specify binary parameters and waveform, horizon, and metadata outputs; feature datasets provide resolution differences, mode-resolved differences for ℓ=2–8, and N=2-versus-N=3/4 extrapolation comparisons.The synthetic datasets contain 1,500 simulation entries for the resolution and mode comparisons and 1,200 for extrapolation comparisons.Source: §7 · p. 48
  • A median waveform difference of 4×10^-4 between the two highest resolutions summarizes the catalog’s overall numerical error, with most differences between 10^-4 and 10^-2.The reproduced distribution is approximately log-normal, matching the reported Figure 6 median.Source: §7 · pp. 48–49
  • Median waveform differences rise from about 3×10^-4 at ℓ=2 to about 1.5×10^-3 at ℓ=8, while scatter broadens and overall accuracy remains dominated by ℓ=2.The higher-ℓ modes have larger errors but smaller amplitude contributions, informing mode truncation in waveform modeling.Source: §7 · p. 49
  • Median differences are 2×10^-5 for N=2 versus N=3 and 5×10^-5 for N=2 versus N=4, both more than an order of magnitude below the 4×10^-4 resolution-error median.The comparisons support the conclusion that extrapolation errors do not dominate waveform errors.Source: §7 · p. 49

Local Analysis of Catalog-Scale Numerical Accuracy in Synthetic SXS Binary Black Hole Data

  • The local analysis builds a reproducible pipeline for distribution summaries and report figures, then constructs a quality index to stratify simulations by combined numerical difficulty.Its three questions concern catalog-wide resolution error, mode-dependent error, and extrapolation-order stability.Source: Local Analysis of Catalog-Scale… · p. 50

1. Context and Goal

  • The local context motivates explicit catalog error diagnostics, attention to harder-to-model subdominant modes, and accuracy sufficient for surrogate-model calibration.It also frames the analysis around global resolution errors, modal degradation, extrapolation convergence, and joint quality across simulations.Source: §1 · p. 50

2. Data and Methodology … 2. Background

The analysis finds mostly low waveform disagreement in the synthetic catalog, with higher errors at larger harmonic indices and nonuniform sensitivity to extrapolation order. The accompanying geometry material describes IMO-AG-30 and AlphaGeometry’s combination of symbolic deduction with language-model-proposed constructions.

2. Data and Methodology

  • The analysis uses three datasets covering 1,500 simulations each for resolution and mode errors and 1,200 for extrapolation, then builds a composite index for the common 1,200.The index combines log-scaled resolution error, median and maximum mode error, and extrapolation differences; it is a local ranking construct, not a physical observable or reproduction of SXS catalog labels.Source: §2 · p. 50

3.1 Catalog-wide resolution accuracy; 3.2 Accuracy loss at higher spherical-harmonic modes

  • Median waveform difference is 4.25 x 10^-4 (90th percentile 2.06 x 10^-3), while median error rises from 3.00 x 10^-4 at l=2 to 2.27 x 10^-3 at l=8.The resolution-error distribution has a rare tail reaching 4.07 x 10^-2; the l=8 to l=2 median-error ratio is 7.57.Source: §3.1 · p. 51, §3.2 · p. 51, §4 · p. 52

3.3 Extrapolation-order stability; 3.4 Joint quality stratification

  • N=2 vs N=4 disagreement is larger in 72.2% of simulations, with median ratio 2.67 and weak correlation (r = 0.036); the quality tiers consistently worsen across all three diagnostics.The tier ordering links worse composite quality with larger resolution disagreement, larger high-mode error, and worse extrapolation stability.Source: §3.3 · pp. 51–52, §3.4 · p. 52

4. Interpretation; 5. Claim Discipline and Limits

  • The evidence supports distributional and comparative claims only: most supplied-resolution differences are small, higher modes are less accurate, and the harder extrapolation comparison exposes additional case-dependent uncertainty.The inputs lack physical simulation parameters and waveform time series, so the study does not establish astrophysical parameter-space coverage, surrogate calibration performance, physical causes of error, mode mixing, or remnant-model accuracy.Source: §4 · p. 52, §5 · pp. 52–53

6. Reproducibility

  • The analysis is reproducible from code/analyze_catalog_accuracy.py, with summary metrics, mode statistics, quality summaries, and figures saved under the listed output and report paths.The stated run command is python code/analyze_catalog_accuracy.py.Source: §6 · p. 53

7. Conclusion; Autonomous Neuro-Symbolic Reasoning for Olympiad Geometry: Analysis of AlphaGeometry on the IMO-AG-30 Benchmark; 1. Introduction

  • On IMO-AG-30, AlphaGeometry solves 25/30, matching average gold-medalist performance, while DD+AR solves 14/30; AlphaGeometry uses 100M+ synthetic examples without human-written proof training.The benchmark contains 30 geometry problems from IMO competitions since 2000; the provided task description frames the output as machine-verifiable, human-readable Euclidean proofs.Source: §7 · p. 54, §1 · p. 55

1.1 Research Questions; 2.1 The IMO-AG-30 Benchmark

  • The geometry analysis asks which structural features predict difficulty, where symbolic reasoning needs a language model, how proof complexity varies, and what makes unsolved problems hard.IMO-AG-30 problems encode point constructions, geometric constraints, and a proof goal in a formal language intended to support machine-verifiable, human-readable proofs.Source: §1.1 · p. 55, §2.1 · pp. 55–56

2.2 Symbolic Deduction: DD+AR; 2.3 AlphaGeometry; 1. Introduction

  • DD+AR exhaustively applies 44 geometric inference rules to a fixed point, while AlphaGeometry alternates deduction with language-model proposals for auxiliary constructions.Its 1-billion-parameter transformer was trained on 100 million synthetic geometry statement-proof pairs generated through random construction and retrograde analysis, avoiding human-annotated proof corpora; AlphaGeometry represents the hybrid approach described by Trinh et al. (2024).Source: §1 · p. 55, §2.2 · p. 56, §2.3 · p. 56

3. Dataset Analysis

IMO-AG-30 covers geometry problems across multiple competition years, with diverse proof goals and construction demands. Its composite complexity score correlates with proof difficulty measured by the need for LM-assisted auxiliary constructions.

3.1 Overview

  • Figure 1 shows congruence goals at 40%, collinearity at 23%, and concyclicity at 17%; problem complexity has no clear upward trend, and construction counts center around 8–9.The benchmark spans 30 problems from 21 competition years (2000–2022), with 8 missing years. It includes seven goal types; construction counts range from 4 to 15 per problem (mean 8.7, std 2.5).Source: §3.1 · pp. 56–57

3.2 Complexity Score

  • The composite complexity score combines point counts, constraint clauses, and weighted high-complexity constructions, ranging from 13.0 to 37.9 and correlating with LM-assisted auxiliary-construction needs.The construction weights include circles (+3.0), incenter (+2.5), orthocenter and reflections (+2.0 each), and angle bisectors (+1.5).Source: §3.2 · p. 57

3.3 Construction Primitive Usage

  • on_line is the most frequent construction primitive, while collinearity proofs use more line intersections and orthocenter constructions and congruence proofs rely more on circles and midpoints.on_line appears in nearly every problem; on_circle, midpoint, foot, and reflect are the next most common primitives.Source: Figure 6 · p. 57, §3.3 · p. 57

4. Methods … 3. Methodology

AlphaGeometry combines language-model-proposed auxiliary constructions with formally verified symbolic deduction, reaching 25/30 on IMO-AG-30; the supplied passages also describe an ECAT battery-parameter framework using an ANN surrogate and genetic algorithm. Synthetic geometry training and surrogate-based battery optimization are presented as ways to address data scarcity and computational cost, respectively.

4.1 Symbolic Reasoning Engine (DD+AR)

  • The LM proposes auxiliary points but does not deduce: DD+AR applies implemented geometric rules to a fixed point, yielding machine-verifiable proofs.The described rules include perpendicular-to-parallel, cyclic-to-equal-angle, congruent-distances-to-cyclic, collinearity extension, and parallel transitivity.Source: §4.1 · p. 57, §4.2 · p. 58

4.3 Training Data Generation

  • ~100M random geometric constructions yield ~100M proof traces and ~90M derived theorems, from which retrograde analysis extracts theorem-proof pairs for synthetic LM training.The approach addresses a scarcity of fewer than 10,000 known formalized geometry proofs; the passages describe synthetic pairs as teaching useful construction patterns without human-labeled examples.Source: Figure 7 · p. 58, §4.3 · pp. 58–59

5.1 Benchmark Performance

  • 25/30 problems (83%) were solved by AlphaGeometry, matching the human gold-medalist threshold and improving on DD+AR’s 14/30 (47%).The five unsolved problems had complexity scores >=29.6 and required multiple interacting auxiliary constructions.Source: §5.1 · p. 59

5.2 Complexity and Solvability

  • 77% accuracy for DD+AR solvability and 67% for AlphaGeometry solvability were achieved by logistic regression using complexity score alone.Problems requiring 4+ auxiliary constructions were generally unsolved; the figure groups problems as DD+AR-solvable at complexity <=22, LM-assisted at 18-32, and unsolved at >=29.Source: §5.2 · p. 60, Figure 3 · p. 60

5.3 Proof Length and Auxiliary Constructions

  • ~44 steps was the mean proof length for LM-assisted problems, versus ~30 for DD+AR-only problems; overall proof lengths averaged 41.7 steps (std 9.4, range 25-58).Most LM-assisted proofs used 1-2 auxiliary constructions, with a maximum of 4; the five unsolved problems span IMO years 2008, 2011, 2017, 2019, and 2022, with no clear temporal trend.Source: §5.3 · p. 60, §5.4 · p. 60

6.3 Unsolved Problems

  • The five unsolved problems feature high point counts, nested reflections, multiple interacting circles, or trigonometric equalities that the current rules do not handle.Proposed directions include trigonometric cevian rules, improved chaining of auxiliary constructions, and wider proof-search beams; the passages also characterize DD+AR as providing correctness guarantees while the LM supplies construction suggestions.Source: §6.3 · p. 61, §6.1 · p. 61

3.2 Parameter Space and LHS Design

  • 500 LHS samples cover the 11-dimensional ECAT parameter space, with reaction rates and diffusivities sampled in log-space; the model uses a thermally coupled single-particle battery representation.The passages describe 11 parameters spanning geometric, kinetic, transport, and thermal properties, and motivate the framework as addressing costly parameter identification in a coupled electrochemical-aging-thermal model.Source: §3.1 · p. 65, §1 · p. 65, §3.2 · p. 66

3.4 Multi-Objective Genetic Algorithm

  • ~100x speedup over direct simulation is reported when the ANN surrogate replaces the SPM simulator for MMGA fitness evaluation.The surrogate maps 11 parameters to a 200-point discharge voltage curve; the genetic algorithm uses a population of 100, 200 generations, and a weighted fitness of voltage RMSE (70%) and MAE (30%).Source: §3.4 · pp. 66–67, §3.3 · p. 66

4. Experimental Data · 7. Conclusion

The MMGA framework combines Latin Hypercube Sampling, an ANN surrogate, and multi-objective genetic optimization to identify parameters for ECAT-coupled battery models. It reports voltage-prediction accuracy on NASA and CS2 data, faster optimization, and cross-dataset transfer, while noting limitations and future extensions.

4. Experimental Data

  • NASA, CALCE CS2_36, and Oxford data provide controlled room-temperature discharge, commercial NCM aging, and dynamic urban-driving validation scenarios.NASA includes four 18650 cells; B0005 completed 168 cycles, and cycle 293 serves as a reference curve. CS2_36 files cover cycles 10, 18, 24, and 28, while Oxford’s first drive cycle has 3,145 transient-load data points.Source: §4.1 · p. 67, §4.2 · p. 67, §4.3 · p. 67

5.2 ANN Surrogate Model Performance

  • The ANN surrogate achieves 0.284 V validation RMSE (median 0.096 V), while MMGA fitness reaches 0.165 for NASA and 0.197 for CS2.ANN training converges within 200 epochs, with best validation loss 0.113 at epoch 150; MMGA converges rapidly in its first 50 generations.Source: §5.2 · p. 68, §5.3 · p. 68

5.4 Voltage Prediction Accuracy

  • NASA-optimized parameters yield RMSE 0.176 V and MAE 0.138 V, versus RMSE 0.212 V and MAE 0.162 V for CS2; cross-validation shows reasonable shape agreement but imperfect transfer.The transfer mismatch is attributed to differences in chemistry and test conditions; identified parameters remain within physically plausible ranges.Source: §5.4 · p. 69, §5.6 · p. 70

5.5 Identified Parameters

  • Sensitivity analysis links maximum concentrations to voltage throughout discharge, reaction rates to initial voltage drop, particle radii to discharge slope, and heat transfer to minimal voltage variation.NASA optimization favors smaller particle radii and moderate reaction rates; CS2 favors larger radii and higher positive-electrode reaction rates, while both reach the 5 W/m2K heat-transfer lower bound.Source: §5.5 · pp. 69–70, §5.7 · p. 71

6.1 Computational Efficiency

  • ANN-assisted MMGA reduces 20,000 fitness evaluations from approximately 200 seconds of direct simulation to 2 seconds, with the full pipeline completing in under 40 seconds.A single SPM simulation takes approximately 0.01 seconds versus 0.0001 seconds for an ANN forward pass; this speedup makes frequent parameter updates for battery digital twins feasible.Source: §6.1 · p. 71, §6.4 · p. 72

6.2 Model Limitations

  • The approach is limited by SPM’s omitted electrolyte gradients and spatial variation, ANN approximation error, weak identifiability of low-sensitivity thermal parameters, and empirical chemistry-specific OCV fits.The reported ANN validation RMSE is 0.284 V; higher training-data volume or more sophisticated architectures could reduce surrogate error.Source: §6.2 · p. 71

6.3 Comparison with Literature

  • Against Li et al.’s 9 mV RMSE using direct P2D simulation, the framework reports 176 mV RMSE but identifies 11 parameters in under 40 seconds, unlike Forman et al.’s three-week cluster run for 88 parameters.The study presents rapid identification as useful for battery digital twins requiring frequent updates.Source: §6.3 · p. 72, §6.4 · p. 72

7. Conclusion

  • 0.176 V (NASA) and 0.212 V (CS2) voltage-prediction RMSEs, ~100x speedup, physically plausible parameters, and cross-dataset transferability summarize the framework’s reported results.The conclusion identifies full P2D models, multimodal experimental data, and adaptive sampling as future directions.Source: §5.4 · p. 69, §6.1 · p. 71, §7 · p. 72

Related papers

Loading 2606.07591v5…