Probing Scientific General Intelligence of LLMs with Scientist-Aligned Workflows

• SGI is defined as autonomous navigation of the complete, iterative scientific-inquiry cycle, structured by the Practical Inquiry Model’s Deliberation, Conception, Action, and Perception quadrants.The framework maps these quadrants to knowledge search and evaluation, idea generation, experimental implementation, and interpretation of results.Source: §1 · p. 4
• Existing benchmarks assess fragments of scientific ability, while SGI-Bench covers all four inquiry capabilities through deep research, idea generation, dry/wet experiments, and experimental reasoning.The paper argues that common evaluations are narrow, often expert-insufficient, and treat inquiry as static closed-domain question answering.Source: §1 · pp. 4–5
• SGI-Bench contains over 1,000 expert-curated samples drawn from ten disciplines, and its agent-based evaluation framework coordinates four specialized stages.Topics originate from Science’s 125 Big Questions; construction involved multi-round domain-expert collaboration and quality control, while evaluation uses tools and discipline-specific metrics.Source: §1 · p. 5
• Deep-research exact match remains below 20%; ideas often lack feasible detail, dry experiments fail scientific correctness, wet plans misorder steps, and comparative multimodal reasoning remains weak.The paper also identifies weak numerical robustness, procedural discipline, multimodal grounding, and self-corrective reasoning as broader limitations.Source: §1 · pp. 5–6
• Even the best closed-source model scores around 30/100 on SGI-Score, and closed-source models hold only a marginal advantage over open-source models.Across tasks, current models exhibit pockets of success but fall short of the integrated reasoning required for scientific intelligence.Source: §1 · pp. 5–6
• Preliminary test-time scaling and reinforcement-learning experiments suggest that minimal unlabeled feedback can improve hypothesis formation and reasoning.The introduction frames scientific intelligence as a capacity that may develop through iterative, self-reflective reasoning rather than a static property.Source: §1 · p. 6

2. Scientific General Intelligence: Concept and Operational Definition

• SGI-Bench operationalizes discovery as four interdependent stages and scientist-aligned tasks, drawing on scientists and spanning ten domains inspired by Science’s 125 Questions.Source: §2 · pp. 6–7

2.1.1. Scientific Deep Research

• Deep Research is literature-inquiry-centric, combining multi-hop retrieval with meta-analytic synthesis; it targets data, properties, micro-experiments, and macro-experiments within a reproducible scope.Source: §2.1.1 · pp. 7–8

2.1.2. Idea Generation

• Because broad hypothesis evaluation requires extensive expert review, the benchmark scores methodological design—data, procedures, evaluation, and expected outcomes—and decomposes ideas by effectiveness, novelty, detail, and feasibility.Source: §2.1.2 · p. 10

2.1.3. Dry/Wet Experiment

• Dry experiments use code completion and wet experiments protocol design from a predefined action space; these constraints retain core reasoning while enabling reproducible, controlled evaluation.Source: §2.1.3 · pp. 11, 13

2.1.4. Experimental Reasoning

• Experimental reasoning tests data-analysis interpretation: models extract visual or numerical cues across five modalities, compare conditions, and identify causal or descriptive patterns.Source: §2.1.4 · p. 14, Table 3 · p. 14

2.2. Multi-Dimensional Metrics

• Evaluation uses task-specific multidimensional metrics rather than a one-size-fits-all binary judgment.Source: §2.2 · p. 15

2.2.1. Metrics of Scientific Deep Research

• Deep Research metrics pair exact-match final-answer correctness with LLM-judged step-level accuracy, capturing both final precision and reasoning fidelity.Source: §2.2.1 · pp. 15–16

2.2.2. Metrics of Idea Generation

• Idea quality covers effectiveness, novelty, detailedness, and feasibility; subjective scores use six votes from three LLM judges, while effectiveness averages keyword hit rate and judge win rate.Source: §2.2.2 · p. 16
• Idea Generation is assessed through novelty, detailedness, and feasibility, combining similarity-based or completeness measures with subjective or expert-template comparisons.Detailedness checks for core idea, implementation steps and order, dataset, evaluation metrics, and expected outcome, while penalizing semantic redundancy; feasibility compares an idea’s implementation graph with an expert template.Source: §2.2.2 · pp. 16–17

2.2.3. Metrics of Dry/Wet Experiment

• Dry Experiment metrics distinguish unit-test success, runtime efficiency, and error-free execution, while Wet Experiment metrics assess action-sequence fidelity and parameter correctness.Dry tasks ask models to complete masked code functions in problems with five unit tests; Wet tasks require correct laboratory action order and parameter settings.Source: §2.2.3 · pp. 17–18

2.2.4. Metrics of Experimental Reasoning

• Experimental Reasoning evaluates multimodal questions with at least 10 answer candidates using both exact-match answer accuracy and judged validity of step-by-step reasoning.Reasoning Validity is scored from 0 to 10 by an LLM judge against reference reasoning, with the final score averaged across test samples.Source: §2.2.4 · p. 19

2.3. Scientist-Aligned Data Construction

• Experts from diverse disciplines helped identify research directions drawn from Science’s 125 important questions and prominent high-impact fields, yielding 75 directions across 10 scientific domains.The collected expert-provided material included scientific texts and images covering background knowledge, methods, procedures, data analysis, experiments, visualizations, and observations.Source: §2.3 · p. 19
• More than 100 discipline-specific Master’s and PhD annotators constructed questions from source material, with expert review, layered cleaning, and difficulty filtering.Questions were traceable to original sources and independently drafted and refined; rule-based, model-based, and expert checks removed invalid items, and questions answerable by more than half of six high-performance models were excluded.Source: §2.3 · pp. 19–20

2.4. Data Distribution

• SGI-Bench contains 318 Scientific Deep Research, 315 Idea Generation, 271 Dry Experiment, 68 Wet Experiment, and 291 Experimental Reasoning questions.The benchmark also categorizes research targets, masked code functions, image modalities, and reasoning types; Wet Experiments cover a subset of disciplines, including Biology and Chemistry.Source: §2.4 · pp. 20–21

3.1. Question Selection

• A questioning agent filters SGI-Bench questions by discipline, task, and evaluation intent, or selects all questions when no query is provided.This stage defines the evaluation scope.Source: §3.1 · p. 22

3.2. Metric Customization

• A customization agent generates metrics from user queries and selected questions, then combines them with predefined scientist-aligned metrics suited to each question type.It can use web search and PDF parsing to formalize evaluation intent and add evaluation dimensions.Source: §3.2 · pp. 22–23

3.3. Inference and Evaluation

• The framework runs target models on selected questions with tools, then a Science Eval Agent scores their responses using finalized metrics and provides evidence-grounded rationales.Rationales draw on reference answers, question context, and supplementary information retrieved through tools when needed.Source: §3.3 · pp. 23–24

3.4. Report Generation

• A reporting agent aggregates evaluation intents, finalized metrics, and results to quantify and visualize performance across selected questions, while contextualizing findings in scientific discovery.The report is intended to support actionable insights and informed decisions.Source: §3.4 · p. 24

4.1. Evaluation Setup

• The evaluation covered open- and closed-weight LLMs and agents, using temperature 0 for configurable models and a standard zero-shot, task-specific prompt template across tasks.The evaluations used scientist-aligned metrics to assess systems throughout the scientific discovery workflow.Source: §4.1 · p. 25

4.2. Overview

• 33.83 was the top SGI-Score (Gemini-3-Pro); Claude-Sonnet-4.5 (32.16) only narrowly edged Qwen3-Max (31.97), while task-specific weaknesses persisted.The overview characterizes SGI-Score as typically 30±5 and notes that closed-source systems had only a marginal edge over leading open-source systems.Source: §4.2 · p. 25, Table 5 · p. 26

4.3. Scientific Deep Research

• EM stayed around 10% and seldom exceeded 20%; multiple systems topped 50% SLA (best around 65%), while agent and LLM performance distributions overlapped.The best LLMs matched or exceeded the best agents on EM, and Grok-4 scored lower than Grok-3 on both EM and SLA.Source: §4.3 · pp. 25–26, Figure 12 · p. 27, Figure 13 · p. 28
• Across deep-research categories, performance rarely exceeded 30%, with Data and Properties weaker than Micro- and Macro-experiments, where structured protocols reduce retrieval uncertainty.Data- and property-focused questions often require numerical details scattered across sources, while experiment-oriented tasks more often provide structured protocols or clearer outcomes.Source: §4.3 · pp. 27–28, Figure 14 · p. 28

4.4. Idea Generation

• 55.40 was GPT-5’s highest Idea Generation average, but feasibility lagged (GPT-5 18.87; o3 best at 22.90) despite high novelty and detailedness.Proposals often omit executable details such as data acquisition and preprocessing, hyperparameters, resource assumptions, algorithm choices, and integration steps; o3 scored 73.74 and GPT-5 76.08 on Novelty, while GPT-5 scored 85.72 on Detailedness.Source: §4.4 · pp. 28–29, Table 6 · p. 30

4.5.1. Dry Experiment

• 42.07% PassAll@1 fell to 36.64% PassAll@5; although many top models exceeded 90% SER, execution without errors did not ensure scientific correctness.Numerical calculation and simulation functions were harder than data processing and predictive modeling, which allow more valid implementations; methodological choices such as integration strategy could substantially affect scientific outcomes.Source: §4.5.1 · pp. 31–32, Table 7 · p. 32, Figure 17 · p. 32

4.5.2. Wet Experiment

• Wet-protocol sequence similarity remained low: the best closed-source model scored around 35.5 versus below 30 for the best open-source model, while parameter accuracy was only moderate.Parameter accuracy peaked around 40.6, but some closed-source models were near 20.7; recurring errors included unnecessary or omitted steps, wrong ordering, and failures in temporal coordination, branching, and sample bookkeeping.Source: Figure 20 · p. 34, §4.5.2 · pp. 34–35, Figure 21 · p. 35

4.6. Experimental Reasoning

• Closed-source models reached higher Experimental Reasoning scores than the best open-source models (MCA 41.9 vs. 37.8; RV 71.3 vs. 52.3), but comparative reasoning remained a persistent weakness.Most models scored higher on Reasoning Validity than Multi-choice Accuracy; performance also varied by discipline, with astronomy strongest and materials, life, and Earth sciences weaker.Source: §4.6 · p. 36
• The narrow performance gap among leading models underscores that scale alone is insufficient; progress requires better multimodal grounding, finer-grained visual understanding, and training aligned with scientific inquiry.Source: §4.6 · p. 37

5.1.1. Methodology

• TTRL optimizes idea dissimilarity to real-time retrieved literature alongside a format reward; Qwen3-8B uses GRPO [1] with high-temperature sampling.The format reward enforces <think>...</think> and <answer>...</answer> structure; the composite reward is an unweighted sum of format and novelty components.Source: §5.1.1 · pp. 38–39, §5.1.2 · p. 39

5.1.3. Experimental Results

• Novelty scores rose from 49.36 to 62.06 without ground-truth labels, as online retrieval feedback drove test-time self-improvement.Format reward rapidly saturated near 1.0, after which idea reward rose steadily during training.Source: Figure 27 · p. 40, §5.1.3 · p. 40

5.1.4. Case Study of TTRL

• After TTRL, the RNA-structure proposal adds a dual-branch transformer, differentiable physics engine, and confidence-aware uncertainty module instead of generic component assembly.The pre-training response combined contact-map prediction with Rosetta energy functions; the post-training response was more specific and technically distinct.Source: Figure 28 · p. 40, §5.1.4 · p. 41

5.2.1. Retrieve–Browse Loop Analysis

• Tool calls concentrate on retrieval: web_search accounts for 539 calls (33.98%), while Qwen3-Max uses fewer web_search and visit_webpage calls than GPT-4.1 for comparable coverage.GPT-4.1 makes 168 web_search and 110 visit_webpage calls, versus Qwen3-Max’s 61 and 59, respectively.Source: §5.2.1 · p. 41

5.2.2. Tool Efficiency Analysis

• visit_webpage latency spans 5.37s to 114.29s across models, a 21.28× spread, making browser execution a major bottleneck beyond model reasoning.The passage attributes the latency to network I/O, DOM parsing, and event replay, and notes a 7.59× spread for wikipedia_search.Source: §5.2.2 · p. 41

5.2.3. Reasoning Cost Analysis

• python_interpreter latency ranges 5.48–52.94s across models, reflecting a trade-off between fast first-attempt execution and slower iterative error correction.GPT-4o is fastest at 5.48s, while DeepSeek-R1 is slowest at 52.94s; the passage recommends aligning routing, retries, and verification depth with model behavior.Source: §5.2.3 · p. 42

5.3.1. User-customized Metric

• SGIEvalAgent’s customized rigor rubric assesses six dimensions, from pipeline coherence and literature grounding to reproducibility and risk awareness, then aggregates 0–10 aspect scores with equal weights.The aggregate maps linearly to 0–100 for visualization without changing rank order, and customized metrics supplement rather than replace standardized SGI-Bench metrics.Source: §5.3.1 · pp. 42–43

5.3.2. Automated Evaluation Report

• Under the customized rigor rubric, closed-source models generally score higher, while open-source leaders make progress that narrows the gap.The report interprets rigor as structured, well-grounded, verifiable plans rather than fluent narratives, and notes measurable gains across iterations within model families.Source: §5.3.2 · p. 43

6.1. Fragmentation Across the Four Quadrants of SGI

• Deep Research remains brittle: most systems achieve only ~10% Exact Match (EM), the best rarely exceed 20% EM, and Data and Properties questions rarely exceed 30% accuracy.Several agentic systems reach ≈50% Step-Level Accuracy (SLA), despite low EM; the gap points to failures in long-chain coherence and numerical aggregation over scattered literature.Source: §6.1 · p. 44
• Idea Generation is weakest in feasibility and implementation detail: feasibility reaches only 22.90 (o3), while the highest Effectiveness is 51.36 (GPT-5.2-Pro).Proposals often omit parameters, resources, step ordering, or reproducible, executable procedures, so fluent ideation does not reliably yield testable plans.Source: §6.1 · p. 44
• Dry-experiment accuracy remains limited despite executable code: best PassAll@1 is 42.07% (Claude-Sonnet-4.5), while best PassAll@5 is 36.64% (Gemini-3-Pro).Most frontier models achieve SER > 90%; numerical-calculation and simulation functions are major failure modes. Wet-experiment Sequence Similarity is uniformly low, with redundant, omitted, or misordered protocol steps.Source: §6.1 · pp. 44–45
• In Experimental Reasoning, closed-source models outperform open-source ones, and Comparative Reasoning is weakest; Reasoning Validity exceeds Multi-choice Accuracy across nearly all models.Signal Perception and Causal Reasoning are relatively stronger; astronomy and chemistry are easier than materials, life, and Earth sciences.Source: §6.1 · p. 45

6.2. Implications from Test-Time RL and Tool-Integrated Reasoning

• TTRL raises Qwen3-8B’s novelty score from 49.36 to 62.06 without labeled supervision, while retrieval dominates tool use and contributes substantially to agent latency.web_search accounts for 539 calls (33.98%) and visit_webpage for 385 calls (24.27%); visit_webpage latency ranges from 5.37s to 114.29s across models, motivating better routing, extraction, and verification policies.Source: §6.2 · pp. 45–46

6.3. Future Directions Toward Scientific General Intelligence

• Future work should strengthen evidence aggregation, feasibility-focused planning, numerical scientific code, branch- and time-aware protocols, comparative multimodal reasoning, multi-objective TTRL, and efficient, trustworthy tool ecosystems.Proposed methods include uncertainty-calibrated synthesis, parameter-complete plans, stability-aware numerical training, stateful lab simulators, cross-image alignment, rewards beyond novelty, and selective retrieval.Source: §6.3 · p. 46

6.4. Limitations

• SGI-Bench incompletely represents scientific practice: it probes four workflow stages, centers Deep Research on literature inquiry and Idea Generation on methodology design, and limits dry-code and wet-action coverage.The benchmark excludes broader disciplinary integration and risk/safety assessment, reserves open-form research evaluation for future versions, and uses Python-only dry experiments and an early-stage wet-action abstraction.Source: §6.4 · p. 47
• Multiple-choice Experimental Reasoning constrains open-form explanations, and the current PIM-grounded benchmark primarily captures deductive discovery while leaving inductive discovery outside its scope.Inductive discovery from observational data and unexpected phenomena is identified as an important direction for benchmark expansion.Source: §6.4 · pp. 47–48

7.1. Benchmarks in Different Disciplines

• Discipline-specific benchmarks span physical, chemical, life, earth, remote-sensing, and other sciences, while ATLAS [105] tests complex reasoning rather than the full discovery process.These benchmarks primarily evaluate depth within isolated disciplines; ATLAS [105] extends across fields but focuses on frontier-model reasoning and logical application tasks.Source: §7.1 · p. 48

7.2. Benchmarks for Different Scientific Tasks

• Cross-disciplinary benchmarks assess stages from idea generation and literature analysis to survey writing, tool use, coding, planning, and research workflows.Examples include MOOSE-Chem2 [37] and AI Idea Bench 2025 [106] for ideas, SciAssess [107] and SFE [45] for analysis, and DeepResearch Bench [33] and Scientist-Bench [110] for surveys and workflows.Source: §7.2 · pp. 48–49
• Existing process-oriented benchmarks cover partial dimensions but lack systematic lifecycle evaluation; DISCOVERYWORLD [119] and LLM-SRBench [120] also have scope limitations.DISCOVERYWORLD uses gamified simulations that do not fully reflect real research complexity, while LLM-SRBench focuses on equation discovery. The related-work section calls for a multidisciplinary benchmark connecting the long-chain research workflow.Source: §7.2 · p. 49

8. Conclusion

• The paper defines SGI as navigating the Practical Inquiry Model’s iterative cycle and operationalizes it through four scientist-aligned task families in SGI-Bench.The benchmark is complemented by an agentic evaluation framework and multi-metric protocol for scalable, transparent, domain-faithful assessment.Source: §8 · p. 49
• Across SGI-Bench, models show gaps between linguistic fluency and integrated scientific cognition, including low deep-research exact match (10–20%) and persistent multimodal comparative-reasoning challenges.Models are step-aligned but brittle in quantitative reasoning; generated hypotheses are underspecified and infeasible; dry-experiment code executes but PassAll@k remains low; wet protocols omit or misorder steps; causal reasoning outperforms comparative reasoning.Source: §8 · p. 49
• Test-Time Reinforcement Learning with novelty rewards improves idea generation without reference answers.Source: §8 · p. 49
• SGI-Bench provides a foundation for systematically advancing SGI, while the paper identifies capabilities and tools needed for systems capable of genuine scientific discovery.The proposed path combines numerically robust reasoning, planning-aware conception, executable experimentation, comparative multimodal inference, dynamic test-time learning, and efficient tool ecosystems.Source: §8 · pp. 49–50

A.2. Disciplines and Research Directions Overview

• Table 9 describes an overview of 10 scientific domains and representative research topics curated for scientist-aligned SGI-Bench workflows.Source: Table 9 · p. 59
• Chemistry examples span target-based and de novo drug design, synthesis-pathway planning, and molecular-property prediction.The descriptions cover computationally designing molecules, planning practical multistep synthesis routes, and predicting properties such as toxicity, solubility, and bioactivity.Source: §A.2 · p. 60
• Earth and energy topics include seismic and climate measurements, emissions estimation, power-flow optimization, and renewable-energy forecasting.Examples include combining satellite measurements with transport models for emission inversion and forecasting wind and solar output from meteorological data.Source: §A.2 · pp. 60–61
• Information research directions cover multimodal understanding, dialogue systems, code generation, and phase-free reconstruction of sensor spatial characteristics.The sensor task reconstructs spatial sensitivity patterns using measurement magnitudes without phase information.Source: §A.2 · p. 62
• Life-science examples include biomarker and neoantigen discovery, RNA structure prediction, and early pancreatic-cancer screening and risk stratification.The overview describes biomarker analysis using high-throughput biological data, identifying mutation-derived peptides for immunotherapies, and developing tools for earlier detection and patient risk classification.Source: §A.2 · pp. 62–63
• Other listed directions span materials and mathematical methods, neuroscience decoding and segmentation, and computational physics and deep PDE solving.Examples include chloride solid-state electrolytes, differential privacy, visual and motion decoding, neuron segmentation, condensed-matter simulation, and physics-informed neural networks.Source: §A.2 · pp. 64–66

Astronomy and Chemistry

• Astronomy estimates the host-environment magnetic field at 46 μG, while the chemistry example identifies Azulene’s aromatic π-electron count as 10.The astronomy steps derive the host-frame dispersion measure before reporting the field estimate; the chemistry steps treat Azulene’s rings as one conjugated system and apply Hückel’s rule.Source: §A.3.1 · pp. 68–69

Earth

• The NO2 example reports Ω0=3.73, no peak over 0–4 hours, and P(4)−E(4)=5.59 after finding concentration rises monotonically.Because no peak occurs in the interval, the example compares photochemical production and emissions at t=4.Source: §A.3.1 · p. 70

Energy

• The solar-collector example gives baseline Fr=0.99 and η=0.59, and a minimum mass flow of 0.05 kg/s for η≥0.58.The calculation uses the stated operating conditions and rounds the results to two decimals.Source: §A.3.1 · p. 71

Information

• For the near-field reconstruction example, the worked solution estimates 4,480 supplementary samples for a cluster, assuming five evenly sized clusters and one-third deep-region points.It applies 24 supplementary samples per deep-region point and 8 per shallow-region point.Source: §A.3.1 · p. 72

Life

• The enhancer example predicts H3K27ac signal at 67% of wild type when both motifs’ flanks change from GGG to CTC.Its calculation compares total stated affinities of 9 for wild type and 6 after the changes.Source: §A.3.1 · p. 73

Material

• The materials example predicts that thermopower increases as relative humidity rises from 50% to 70%, without changing the sample’s p-type character.The passage attributes the increase to greater water absorption.Source: §A.3.1 · pp. 73–74

Math

• The Legendre-collocation example obtains β²=18 as the condition for a non-trivial solution.The worked steps reduce the system to a determinant condition.Source: §A.3.1 · pp. 74–75

Neuroscience

• The neuroscience prompt notes that non-motor brain regions can contribute useful classifier features during motor imagery, particularly with visual feedback or focused attention.The supplied passage ends before identifying the requested cerebral lobe.Source: §A.3.1 · p. 75

Scientific Deep Research

• The motor-imagery answer identifies the occipital lobe as the location of the primary visual cortex and links its visual processing to visual feedback.It concludes that the occipital lobe is the answer.Source: §A.3.1 · p. 75
• For the LaFeAsO prompt, the worked answer reports t_eff = −0.1 eV, Δ(k) = 0.04, and Tc = 33.14 K.The gap calculation assumes ξ(k′) = −2t1 cos kx − 2t1 cos ky, Δ(k′) = Δ0, and tanh ≈ 1 at T = 4.2 K; the calculated Tc is described as aligning with theoretical predictions and compared with an experimental Tc of 26 K.Source: §A.3.1 · p. 76

Astronomy

• The astronomy proposal equips the Palomar 48-inch Schmidt telescope with a custom CCD mosaic, optimized scheduling, and a robust data system for rapid, wide-field transient discovery.The proposed ZTF system is described as delivering an order-of-magnitude survey-speed improvement and a real-time, feature-rich alert stream.Source: §A.3.2 · p. 78

Chemistry

• The chemistry proposal represents molecules as junction trees of valid substructures and uses a two-stage VAE to generate scaffolds before assembling molecular graphs.The approach combines tree- and graph-based encodings, feasibility-checked decoding, and subgraph assembly for property-driven optimization.Source: §A.3.2 · p. 81
• JT-VAE achieved 100% molecular validity versus SDVAE’s 43.5% and Atom-by-Atom LSTM’s 89.2%, with 76.7% reconstruction accuracy.For constrained property optimization, it achieved scores up to 5.3 versus 4.04 from SD-VAE and over 80% success with >0.4 similarity.Source: §A.3.2 · p. 82

Earth

• The grassland proposal systematically compares structure-from-motion volumetric modeling with GNDVI regression and classification to estimate forage biomass and vegetation cover.It uses UAV imagery and field measurements collected over 14 dates to assess the methods across a growing season.Source: §A.3.2 · pp. 83–84
• Volumetric modeling achieved R2 = 0.93 (fresh), 0.94 (dry), and RMSE 0.072 kg/m2 (fresh), 0.013 kg/m2 (dry); GNDVI regression reached R2 = 0.80 (fresh), 0.66 (dry).The study compared three UAV approaches across a full growing season in temperate grasslands: volumetric modeling, GNDVI regression, and GNDVI classification.Source: §A.3.2 · pp. 83, 85

Example of Idea Generation in Energy

• 2.37% MAPE and 3.79% RMSE are expected for an LSSVM–RBFNN hybrid, outperforming standalone models and simple averaging.The proposal dynamically weights model forecasts using grey-relational analysis and Weibull-segmented wind-speed regimes, aiming to reduce retraining overhead in 15-minute-ahead prediction.Source: §A.3.2 · pp. 85–87

Example of Idea Generation in Information

• 72.2 on MMMU, 79.0 on MathVista, 91.4 on RefCOCOg, 90.9% on GUI grounding, and 48.4 on VSI-Bench are expected for InternVL3-78B.The proposal uses native joint vision, language, and video pre-training, Variable Visual Position Encoding, advanced post-training, and test-time scaling; it also targets open-source reproducibility.Source: §A.3.2 · pp. 88–89

Example of Idea Generation in Life

• 0.96 Å r.m.s.d.95 median backbone accuracy on CASP14 and 1.5 Å all-atom accuracy are reported for AlphaFold, exceeding the next-best method by over 1.8 Å.The proposal jointly embeds MSAs and pairwise residue features, iteratively refines structures with Evoformer and Invariant Point Attention, and uses self-distillation and per-residue confidence estimates.Source: §A.3.2 · pp. 91–92

Idea Generation

• The MOF proposal combines AI, high-throughput platforms, and literature knowledge extraction in a closed-loop self-driving laboratory for data-driven synthesis, characterization, and inverse design.It addresses prior approaches that automated isolated steps or applied AI to isolated tasks without robust feedback, adaptation, or broad generalization.Source: §A.3.2 · p. 93
• For MOF discovery, the proposal links robotic experimentation and structured data with Transformer/LLM models and feedback-driven planning that iteratively updates strategies from outcomes.The context identifies sparse, noisy, or unstandardized data and hardware and sample-handling issues as obstacles to autonomous discovery.Source: §A.3.2 · pp. 93–94
• The SSSP proposal uses Δ-stepping to process nodes in distance buckets, relaxing light edges in parallel and deferring heavy edges to balance parallelism and efficiency.It proposes tuning Δ to obtain linear average-case work and scalable parallelism for a broad graph class, with extensions to distributed memory and arbitrary edge weights.Source: §A.3.2 · p. 96
• The EEG proposal introduces EEG Conformer, combining temporal-spatial convolutions for local features with multi-head self-attention for global dependencies in end-to-end raw-signal decoding.Class Activation Topography is proposed to map activations to brain regions and improve interpretability.Source: §A.3.2 · p. 98

Question

• The example prompt asks a researcher to generate a novel, detailed proposal from supplied scientific context.The related-work context contrasts domain-specific models with the goal of zero-shot generalization across chemical domains.Source: §A.3.2 · p. 100

Limitation

• The motivating gap is that existing MLIPs often specialize in one domain or require fine-tuning, while scaling can reduce efficiency and limited data diversity constrains universality.The proposal targets generalization across materials, molecules, and catalysis despite heterogeneous DFT settings.Source: §A.3.2 · p. 100

Reference Answer

• The reference proposal introduces UMA, a family of universal MLIPs trained on nearly 500M multi-domain structures, using MoLE to scale capacity without inference overhead.Unified embeddings and referencing support multi-DFT-task learning, with empirical scaling laws guiding model and dataset sizing.Source: §A.3.2 · p. 101

ImplementationSteps

• The proposal’s implementation aggregates and normalizes datasets, adds global charge, spin, and DFT-task embeddings, and uses staged training, scaling-law-based model selection, and cross-domain evaluation.It describes eSEN-based graph networks with MoLE layers, BF16 force pretraining, FP32 conservative-head fine-tuning, and graph parallelism, FSDP, and activation checkpointing.Source: §A.3.2 · p. 101

Data

• UMA combines five datasets covering about 459M structures and >30B atoms, with near-complete elemental coverage and varied DFT settings.The datasets span bulk materials, molecules, catalysis, molecular crystals, and MOFs.Source: §A.3.2 · p. 102

ExpectedOutcome

• UMA is expected to improve AdsorbML success rate by up to 25% and reduce OC20 adsorption-energy error by about 80% versus prior SOTA, while supporting simulations of >100k atoms without inference penalty.The expected outcome also describes reliable, energy-conserving predictions across chemical domains and accuracy that matches or surpasses specialized models in zero-shot and fine-tuned settings.Source: §A.3.2 · p. 102

A.3.3. Dry Experiment

• ZTF is presented as a pathfinder for next-generation surveys through its prototype alert stream and extensive time-domain data products.The supplied text labels the example as a dry experiment in astronomy but does not describe the main code with incomplete functions.Source: §A.3.3 · pp. 102–103
• ZTF’s 47.7-square-degree camera field supports survey speed over an order of magnitude faster than its predecessor.The camera uses 16 large-format CCDs; median image quality is approximately 2.0 arcseconds FWHM across g, r, and i bands.Source: §A.3.3 · p. 102
• ZTF’s data system processes images near real time, performing calibration and ZOGY image differencing to detect transient and variable sources.Its distributed alerts include contextual information and machine-learning-based Real-Bogus scores.Source: §A.3.3 · p. 103
• A robotic scheduler selects fields on a fixed grid with minimal dithering, while the camera enables 8.2-second readout with median noise of 10 electrons.These systems support efficient survey coverage and rapid data collection.Source: §A.3.3 · p. 103
• ZTF surveys the Northern sky at three-day cadence and the Galactic Plane nightly, typically observing twice per night in g and r bands.Their moderate depth and high cadence complement future facilities by enabling early discovery and characterization of bright transients.Source: §A.3.3 · p. 103
• The facility supports transient, variable-star, and solar-system studies, including supernova discovery, multi-messenger follow-up, asteroid tracking, and comet activity.It can detect point-like and streaked moving objects, link detections into orbits, and report them to the Minor Planet Center.Source: §A.3.3 · p. 103

Answer

• The periodogram code converts ordinary frequencies to angular frequencies, centers magnitudes, computes normalized Lomb–Scargle power, and returns the period at maximum power.It searches periods from the configured minimum to maximum and uses the power argmax to select the best period.Source: Answer · p. 109
• China’s warm-season 4MDA8 averages around 86 ppb, while high-ozone events and vegetation-exposure metrics are elevated relative to other monitored regions.The passage also reports comparable median warm-season concentrations across regions, but greater frequency and magnitude of high-ozone events in China.Source: Answer · p. 110
• Ozone exposure metrics across Chinese cities rose approximately 3.7% to over 15% annually from 2013 to 2017, despite reductions in several primary pollutants.The passage attributes the rise to complex photochemistry, including possible ozone enhancement when NOx or PM2.5 falls in VOC-sensitive eastern China, alongside rising VOC emissions and hotter, drier summers.Source: Answer · p. 110
• The findings characterize China as a global surface-ozone hotspot and call for targeted VOC controls and better understanding of chemical and meteorological influences.Hotspots include the North China Plain, Yangtze River Delta, and Pearl River Delta; ozone peaks in late spring and early summer, with some western areas also affected.Source: Answer · p. 110

Main Code with Incomplete Functions

• The main-code subsection begins with a Python 3 interpreter declaration and a UTF-8 encoding declaration.The supplied code also includes the periodogram implementation described above.Source: Answer · p. 109, Main Code with Incomplete Functions · p. 112

Answer

• The ozone-metric code filters W126 inputs to daytime hours and calculates MDA8 from rolling eight-hour means, requiring at least 18 daily observations.The shown W126 example returns 954.16 ppb-hours for constant 80 ppb data; the MDA8 rolling mean uses a minimum of six observations per window.Source: Answer · pp. 116–117

Example of Wet Experiment in Life

• Clinical studies of the anti-PD-L1 antibody found safety and tolerability across doses, manageable adverse events, and increased proliferating CD8+ T cells and IFN-𝛾.Objective responses occurred in NSCLC, melanoma, and renal cell carcinoma; response rates and progression-free survival were associated with pre-treatment PD-L1 expression on tumor-infiltrating immune cells.Source: §A.3.4 · p. 117
• Responding tumors showed increased immune infiltration, tumor necrosis, and PD-L1 and IFN-𝛾 upregulation, while non-responders showed ignorance, non-functional responses, or excluded infiltrates.Blood IFN-𝛾-inducible chemokines and activated cytotoxic T cells increased early but did not clearly distinguish responders from non-responders; the findings support pre-existing antitumor immunity as a response basis.Source: §A.3.4 · pp. 117–118
• The cancer workflow spans tissue fixation, paraffin embedding and sectioning, antibody staining and visualization, IHC scoring, gene-expression analysis, blood assays, treatment, biopsies, and response evaluation.The listed assays include ELISA, FACS, and fluorescence detection.Source: §A.3.4 · pp. 118–119

Example of Wet Experiment in Material

• −8.18 mV K−1 (𝑛-type) and 9.62 mV K−1 (𝑝-type) thermopowers were reported for the optimized I−/I−3 + 2 wt% MC + 0.3 M KCl electrolyte.At a 15 ◦C temperature difference, single 𝑛-type and 𝑝-type cells achieved maximum power outputs of 27.78 𝜇W and 80.47 𝜇W, respectively.Source: §A.3.4 · p. 120
• The thermocell workflow prepares iodide/triiodide electrolyte with methylcellulose and KCl, then controls electrode temperatures and measures voltage, current, power, and electrolyte properties.Characterization steps include UV-Vis spectroscopy, cyclic voltammetry, FTIR spectroscopy, and ionic-conductivity measurement.Source: §A.3.4 · pp. 121–122

Example of Wet Experiment in Physics

• The physics example covers digital-filter quantization noise, absolute-stability analysis of nonlinear discrete-time systems, and chaotic attractors in reciprocal circuits.Its workflow builds and configures a nonlinear circuit, simulates trajectories, generates phase portraits, identifies attractors, varies parameters, and analyzes bifurcations and periodic orbits.Source: §A.3.4 · pp. 123–124

A.3.5. Experimental Reasoning

• The astronomy example estimates chirp mass by fitting a slope to transformed time–frequency ridge points, then uses the noise-weighted integral and network SNR of 24 to solve for luminosity distance.The prompt specifies detector factor F = 1 and asks for distance in rounded Mpc.Source: §A.3.5 · pp. 126–128
• The chemistry example identifies reaction-center extraction by subgraph pattern matching and GNN compatibility scoring; successful retrosynthesis requires structural matching and reaction-mechanism alignment.The illustrated successful predictions preserve molecular topology and agree with known reaction mechanisms.Source: §A.3.5 · pp. 130–131
• Option B is correct: Pacific equatorial cooling coincides with strong El Niño, Pacific Southern Hemisphere warming begins earlier, and Indian Ocean anomalies are most variable at 20°S–40°S.The equatorial Atlantic remains mostly near zero during strong El Niño, and the Indian Ocean shows the most frequent anomaly shifts in the 20°S–40°S band.Source: §A.3.5 · pp. 134–135
• TES4 falls to about 20–30 kWh during independent operation but sustains 90–100 kWh in cooperative mode as microgrids without storage share surplus solar thermal energy.The cooperative-mode plateau is especially evident during daytime hours on Days 3–5.Source: §A.3.5 · pp. 136–137
• Option 3 is fully consistent with the coupler, 2 GHz measurements, field sources, 0.5-mm reconstructions, material coding, and magnetic-field components.The reasoning distinguishes the 4-mm measurement plane from reconstruction distance and maps image (b) to |H_x| variation and (c) to |H_γ| variation.Source: §A.3.5 · pp. 140–141

I. CD24;B;B J. CD24;C;A

• For targeting non-tumor cells, the reasoning selects Siglec-10 and siRNA strategy C as the deeper intervention, delivered by antibody-free passive nanoparticles (answer E).Strategies A and B block surface signaling directly, whereas strategy C suppresses target-protein expression at the nucleic-acid level.Source: §I · pp. 144–145

I. Li10SiP2S12 J. Li6PS5Cl

• The largest Li-ion probability-density region identifies Li10GeP2S12 as the material with the best Li-ion conductivity (answer A).The reasoning selects the material by comparing the sizes of the Li-ion probability-density regions.Source: §I · pp. 146–147

A.4. Supplementary Evaluation Results

• Deep-research supplements break down scores by category and subject for both LLMs and agents, with combined subject-wise performance also shown.The tables note that unequal question counts across categories prevent directly averaging the displayed values to obtain overall performance.Source: Figure 31 · p. 151, Table 10 · p. 152, Table 11 · p. 152
• Idea-generation supplements report subject-wise scores, while warning that unequal question counts prevent direct averaging to obtain overall performance.Subject-wise scores are presented in both a figure and a table.Source: Figure 32 · p. 151
• Wet-experiment supplements break down subject-wise performance using Action Sequence Similarity (SS) and Parameter Accuracy (PA), with unequal question counts limiting direct averaging.The subject-wise figure and table report the two wet-experiment measures.Source: Figure 33 · p. 151, Table 18 · p. 156
• Dry-experiment supplements report completion scores by function type and subject, with unequal question counts precluding direct averaging into overall performance.The category table covers six function types.Source: Table 12 · p. 153, Table 17 · p. 155
• Experimental-reasoning supplements report multi-choice accuracy by reasoning type and scientific subject, noting that unequal question counts prevent direct averaging into overall performance.Reasoning types include signal, attribute, comparative, and causal reasoning.Source: Table 13 · p. 153, Table 19 · p. 156