PaperlayerView exact revision on arXiv

Source-linked AI summary

Assessing Language Models for Scientific Applications: Inertial Confinement Fusion as a Use Case

P. B. Radha, D. D. Meyerhofer, M. Bhattarai, B. Cheng, L. Diaz, C. L. Rousculp, A. G. Seaton, G. Gupta, P. Gurumurthy, V. Gutta, A. Jhunjhunwala, A. Vem, P. Cook, G. Rowe

arXiv:2610.38024v1cs.AI

TL;DR

Broad scientific benchmarks do not establish whether language models can perform the integrated reasoning required in specialized domains such as ICF. The paper builds an expert-curated ICF benchmark and physics-aware evaluation, finding gains from specialization alongside persistent critical errors and model-dependent retrieval benefits.

  • Problem

    General scientific benchmarks do not necessarily show whether models can integrate the specialized knowledge, physical reasoning, calculations, and assumptions needed by practicing scientists.

    Source: §I · p. 2

  • Method

    The authors evaluate language models on an expert-curated 250-question ICF benchmark using physics-aware checks and blinded language-model judges, including specialized and retrieval-assisted models.

    Source: A. Blinded, physics-aware judging · p. 7, §VI · p. 17

  • Results

    Prospero improved 10.7 percentage points over its base model with gains across question types, while frontier models had the highest aggregate scores and critical physics errors persisted, especially in numerical calculations.

    Source: §VI · p. 17

  • Takeaways & Limitations

    Evaluate existing domain capabilities and retrieval benefits before substantial additional training, and retain expert assessment because useful assistance can coexist with consequential reasoning errors.

    Source: §VI · p. 17

    Benchmark gains alone do not establish scientific reliability, and retrieval benefits differed between the tested smaller and frontier models.

    Source: §VI · p. 17

Abstract

from arXiv · show

Advancing language models for specialized scientific applications requires evaluations of scientific fluency, physical reasoning, and quantitative analyses. Using inertial confinement fusion (ICF) as a test case, we have developed an expert-curated benchmark of 250 questions spanning qualitative reasoning, numerical calculation, symbolic derivation, and equation formulation. Responses have been evaluated using deterministic numerical and symbolic checks together with language-model judges. We compared a variety of out-of-the-box closed-source and open-source large language models with Prospero, a model specialized through combined domain-adaptive pretraining and supervised fine-tuning for ICF. We also conducted initial retrieval-assisted evaluations. Prospero improved its aggregate score by 10.7 percentage points over its base model, with gains across all question types. Frontier models achieved the highest aggregate scores, but critical physics errors persisted, particularly in numerical calculations. We note that domain-specific performance can improve through specialization, but benchmark gains alone do not establish scientific reliability. Retrieval improved performance for a smaller model but yielded no statistically significant improvement for a frontier model. Two scientific stress tests further illustrate useful capabilities and the need for expert assessment of physical assumptions and model validity. Based on these findings, we recommend evaluating existing domain capabilities and retrieval benefits before committing to substantial additional specialization.

I. INTRODUCTION

General scientific benchmarks do not establish whether language models can integrate the specialized knowledge and physical reasoning needed in ICF. The paper uses ICF to examine model capabilities and improvement pathways, including a heterogeneous expert-curated benchmark.

I. INTRODUCTION

  • General benchmarks such as GPQA and MMLU may not reflect whether a model can integrate specialized knowledge, quantitative calculation, physical-model selection, and assumptions required by practicing scientists.A model can perform well on broad scientific benchmarks yet fail on questions requiring integrated domain-scientist knowledge.Source: §I · p. 2
  • ICF spans interacting physics and engineering constraints, while experimental opportunities are limited and multimodal data can have uncertain uncertainties.These conditions make ICF a test of whether models can integrate knowledge across physics regimes with limited, uncertain data.Source: §I · p. 2
  • The study compares off-the-shelf models, Prospero specialized through combined DAPT and SFT, and retrieval-assisted models; the combined training requires substantial compute and expert data curation.The study does not isolate the contributions of DAPT and SFT, and proposes retrieval, scientific tools, and reinforcement learning as a future research direction.Source: §I · p. 3

II. ICF BENCHMARK

  • The expert-curated golden-set contains 250 question-and-answer pairs spanning fundamental physics through specialized high-energy-density physics and ICF topics.Two independent ICF researchers curated each pair, resolving disagreements through discussion and adding context where needed without giving away the answer.Source: §II · p. 4
  • The benchmark covers qualitative reasoning, numerical answers, symbolic derivations, and equation formulation, with question formats designed to test different forms of physics reasoning.Categories overlap because ICF processes are coupled; fuel compression, for example, depends on several aspects of the physics and appears in multiple categories.Source: §II · pp. 4–5

III. EVALUATION METHODOLOGY

The evaluation combines repeated model responses, validity checks, physics-aware equivalence criteria, and blinded judgments by three independent judges. Agreement measures and ranking stability provide checks on the judging process.

III. EVALUATION METHODOLOGY

  • Each of 22 models was evaluated by three judges across three independent attempts per question, for 16,500 planned answer calls.Three attempts were chosen as an ad hoc way to account for probabilistic model outputs; the evaluation used 10,000 bootstrap resamples for 95% confidence intervals.Source: §III · pp. 5–6
  • Responses pass a validity gate before physics checks and judging; non-valid answers receive zero, while numerical and symbolic answers receive tolerance, unit, and computer-algebra checks.Valid responses are judged blindly by three independent LLMs for correctness, causal explanation, relevance, completeness, contradictory derivations, and critical physics errors.Source: §III · p. 6, A. Blinded, physics-aware judging · p. 7

A. Blinded, physics-aware judging

  • Scoring uses form-specific physics checks because numerical, symbolic, equation, and qualitative answers can be equivalent despite differing text.A critical error caps a score at 40%, and a correct value supported by an invalid or contradictory derivation is capped at 70%; non-valid responses receive zero.Source: A. Blinded, physics-aware judging · p. 7
  • 59.8–76.0% rubric-point agreement and 82.0–87.8% critical-error agreement were observed between judge pairs.The independently rated answers also showed relatively high Pearson correlations, which the authors interpret as suggesting effective rubric use.Source: A. Blinded, physics-aware judging · p. 8, Figure 4 · p. 8
  • The three judge families produced the same top-three model order, and no model in the consensus top ten differed by more than two ranking positions.The authors also report agreement with judge assessments on 30 judgments sampled across ten categories.Source: A. Blinded, physics-aware judging · pp. 8–9

B. Domain-adaptive pretraining (DAPT) and supervised fine-tuning (SFT)

The study combined domain-adaptive pretraining (DAPT) and supervised fine-tuning (SFT) to specialize an open-weight model, tracking general-science performance during training. Broader SFT recovered performance after an intermediate decline.

B. Domain-adaptive pretraining (DAPT) and supervised fine-tuning (SFT)

  • DAPT exposed Llama-3.3-Nemotron-49B-Instruct-128K to about 1.3 million scientific documents, while SFT used 1,048,592 samples, including 180,253 ICF Physics Q&A pairs.The ICF papers used to generate SFT questions were excluded from DAPT; 450 curated ICF question-and-answer pairs were deemed rigorous and pertinent.Source: B. Domain-adaptive pretraining (DAPT)… · p. 9
  • As ICF-token continuous pretraining increased, GPQA performance declined alongside more unanswered questions, then improved after broader SFT data were restored.The figure describes an intermediate training strategy, not the final blend.Source: B. Domain-adaptive pretraining (DAPT)… · p. 10, Figure 7 · p. 10
  • Prospero achieved nearly 16% higher GPQA performance than its base model after an optimal SFT blend was identified.Similar improvements were observed for other standardized benchmarks.Source: B. Domain-adaptive pretraining (DAPT)… · p. 10

A. Aggregate model performance

Frontier models led the aggregate benchmark, while Prospero improved over its base model. Results varied across ICF categories and question types; retrieval helped the smaller model tested but did not significantly improve the frontier model.

A. Aggregate model performance

  • Claude Sonnet 5 scored 69.2, ahead of Inkling Small (65.9), GPT-5.4 (63.2), and Nemotron Ultra (58.3); Prospero improved 10.7 percentage points over its base model.The top-four intervals separate a small leading group from a long tail, but the authors caution that the results are not an ICF leaderboard.Source: Figure 8 · p. 11, A. Aggregate model performance · p. 11

B. Segmented Model Performance

  • Claude Sonnet 5 led eight of ten ICF subareas, while ICF Concepts, Hydrodynamics, and Engineering were the hardest categories for the best system in each slice.The highest leader scores were in Plasma Equations and Transport Properties, the two smallest categories, with 14 and 11 questions respectively.Source: B. Segmented Model Performance · pp. 11–12, Figure 9 · p. 12
  • Numerical calculations were the hardest question type for every model, with critical errors in 58–71% of numerical responses; Prospero exceeded its base on all question types.Prospero’s gains were +18.4 on qualitative questions, +11.3 on equation formulation, +6.4 on symbolic reasoning, and +4.4 on numerical questions.Source: B. Segmented Model Performance · p. 12, Figure 10 · p. 13

C. Detailed performance of Prospero versus the base model

D. Retrieval helps when it fills a knowledge gap

  • Gemma 4’s mean rose from 34.7 closed-book to 48.5 with URSA and 49.4 with Open Deep Research, while GPT-5.4’s retrieval variations were not statistically significant.For Gemma 4, the difference between retrieval systems was 0.94 points, with a paired 95% interval spanning −3.70 to +5.80.Source: Figure 12 · p. 14, D. Retrieval helps when it fills a… · p. 14

V. DISCUSSION

The discussion identifies limits in benchmark coverage and objectivity, alongside practical costs and unresolved questions around domain adaptation. It also describes provenance and continual-learning challenges for scientific models.

A. Limits of a domain benchmark

  • The golden-set covers only a small fraction of ICF phenomena, regimes, and questions, while reducing the multiphysics domain to Q&A pairs and expert-authored questions may bias coverage toward established knowledge.Qualitative questions also make objective scoring more challenging.Source: A. Limits of a domain benchmark · p. 14
  • Stress-test examples show that useful scientific assistance can require collating references, multiphysics calculations, and tools such as visual inspection, Python, and symbolic manipulation, with subject-matter expertise.The discussion concludes that effective use of LLMs for science requires subject-matter expertise.Source: A. Limits of a domain benchmark · p. 15

B. Benefits and costs of domain adaptation and fine-tuning

  • Training on peer-reviewed literature introduces publication bias toward positive results and underrepresents null results and experimental failures that may help identify knowledge gaps.These underrepresented experiments may contain valuable information for scientific discovery.Source: B. Benefits and costs of domain… · p. 15
  • Because DAPT and SFT were not studied separately, DAPT’s value without an SFT-only comparison remains unknown, and its added compute may not be justified for an already domain-aware model.Performance declined during an intermediate training run but recovered with supervised fine-tuning.Source: B. Benefits and costs of domain… · p. 15
  • DAPT makes claims difficult to trace to their source, while slow experimental data generation and evolving regimes complicate updates without losing scientifically valid knowledge.Earlier lessons may not hold under new conditions, and new experiments can challenge existing understanding.Source: B. Benefits and costs of domain… · p. 15

C. Retrieval and/or reinforcement learning versus weight adaptation

The discussion weighs retrieval against weight adaptation and argues for systems that combine evidence, scientific tools, expert review, and validation. The conclusions recommend assessing existing capabilities and retrieval benefits before substantial additional training.

C. Retrieval and/or reinforcement learning versus weight adaptation

  • Language models could support scientific work from hypothesis generation through post-shot analysis, but the central question is whether they can guide experiments that yield new information or improve results.The discussion frames this as a question about using the available evidence.Source: C. Retrieval and/or reinforcement… · p. 16
  • For ICF, retrieval requires multimodal sources such as articles, shot reports, target specifications, diagnostic calibrations, and simulation results, and can preserve provenance better than weight adaptation.Retrieval-augmented generation supplies relevant information at inference time.Source: C. Retrieval and/or reinforcement… · p. 16
  • Retrieval may be most useful for smaller or domain-limited models, but it cannot resolve inconsistent evidence, including poorly documented null results and failed shots.Physicists must judge whether retrieved evidence is relevant and mutually consistent.Source: C. Retrieval and/or reinforcement… · p. 16
  • The authors did not evaluate reinforcement learning and propose a future multi-agent pipeline to retrieve shot records, run physics calculations, and revise target recommendations against a defined goal.They note that predictions can be checked on existing facilities.Source: C. Retrieval and/or reinforcement… · pp. 16–17

VI. CONCLUSIONS

  • The conclusions report that Prospero gained 10.7 percentage points over its base, yet frontier models retained critical physics errors and benchmark gains did not establish scientific reliability.The authors recommend assessing domain capabilities, failure modes, and retrieval benefits before substantial additional training.Source: §VI · p. 17

Appendix A: Prospero performance comparison with base model

The appendix compares selected Prospero answers with its base model, showing both substantial improvement and clear regressions across ICF problems. The examples span scaling, force calculations, fusion reactivity, plasma instabilities, and other physics tasks.

1. Examples of improved performance of Prospero compared to base

  • 67.6% of questions had improved answers, with the base usually incorrect and Prospero succeeding in all three attempts.The examples were randomly selected from this group.Source: §1 · p. 22
  • Prospero correctly canceled the common inverse-square dependence for the proton force ratio and recognized that the D-He3 to D-T reactivity-peak ratio is the cube root of four.For the fusion comparison, it recognized approximately equal reduced masses, leaving only the charge factor to change.Source: §1 · p. 22
  • Prospero identified Stimulated Raman Scattering, Stimulated Brillouin Scattering, and Two Plasmon Decay as the three-wave instabilities and described their daughter waves.The base substituted four-wave mixing for Two Plasmon Decay.Source: §1 · p. 22
  • Prospero applied hydrodynamic similarity to scale target radius, shell thickness, and pulse duration by the cube root of the energy ratio.The base treated the problem as underdetermined and proposed a square-root radius scaling under an unsupported fixed-pulse assumption.Source: §1 · p. 23

2. Examples of Prospero regression compared to base

  • 21.2% of questions had poor answers, with Prospero introducing physics or units errors where the base was usually correct.The examples were randomly selected from this regression group.Source: §2 · p. 23
  • Prospero’s regressions included unstable collision-rate derivations instead of the base’s Nv/4 and loss of the logarithmic mass dependence in shell-velocity attempts.For shell ablation, the base obtained u = uexln(M0/M) in two attempts, while Prospero double-counted momentum in one attempt and lost the logarithmic dependence in another.Source: §2 · p. 23
  • Prospero’s preheat calculation regression left the answer unit unspecified, suggested MJ, and included a critical numerical error, unlike the base’s approximately correct 42 J.The question specified the units, so the answer unit should have been definable.Source: §2 · p. 24
  • The Rayleigh–Taylor stress test separates formula recall from model selection by requiring regime recognition, dimensional checks, and late-time amplitude calculation.The appropriate response must identify when growth becomes nonlinear and calculate the late-time amplitude.Source: §2 · p. 24

1. Analytic solution

This section presents linear Rayleigh–Taylor interface growth and piecewise alternatives for the transition to nonlinear growth. The piecewise approaches hold the amplitude growth velocity constant after the transition, with the figure comparing three modeled solutions.

1. Analytic solution

  • For an inviscid, incompressible single Fourier mode, the section considers the linearized interface displacement and assumes the interface is initially at rest.The initial condition is ˙η(0) = 0.Source: §1 · p. 25
  • The piecewise models transition from linear growth to constant-velocity nonlinear growth at a transition time, with two approaches differing in how that time is defined.One approach matches the transition time at which bubble velocities match.Source: §1 · p. 25
  • Fig. 13 compares the purely hyperbolic-cosine solution with two piecewise methods using different terminal bubble velocities.The alternatives use v(t0.1) and vNL, respectively.Source: Figure 13 · p. 26, §1 · p. 26
  • For the Fig. 13 piecewise model, the threshold ηs/λ = 0.1 gives ts ≈ 1.86 ns for the stated parameters, after which velocity is held constant at its value at ts.The threshold corresponds to kηs = 0.2π ≈ 0.63.Source: §1 · p. 26

2. Summary of model responses

The strongest responses handled linear Rayleigh–Taylor growth but miscalculated the nonlinear amplitude; other answers made regime, wavelength, acceleration, or unit errors. Retrieval improved discussion of the linear-to-nonlinear boundary without retrieving the answer, while an expanded reasoning setup could reverse-engineer a solution.

2. Summary of model responses

  • Other responses either used linear solutions at both times or made wavelength, acceleration, or time-unit errors, sometimes producing significantly incorrect amplitudes.The results indicate that models are ICF-aware but still require expert review.Source: §2 · p. 27
  • None of the top four models correctly calculated the nonlinear amplitude, although some flagged nonlinearity and the strongest responses warned that the 3.3 ns extrapolation exceeded linear theory.Those responses derived linear growth correctly but calculated the extrapolated value incorrectly.Source: Table IV · p. 27, §2 · p. 27
  • Retrieval improved discussion of the linear-to-nonlinear boundary but did not retrieve the answer; an expanded reasoning setup reverse-engineered a solution using a different bubble-velocity model.The alternative prompt did not provide the references motivating that model.Source: §2 · pp. 27–28

Appendix C: Capability assessment of LLMs at systematic physicist-like derivation and thinking

The p–¹¹B stress test asks models to assess when self-sustaining ignition is possible using supplied references, coupled physics, analytic constraints, and executed numerical work. It requires verification, optimization, sensitivity analysis, and an explicit distinction between necessary ignition criteria and sufficient conditions.

Appendix C: Capability assessment of LLMs at systematic physicist-like derivation and thinking

  • The task asks when a self-sustaining p–¹¹B plasma state exists, using supplied references to formulate ignition equations and solve coupled plasma variables without prescribing absolute density.It asks models to determine whether density enters the result and what additional quantities must be specified.Source: Appendix C · p. 28
  • Models must compare numerical results with analytic constraints, investigate disagreements, and verify executed scripts against references rather than relying on untested calculations.The task also requests sensitivity plots and identification of assumptions and uncertain inputs affecting the inferred boundary.Source: Appendix C · p. 28
  • The requested deliverables include a derivation, constraints, code, machine-readable results, and figures, while treating the analytic ignition criterion as necessary but not automatically sufficient.If the nominal model lacks physical solutions, the prompt asks for the smallest meaningful modification and its admissible range and effect on optimized conditions.Source: Appendix C · p. 28
  • The deliberately vague prompt requires deriving the ignition criterion, retrieving and checking equations, calculating reactivity, deriving an areal-density constraint, selecting optimization variables, and writing and debugging Python.It also calls for parameter optimization and sensitivity analyses.Source: Appendix C · p. 29

1. Analytic solution

The analytic model frames ignition as an alpha-heating power balance against radiation and conduction losses. It expresses alpha power using reactant densities and reactivity, then incorporates radiation and heat-conduction loss terms.

1. Analytic solution

  • The ignition criterion requires deposited alpha-particle power to exceed radiation and heat-conduction losses.The criterion is attributed to Ref..Source: §1 · p. 29
  • Alpha power depends on the deposited-energy fraction, proton and boron densities, reaction Q-value, and fusion reactivity, then is rewritten after substituting for the species densities.The reaction Q-value is 8.58 MeV.Source: §1 · p. 29
  • The loss model includes non-relativistic radiation from Ref. and hot-spot heat conduction, with the conduction coefficient proportional to T^5/2/lnΛ.The plasma definitions include electron temperature and density, ionization, and ion density; the equations are substituted and rearranged into a constraint.Source: §1 · pp. 29–30

2. Model results

In the p–¹¹B study, GPT-5.6Sol retrieved and combined reference information to derive conditional ignition results, with sensitivity checks and expert review. The assessment also identifies modeling limits, including an unimplemented radiation-recycling mechanism and omitted self-consistent electron-ion collision calculations.

2. Model results

  • 2.44% was the maximum discrepancy between the model’s calculated fusion reactivity and its visual check against the reference figure.The model converted proton energies to the center-of-mass frame and integrated over the Maxwellian without being explicitly instructed to do so.Source: §2 · p. 30, Figure 14 · p. 31
  • For the analytically optimal boron fraction xB=0.0821, the script identified solutions across fusion-to-radiation power ratios; Fig. 15 shows no solution at higher electron-to-ion temperature ratios.The model also identified the requirement that the denominator be positive.Source: §2 · p. 30, Figure 15 · p. 31
  • The model flagged relativistic electrons and applied a multiplier to radiation losses, while identifying that radiation recycling would lower the threshold without specifying how to implement it.The radiation-loss and recycling multipliers are functionally equivalent, though the model treated them separately to illustrate the concepts.Source: §2 · p. 31
  • The ignition threshold was conditional because self-consistent electron-ion collision calculations were omitted, and sensitivity analysis covered alpha stopping power, radiation losses, and fusion reactivity.ICF scientists checked consistency with the assumptions supplied to the model.Source: §2 · pp. 31–32, Figure 16 · p. 32
  • GPT-5.6Sol was judged plausibly capable of systematic derivation, scripting, execution, and identifying limiting assumptions, but scientists still needed to assess its dense output.The authors describe the example as illustrating a minimal tool set for assisting scientists.Source: §2 · pp. 32–33

Related papers

Loading 2610.38024v1…