Physics as the label for measuring and correcting materials reasoning in multimodal models

1 Introduction

• Across materials tasks, models are reported to hallucinate invalid properties, while existing evaluations score final answers and do not measure reasoning-chain consistency against materials physics.Related approaches verify final proposals, DFT execution, visual grounding, or generic programs, rather than the physical consistency of materials reasoning chains.Source: §1 · pp. 2–3
• Materials data supplies checkable constraints, which MatPCR turns into a label-free benchmark with physical oracles, PCR, CGSV, and a reusable fine-tuned verifier.The stated scope includes image inputs for VLMs and structure or text inputs for LLMs.Source: §1 · p. 3
• Unlike answer-scoring materials benchmarks, physical-law evaluations, discovery agents, and general verifiers, MatPCR checks intermediate-chain consistency with materials-specific, DFT-grounded oracles.The paper distinguishes it from contemporaneous diffraction and DFT efforts that use final-answer labels or exact-match rewards rather than chain-level evaluation.Source: §1 · p. 3
• MatPCR differs from the authors’ earlier resources by measuring physical consistency of reasoning chains and acting on flagged violations, rather than scoring datasets, predictions, or text factuality.The earlier work covers crystalline-material resources, nanomaterial property prediction, hallucination benchmarks, and physics-grounded dynamic-systems benchmarks.Source: §1 · p. 4

2.1 Oracle validity: positive control and precision audit

• All six oracle families recovered every planted violation, with a maximum false-negative rate of 0.0000; audited precision ranged from 77.2% for XRD to 95.3% for stability.After revision, the Raman oracle’s audited precision was 92.7% across 495 fired cases; the magnetization check was also changed to use an absolute-tolerance zero check.Source: §2.1 · p. 5, Table 1 · p. 7

2.2 Physical consistency of current models

• Overall PCR ranges from 0.555 (Qwen3-VL-8B-Instruct) to 0.774 (GPT-5.6-Terra), with microscopy rates of 0.67–1.00 and spectra rates of 0.07–0.83.The evaluation covers nine models and six constraint types; the pooled DFT column can conceal differences among its three constraint types.Source: §2.2 · p. 7, Table 2 · p. 8, Figure 3 · p. 13

2.3 Accuracy versus consistency

• Accuracy and PCR disagree on a mean of 22.2% of items across 12 model–constraint cells, so PCR is not answer accuracy under another name.Diffraction often yields wrong-but-consistent claims under its looser tolerance, while stability comparisons are affected by disagreement between the near-hull threshold and Materials Project’s is_stable label.Source: §2.3 · pp. 8–9

2.4 Effect of self-verification

• CGSV significantly raises pooled PCR for all four models and has no regressions, while unaided self-refinement and equal-compute re-prompting show no significant change.For GPT-4o, PCR rises from 0.434 to 0.593 (p = 7.6 × 10^-6); both matched controls show nonzero regressions.Source: §2.4 · pp. 9–10, Table 3 · p. 11, Figure 3 · p. 13
• After CGSV, PCR gains do not establish improved physical correctness: post-CGSV accuracy is 29.8% on diffraction and 9.9% on spectra.Each item exposed one checkable claim before and after revision, ruling out claim removal as the source of the gain; the results remain consistent with retreat to values the oracle accepts.Source: §2.4 · p. 10

2.5 Verifier for violation prediction

• The verifier separates violations in distribution (AUC = 0.850), but no held-out constraint type shows transfer that survives correction for six comparisons.Metallicity’s nominal interval excludes chance before correction, but not after it; its corrected interval is [0.487, 0.673].Source: §2.5 · p. 11, Table 4 · p. 12, Figure 3 · p. 13

3 Discussion

• Overall PCR ranges from 0.555 to 0.774 across nine models, while spectra and diffraction remain hard and highly model-dependent.The reported range rules out a benchmark where every evaluated model already satisfies the checked physics.Source: §3 · p. 14
• Accuracy and PCR disagree on a mean 22.2% of items across 12 model–constraint cells, showing that PCR is not answer accuracy under another name.Diffraction can be oracle-consistent while measurably wrong under a tighter accuracy criterion; stability can also be correct under one definition yet flagged under another.Source: §3 · p. 14
• CGSV significantly raises PCR for all four tested models and never regresses a previously consistent item; unaided refinement and equal-compute re-prompting do not reproduce the gain.The authors attribute the gain to the oracle signal rather than extra inference-time compute alone.Source: §3 · p. 14
• Five of six held-out constraint types are indistinguishable from chance after correction, so the verifier’s in-distribution performance does not establish cross-type transfer.The nominal metallicity exception does not survive correction for six comparisons.Source: §3 · pp. 14–15
• PCR is a necessary-not-sufficient reliability signal because the oracles cover only physically checkable errors, and the verifier predicts oracle signals rather than first-principles ground truth.GPT-5.6-Terra improves over GPT-4o (0.774 vs 0.687), but release date and marketed capability tier are not reliable proxies for physical-consistency performance in this sample.Source: §3 · pp. 14–15

4.1 Data sources

• The benchmark draws on experimental diffraction patterns, public SEM images and Raman spectra, and public first-principles data; reported DFT references come from Materials Project, not author-run calculations.SimXRD-4M was excluded because its schema could not be resolved confidently.Source: §4.1 · p. 15

4.2 Tasks, modalities, and exposed constraints

• Items pair an input with a question requiring at least one checkable quantity, spanning image inputs for VLMs and structure or text inputs for LLMs.Examples include micrographs and diffraction images, or crystal structures paired with property questions.Source: §4.2 · p. 16
• The evaluation parses reasoning chains for stated quantities, such as peak positions, scale, feature sizes, stability claims, or property values, and tests them against matching oracles.An item is checkable when its chain exposes at least one such quantity.Source: §4.2 · p. 16

4.3 Programmatic geometric oracles

• Diffraction claims are converted from 2θ to d-spacing using Bragg’s law, then checked against the stated structure’s reference peaks and the realizable 2θ window.Claimed peaks must fall within tolerance of the reference pattern.Source: §4.3 · p. 16
• For microscopy, the oracle derives nanometres per pixel from the detected scale bar and rejects feature sizes larger than the field of view.Source: §4.3 · p. 16
• A spectral claim passes only when a nearby genuine interior local maximum exceeds a prominence threshold relative to signal dynamic range.The criterion was revised after a precision audit found a windowed-maximum check too permissive on noisy, baseline-drifting Raman spectra.Source: §4.3 · p. 16

4.4 DFT-grounded property oracles

• Stability claims use energy above the convex hull, with E_hull ≤ 0.05 eV atom−1 treated as consistent with a stable claim under the benchmark’s operational near-hull criterion.The threshold is a screening criterion, not strict thermodynamic stability; on 3,000 sampled entries it admitted 571 structures labeled unstable by Materials Project as near-hull.Source: §4.4 · p. 16
• Computed band gaps ≤0.05 eV are classified as zero/near-zero, gaps ≥0.20 eV as nonzero, and the intervening range is abstained from and excluded from PCR.The thresholds were fixed before evaluation, and the abstention band excluded 125 of 3,000 sampled materials in positive-control testing.Source: §4.4 · pp. 16–17
• All three property checks use Materials Project’s standardized computational references, designed around known DFT errors rather than treated as experimental ground truth.Positive-control testing reported 0% false-negative and false-positive rates for the property thresholds and stability sensitivity grid on real Materials Project structures.Source: §4.4 · pp. 16–17
• The magnetism oracle classifies absolute total magnetization per atom using a 0.05 μB atom−1 threshold, then checks magnitude within one order of magnitude only for finite-net-magnetization cases.A near-zero net value does not rule out antiferromagnetic or compensated ferrimagnetic ordering.Source: §4.4 · p. 17

4.5 The Physical-Consistency Rate

• PCR is the fraction of checkable items with zero oracle-detected violations.An item is checkable when it exposes at least one physically checkable constraint.Source: §4.5 · p. 17
• The benchmark exposes exactly one claim per item, so each constraint type’s violation rate equals 1 − PCR.The general rate is retained for chains that expose multiple claims.Source: §4.5 · p. 17
• PCR is distinct from answer accuracy: a model can be accurate yet inconsistent, or consistent yet wrong.The gap between the two measures is also reported.Source: §4.5 · p. 17

4.6 Constraint-Grounded Self-Verification

• CGSV presents oracle-detected violations to the model and lets it revise its answer for up to K rounds or until no violation remains.The correction signal comes from the oracle rather than unaided model self-verification.Source: §4.6 · pp. 17, 19
• The loop treats PCR gain as a positive control for whether the oracles capture real errors, and evaluates the change with a paired test.Revisions may introduce new violations, which are measured rather than assumed away.Source: §4.6 · pp. 17, 19
• Materials Project-grounded checks compare model stability, band-gap, and net-magnetization claims with computed references.The examples include a structure far above the 0.05 eV atom−1 operational near-hull threshold; the magnetization check does not determine complete magnetic ordering.Source: Figure 4 · p. 18

4.7 Released physical-consistency verifier

• The verifier predicts whether a physical violation is present from the input and chain when an oracle’s reference data are unavailable at inference.Training labels come from the oracles, without human annotation.Source: §4.7 · p. 19
• Its labels are deterministic physics signals rather than human-preference or model-judged correctness labels.Where a reference exists, the oracle is hard to game through surface-form changes.Source: §4.7 · p. 19
• The released verifier is evaluated against held-out oracle judgments using precision, recall, F1, ROC AUC, and calibration.Source: §4.7 · p. 19

4.8 What oracle error does to the reported rate

• PCR − PCR⋆ = τ(1 − d̂) − (1 − τ)f̂: missed violations and false alarms displace measured PCR in opposite directions.Here τ is the true inconsistency rate; d̂ and f̂ are oracle recall and false-alarm rate.Source: §4.8 · p. 19
• The oracle-validity gate uses planted violations at 10%, 30%, and 50%, audits precision on fired cases, and precedes model evaluation.The precision audit estimates false-discovery rate, not the false-alarm rate; estimating the latter when recall is imperfect also requires sampling oracle-silent items.Source: §4.8 · pp. 20–21
• 22.2% mean disagreement across twelve model–constraint-type cells shows that PCR and answer accuracy are distinct axes.The figure reports the percentage of items where the two axes differ across four evaluated models.Source: Figure 5 · p. 20
• With complete detection, PCR cannot overstate true consistency; with a sound oracle, PCR cannot understate it, and the absolute error is bounded by (1 − d̂) + f̂.Source: §4.8 · p. 21
• Source-level percentile bootstrap intervals capture sampling uncertainty separately from exact fixed-item oracle bias.The verifier’s substitution guarantee adds measurable oracle–verifier disagreement and is presented for in-distribution use, not transfer across constraint types.Source: §4.8 · p. 21

Model coverage

• The later-generation refresh adds Claude Opus 4.6, GPT-5.6-Terra, Qwen3-VL-8B-Instruct, and Qwen3-VL-8B-Thinking, while the original pass includes one open VLM and three API models.The refresh was one to three provider releases behind the newest models available at collection; a third pass added flagship GPT-5.6-Sol to test it against GPT-5.6-Terra.Source: §4.9 · pp. 21–22
• MatterChat was evaluated on three DFT constraint types using 120 shuffled Materials Project structures, and no domain-specific model was evaluated on the image arms.MicroscopyGPT's released interface lacks scale-bar reading and feature-size estimation, so it was not a fit for that image task.Source: §4.9 · p. 22

Experimental protocol

• Each model-item condition uses three seeds, with PCR confidence intervals from 10,000 bootstrap resamples; CGSV uses the same decoding and two rounds.The local model uses greedy decoding, while API models use provider defaults; paired before/after comparisons use exact McNemar tests.Source: §4.9 · p. 22

Domain-specific baseline

• MatterChat's PCR is 0.783 for stability, 0.795 for metallicity, and 0.850 for magnetism on DFT inputs, but image arms lack a domain-specific evaluee.The authors describe this as a coverage gap; MicroscopyGPT's task and interface do not support scale-bar reading or feature-size estimation.Source: §4.10 · p. 23

Verifier baselines

• The fine-tuned verifier reaches in-distribution AUC 0.850, but it, the LLM judge, and self-confidence are not reliable held-out detectors across six constraint types.The LLM judge has in-distribution AUC 0.447 and self-confidence AUC 0.315; held-out splits are weak, and severe miscalibration makes the verifier's default threshold unreliable there.Source: §4.10 · pp. 23–24

Ablations and robustness

• Diffraction model rankings are sensitive to both tolerance and prompt wording, while spectra rankings are more robust and stability rankings are mixed.Across tested tolerances, diffraction rank agreement with the default is 0.0; under independent prompt wording, diffraction rank agreement is 0.5, spectra 0.833, and stability 0.667.Source: §4.10 · pp. 24–25

Human agreement and error analysis

• The precision audits use a single annotator, and error taxonomies draw only on human-confirmed diffraction and microscopy violations.The reported dominant errors are gridline or scale misreading for diffraction and scale-bar unit or conversion errors for microscopy.Source: §4.10 · p. 25

Contamination and statistics

• Across 2,152 verifier records, no chains state source-specific metadata they were not shown; most model–dataset pairs cannot be cleared of possible contamination using published training cutoffs.The six exact McNemar tests were corrected directly, and the four CGSV effects remain significant after correction.Source: §4.10 · pp. 25–26

Self-verification controls

• CGSV's gains survive Holm and Benjamini–Hochberg correction, while unaided self-refinement and equal-compute re-prompting remain nonsignificant controls.The controls show no significant change: PCR 0.349 → 0.339 (p = 0.615) and 0.358 → 0.342 (p = 0.20); all four CGSV effects survive Holm correction with adjusted p < 10^-3.Source: §4.10 · pp. 24, 26

• Supplementary Note 1 presents standard derivations for the article's oracles and estimators, while Supplementary Table S1 reports per-cell PCR with bootstrap confidence intervals.Source: Supplementary Information · p. 31

Supplementary Note 1: standard derivations

• The supplement identifies its derivations as standard background included for completeness.Source: Supplementary Note 1: standard… · p. 31

1.1 Bragg’s law and 𝑑-spacing

• The diffraction oracle uses first-order Bragg geometry to map a claimed 2θ peak to spacing, rejecting impossible spacings and angles outside the instrument range.For measured angle 2θ, θ is half that angle; physical realizability requires d ≥ λ/2.Source: §1.1 · p. 31

1.2 Scale-bar calibration

• The microscopy oracle converts scale-bar pixels into nanometres per pixel and uses image dimensions to upper-bound reported feature size.Source: §1.2 · p. 31

1.3 Convex-hull stability

• The near-hull oracle treats a phase as stable at E_hull = 0 and within tolerance when E_hull ≤ 0.05 eV atom−1.Source: §1.3 · p. 31

1.4 Percentile bootstrap confidence intervals

• Percentile bootstrap intervals resample items with replacement, or parent sources when items share a source, and report empirical tail quantiles.The interval is consistent as the number of independent items and resamples grows.Source: §1.4 · p. 31

1.5 Concentration of the empirical rate

• Hoeffding’s inequality bounds sampling error for the empirical PCR using independent bounded item outcomes, with the number of independent sources substituted when sources are shared.Source: §1.5 · p. 31

1.6 Exact McNemar test

• The exact McNemar test compares paired before-and-after consistency using discordant items, classified as regressions or fixes.Under the null, each discordant item's direction is equally likely.Source: §1.6 · p. 32

1.7 Distribution-free bound on the total error of a reported rate

• The total-error bound combines oracle-bias and sampling-error bounds, but its independence assumption is an idealization and its sampling term is looser than the reported bootstrap intervals at n = 90.The supplement also presents oracle-audit precision of 77.2–95.3% and a per-cell PCR table with bootstrap 95% intervals.Source: §1.7 · p. 32, Supplementary Figure S1 · pp. 33–34, Table S1 · p. 34