PaperlayerView exact revision on arXiv

Source-linked AI summary

Quantifying out-of-distribution data severity with Wasserstein distances: benchmarking large language models for alloy property prediction

YuJie Liu, Dezhen Xue, Jun Sun, Xiangdong Ding, Zhibin Gao, Turab Lookman

arXiv:2611.03767v1cs.LGcond-mat.mtrl-sci

TL;DR

The paper asks whether general-purpose LLMs can rival domain-specific models for alloy-property prediction. It evaluates LLMs across alloy datasets and combines numerical priors with similarity-retrieved examples. The LLMs match or often outperform domain-specific models, while high-similarity references can deliver comparable performance with a small fraction of the data.

  • Problem

    Whether general-purpose LLMs can rival materials-specific models for alloy-property prediction remains an open question, particularly given data scarcity and alloy complexity.

    Source: §1 · p. 2

  • Method

    The study benchmarks general-purpose LLMs across alloy datasets and guides them with numerical priors and composition- and processing-similar reference examples.

    Source: §1 · p. 2, §2.1 · p. 4

  • Results

    General-purpose LLMs match or often outperform domain-specific models, and 5% similarity-selected references achieve accuracy comparable to much larger random-sampling datasets.

    Source: §1 · p. 3, §2.2 · p. 6

  • Takeaways & Limitations

    The findings support combining general LLMs with numerical baselines and relevant reference data for alloy-property prediction.

    Source: §3 · p. 17

    Heterogeneous steel text descriptions hinder Guided Mode performance, limiting results where processing information is inconsistent.

    Source: §3 · pp. 17–18

Abstract

from arXiv · show

As general-purpose Large Language Models (LLMs), such as Gemini and DeepSeek, exhibit increasingly advanced reasoning capabilities, a critical question arises: how do their property prediction abilities compare to those of domain-specific pretrained materials models? In this study, we evaluate predictions for Ti, steels, high-entropy alloy (HEA) and Al, demonstrating that general-purpose LLMs surprisingly outperform SteelBERT, MatSciBERT, and SciBERT. To further enhance performance, we introduce a hybrid approach that utilizes numerical priors from a prior model, such as CatBoost or SteelBERT, to guide the general LLMs (Deepseek V3.1, Gemini-2.5-Flash). By inputting these initial estimates from a prior model as prompts, the LLM acts as a reasoning corrector, achieving accuracy superior to the prior model and "unguided mode". Furthermore, we investigate the impact of data strategy using a similarity-based metric, revealing that data quality dominates quantity; using merely 5% of high-similarity reference data of training set as input yields performance similar to the results of using 100% of training set. Additionally, the LLMs demonstrate excellent uncertainty calibration, where self-reported "high" confidence strongly correlates with minimal deviation from ground truth. Our findings suggest that synergizing general-purpose LLMs with prior model numerical baselines and high-quality data curation creates a cost-effective and accurate paradigm for materials discovery, reducing the need for expensive domain-specific pretraining.

1 Introduction

The Introduction asks whether general-purpose LLMs can rival or surpass domain-specific materials models in alloy-property prediction. It presents guided numerical priors and similarity-based reference selection as ways to improve prediction, while noting remaining limitations in output consistency and heterogeneous data.

1 Introduction

  • The Introduction describes domain-specific pretrained models, including SteelBERT, MatSciBERT, and SciBERT, as tools for materials knowledge encoding and downstream prediction tasks.The broader approach has emphasized extensive specialized literature, fine-tuning, and retrieval-augmented generation.Source: §1 · pp. 1–2
  • The study asks whether general-purpose LLMs can rival or surpass materials-property models, given data scarcity and the compositional and processing complexity of metallic alloys.Prior context-only LLM regression can suffer from mode collapse, arithmetic weakness, and difficulty extrapolating.Source: §1 · p. 2
  • The work compares general-purpose LLMs with pretrained materials models across Ti, HEA, Al, and steel datasets, and introduces a guided mode using numerical priors as prompts.The LLM is positioned as a reasoning-based corrector rather than a standalone calculator.Source: §1 · p. 2
  • General-purpose LLMs match or often outperform domain-specific models, with Ti-alloy UTS R² up to 0.886 and elongation R² = 0.798.The Introduction also reports that higher self-reported confidence is associated with a larger proportion of low-error predictions.Source: §1 · p. 3
  • LLM outputs remain stochastic even at zero temperature; heterogeneous steel descriptions can also reduce performance, and confidence calibration relies on prompt engineering that may not generalize.The authors point to standardized data formats as a need for heterogeneous datasets.Source: §1 · p. 3

2 Results

General-purpose LLMs outperform or rival traditional and domain-specific baselines on alloy property prediction, while similarity-based retrieval and numerical priors can improve predictions. Their self-reported confidence tracks accuracy, though reproducibility and steel processing descriptions remain challenges.

2.1 Hybrid framework for Materials properties by few-shot prompts

  • The framework retrieves composition- and processing-similar samples, optionally supplies a prior-model estimate, and prompts the LLM to assess plausibility and correct predictions.Its reasoning protocol establishes a reference baseline, checks metallurgical constraints, then adjusts estimates through interpolation or limited extrapolation.Source: §2.1 · pp. 4–5

2.2 Data quality dominates quantity for LLM context

  • Using 5% semantically similar references yields R² > 0.8 for Ti and RMSE < 65 MPa for Al, comparable to random sampling with the full dataset.Retrieved examples cluster near target samples in embedding space and their UTS distributions can align with the target’s ground truth.Source: Figure 2 · p. 6, §2.2 · pp. 6–8

2.3 Reproducibility with LLMs

  • Gemini-2.5-Pro exceeds standard MLP baselines, with R² 0.743–0.772 versus 0.667–0.691, but is less reproducible than Gemini-2.5-Flash across three runs.At zero temperature, Flash gives identical outputs for 122 of 136 samples; Pro is fully consistent for 34 and varies on 102.Source: Figure 4 · p. 8, §2.3 · pp. 9–10

2.4 Comparative Assessment: Materials Specific vs. General-Purpose LLMs

  • Gemini-2.5-Pro reaches R² 0.886 for Ti-alloy UTS versus 0.835 for traditional ML, and R² 0.798 for elongation versus 0.651.Across other alloy tasks, general LLMs show similar trends, although the best LLM can have lower R² than traditional models while achieving lower MAE.Source: Figure 5 · p. 10, §2.4 · pp. 11–12, Figure 6 · p. 11

Enhancement of LLM performance using initial model estimates

  • For Ti UTS, Guided Gemini-2.5-Pro at 5% retrieval lowers MAE from 87.72 to 73.48 MPa and raises R² from 0.853 to 0.889 versus Unguided mode.Five-percent retrieval is comparable to the full dataset in most tested cases; gains are lower for steel, whose processing descriptions are less structured.Source: Figure 7 · p. 12, Enhancement of LLM performance using… · p. 14

2.5 “Confidence” from LLMs

  • High-confidence predictions consistently have higher accuracy than medium-confidence predictions, with most high-confidence samples within 0–5% relative error.In HEA, multi-target learning attains UTS R² = 0.783 and performs comparably to or better than single-target prediction in several cases.Source: Figure 9 · p. 15, §2.5 · pp. 15–16

3 Discussion

The discussion argues that general-purpose LLM reasoning can rival specialized models and that guided prediction can benefit from relevant reference data. It also highlights confidence calibration, model-generation gains, and a steel-data limitation.

3 Discussion

  • Figure 10 compares Gemini-3.0-Flash and Gemini-2.5-Flash on steel UTS and elongation, grouping predictions by reported confidence and showing relative-error distributions.The supplied discussion reports improved performance with Gemini-3.0-Flash but does not provide the figure's numerical values.Source: Figure 10 · p. 16, §3 · p. 17
  • Guided Mode uses a numerical prior to anchor the LLM as a reasoning corrector, with the hybrid approach reported to capture microstructural dependencies often missed by regressors.The discussion frames nuanced, physics-informed reasoning over relevant context as a bottleneck distinct from domain-specific pretraining.Source: §3 · pp. 16–17
  • Using only 5% of a dataset can achieve state-of-the-art accuracy in the reported similarity-based retrieval analysis, motivating active retrieval over exhaustive heterogeneous data collection.The discussion frames high-fidelity experimental data as scarce and expensive.Source: §3 · p. 17
  • Gemini-2.5-Pro is described as more accurate but less deterministic than Flash, prompting a consensus-based inference strategy; confidence correlates with error and can flag potential hallucinations on novel materials.The paper attributes output variability to inference computations including floating-point rounding and distributed execution differences.Source: §3 · p. 17
  • Gemini-3.0-Flash improves steel predictions over Gemini-2.5-Flash, with higher R² and lower MAE for both UTS and elongation, especially at high confidence.Self-reported confidence remains correlated with actual error across model versions.Source: §3 · p. 17
  • Steel is a boundary case: heterogeneous text descriptions hinder Guided Mode performance, and the authors identify standardized, machine-readable processing data as a future improvement target.They also suggest adding multimodal priors such as micrograph images.Source: §3 · pp. 17–18

4 Methods

The methods assemble four alloy-property datasets, establish tuned machine-learning baselines, and describe BERT-embedding regressors and LLM access. The evaluation uses an 8:2 train-test split and nine-fold cross-validation.

4.1 Data preparation

  • The study uses 233 titanium, 831 aluminum, 491 high-entropy alloy, and 677 steel samples, with composition, processing, and mechanical-property data.Titanium data combine literature and XJTU laboratory measurements; steel data come from Reference4.Source: §4.1 · p. 18

4.2 Model selection and hyperparameter optimization

  • Five ML algorithms are evaluated with an 8:2 train-test split and nine-fold cross-validation, with Optuna tuning across 30 trials per model.Performance is quantified using RMSE and R²; the reported baseline is average performance across trials.Source: §4.2 · pp. 18–19
  • SciBERT, MatSciBERT, and SteelBERT separately encode composition and processing into 768-dimensional vectors, which are concatenated as input to an ANN.The ANN uses MSE loss and AdamW with linear learning-rate decay; architecture and training hyperparameters are optimized using Optuna.Source: §4.2 · p. 19

4.3 The use of LLMs

  • Gemini models are accessed through Google's free-tier API and DeepSeek through its official API, using a unified LiteLLM invocation interface.Source: §4.3 · p. 19

Corresponding author

Correspondence is listed for Turab Lookman or Zhibin Gao.

Loading 2611.03767v1…