Source-linked AI summary
Fine-Tuned Language Models Generate Stable Inorganic Materials as Text
Nate Gruver, Anuroop Sriram, Andrea Madotto, Andrew Gordon Wilson, C. Lawrence Zitnick, Zachary Ulissi
TL;DR
Materials science has relatively few known stable materials, while crystal data span diverse structures and properties. This paper fine-tunes pretrained LLMs on crystal strings and uses task-specific prompts for several generation tasks. The models generate materials predicted stable at higher rates than baseline methods, and larger models better capture crystal symmetries.
Problem
Materials science has relatively few known stable materials, and material data span composition, structure, and complex properties.
Source: §1 · p. 1Method
The paper fine-tunes pretrained LLMs on crystal-string representations and uses task-specific prompts for unconditional generation, text-conditioning, and infilling.
Source: §4 · p. 4Results
Takeaways & Limitations
The authors conclude that parameter-efficient instruction tuning and minimal task-specific modeling can make LLMs state-of-the-art generative models for atomistic domains, with multitask capabilities in one sampling paradigm.
Source: §6 · p. 9LLMs can be sensitive to prompt details and number tokenization, may generate unphysical elements or structures, and training the largest models can be prohibitively expensive.
Source: §6 · p. 9
Abstract
from arXiv · showhide
We propose fine-tuning large language models for generation of stable materials. While unorthodox, fine-tuning large language models on text-encoded atomistic data is simple to implement yet reliable, with around 90% of sampled structures obeying physical constraints on atom positions and charges. Using energy above hull calculations from both learned ML potentials and gold-standard DFT calculations, we show that our strongest model (fine-tuned LLaMA-2 70B) can generate materials predicted to be metastable at about twice the rate (49% vs 28%) of CDVAE, a competing diffusion model. Because of text prompting's inherent flexibility, our models can simultaneously be used for unconditional generation of stable material, infilling of partial structures and text-conditional generation. Finally, we show that language models' ability to capture key symmetries of crystal structures improves with model scale, suggesting that the biases of pretrained LLMs are surprisingly well-suited for atomistic data.
1 INTRODUCTION
The introduction motivates LLMs for materials science because known stable materials are relatively few and their data span diverse properties and structures. The paper fine-tunes LLaMA-2 on crystal strings and reports stable-crystal generation alongside text-conditioned generation and infilling.
1 INTRODUCTION
- Materials science is a promising application because stable-material examples are relatively few and their data include composition, structure, and complex properties.The paper also motivates a text interface for specifying desired properties or constraints.Source: §1 · p. 1
- Fine-tuned LLMs generate stable three-dimensional crystal structures as text and predict stability at higher rates than baseline methods.The evaluation uses learned ML potentials and gold-standard DFT calculations.Source: §1 · p. 1
- The approach fine-tunes base LLaMA-2 models on a database of known inorganic materials using low-rank adapters.The paper describes LLMs as useful for learning patterns from diverse data and with relatively few examples.Source: §1 · p. 1, Figure 1 · p. 2
- Text prompting supports unconditional generation, text-conditioned generation, and infilling of existing structures.Infilling and text-conditioning can be used to optimize existing materials' properties.Source: §1 · p. 2, Figure 1 · p. 2
2 RELATED WORK
Generative models for crystals must handle both discrete atom identities and continuous positions, while also accounting for crystal symmetries. The paper positions pretrained LLM fine-tuning as a flexible alternative to diffusion and from-scratch sequence models.
2 RELATED WORK
- Crystal generation must jointly address discrete atom identities and continuous positions, as well as symmetries that constrain neural-network functions.Unit-cell translations can shift and wrap atoms without changing the represented structure.Source: §2 · p. 2
- Diffusion approaches address crystal generation with separate or unified representations and often build symmetry constraints into graph-neural-network architectures.CDVAE reconstructs composition from a shared latent space and generates positions with denoising diffusion; later work also diffuses lattice parameters and atom identities [Xie et al. (2021); Jiao et al. (2023); Zeni et al. (2023); Yang et al. (2023)].Source: §2 · p. 2
- Prior sequence models encode atom identities and coordinate digits as tokens, using data augmentation to encourage invariances rather than architectural symmetry guarantees [Flam-Shepherd & Aspuru-Guzik (2023)].Those models were trained from scratch; another sequence-model effort trained on millions of CIF strings [Antunes et al. (2023)].Source: §2 · pp. 2–3
- This work uses pretrained LLMs without crystal-specific tokenization or millions of auxiliary structures, and reports improved symmetry learning with larger models.The authors say the approach can extend to multiple crystal tasks and potentially other atomistic modalities without changing the underlying model or training procedure.Source: §2 · p. 3
3 BACKGROUND
The background explains how language models generate token sequences and how crystals and their stability are represented. It introduces energy above hull as a composition-aware measure of material stability.
3 BACKGROUND
- LLMs generate sequences through next-token prediction, with temperature and nucleus size modifying the sampling distribution.Nucleus sampling discards tokens outside the cumulative probability mass specified by p.Source: §3 · p. 3
- LLaMA-2 tokenizes numbers as individual digits, a property the authors use because it supports a natural representation of three-dimensional coordinates.Byte pair encoding generally assigns tokens to common substrings, but can split numbers into irregular groups that complicate arithmetic.Source: §3 · p. 3
- A crystal is represented by a unit cell with lattice dimensions and angles, plus each atom's element identity and three-dimensional coordinates.Coordinates may be absolute or fractional; the complete crystal tuple is given in Equation 1.Source: §3 · p. 3
- DFT is the gold standard for energy prediction but can be prohibitively expensive, motivating deep-learning potentials that approximate DFT solutions.DFT often scales as O(n^3) with system size.Source: §3 · p. 3
- Energy above hull measures a crystal's distance from the minimum observed energy for its elemental composition; Ehull < 0.1 eV/atom is often considered metastable and potentially practical.Crystals with Ehull < 0 are stable by definition.Source: §3 · p. 4, Figure 2 · p. 4
4 METHOD
The method fine-tunes a pretrained LLM on string-formatted crystals and uses task-specific prompts to support multiple generation tasks. Random translations augment training data to encourage translation invariance.
4 METHOD
- The method fine-tunes a pretrained LLM on crystal strings, using its generalizable pattern biases and text input to support conditional generation and infilling.The authors say the approach could be extended to proteins, nucleic acids, or small molecules.Source: §4 · p. 4
- Crystal tuples are serialized with fixed precision: lattice lengths use one decimal place, angles are integers, and fractional coordinates use two digits.Coordinates are space-separated, while other crystal components are newline-separated; LLaMA-2 is chosen partly because it tokenizes numbers digit by digit.Source: §4 · p. 4
- Task-specific prompts support unconditional generation, text-conditioned generation, and infilling; training samples two-thirds generation tasks and one-third infilling.For infilling, a randomly chosen element is replaced by [MASK], and the model predicts its identity; prompts are excluded from the generative loss.Source: §4 · pp. 4–5
- Random uniform translations of fractional coordinates encourage translation invariance, while atom ordering is not augmented because it may convey useful grouping information.Translations modulo lattice boundaries preserve the crystal structure, and atom ordering is permutation-invariant at the structure level.Source: §4 · p. 5
5 EXPERIMENTS
The experiments evaluate fine-tuned LLaMA-2 for unconditional stable-crystal generation, symmetry learning, diversity and novelty, and sampling speed, then test prompted generation and infilling.
Datasets and evaluation
- Experiments use MP-20 for unconditional generation and a 120,000-crystal Materials Project collection for prompted generation, evaluating validity, diversity, and estimated stability.Validity covers non-overlapping atomic radii and net-neutral composition; M3GNet and VASP provide stability estimates.Source: §5 · pp. 5–6, Table 1 · p. 6
Unconditional generation
- LLaMA models generate high percentages of physically valid and stable materials, while unconstrained element generation can produce imaginary identities such as “Ln.”Restricting element-identity tokens can avoid these hallucinations.Source: Table 1 · p. 6, §5 · p. 6
- Fine-tuned LLaMA-2 70B generates more metastable and stable materials than CDVAE, and its stable samples are more diverse.Figure 4 compares stability estimates from M3GNet and VASP; VASP is run only on structures M3GNet predicts stable.Source: Figure 4 · p. 7
Symmetry learning
- Translation invariance and stable-material generation increase together across model sizes, with larger models learning invariances from augmentations more effectively.The proposed IPT measure assesses invariance through changes in perplexity under transformations.Source: Figure 5 · p. 7, §5 · p. 7
Diversity, novelty, and sampling speed
- Among M3GNet-judged metastable samples, LLaMA-2 often matches or exceeds CDVAE in diversity and achieves high novelty, while larger models have less structural but more compositional novelty.Smaller LLaMA models are often significantly faster than CDVAE at generating metastable samples.Source: §5 · pp. 7–8, Figure 6 · p. 8
Text-conditioned generation
- Text prompts generate materials with requested compositions, space groups, or stability, though complex formulas and space groups challenge the model.Composition is met the majority of the time; space-group conditioning is more difficult, and binary stability conditioning is the most challenging.Source: §5 · p. 8, Figure 7 · p. 9
Infilling Existing Materials
- For existing-material mutations, LLaMA-2 70B-guided element swaps produce stable structures at a higher rate than uniform template swaps without sacrificing stable-sample diversity.The infilling procedure uses a model-proposed element distribution and relaxes mutated structures with M3GNet.Source: §5 · pp. 8–9, Figure 7 · p. 9
6 DISCUSSION
Fine-tuned LLMs generate plausible stable materials verified by DFT, supporting their use as generative models for atomistic domains. The discussion highlights evaluation, conditional generation, and training-cost limitations as areas for further work.
6 DISCUSSION
- DFT-verified generation of plausible stable materials supports LLMs as state-of-the-art atomistic generative models, with parameter-efficient tuning and minimal task-specific modeling.The authors argue that generating near-stable structures may make downstream materials-discovery tasks easier.Source: §6 · p. 9
- The authors advocate evaluating generative models with energy-above-hull metrics tied to generating stable or metastable materials.They note that only a small subset of the combinatorially large hypothetical-material space is stable or metastable.Source: §6 · p. 9
- Prompt details and number tokenization can affect LLM behavior, while leveraging scientific and chemistry literature successfully remains a major challenge.Training the largest models can also be prohibitively expensive for some compute budgets.Source: §6 · p. 9
- Conditional generation remains an area for improvement, and the authors suggest exploring classifier-free guidance, PPLM variants, and Bayesian optimization.These approaches are proposed for property-directed, sample-efficient, and uncertainty-aware design.Source: §6 · p. 9
A TRAINING DETAILS
The appendix describes numerical-formatting choices and stochastic prompts used to train the models for generation and infilling. It also explains the use of LLaMA-2’s existing tokenizer rather than a crystal-specific vocabulary.
A.1 NUMERICAL FORMATTING
- The method reuses LLaMA-2’s tokenizer for both encoded crystals and text instead of introducing a crystal-specific vocabulary.Experiments with added crystal tokens were harder to train and did not improve on the shared tokenizer.Source: §A.1 · p. 13
- The authors identify coordinate formatting and translation or permutation augmentation as choices that affect validity, and report comparing formatting styles.The supplied passage does not specify the comparison’s validity values.Source: §A.1 · p. 13
A.2 TRAINING WITH STOCHASTIC PROMPTS
- Two-thirds of training prompts request bulk-material generation and may include randomly selected crystal properties.The remaining third uses an infilling task, with one randomly selected element masked and then generated as text.Source: §A.2 · pp. 13–14
A.3 EXTENDED MATERIALS PROJECT DATASET
For text-conditional generation, the training data were expanded with additional Materials Project crystals while retaining the original validation and test splits.
A.3 EXTENDED MATERIALS PROJECT DATASET
- The extended training set contains 127609 crystal structures after filtering out unit cells with more than 30 atoms; original validation and test splits remain unchanged and excluded from the expanded training set.The added Materials Project data were current as of April 2023.Source: §A.3 · p. 14
A.4 TRAINING HYPERPARAMETERS AND DETAILS
Training details vary across LLaMA-2 sizes, reflecting compute limits and a choice to report each size’s best trainable model. The appendix also explains how text pretraining supports tractable scaling and text conditioning.
A.4 TRAINING HYPERPARAMETERS AND DETAILS
- The reported LLaMA-2 models use batch size 256 for 65 epochs at 7B and 44 epochs at 13B, versus batch size 32 for 21 epochs at 70B.All three use a cosine-annealed learning rate of 0.0005 and LoRA rank 8 with alpha 32.Source: §A.4 · p. 14
- Compute limits led to differing batch sizes and epoch counts, so the authors report the best model they could train at each size rather than matching these settings.They state that ideally all models would use the largest batch size and the same maximum epoch count.Source: §A.4 · p. 14
A.5 ROLE OF TEXT PRETRAINING
- Using pretrained models with LoRA makes large-model scale tractable while limiting overfitting through a relatively small number of trainable parameters.The authors state that training models up to 70B from scratch on their data would be impractically expensive or computationally infeasible.Source: §A.5 · p. 14
- Text pretraining provides text conditioning without additional training for it, enabling prompts to specify desired material properties.The authors say achieving comparable text understanding from scratch would require substantially more data and effectively training a general-purpose language model.Source: §A.5 · p. 14
B MODEL EVALUATION
Sampling temperature and nucleus size trade coverage for validity and stability, with LLaMA-2 70B showing the best stated trade-off. Stability estimates combine structure filtering with ML-potential or DFT energy-above-hull checks and consistent energy corrections.
B.1 EVALUATION WITH ML POTENTIALS AND DFT
- Both ML-potential and VASP total energies use the MP2020 compatibility scheme to maintain formation-energy consistency across DFT and DFT+U calculations.VASP candidates receive one relaxation with default MPRelaxSet parameters; M3GNet was trained on Materials Project VASP total energies.Source: §B.1 · p. 15
B.2 STABILITY CHECKS AND PERCENTAGES
- The metastability percentage first excludes samples failing structure and composition checks, then counts M3GNet-relaxed samples with Ehull < 0.1; the VASP percentage uses M3GNet-screened candidates and Ehull < 0.0.The VASP percentage is calculated only for materials identified as metastable by M3GNet.Source: §B.2 · p. 15
B.3 TRADE-OFFS IN SAMPLING
- LLaMA-2 70B shows the best stated trade-off between coverage and stability, while lowering temperature or nucleus size improves validity/stability at a cost to held-out-set recall.The authors suggest that these settings concentrate sampling around high-likelihood structures, which are more likely to be valid or stable.Source: §B.3 · p. 15, Figure 8 · p. 16
- LLaMA-2 70B may better match both stability-related properties and the full, diverse structure distribution, but this interpretation is presented as tentative.The authors associate the coverage decrease with sampling distributions collapsing around high-likelihood structures.Source: §B.3 · pp. 15–16
B.4 “HALLUCINATION” EXAMPLES
The appendix presents generated crystal records, describes a perplexity-based analysis of transformation response, defines diversity and novelty measures, and compares sampling throughput and computational overhead.
B.4 “HALLUCINATION” EXAMPLES
- Generated examples are presented as crystal records containing formula and cell parameters, with the displayed entries specifying P1 symmetry.The examples include entries labeled Cu2N, Ln3BO4, MandeGd2O4, Gro15Nd4, L3Li, and LeB7 (NO3)2.Source: §B.4 · pp. 16–17
B.5 INCREASE IN PERPLEXITY UNDER TRANSFORMATION (IPT)
- The IPT analysis measures transformation-related perplexity changes across 500 test datapoints and 20 random translations per datapoint, normalizing by mean perplexity over transformations.Translations are sampled as fractional coordinates and respect periodic boundary conditions; the metric is used to highlight differences between base models.Source: §B.5 · p. 18
B.6 DIVERSITY AND NOVELTY CALCULATION
- Diversity is measured by pairwise sample distance, while novelty compares each sample with its nearest training-set element using structural and composition distance cutoffs of 0.1 and 2.Metrics are calculated on M3GNet-filtered metastable samples and normalized by corresponding test-set values.Source: §B.6 · p. 18
B.7 SAMPLING SPEED
- LLaMA-2 13B has comparable computational overhead to prior work and higher stable-material throughput than CDVAE, while LLaMA-2 70B has slightly higher overhead.The AWS comparison uses an 8-A100 instance benchmark and adjusts CDVAE's RTX2080 Ti runtime for GPU count and hardware.Source: §B.7 · p. 19
C TEMPLATE METHOD BASELINE
The template baseline builds element substitutions from ionic-radius similarity within oxidation states, then applies those substitutions to existing structures. Its language-model variant replaces uniform substitution choice with context-conditioned sampling.
C TEMPLATE METHOD BASELINE
- The swap table maps elements to physically plausible substitutions, using similar ionic radii in the same oxidation state to constrain candidate mutations.The code constructs this table for both the template method and LLM-guided sampling.Source: C TEMPLATE METHOD BASELINE · pp. 19–20
- The basic template method proposes structures by replacing species in an existing structure, with uniform sampling over available swaps.The implementation describes the procedure as a way to propose local mutations around an existing material.Source: C TEMPLATE METHOD BASELINE · p. 20
- The LLM-guided variant uses logits from an infilling prompt instead of uniform swap selection, allowing proposed mutations to use context from the rest of the crystal structure.The sampling distribution is modulated by temperature τ.Source: C TEMPLATE METHOD BASELINE · pp. 19–20