Source-linked AI summary
Crystal Structure Generation with Autoregressive Large Language Modeling
Luis M. Antunes, Keith T. Butler, Ricardo Grau-Crespo
TL;DR
Crystal structure prediction needs plausible candidate structures, but existing approaches are computationally expensive. CrystaLLM autoregressively models CIF text to generate inorganic crystal structures, and produces valid, sensible structures across known classes while generalizing to unseen cases. Its scope is constrained when structural templates are rare, under-represented, or absent from training.
Problem
Crystal structure prediction is often computationally expensive, creating a need for effective candidate structures to initialize searches for inorganic materials.
Method
CrystaLLM is a Transformer-based, decoder-only language model trained autoregressively on millions of CIF files and used to generate new files by sampling successive tokens.
Results
CrystaLLM generates valid and sensible inorganic crystal structures across a variety of known classes and can generate structures matching unseen compounds or structures.
Takeaways & Limitations
CIF-language modeling provides a flexible crystal-generation tool that can condition generation on structure information such as composition and space group.
Takeaways & Limitations
Generation difficulties arise for rare or under-represented templates, multiple oxyanion arrangements, and CIF files with relatively many tokens.
Abstract
from arXiv · showhide
The generation of plausible crystal structures is often the first step in predicting the structure and properties of a material from its chemical composition. Quickly generating and predicting inorganic crystal structures is important for the discovery of new materials, which can target applications such as energy or electronic devices. However, most current methods for crystal structure prediction are computationally expensive, slowing the pace of innovation. Seeding structure prediction algorithms with quality generated candidates can overcome a major bottleneck. Here, we introduce CrystaLLM, a methodology for the versatile generation of crystal structures, based on the autoregressive large language modeling (LLM) of the Crystallographic Information File (CIF) format. Trained on millions of CIF files, CrystaLLM focuses on modeling crystal structures through text. CrystaLLM can produce plausible crystal structures for a wide range of inorganic compounds unseen in training, as demonstrated by ab initio simulations. The integration with predictors of formation energy permits the use of a Monte Carlo Tree Search algorithm to improve the generation of meaningful structures. Our approach challenges conventional representations of crystals, and demonstrates the potential of LLMs for learning effective 'world models' of crystal chemistry, which will lead to accelerated discovery and innovation in materials science.
1 Introduction
Crystal structure prediction is computationally expensive, motivating candidate-generation methods that can seed CSP searches. CrystaLLM models inorganic crystal structures directly as CIF text and generates plausible structures, including compounds and structure combinations not seen during training.
- Motivation: Crystal structure prediction commonly uses computationally expensive methods, so sensible initial candidates can reduce the computation required for searching.Candidate generation is therefore an important first step in accelerating CSP workflows.
- Approach: CrystaLLM addresses an unconsidered application of LLMs by training directly on textual CIF representations of inorganic crystal structures.Earlier materials applications focused on natural-language LLMs, chemical subject matter, or expanded chemical compositions rather than textual crystal structures.
- Capabilities: CrystaLLM reliably generates valid CIF syntax and physically plausible structures across many inorganic crystal classes, despite their periodicity, symmetry, and compositional complexity.The model was trained on millions of CIF files representing inorganic solid-state materials.
- Approach: The Transformer decoder-only model autoregressively predicts subsequent CIF tokens and generates new files by repeatedly sampling tokens conditioned on the accumulated sequence.Its vocabulary includes symbols for atoms, space groups, and numeric digits, with learned representations emerging through training.
- Generalization: The model generalizes to unseen formulas, elements, and formula–space-group combinations, including matching held-out structures for Ba2MnCr and CsCuTePt.Ba2MnCr matched on the first prompted attempt, while CsCuTePt matched on the third prompted attempt.
- Generalization: On a 144-compound pyrochlore space, all three generations for each of 10 compounds unseen in training produced valid pyrochlore structures.The cell composition and space group were included in the prompt.
3 Discussion
CrystaLLM demonstrates that autoregressive modeling of CIF text can generate plausible inorganic crystal structures, while further task-specific tuning is needed for CSP use.
- Model capabilities: The model generates valid and sensible three-dimensional atomic arrangements while capturing relationships between space-group symbols and structural symmetries.It generates xyz coordinates digit-by-digit and produces structures across several known classes.
- Representation choice: Using CIF text directly avoids designing an intermediate representation and allows the model to learn the format’s grammatical structure.The authors chose CIF because it is not particularly verbose and inter-conversion could introduce errors.
- Relation to prior approaches: Compared with template-based generation approaches, CrystaLLM models crystal structures without relying solely on predefined structural templates and constrained substitutions.Existing approaches commonly substitute atoms and adjust cell parameters under a specified space group.
- CSP applicability: Sensible structure generation alone does not make CrystaLLM suitable for CSP; the model requires fine-tuning for advanced tasks.Potential fine-tuning methods include supervised or reinforcement learning, depending on the task.
- CSP applicability: Formation-energy predictors could provide thermodynamic feedback for reinforcement learning, but the evaluator must estimate energy quickly enough to avoid time-consuming DFT calculations.The proposed procedure targets low-energy structures through feedback from an external energy evaluator.
- Limitations: The current dataset excludes site-occupancy disorder and combines CIF structures generated with different levels of theory.Consequently, the model cannot generate disordered structures and may learn inconsistent composition–structure relationships in some cases.
- Future directions: The authors identify crystal structure generation as a potential tool for large-scale materials discovery and plan property-prediction fine-tuning.They specifically mention lattice thermal conductivity prediction, where experimental data are relatively scarce.
4 Methods
The methods assemble and standardize a large CIF corpus, train an autoregressive Transformer language model, validate generated structures, and evaluate benchmark and search procedures.
- Dataset: The dataset combines structures from Materials Project, OQMD, and NOMAD, originally optimized using density functional theory simulations.Duplicate structures sharing cell composition and space group were resolved by selecting the lowest volume per formula unit.
- Dataset: 2.3 million structures were converted to CIF files with pymatgen and randomly split into training, validation, and test sets.The conversion used a symmetry-finding tolerance of 0.1 Å and rounded floating-point values to four decimal places.
- Preprocessing: The standardized CIF corpus used a 371-symbol vocabulary and contained 768 million training tokens after tokenization.The vocabulary included CIF tags, space-group symbols, element symbols, digits, and punctuation.
- Generative pre-training: Generative pre-training maximizes the likelihood of each token conditioned on the preceding context window.The vocabulary V and ordered token sequence U define the training inputs.
- Generative pre-training: A decoder-only Transformer is trained by minimizing negative log-likelihood with stochastic gradient descent.The model has 25 million parameters, eight layers, eight attention heads, and an embedding size of 512.
- Structure validation: CIF validity requires consistency among the declared space group, generated geometry, bond lengths, and atom-site multiplicities.Space-group consistency is checked with pymatgen’s SpacegroupAnalyzer and spglib, while bond lengths are assessed using Voronoi-based neighbors.
- Structure validation: Structure matching uses pymatgen’s StructureMatcher after reducing both structures to primitive cells and scaling them to equivalent volume.The stated default tolerances are 0.2 fractional length, 0.3 Å site, and 5 degrees angle.
- Evaluation: Benchmark evaluation covers Perov-5, Carbon-24, MP-20, and MPTS-52 under benchmark-only and large-dataset training scenarios.Both small and large model architectures are used in the benchmark-only scenario.
8 Author Contributions
The project was conceived, experimentally conducted, and initially drafted by L.M.A., with experiment design, calculations, supervision, and manuscript review shared among the authors.
- Contributions: L.M.A. conceived the project, performed the experiments, and drafted the manuscript.L.M.A. and R.G.-C. designed the experiments.
- Contributions: R.G.-C. carried out the DFT calculations, while R.G.-C. and K.T.B. supervised and guided the project.All authors reviewed, edited, and approved the manuscript.
arXiv:2307.04340v3 [cond-mat.mtrl-sci] 12 Feb 2024
The paper lists Luis M. Antunes, Keith T. Butler, and Ricardo Grau-Crespo as its authors.
- Authors: The authors are Luis M. Antunes, Keith T. Butler, and Ricardo Grau-Crespo.
1. CIF Syntax Standardization and Tokenization
CrystaLLM standardizes CIF files into a fixed syntax enriched with atomic properties, then tokenizes the resulting text into model-readable sequences.
- Syntax standardization: Every dataset CIF uses an identical tag arrangement, despite CIF allowing optional tags and flexible ordering.The standardized syntax also includes several tags outside the official CIF specification.
- Pre-processing: CIF files are normalized through pymatgen conversion, rewriting, symmetry removal, property insertion, and numerical rounding.The preprocessing replaces the data tag with cell composition, removes symmetry operators, adds electronegativity and radii, and rounds values to four decimal places.
- Example: A pre-processed PbTe example illustrates the standardized CIF representation with composition, lattice, space-group, and atomic-property fields.The example uses Pmma symmetry and includes electronegativity, atomic radius, and ionic radius values.
- Tokenization: The custom vocabulary represents irreducible CIF elements including numeric digits, atomic symbols, space-group symbols, and CIF tags.Tokenization parses each standardized CIF into a sequence of numbers that the model can process.
- Post-processing: Post-processing removes the custom atomic-property loop and restores symmetry-equivalent site identifiers and positions matching the printed space group.This reconstructs required structural information after token generation.
2. Model Architecture and Generative Pre-training
The model autoregressively trains a GPT-style transformer on tokenized CIF sequences and generates files through temperature-scaled top-k sampling.
- Architecture: CrystaLLM uses a GPT-style transformer with multi-head self-attention and feed-forward networks to model token sequences autoregressively.Token indices are embedded with learned token and positional representations before passing through transformer blocks.
- Training: Training iteratively samples dataset sequences, computes loss, and backpropagates errors using AdamW with cosine learning-rate decay from 10^-3 to 10^-4.Gradients are clipped at norm 1.0, with 40 accumulation steps and batches of 32 sequences per step.
- Model configurations: The small model has 25 million parameters, while the large model has 200 million parameters and uses a larger transformer configuration.The large model is trained for 48,000 iterations because of computational resource and time constraints.
- Generation: Top-k generation temperature-scales final-position logits, retains the k most likely tokens, normalizes their probabilities, and samples the next token.Sampling proceeds iteratively until two consecutive newline tokens terminate the CIF sequence.
3. Validation of Generated CIF Files
Generated CIF files are accepted only when their printed chemistry, site multiplicities, bond lengths, and space-group information are mutually consistent.
- Consistency checks: Validation first checks that chemical formulas in the file agree with the cell composition and with one another.The chemical formula sum and structural formula tags are compared against the first-line cell composition.
- Consistency checks: Printed atom-site multiplicities must be consistent with the cell composition.The check uses the atom-site type-symbol and symmetry-multiplicity fields.
- Geometric checks: Bond-length reasonableness is assessed using Voronoi nearest neighbors, electronegativity differences, and ionic or covalent radii.The resulting score B ranges from 0 to 1 and is compared with an acceptability threshold.
- Geometric checks: The bond-length threshold is set to 1.0, requiring all bond lengths to lie within 30% of expected values.Space-group consistency is checked with pymatgen’s SpacegroupAnalyzer using spglib.
- Validation procedure: Algorithm 1 returns True or False after sequentially applying formula, multiplicity, bond-length, and space-group checks.Any failed check causes the generated structure to be rejected.
4. Monte Carlo Tree Search Decoding
MCTS improves CIF sampling by exploring token-sequence trees, evaluating completed structures, and propagating structure-quality scores through promising paths.
- Tree search: MCTS treats token sequences as paths through a tree and balances exploitation of strong sequences with exploration of potentially better ones.Each node represents a token and each edge represents a transition to the next token.
- Selection: At each level, PUCT selects the most promising node using accumulated scores, visit counts, model priors, parent visits, and an exploration constant.The CrystaLLM probability for the child token supplies the prior probability.
- Expansion and rollout: Untried children are expanded using the model’s top-k predicted tokens, with probabilities above 0.99 producing a sole child.Rollout then samples repeatedly from the selected node until a completed structure is obtained.
- Evaluation: Completed CIFs are validated, and valid structures are scored using ALIGNN-predicted formation energy per atom.The search accumulates visit counts and scores for nodes selected during each simulation.
- Scoring: Valid-structure rewards are computed from the statistics of predicted formation energies collected during the search.The reward uses the mean, standard deviation, and responsiveness constant λ to map energy predictions into the search score.
- Scoring: Invalid structures receive -1, while otherwise valid structures with unreasonable bonds receive a negative score proportional to the number of unreasonable bonds.The overall score is piecewise, using the valid-structure reward or bond-length reasonableness score as appropriate.
- Algorithm: Algorithm 2 runs a fixed number of simulations over a tree initialized from a prompt and returns the valid sequences collected during search.Each simulation performs selection, expansion, rollout, evaluation, and backpropagation.
Supplementary Tables
The supplementary tables document Challenge Set compositions, model-predicted formation energies, generation performance, and MCTS results under prompts with or without space-group information.
- Challenge Set compounds are listed with their sources and ALIGNN-predicted formation energies per atom.
- Small- and large-model performance is reported separately for prompts without and with the space group.
- Additional tables report MCTS results for the small model under both prompting conditions.
Supplementary Figures
The supplementary figures visualize dataset composition, training behavior, and learned representations of digits, atoms, and space groups, alongside MCTS generation validity over time.
- Dataset plots show element frequencies, unit-cell formula-unit counts, and composition sizes.Oxygen, copper, and lithium are the most abundant elements, while most structures have Z values of 1–4.
- Training and validation losses for the small model are plotted across training iterations.The validation loss continues improving after 90,000 iterations, although the train–validation gap grows.
- Learned atom and space-group vectors are visualized using dimensionality reduction and cosine-similarity plots.
- Learned numeric digit vectors are shown through PCA, cosine-similarity, and Euclidean-distance plots.
- MCTS generates more valid structures than random sampling, with validation rates increasing over time in some cases.The comparison covers 1,000 iterations without a space group.