PaperlayerView exact revision on arXiv

Source-linked AI summary

Crystal Diffusion Variational Autoencoder for Periodic Material Generation

Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, Tommi Jaakkola

arXiv:2110.06197v3cs.LGcond-mat.mtrl-sciphysics.comp-ph

TL;DR

Stable-material generation must satisfy both local quantum-mechanical energy minima and global bonding preferences, which existing methods inadequately capture. CDVAE uses a physics-informed diffusion VAE with periodic and symmetry-aware representations, and significantly outperforms prior methods in reconstruction, generation, and property optimization.

  • Problem

    Stable-material generation must satisfy local quantum-mechanical energy minima and global bonding preferences, while existing methods inadequately incorporate these constraints and invariances.

  • Method

    CDVAE combines a variational autoencoder with a noise-conditional score decoder that denoises coordinates toward energy minima and updates atom types according to neighboring bonding preferences.

  • Results

    CDVAE significantly outperforms past methods on reconstruction, stable-material generation, and property optimization tasks.

  • Takeaways & Limitations

    The generation and property-optimization tasks can directly support producing new materials for subsequent verification with quantum-mechanical simulations and experiments.

Abstract

from arXiv · show

Generating the periodic structure of stable materials is a long-standing challenge for the material design community. This task is difficult because stable materials only exist in a low-dimensional subspace of all possible periodic arrangements of atoms: 1) the coordinates must lie in the local energy minimum defined by quantum mechanics, and 2) global stability also requires the structure to follow the complex, yet specific bonding preferences between different atom types. Existing methods fail to incorporate these factors and often lack proper invariances. We propose a Crystal Diffusion Variational Autoencoder (CDVAE) that captures the physical inductive bias of material stability. By learning from the data distribution of stable materials, the decoder generates materials in a diffusion process that moves atomic coordinates towards a lower energy state and updates atom types to satisfy bonding preferences between neighbors. Our model also explicitly encodes interactions across periodic boundaries and respects permutation, translation, rotation, and periodic invariances. We significantly outperform past methods in three tasks: 1) reconstructing the input structure, 2) generating valid, diverse, and realistic materials, and 3) generating materials that optimize a specific property. We also provide several standard datasets and evaluation metrics for the broader machine learning community.

1 INTRODUCTION

CDVAE addresses crystal generation by learning stable-material distributions while incorporating physical stability constraints and the invariances of periodic structures. Its decoder uses diffusion-based denoising with a physical inductive bias, alongside curated datasets and evaluation tasks for material generation.

  • Challenge: Stable materials occupy a low-dimensional subspace because atomic coordinates must reach quantum-mechanical local energy minima and satisfy specific interatomic bonding preferences.Material stability requires both local energetic equilibrium and globally appropriate bonding between atom types.
  • Challenge: Material generation must model periodic-boundary interactions and respect permutation, translation, rotation, and periodic invariances.These requirements distinguish periodic materials from simpler molecular representations.
  • Method: CDVAE learns a variational autoencoder that generates stable 3D materials directly from latent representations, using denoising of perturbed stable structures to promote stability.The decoder is a noise conditional score network, and the approach avoids intermediate graph representations.
  • Physical inductive bias: The decoder’s learned gradient field is connected theoretically to a harmonic force field that estimates atomic forces when coordinates deviate from equilibrium positions.This connection supplies a physical inductive bias for generating stable materials.
  • Contributions: The work curates 3 standard datasets from quantum-mechanical simulations and defines physically meaningful tasks and metrics for material generation.These resources are intended to support broader machine learning research on material generation.

2 RELATED WORK

Prior material-generation methods either represent structures as voxels or vectors, but neither approach incorporates stability as an inductive bias and both have important validity or invariance limitations. Alternative quantum-mechanical searches and force-field methods are computationally or data intensive, motivating the paper’s diffusion-based approach.

  • Quantum mechanical search of stable materials: Quantum-mechanical discovery requires expensive random searches and simulations, while existing sampling and evolutionary methods generally have low success rates.The passage also lists element substitution in known materials among state-of-the-art approaches.
  • Material generative models: Existing material generative models mainly use voxel images or vectorized atom coordinates, types, and lattices, and neither approach incorporates stability as an inductive bias.Voxel decoding often produces low validity, while vector-based models are generally not invariant to Euclidean transformations.
  • Material generative models: Voxel-based models often yield low validity when decoding images into atom types and coordinates and are not rotationally invariant.These limitations are associated with prior voxel-image approaches to material generation.
  • Material generative models: Force-field generation by energy minimization is conceptually similar to the decoder but requires additional quantum-mechanical force data that is expensive to obtain.The method trains a force field from quantum-mechanical forces before minimizing energy to generate stable materials.

3 PRELIMINARIES

The preliminaries represent crystals as periodic unit cells and multi-graphs, define composition and structural invariances, and formulate stable-material generation as learning from observed materials under physical constraints.

  • Crystal representation: A periodic crystal is described by atom types, atom coordinates, and a lattice defining a repeating unit cell that tiles three-dimensional space.Integer translations of the lattice generate the infinite periodic structure.
  • Composition: Material composition records the percentages of chemical elements, with the component values summing to one.For diamond, the carbon component is one and all other element components are zero.
  • Invariances: Crystal structures are invariant to atom permutation, coordinate translation, joint rotation of coordinates and lattice, and alternative unit-cell choices.These invariances ensure equivalent representations describe the same material.
  • Multi-graph representation: Materials can be represented as directed multigraphs whose nodes are atoms and whose edges encode bonds across translated periodic cells.Each edge specifies a source atom, target atom, and integer lattice translation.
  • Stable-material generation: Stable materials must satisfy local energy-minimum conditions for coordinates and lattice and global stability against decomposition, motivating generative learning from experimentally observed materials.The learned model is intended to generate candidates for later quantum-mechanical verification.

4 PROPOSED METHOD

CDVAE encodes stable periodic materials into a latent representation, predicts composition, lattice, and atom count, then uses a periodic equivariant decoder with annealed Langevin dynamics to denoise coordinates and atom types. Its coordinate updates are mathematically equivalent to a spring-like harmonic force field when noise is small.

  • Conditional decoding: The decoder denoises atomic coordinates and types from structures initialized using latent-predicted composition, lattice, and atom count.Noise schedules are conditioned on predicted aggregated properties, constraining the decoder’s denoising task.
  • Material generation: Generation samples z, predicts c, L, and N, randomly initializes a periodic structure, and applies annealed Langevin dynamics to update coordinates and atom types.The process simultaneously refines the structure toward improved local and global stability.
  • Model training: CDVAE jointly trains a periodic GNN encoder, an aggregated-property predictor, and a periodic GNN decoder on stable materials.The encoder produces z, the predictor estimates composition c, lattice L, and atom count N, and the decoder denoises noisy structures conditioned on z.
  • Periodic equivariance: The model represents periodic-boundary interactions with multi-graphs and uses SE(3)-adapted graph networks to preserve permutation, translation, rotation, and periodic invariances.The decoder’s coordinate score is rotation equivariant, while periodicity is handled through shortest-image displacement targets.
  • Physical interpretation: When noise is small, the Langevin coordinate update α_js_X,t is mathematically equivalent to a harmonic force field F(˜X) = −k(˜X − X).Here X is the equilibrium atomic position and k is a force constant, giving the update a spring-like physical interpretation.

5 EXPERIMENTS

The experiments establish three standardized material-generation tasks—reconstruction, generation, and property optimization—using curated datasets and physically meaningful evaluation metrics. CDVAE reconstructs stable structures effectively, generates valid materials, and outperforms or matches baselines on property optimization depending on the dataset.

  • Datasets: The benchmark uses Perov-5, Carbon-24, and MP-20, spanning compositionally varying perovskites, structurally varying carbon materials, and realistic inorganic materials.Perov-5 contains 18,928 materials, Carbon-24 contains 10,153, and all datasets use a 60-20-20 random split.
  • Reconstruction: CDVAE achieves lower reconstruction RMSE than every competing model, although FTCP has a higher match rate.StructureMatcher evaluates matches while accounting for material invariances, and SE(3) invariance permits translated or rotated reconstructions.
  • Generation: CDVAE achieves higher validity than FTCP, Cond-DFC-VAE, and P-G-SchNet, while G-SchNet achieves similar validity.Generation is evaluated with physically meaningful metrics for validity, property statistics, and diversity without requiring computationally prohibitive quantum-mechanical calculations.
  • Property optimization: CDVAE significantly outperforms FTCP on property optimization and performs similarly to Cond-DFC-VAE on Perov-5, while all models perform poorly on Carbon-24.G-SchNet and P-G-SchNet are incapable of property optimization, and Cond-DFC-VAE cannot work for Carbon-24 or MP-20.

6 CONCLUSIONS AND OUTLOOK

The CDVAE significantly outperforms prior methods in reconstruction, material generation, and property optimization. Generation and property optimization are especially important because their outputs can be verified through QM simulations and experiments.

  • Conclusions: CDVAE significantly outperforms past methods on reconstruction, generation, and property optimization.The model generates periodic structures of stable materials.
  • Conclusions: Generation and property optimization matter more for material design than reconstruction because they can directly produce new materials for verification.The proposed verification routes are QM simulations and experiments.
  • Outlook: An improved version of CDVAE for conditional generation was recently published, but its code had not been released.The cited improved version is Gebauer et al. (2021).

A PROOF FOR THE CONNECTION TO A HARMONIC FORCE FIELD … B.2 MULTI-GRAPH CONSTRUCTION

The proof shows that, under small-noise assumptions, the coordinate force in the decoder becomes a harmonic restoring force. Implementation details use a unique reduced lattice parameterization and periodic graph construction, with K = 20 for decoder graphs.

  • A PROOF FOR THE CONNECTION TO A HARMONIC FORCE FIELD: The derivation assumes Equation 2’s loss can be minimized to zero when the noises are small.
  • B IMPLEMENTATION DETAILS: The force term is first expressed through the score function and noise-dependent scaling in the Langevin dynamics.
  • A PROOF FOR THE CONNECTION TO A HARMONIC FORCE FIELD: For sufficiently small noise that atoms do not cross periodic boundaries, dmin(X, ˜ X) = X −˜ X.
  • A PROOF FOR THE CONNECTION TO A HARMONIC FORCE FIELD: The resulting coordinate force is αjsX( ˜ A, ˜ X, L|z; σA,j, σX,j) = −k( ˜ X −X), ∀j > J.
  • B.1 PREDICTION OF LATTICE PARAMETERS: The method computes a unique Niggli reduced lattice for each material and predicts six lattice parameters instead of the rotation-variant lattice matrix.
  • B.1 PREDICTION OF LATTICE PARAMETERS: Lattice-vector lengths are normalized by 3√ N, where N is the number of atoms, to align scales across materials of different sizes.
  • B.2 MULTI-GRAPH CONSTRUCTION: The encoder builds multi-graph edges with CrystalNN, while the decoder uses an on-the-fly periodic KNN graph because CrystalNN is too slow.K = 20 in all experiments.

B.3 GNN ARCHITECTURE

The architecture uses a periodicity-adapted DimeNet++ encoder and GemNet-dQ decoder to preserve the required SE(3) invariance and equivariance. Hidden dimensions are reduced to 128 for faster training, while the encoder has 2.2 million parameters.

  • Encoder: The encoder is DimeNet++ adapted for periodic structures and is SE(3) invariant to the input structure.The implementation follows Klicpera et al. (2020a;b).
  • Decoder: The decoder is GemNet-dQ and outputs a vector per node that is SE(3) equivariant to the input structure.The implementation follows Klicpera et al. (2021).
  • Implementation: The model uses Open Catalysis Project implementations with hidden dimensions reduced to 128 for faster training.The encoder has 2.2 million parameters.

C DATASET CURATION … D EXPERIMENT DETAILS

The paper curates three crystal-structure datasets spanning perovskites, carbon structures, and experimentally stable materials, with dataset-specific composition, stability, and relaxation criteria. All included structures in Perov-5 and Carbon-24, and the selected MP-20 materials, are relaxed using DFT.

  • C.1 PEROV-5: Perovskite materials follow the general formula ABX3 and ideally adopt a cubic structure with A at corners, B body-centered, and X face-centered sites.Perovskites have broad applications, and Perov-5 is curated from an open water-splitting database.
  • C.1 PEROV-5: Perov-5 includes all 18928 materials from its source database, allowing nonradioactive metals at A and B sites and one or more elements from O, N, S, and F at X sites.Multiple different X atoms may occur in one material.
  • C DATASET CURATION: All Perov-5 materials are relaxed using DFT, and their relaxed structures can deviate significantly from ideal perovskite structures.A significant portion are not thermodynamically stable and will decompose.
  • C.2 CARBON-24: Carbon-24 is derived from AIRSS carbon structures generated at 10 GPa, selecting the 10% with the lowest energy per atom from the original dataset.The resulting dataset contains 10153 DFT-relaxed structures; diamond is the most stable structure at 10 GPa, while most others are unstable or unsynthesized.
  • C.3 MP-20: MP-20 includes almost all experimentally stable Materials Project materials with unit cells containing at most 20 atoms and originally sourced from ICSD.These materials represent most experimentally known materials within that unit-cell-size limit.
  • C.3 MP-20: MP-20 selects materials with energy above the hull smaller than 0.08 eV/atom and formation energy smaller than 2 eV/atom, without limiting unique elements per material.All MP-20 materials are DFT-relaxed, and most are thermodynamically stable and synthesized.

D.1 REASONS FOR THE UNSUITABILITY OF SOME METRICS FOR SPECIFIC DATASETS · D.2 COMPOSITION VALIDITY CHECKER · D.3 NON-GAUSSIAN STATISTICAL STRUCTURE OF MATERIALS

The appendix explains why certain metrics are unsuitable for specific datasets, introduces an alloy-aware composition validity checker, and argues that material distributions are substantially non-Gaussian. These observations motivate dataset-specific evaluation and caution against naive latent-space sampling.

  • D.1 REASONS FOR THE UNSUITABILITY OF SOME METRICS FOR SPECIFIC DATASETS: Property statistics based on earth mover’s distance between generated and ground-truth property distributions are not meaningful for ground-truth data.This limitation applies to the statistics reported in Table 2.
  • D.1 REASONS FOR THE UNSUITABILITY OF SOME METRICS FOR SPECIFIC DATASETS: Perov-5 materials share the same structure, so requiring higher structure diversity is not meaningful.
  • D.1 REASONS FOR THE UNSUITABILITY OF SOME METRICS FOR SPECIFIC DATASETS: Carbon-24 materials all have carbon-only compositions, making higher composition diversity inappropriate; composition validity is omitted because all models achieve ∼100%.
  • D.2 COMPOSITION VALIDITY CHECKER: The modified SMACT checker evaluates charge neutrality using possible element charges while treating compositions containing only metals as valid for alloys.The original checker is unsuitable because metal alloys can mix in almost any combination.
  • D.2 COMPOSITION VALIDITY CHECKER: ∼90% composition validity from the modified checker versus ∼50% from the original checker better reflects ground-truth MP-20 materials.The paper notes that these empirical checkers do not replace high-fidelity stability evaluation through QM simulations.
  • D.3 NON-GAUSSIAN STATISTICAL STRUCTURE OF MATERIALS: Material datasets are biased toward particular groups, exemplified by the abundance of lithium-containing materials in MP-20 from its battery-research origins.
  • D.3 NON-GAUSSIAN STATISTICAL STRUCTURE OF MATERIALS: A larger β in Equation 9 causes the decoder to underfit the data distribution, supporting the conclusion that ground-truth materials have far-from-Gaussian statistical structure.
  • D.3 NON-GAUSSIAN STATISTICAL STRUCTURE OF MATERIALS: Sampling from N(0, 1) may produce out-of-distribution materials because the ground-truth material distribution is far from Gaussian.

D.4 HYPERPARAMETERS AND TRAINING DETAILS · E VISUALIZATION OF MULTIPLE RECONSTRUCTED STRUCTURES · F SAMPLING SPEED FOR MATERIAL GENERATION

CDVAE training uses balanced loss-term weights, dataset-specific KL regularization, and scheduled optimization, while generation applies Langevin dynamics at multiple noise levels. Reconstructions vary across random seeds, and CDVAE’s sampling speed is compared with faster but less capable alternatives.

  • D.4 HYPERPARAMETERS AND TRAINING DETAILS: The total objective combines aggregation, decoder, and KL losses as L = LAGG + LDEC + LKL.The expanded objective weights coordinate, lattice, neighbor, composition, atom-type, and KL terms.
  • D.4 HYPERPARAMETERS AND TRAINING DETAILS: For all three datasets, loss weights are λc = 1, λL = 10, λN = 1, λX = 10, and λA = 1.These values are chosen to keep each loss term at a similar scale.
  • D.4 HYPERPARAMETERS AND TRAINING DETAILS: Validation-loss selection uses β ∈ {0.01, 0.03, 0.1}, with β = 0.01 for Perov-5 and MP-20 and β = 0.03 for Carbon-24.The model with the best validation loss is selected.
  • D.4 HYPERPARAMETERS AND TRAINING DETAILS: Training starts at learning rate 0.001, reduces it by 0.6 after 30 validation epochs without improvement, and floors it at 0.0001.The same training schedule is described for the model training procedure.
  • D.4 HYPERPARAMETERS AND TRAINING DETAILS: Generation uses ϵ = 0.0001 and runs Langevin dynamics for 100 steps at each noise level.The sampling procedure applies these settings during material generation.
  • E VISUALIZATION OF MULTIPLE RECONSTRUCTED STRUCTURES: Reconstructions from the same latent z differ when three Langevin-dynamics samplings use different random seeds.Figure 5 visualizes the resulting multiple reconstructed structures.
  • F SAMPLING SPEED FOR MATERIAL GENERATION: Sampling speed is summarized for generating 10,000 materials, with FTCP significantly faster but producing very poor material quality.Cond-DFC-VAE is faster than CDVAE on Perov-5 but has lower quality, supports only cubic systems, and may scale poorly for larger materials because density-map computation grows cubically.

G COVERAGE METRICS FOR MATERIAL GENERATION

The paper defines coverage metrics for comparing generated and ground-truth material ensembles using structure and composition fingerprint distances. Coverage requires both distances to fall below thresholds, while recall and precision assess ground-truth recovery and generated-material quality.

  • Metric definitions: Six metrics compare generated materials with ground-truth test materials.The metrics are inspired by established work and include coverage, average minimum distances, recall, and precision variants.
  • Metric definitions: Structure distance uses Euclidean CrystalNN fingerprint distance, while composition distance uses normalized Magpie fingerprint distance.Separate thresholds, δstruc. and δcomp., are defined for the two distances.
  • Metric interpretation: Recall measures correctly predicted ground-truth materials, whereas precision measures the fraction of generated materials that are high quality.COV-P, AMSD-P, and AMCD-P swap the generated and ground-truth material sets in the corresponding equations.
  • Metric rationale: COV requires both structure and composition distances to remain within their thresholds, making it more meaningful than AMSD or AMCD alone.A material structurally close to one ground-truth material but compositionally close to another does not count as meaningful coverage.
  • Threshold selection: For Perov-5 and Carbon-24, thresholds are δstruc. = 0.2 and δcomp. = 4; for MP-20, they are δstruc. = 0.4 and δcomp. = 10.Figures 6–8 vary both thresholds and show their effects on COV-R and COV-P across the three datasets.
Loading 2110.06197v3…