PhononBench:A Large-Scale Phonon-Based Benchmark for Dynamical Stability in Crystal Generation

Introduction

• Dynamical stability concerns whether a structure remains in a local potential well under small perturbations, but systematic phonon validation is computationally demanding.Thermodynamic stability metrics such as Ehull do not fully assess this property.Source: Introduction · p. 2
• MatterSim's phonon predictions across more than 10,000 materials have accuracy comparable to DFT, supporting its use for high-throughput phonon analysis.Its dynamical-stability classification reaches a 95% true-positive rate at a tiny fraction of full DFT workflow cost.Source: Introduction · p. 3
• 32.15% is the average dynamical-stability rate across models, while MatterGen reaches 45.05%.The reported rates show that dynamical stability remains limited even for the top-performing model.Source: Introduction · p. 3
• PhononBench assesses 133,838 structures generated by 7 models, providing the first large-scale evaluation of dynamical stability in AI-generated materials.Source: Introduction · p. 3
• Under a strict -0.001 THz threshold, 32,995 structures are fully phonon-stable across the Brillouin zone.The paper presents these structures as a pool of candidate materials for subsequent design and discovery.Source: Introduction · p. 3
• In case studies, MatterGen's band-gap-conditioned generation reaches 41.0% stability at a 0.5 eV target, while cubic space-group-controlled systems reach up to 53.4%.The average stability rate for space-group-controlled generation is 44.7%.Source: Introduction · p. 3

Systematic Evaluation of Dynamical Stability in Crystal Generation Models

• At -0.1 THz, MP20 MatterGen (45.05%), DiffCSP (43.94%), and InvDesFlow-AL (43.78%) lead the benchmark, while the best LLM/Transformer configuration, LLaMA2-70B, reaches 32.50%.On Alex20, MatterGen reaches 57.89% and InvDesFlow-AL 52.65%; stability also depends on the evaluation threshold. The study defines stability ratio using successfully relaxed structures and reports that it converges above approximately 4,000 samples, except for CrystaLLM.Source: Systematic Evaluation of Dynamical… · p. 4, Figure 1 · p. 6
• Under the strictest phonon threshold of -0.001 THz, formula-prompted CrystaLLM-large models reach 18.4% and 19.6% stability, while LLaMA2-70B reaches 21.7%, rising to 23.9% under certain sampling configurations.CrystaLLM-small’s unprompted stability was 3.0%: 58 of 1,951 successfully converged structures were dynamically stable.Source: Systematic Evaluation of Dynamical… · p. 5

Dynamical Stability Analysis of Space-Group-Constrained Crystal Generation

• At -0.1 THz, 3,923 of 8,642 CrystalFormer structures are stable (45.39%); cubic structures reach 53.4% stability, compared with 39.3% for triclinic structures.The results associate higher symmetry with greater stability, while the space-group constraints reduce novelty relative to models without those constraints. Generated crystal-system and elemental-composition distributions resemble those of Materials Project, with ternary and quaternary compounds making up more than half of the generated materials.Source: Dynamical Stability Analysis of… · pp. 5, 7, 8, Figure 2 · p. 7

Dynamical Stability Analysis of Property-Constrained Crystal Generation

• For band-gap-conditioned MatterGen, stability is highest at Eg = 0.5 eV (41.0%) and lowest at Eg = 4.5 eV (19.4%), with an overall stability ratio of 26.8%.The generated materials retain compositional diversity similar to Materials Project, but the three illustrated Eg = 0.5 eV structures all show pronounced imaginary phonon frequencies. The study will release phonon calculations and optimized structures for the 33,210 evaluated materials.Source: Dynamical Stability Analysis of… · pp. 8–9, Figure 3 · p. 9

Physical Origins of Dynamical Instability

• At -0.1 THz, instability is associated with electronegative elements, increasing compositional complexity, and low coordination, while formation-energy sign has only a weak overall correlation with instability.Instability probabilities exceed approximately 65% for structures containing elements including F, O, Cl, H, Br, and I; they decline from approximately 84% at average coordination number near 1.5 to approximately 38% near 9. Structures with positive formation energy have a 66.62% instability probability versus 53.21% for negative formation energy; insulating systems show more moderate instabilities than metallic systems, with the difference diminishing for stronger instabilities.Source: Physical Origins of Dynamical… · pp. 10–11

Dynamically Stable Crystal Structures

• The benchmark identifies 32,995 dynamically stable crystals, and MatterSim achieves 97.62% true stable rate and 88.07% accuracy at -0.001 THz on a 120-structure DFT validation set.MatterSim's false unstable rate is 2.38%, lower than those of the compared models, and the validation results remain consistent across frequency thresholds. Oxygen appears most often in the stable-crystal set (10,818 occurrences), followed by lithium (6,402) and fluorine (5,653).Source: Dynamically Stable Crystal Structures · p. 11, Table 2 · p. 13, Figure 5 · p. 14

Summary of Model Architectures and Inference Performance

• CrystalFlow generates 333.3 crystals per minute, the highest reported throughput, while MatterGen and CrystalFormer generate 13.2 and 12.0 crystals per minute, respectively.Speeds are normalized to an equivalent batch size of 200; CrystalFormer was tested using its CPU version. CrystalFlow's throughput is reported as more than an order of magnitude above the slowest model, supporting large-scale screening and iterative optimization.Source: Summary of Model Architectures and… · pp. 14–15
• GNNs dominate the comparison; MatterGen is largest at 53.7M parameters, while CrystalFormer is the lightest at 4.8M.CrystalFlow has 20.9M parameters; InvDesFlow-AL and DiffCSP each have 12.3M, and CrystaLLM (Small) has 26.1M.Source: Summary of Model Architectures and… · p. 16

Discussion

• Prompting and model scale can improve LLM-based generation; CrystaLLM-large and LLaMA2-70B reach up to 32.5% dynamical stability, exceeding CrystalFormer’s 15.5% under the same setting.The discussion also reports an approximately 18% gain for prompted LLaMA2-70B over its no-prompt baseline.Source: Discussion · pp. 16–17
• Novelty-oriented models can generate lower-symmetry structures that increase subsequent DFT difficulty or instability, while symmetry constraints can reduce novelty.The discussion also notes that CrystalFormer’s symmetry constraints make low-symmetry, potentially stable materials harder to generate.Source: Discussion · pp. 16–17
• Direct Preference Optimization could use phonon-stable and unstable structure pairs to learn stable-material preferences and avoid unstable structures.The discussion presents this as a more tractable alternative to reinforcement learning with near-instantaneous phonon-based feedback.Source: Discussion · p. 17
• The authors propose combining explicit stability-driven optimization with implicit semantic guidance as a scalable pathway toward dynamically stable crystal generation.They suggest stability-focused prompts could steer generation without additional computational overhead.Source: Discussion · p. 18
• MatterSim’s broader training coverage, including perturbed configurations, may help explain its phonon-prediction performance by capturing local potential-energy-surface curvature.The authors identify off-equilibrium training configurations as a future direction for improving phonon-related predictions.Source: Discussion · p. 18

Methods

• Phonon calculations require reliable force predictions under small atomic displacements because phonon spectra depend on second derivatives of the potential-energy surface.The method obtains phonon frequencies by diagonalizing the dynamical matrix.Source: Methods · p. 18
• MatterSim-v1 is selected for large-scale phonon evaluation based on reported benchmark accuracy and faster inference than recently released uMLIPs.The cited benchmark reports approximately 95% accuracy in dynamical-stability classification; MatterSim’s efficiency supports calculations for more than 100,000 structures.Source: Methods · pp. 18–19
• The workflow relaxes structures with symmetry preserved, evaluates forces on displaced supercells, constructs force-constant matrices, and assesses stability from phonon modes.It uses 2 × 2 × 2 supercells, 0.01 Å displacements, a 0.005 eV/Å force-convergence criterion, and a −0.1 THz threshold unless otherwise specified.Source: Methods · p. 19

Appendix Contents

• The supplementary contents list analyses of phonon-model accuracy and efficiency, predictions on 9,958 Materials Project materials, DFT validation, and instability origins across thresholds.These are listed as appendix topics rather than reported findings in the supplied passages.Source: Appendix Contents · p. 25
• Other listed supplements cover training-data scale, LLM prompting and scaling, generation and stability summaries, and convergence of the dynamical-stability rate.The contents also list crystal relaxation and stability statistics across band-gap ranges.Source: Appendix Contents · p. 25

A.1 Phonon Prediction on 9958 MP Materials Using SOTA Models

• On the full benchmark, PET-OAM-XL achieves TS = 63% and TU = 91%, indicating weaker stable recognition than its leading leaderboard position might suggest.The section explains that the Matbench-Discovery leaderboard evaluates energy and force accuracy on near-equilibrium structures, whereas phonons depend on second derivatives and force accuracy under small displacements.Source: §A.1 · pp. 26, 28, Figure 7 · p. 26
• MatterSim achieves TS = 96% and TU = 80% on the 100-structure subset, compared with PET-OAM-XL’s TS = 68% and TU = 98%.PET-OAM-XL’s stable-structure errors include FU = 32%, versus MatterSim’s FU = 4%.Source: §A.1 · pp. 28–29, Table 7 · p. 28
• MatterSim’s discrepancy from PBE phonon results is comparable to the intrinsic variation between PBE and PBEsol on both the full benchmark and the representative subset.The authors place MatterSim’s prediction error within the uncertainty range associated with the choice of exchange-correlation functional.Source: Table 7 · p. 28, §A.1 · p. 29

A.2 Inference Efficiency

• MatterSim completes phonon calculations for 100 structures in 3.99 hours, versus 7.55 hours for PET-OAM-XL and 27.53 hours for eSEN-30M-OAM.The authors report MatterSim as approximately twice as fast as PET-OAM-XL and seven times faster than eSEN-30M-OAM.Source: §A.2 · p. 29, Table 8 · p. 29
• MatterSim’s training set spans about 17 million structures across temperatures of 0–5000 K and pressures up to 1000 GPa, covering a wider thermodynamic space than datasets centered near equilibrium.The authors connect perturbed configurations in this coverage to learning local potential-energy-surface curvature relevant to phonon predictions.Source: §A.2 · p. 29
• The authors choose MatterSim-v1 for balanced phonon-stability prediction, higher inference efficiency, and training data that better captures potential-energy-surface curvature.They describe it as a practical and reliable backbone for large-scale phonon benchmarking.Source: §A.2 · p. 30

A.3 DFT Validation on 120 AI-Generated Materials

• The DFT validation set contains 120 randomly sampled structures, split into 60 stable and 60 unstable materials, spanning diverse chemistry, compositions, bonding, and space groups.Phonons were calculated using DFPT in VASP, with force constants from IBRION = 8 and dispersion relations evaluated using PHONOPY.Source: §A.3 · pp. 30, 34, Table 9 · p. 31, Table 12 · p. 34
• Under the strict -0.001 THz threshold, MatterSim reaches TS = 97.62% and FU = 2.38% on the 120-material OOD dataset; at -1.0 THz, TU reaches 79.66%.Prediction performance varies by only a few percent between commonly used thresholds of -0.1 THz and -1.0 THz.Source: §A.3 · pp. 32–33, Table 10 · p. 32, Figure 9 · p. 32
• At the strict -0.001 THz threshold, PET-OAM-XL has TS = 30.95% and TU = 97.01%, while MatterSim has TS = 97.62% and competitive TU = 74.63%.Across thresholds, MatterSim maintains top-tier accuracy while PET-OAM-XL remains strongly biased toward instability; MatterSim also predicts about 25 structures per hour, versus roughly eight-times-slower eSEN-30M-OAM.Source: §A.3 · pp. 33–34, Table 11 · p. 33

B.1 Element

• Element-resolved analysis finds instability probabilities above 85% for F, O, Cl, H, Br, and I, with proposed causes including directional bonding, weak bonding, and complex coordination.Highly electronegative nonmetals, alkali metals, and some transition metals are associated with distinct instability mechanisms.Source: §B.1 · pp. 34–35, Figure 10 · p. 36

B.2 Chemical-Complexity

• At -0.001 THz, instability probability rises from 55.78% for binary systems to 93.56% for senary systems; at -0.1 THz it rises from 40.73% to 67.93%.The increase is reported across thresholds and attributed to the greater difficulty of satisfying bonding, charge-balance, and geometric constraints in multicomponent systems.Source: §B.2 · pp. 36–37, Figure 11 · p. 37

B.3 Coordination Environments

• Instability probability decreases from approximately 84% at avg_cn ≈ 1.5 to about 38% at avg_cn ≥ 9, with the negative trend persisting across thresholds.The authors associate higher coordination with greater lattice connectivity and rigidity, which suppress soft phonon modes.Source: §B.3 · pp. 37–38, Figure 12 · p. 39

B.4 Formation Energy

• At -0.1 THz, 66.62% of positive-formation-energy materials are dynamically unstable, compared with 53.21% of negative-formation-energy materials; the Pearson correlation is only 0.0133.The authors interpret the modest difference and near-zero correlation as weak overall coupling between thermodynamic and dynamical stability.Source: §B.4 · pp. 39–40, Figure 13 · p. 40, Table 13 · p. 41

B.5 Metal vs. Insulator

• Insulators have a higher fraction of unstable structures at weak thresholds around -0.01 to -0.1 THz, but the difference from metals nearly disappears below approximately -1 THz.The excess instability in insulators is attributed primarily to weak soft phonon modes, including distortions such as ferroelectric instabilities and octahedral tilting.Source: §B.5 · p. 42, Figure 14 · p. 42

B.6 Space Group Symmetry

• At ´0.001 THz, triclinic stability is 17.0% versus 49.2% for cubic; at ´0.1 THz, the rates are 39.3% and 53.4%, respectively.Across the intermediate systems, stability also rises monotonically with symmetry under the strictest threshold.Source: §B.6 · p. 43
• Symmetry operations constrain the Hessian matrix, reducing its independent degrees of freedom and restricting the admissible matrix form.The constraints propagate to the dynamical matrix, where they impose block-diagonal structure according to space-group irreducible representations.Source: §B.6 · p. 43
• The authors attribute lower-symmetry systems’ greater susceptibility to local distortions and soft modes to their more unconstrained degrees of freedom.The benchmark also finds an empirical correlation between higher structural symmetry and fewer imaginary phonon modes.Source: §B.6 · pp. 43–44
• Higher-symmetry systems consistently have higher stability ratios, with the relative ordering largely unchanged as the frequency threshold is relaxed.The trend is described as consistent across crystal systems and thresholds, rather than an artifact of a particular threshold.Source: §B.6 · pp. 43–44, Figure 15 · p. 44

C Effect of Training Data Scale on Dynamical Stability

• At ´0.001 THz, DiffCSP stability rises from approximately 0.108 to 0.312 as training data grows; at ´0.01 THz, it rises from 0.118 to 0.340.Under relaxed ´0.1 THz criteria, stability first decreases from 0.249 to 0.228 at 16,384 structures, then reaches 0.527 at larger scales.Source: C Effect of Training Data Scale on… · p. 45
• Stability gains from increasing dataset size are not proportional or monotonic, and performance gains gradually saturate at larger scales.At intermediate scales, models produce borderline unstable structures; the authors conclude that data scaling alone is insufficient and point to explicit physical constraints or inductive biases.Source: C Effect of Training Data Scale on… · pp. 45–46

D.1 CrystaLLM-large

• The study generated 10,902 structures per CrystaLLM prompting strategy, using MatterGen candidate formulas and space groups to target unexplored chemical space.The two strategies were formula prompting and formula-plus-space-group prompting.Source: §D.1 · p. 46

D.2 LLaMA-2 70B

• At ´0.001 THz, LLaMA2-70B reaches 21.7% stability, while the two large CrystaLLM models reach 18.4% and 19.6%.Across LLaMA2-70B sampling settings, stability exceeds 19% and reaches 23.9%; the authors report that larger models and prompt design improve on the small unprompted CrystaLLM.Source: §D.2 · p. 47
• The authors note that LLM-based generators often require substantially more computation and careful prompt and sampling-strategy tuning than GNN-based generators.They contrast this with GNN models’ atomistic structure and lack of dependence on external prompts.Source: §D.2 · p. 47

E Summary of Large-Scale Crystal Generation and Dynamical Stability

• Across all settings, MatterGen has 11,923 dynamically stable structures among 53,391 relaxed structures, a stability rate of 22.3%.These counts aggregate different training datasets and generation conditions.Source: E Summary of Large-Scale Crystal… · pp. 49–50
• The sector angle encodes each model’s relaxed-structure count, while the darker inner sector’s area represents its fraction of dynamically stable structures.The plot compares six models.Source: E Summary of Large-Scale Crystal… · p. 49, Figure 17 · p. 49
• Because the plot pools training datasets and generation conditions, its rates differ slightly from single-setting main-text analyses.The plot applies a unified phonon-based evaluation to successfully relaxed structures.Source: E Summary of Large-Scale Crystal… · p. 49, Figure 17 · p. 49

F Convergence Analysis of Dynamical Stability Rate

• Above approximately 4,000 tested samples, stability-rate estimation errors become sufficiently small that further sampling negligibly affects model ranking.The convergence curves use 80 bootstrap iterations and show narrowing 95% confidence intervals; CrystaLLM’s slightly smaller effective sample remains insufficient to change its ranking.Source: F Convergence Analysis of Dynamical… · p. 50, Figure 18 · p. 51

G Summary of Crystal Generation

• Table 17 summarizes generation, relaxation, and dynamical-stability counts, with random subsets of up to 10,000 structures selected for relaxation when available.InvDesFlow-AL-Alex20 and MatterGen-Alex20 are larger-sample test cases; InvDesFlow-AL-MP20 stability is estimated by sampling 8,000 DiffCSP-MP20 structures over five trials.Source: G Summary of Crystal Generation · pp. 51–52, Table 17 · p. 52

H Crystal Relaxation and Dynamical Stability Statistics Across Bandgap Ranges

• Table 18 reports relaxation and dynamical-stability counts across bandgap ranges, while Figure 19 shows the elemental distribution of 28,119 dynamically stable crystals.The supplied text describes the scope of these summaries but does not provide their bandgap-specific or elemental values.Source: H Crystal Relaxation and Dynamical… · p. 52, Table 18 · p. 52, Figure 19 · p. 53

• Explicit references and access links were added for every dataset listed in Table 19.Source: I Data Download · p. 53
• Table 19 summarizes the datasets used in this work.Source: Table 19 · p. 53
• All datasets listed in Table 19 are publicly available through the provided links.Source: Table 19 · p. 53