PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable deeper graph neural networks for high-performance materials property prediction

Sadman Sadeed Omee, Steph-Yves Louis, Nihang Fu, Lai Wei, Sourin Dey, Rongzhi Dong, Qinyang Li, Jianjun Hu

arXiv:2109.12283v1cond-mat.mtrl-scics.LG

TL;DR

Materials-property GNNs face an unresolved question of whether greater depth can improve state-of-the-art prediction, while over-smoothing limits their scalability. DeeperGATGNN uses differentiable group normalization and skip connections to train deeper models. It achieves state-of-the-art performance on five of six benchmarks, with improvements up to 10%.

  • Problem

    Whether deeper graph neural networks can push state-of-the-art materials-property prediction remains unstudied, while over-smoothing limits GNN depth.

    Source: §1 · p. 3

  • Method

    DeeperGATGNN combines graph attention, differentiable group normalization, and residual skip connections to train deeper GNNs.

    Source: §1 · p. 3

  • Results

    Five of six benchmark datasets achieved state-of-the-art performance, with mean absolute error reductions up to 10%.

    Source: §1 · p. 3, §5 · p. 15

  • Takeaways & Limitations

    The results support using large-scale, very deep GNNs to address the complexity of mapping crystal structures to materials properties.

    Source: §1 · p. 3, §5 · p. 15

    On the 2D materials dataset, the authors identify limited data or input information as a performance constraint and suggest angular structural information may be needed for further improvement.

    Source: §4 · p. 15

Abstract

from arXiv · show

Machine learning (ML) based materials discovery has emerged as one of the most promising approaches for breakthroughs in materials science. While heuristic knowledge based descriptors have been combined with ML algorithms to achieve good performance, the complexity of the physicochemical mechanisms makes it urgently needed to exploit representation learning from either compositions or structures for building highly effective materials machine learning models. Among these methods, the graph neural networks have shown the best performance by its capability to learn high-level features from crystal structures. However, all these models suffer from their inability to scale up the models due to the over-smoothing issue of their message-passing GNN architecture. Here we propose a novel graph attention neural network model DeeperGATGNN with differentiable group normalization and skip-connections, which allows to train very deep graph neural network models (e.g. 30 layers compared to 3-9 layers in previous works). Through systematic benchmark studies over six benchmark datasets for energy and band gap predictions, we show that our scalable DeeperGATGNN model needs little costly hyper-parameter tuning for different datasets and achieves the state-of-the-art prediction performances over five properties out of six with up to 10\% improvement. Our work shows that to deal with the high complexity of mapping the crystal materials structures to their properties, large-scale very deep graph neural networks are needed to achieve robust performances.

1 Introduction

Materials-property GNNs had not yet shown whether greater depth could push state-of-the-art performance, while over-smoothing limited existing models. DeeperGATGNN combines differentiable group normalization and residual skip connections, and the paper reports state-of-the-art results on five of six benchmarks.

1 Introduction

  • Structure-based materials models tend to predict more accurately than composition-based models, but limited structure data and property labels remain a challenge.Structure-based models can screen known structure repositories and hypothetical crystals with generated structures.Source: §1 · p. 2
  • Over-smoothing limits graph neural network depth; earlier materials GNNs typically used only four or five graph-convolution layers.As layers increase, node representations tend to become similar, making these models difficult to scale in depth.Source: §1 · p. 2
  • Prior deep materials prediction studies used vector representations, leaving deeper graph models for capturing structure–property effects unexamined.The paper identifies whether deeper GNNs can improve state-of-the-art materials prediction as an open question.Source: §1 · pp. 2–3
  • Five of six benchmark datasets achieved state-of-the-art results, with mean absolute error reductions up to 10%.The paper reports this outcome for DeeperGATGNN.Source: §1 · p. 3
  • DeeperGATGNN combines global attention, differentiable group normalization, and residual skip connections to train much deeper graph neural networks.The reported depth reaches 30 or 50 layers, compared with the usual 4–9 graph-convolution layers.Source: §1 · p. 3
  • The authors also apply their depth-scaling strategy to four other GNNs, reporting improvements for some datasets.The variants are DeeperCGCNN, DeeperMEGNet, DeeperMPNN, and DeeperSchNet.Source: §1 · p. 3

2 Methods

The methods combine local and global graph attention with differentiable group normalization and residual skip connections to support deeper materials-property models. They evaluate predictions using regression metrics and cross-validation or hold-out tests.

2.1 Overall architecture for deep graph attention neural networks

  • GATGNN performance begins to decrease beyond seven layers, motivating the use of skip connections and differentiable normalization to address over-smoothing.The authors aim to extract more inter-atomic dependent features with the deeper model.Source: §2.1 · pp. 3–4
  • DeeperGATGNN applies local neighbor attention, then global node attention before pooling, followed by fully connected layers and an output layer.The global attention calculates each node’s overall importance.Source: §2.1 · pp. 3–4, Figure 1 · p. 4

2.2 Differentiable Group Normalization

  • Differentiable group normalization assigns nodes to clusters and normalizes each cluster separately, preserving input embeddings to prevent over-normalization.Separate group means and standard deviations reduce similarity across groups.Source: §2.2 · pp. 4–5

2.3 Skip connections for enabling scalable graph neural networks

  • DeeperGATGNN implements layer-wise residual skip connections in its global-attention graph network.The residual design adds an input to stacked-layer output; the paper describes its implementation as similar to IRNet.Source: §2.3 · p. 5, Figure 2 · p. 6

2.4 Evaluation criteria

  • The evaluation uses MAE, MSE, and R2, with either five-fold cross-validation or hold-out tests depending on the experiment.Baseline and model hyperparameters are provided in supplementary files.Source: §2.4 · p. 6

3 Experimental Results

Across six materials-property datasets, DeeperGATGNN achieves the best results on five, while supporting much deeper networks than the compared GNNs. Parameter studies and latent-space visualizations further characterize its behavior and learned representations.

3.1 Datasets

  • The six datasets cover nanocluster, alloy-surface, bulk, 2D-material, and MOF energy predictions, plus bulk-material band-gap prediction; sample counts range from 3,814 to 37,334.The datasets contain between 1 and 87 element types.Source: §3.1 · p. 6, Table 1 · p. 6

3.2 Performance Comparison of DeeperGATGNN with other state-of-the-art GNN models

  • Across six datasets, DeeperGATGNN achieves the best results on five, with improvements of 0.05% to 10.29% over corrected benchmark results.The comparison uses a single hyperparameter setting across datasets, apart from graph-convolution layer count.Source: §3.2 · p. 7, Table 3 · p. 8

3.3 Parameter study of DeeperGATGNN

  • On Pt-cluster data, 35 graph-convolution layers achieve an MAE of 0.12801, while performance declines as layers increase through 30.The study also finds that increasing dropout degrades performance, batch size 100 performs best among tested sizes, and Softplus outperforms ReLU and Leaky-ReLU.Source: §3.3 · pp. 11–12

3.4 Scalability comparison with existing graph neural networks

  • DeeperGATGNN is the only evaluated model that remains viable at 30 layers; its bulk-formation-energy MAE there is 0.03041, although performance did not improve beyond 15 layers.Other original and modified GNNs become unscalable at 20 or 30 layers; MPNN was excluded from this study because of its high parameter count and training cost.Source: §3.4 · pp. 12–14, Figure 9 · p. 13

3.5 Physical insights from scalable DeeperGATGNN

  • t-SNE plots of alloy-surface latent representations show same-color regions across models, but cluster sizes and distances are not meaningful measures in these plots.The plotted representations are outputs from the first layer after the final graph-convolution layer; colors encode alloy surface energy.Source: §3.5 · p. 14, Figure 10 · p. 14

4 Discussion

The discussion identifies costly training and hyperparameter search, over-smoothing in deep GNNs, and limited data or input information as constraints on materials-property prediction. It describes how normalization and skip connections relate to scalability, while noting that further gains may require additional structural information.

4 Discussion

  • Complex GNN training and extensive hyperparameter searches can leave models under-trained, while the authors report that their DeeperGATGNN achieved SOTA across five datasets with one architecture and minor tuning.The discussion says existing networks did not stagnate until 500 epochs and describes 500-epoch evaluation as costly; it gives an Alloy CGCNN example of 0.06 eV MAE at 250 epochs versus 0.042 eV at 500.Source: §4 · p. 15
  • At 30 convolution layers, existing GNNs except DeeperGATGNN show significant performance degradation; normalization and skip connections help other models, but less consistently.The discussion names SchNet, CGCNN, and MEGNet as models for which the strategy helps but does not work equally well.Source: §4 · p. 15
  • On the 2D materials dataset with 3,814 samples, DeeperGATGNN's MAE of 0.17185 is close to MEGNet's 0.17194, and further improvement may need angular structural information.The cited comparison reports MEGNet with 816,801 parameters and DeeperGATGNN with 25 layers and 913,546 parameters.Source: §4 · p. 15

5 Conclusion

The conclusion presents DeeperGATGNN as a scalable global-attention model for materials-property prediction. It reports state-of-the-art results on five of six benchmarks and notes that its deeper-network strategies also help other GNNs on some datasets.

5 Conclusion

  • Five of six benchmark datasets reached state-of-the-art results, with up to 10% improvement over previous graph neural network results.The conclusion attributes the outcome to a scalable global-attention graph neural network, DeeperGATGNN.Source: §5 · p. 15
  • DeeperGATGNN uses one neural architecture and hyperparameter set, varying only an easy-to-set, sufficiently large graph-convolution layer count.The conclusion presents this as avoiding expensive hyperparameter tuning in practical materials-property prediction.Source: §5 · p. 15
  • Residual skip connections and differentiable group normalization also improve the scalability and performance of MEGNet, SchNet, and CGCNN, but only on special datasets.Source: §5 · p. 15

7 Contribution

The contribution statement assigns roles spanning conceptualization, methodology, software, validation, investigation, and other research and publication tasks.

7 Contribution

  • J.H. is credited with conceptualization, resources, supervision, and funding acquisition.Source: §7 · pp. 15–16
  • Methodology is credited to J.H., S.O., and S.L.; software to S.O. and S.L.; and validation to S.O. and J.H.Source: §7 · p. 15
  • Investigation is credited to J.H., S.O., S.L., N.F., L.W., S.D., R.D., and Q.L.; writing and visualization roles are also listed.Source: §7 · pp. 15–16

Related papers

Loading 2109.12283v1…