PaperlayerView exact revision on arXiv

Source-linked AI summary

Periodic Graph Transformers for Crystal Material Property Prediction

Keqiang Yan, Yi Liu, Yuchao Lin, Shuiwang Ji

arXiv:2209.11807v1cs.LG

TL;DR

Crystal property prediction must represent periodic structures invariantly to artificial cell-boundary choices while preserving lattice-dependent repeating patterns. Matformer uses periodic-invariant graph construction and geometric-distance encoding of lattice patterns, and it consistently outperforms baselines across benchmark tasks. The paper also identifies limitations in existing neighbor-selection settings and points to angular encoding as future work.

  • Problem

    Crystal graphs require periodic invariance to boundary shifts and cell scaling, while also needing explicit encoding of repeating patterns that distinguish lattice sizes and orientations.

  • Method

    Matformer combines periodic-invariant graph construction with explicit periodic-pattern encoding through geometric distances between atoms in neighboring cells.

  • Results

    Matformer consistently outperforms baselines across benchmark tasks, including Materials Project and Jarvis evaluations.

  • Takeaways & Limitations

    The experiments support the importance of both periodic invariance and explicit periodic-pattern encoding for crystal representation learning.

  • Takeaways & Limitations

    Existing neighbor selection by choosing a fixed number of nearest atoms cannot guarantee periodic invariance when distance ties occur.

Abstract

from arXiv · show

We consider representation learning on periodic graphs encoding crystal materials. Different from regular graphs, periodic graphs consist of a minimum unit cell repeating itself on a regular lattice in 3D space. How to effectively encode these periodic structures poses unique challenges not present in regular graph representation learning. In addition to being E(3) invariant, periodic graph representations need to be periodic invariant. That is, the learned representations should be invariant to shifts of cell boundaries as they are artificially imposed. Furthermore, the periodic repeating patterns need to be captured explicitly as lattices of different sizes and orientations may correspond to different materials. In this work, we propose a transformer architecture, known as Matformer, for periodic graph representation learning. Our Matformer is designed to be invariant to periodicity and can capture repeating patterns explicitly. In particular, Matformer encodes periodic patterns by efficient use of geometric distances between the same atoms in neighboring cells. Experimental results on multiple common benchmark datasets show that our Matformer outperforms baseline methods consistently. In addition, our results demonstrate the importance of periodic invariance and explicit repeating pattern encoding for crystal representation learning.

1 Introduction

Crystal property prediction requires representations that handle periodic structures rather than treating crystals like ordinary molecular graphs. Matformer addresses periodic invariance and explicitly encodes repeating patterns, outperforming baselines across benchmark tasks.

  • Motivation: Crystals comprise minimum unit cells repeating on regular 3D lattices, creating representation challenges absent from ordinary molecular graphs.Periodic structures require properties of the repeating crystal to be encoded, not just local molecular-style connectivity.
  • Motivation: Periodic invariance requires representations to remain unchanged under scaling of repeatable cells and shifts of artificial periodic boundaries.The latter boundary-shift requirement was rarely identified explicitly in prior studies.
  • Matformer: Matformer combines periodic-invariant graph construction with explicit encoding of periodic patterns using geometric distances between identical atoms in neighboring cells.The design targets both invariance and representation of lattice-dependent repeating structure.
  • Results: Experiments on Materials Project and Jarvis benchmarks show that Matformer consistently outperforms baseline methods across various tasks.The reported results also demonstrate the importance of periodic invariance and explicit repeating-pattern encoding.

2 Background

A crystal is represented by atom features, positions, and a lattice matrix describing how a unit cell repeats through space. Multi-edge graphs include repeated atomic images within a radius to model interactions across artificial cell boundaries.

  • Crystal representation: A crystal representation consists of atom features A, positions P, and lattice matrix L, which specifies repetition along three lattice vectors.The lattice vectors need not be orthogonal in practical three-dimensional crystals.
  • Crystal representation: The infinite crystal contains translated copies of each unit-cell atom at positions p_i + k_1ℓ_1 + k_2ℓ_2 + k_3ℓ_3 for integer k_1, k_2, and k_3.Repeated copies retain the same atom features as their source atoms.
  • Multi-edge graph construction: In a multi-edge graph, each node represents an atom together with all its duplicates in the infinite crystal.This representation permits graph edges to connect atoms across unit-cell boundaries.
  • Multi-edge graph construction: For every repeated position of atom j within radius r of atom i, the construction adds an edge from j to i with the corresponding Euclidean distance as an edge feature.If m repeated positions fall within the radius, the method creates m edges, capturing cross-boundary interactions.

3 Periodic invariance and periodic pattern encoding for crystals

Crystal representations must be invariant to equivalent choices of unit-cell boundaries while still encoding lattice-dependent repeating patterns. The paper motivates these requirements by showing that boundary-sensitive graphs can misrepresent the same crystal and that multi-edge graphs omit global periodic structure.

  • Periodic invariance: Unit-cell E(3) invariance keeps representations unchanged under joint rotations or reflections of positions and lattice vectors, and translations of positions.This formalizes ordinary geometric invariance for crystal unit cells.
  • Periodic invariance: Different choices of period boundaries can produce different unit-cell position matrices for the same crystal, motivating periodic invariance.The requirement also covers scaling up a repeatable unit cell.
  • Periodic pattern encoding: Matformer adds self-connecting edges that encode six geometric distances sufficient to determine lattice-vector lengths and relative orientations.The construction preserves periodic invariance while representing the lattice geometry.
  • Periodic invariance: A graph construction that treats atoms as single nodes can break periodic invariance when shifting the boundaries changes the resulting graph.The paper identifies this issue as producing different crystal graphs for one physical structure.
  • Periodic pattern encoding: The multi-edge graph captures local atom interactions but does not explicitly encode the periodic patterns of the infinite structure.Without lattice information, crystals can be treated as finite structures similar to molecules.

4 The proposed Matformer

Matformer constructs crystal graphs in two stages: periodic-invariant graph construction followed by explicit encoding of lattice patterns. Its message passing uses edge-wise attention and aggregates multi-edge neighborhood information for node updates.

  • Invariant graph construction: Matformer uses radius-based or fully connected graph construction methods that treat each atom together with its repeated duplicates.Both methods satisfy periodic invariance; the radius-based method is used in the main experiments because of better empirical performance.
  • Invariant graph construction: The fully connected construction connects every node pair with t edges corresponding to the t smallest periodic-image distances.
  • Periodic pattern encoding: Matformer encodes lattice sizes and orientations by adding self-connecting edges based on distances between atoms and their repeated duplicates.
  • Message passing scheme: The message passing scheme computes edge-wise attention, edge-wise value messages, and node updates over constructed crystal graphs.Queries attend to node and edge representations, while values are transformed before aggregation.
  • Message passing scheme: Matformer omits softmax so attention can distinguish nodes with different degrees and the network becomes more efficient.
  • Message passing scheme: Node features are updated by aggregating messages from multi-edge neighborhoods and applying message and feature transformations.The layer uses layer normalization, linear transformations, activation, and batch normalization in the update process.

5 Related work

Prior crystal-learning methods use chemical formulas or 3D graph representations, but periodic invariance and periodic pattern encoding have received limited attention. Matformer is designed specifically for crystals by addressing both components, unlike molecular graph transformers such as Graphormer.

  • Crystal property prediction: Existing crystal-property methods either process chemical formulas with sequence models or represent 3D structures as graphs for GNN learning.
  • Geometric GNNs and graph transformer: Geometric neural networks and Graphormer incorporate geometric information but are primarily designed for non-periodic molecular or general graph settings.
  • Differences with our method: Periodic invariance and periodic pattern encoding are rarely identified and explored in prior crystal-property prediction work.
  • Crystal property prediction: Table 1 compares retrained models using identical training, validation, and test sets with test MAE on The Materials Project dataset.
  • Differences with our method: Matformer differs from Graphormer by considering both periodic invariance and periodic patterns in crystal representation learning.

6 Experimental studies

Experiments across The Materials Project and JARVIS show that Matformer consistently outperforms baselines, including on tasks with limited training data, while also improving efficiency. Ablations support the importance of periodic invariance and explicit repeating-pattern encoding.

  • Benchmark results: Matformer achieves the best performance on all The Materials Project tasks, reducing formation energy by 4.5% relative to the second-best model.It also performs best on Bulk Moduli and Shear Moduli with only 4664 training samples.
  • Benchmark results: Matformer outperforms baselines on all five JARVIS tasks and reduces JARVIS Ehull by 0.012, or 15.8% of ALIGNN.The passage attributes stronger discriminative ability relative to ALIGNN to explicit periodic-pattern encoding.
  • Efficiency: Matformer is three times faster than ALIGNN in total training time and nearly three times faster in inference, while using a lighter model.The comparison uses the JARVIS formation-energy dataset and reports inference over the whole test set.
  • Energy within threshold: Matformer outperforms ALIGNN consistently on all three energy tasks under energy within threshold, with gains mainly from predictions within absolute error 0.01.With 15422 more training samples in The Materials Project than JARVIS, Matformer’s percentage within 0.01 increases by 14.69%.
  • Ablation studies: Breaking periodic invariance with OCgraph causes test MAE to drop by 53% relative to radius-based multi-edge graphs, while Radius also outperforms T-fully.The OCgraph construction produces more than n^2 edges for n atoms in a cell.
  • Ablation studies: Removing periodic pattern encoding worsens test MAE from 0.0325 to 0.0337, while adding angular information increases training time by around three times without much performance gain.The authors suggest that periodic-invariant graph construction and pattern encoding already capture sufficient structural information.

7 Conclusions and discussions

The paper reports consistent improvement over baseline methods and highlights periodic invariance and explicit periodic-pattern encoding as important for crystal representation learning. It also identifies angular-information handling as a limitation and future direction.

  • The angular-information ablation reports training time per epoch, total training time, and test MAE.
  • Matformer outperforms baseline methods consistently on common benchmark datasets.
  • The results demonstrate the importance of periodic invariance and explicit periodic pattern encoding for crystal representation learning.
  • Angular information remains a limitation because incorporating it properly while preserving periodic invariance and low time complexity is left for future work.

A.1 Cases breaking periodic invariance

Two graph-construction strategies can break periodic invariance: boundary-sensitive node construction and distance-only nearest-neighbor selection. These failures arise because shifting boundaries can change graph structure or leave tied neighbors unresolved deterministically.

  • Graphormer’s OC20 construction treats each atom as one node, so shifting periodic boundaries can substantially change the fully connected graph.
  • The illustrated Graphormer graphs differ after a boundary shift, including three blue atoms on the left versus two on the right.
  • Nearest-neighbor construction based only on pairwise distances is nondeterministic when multiple atoms tie at the same distance from the center atom.
  • Distance-only nearest-neighbor selection therefore can produce different neighborhood formations and break periodic invariance.

A.2 Proofs of periodic invariance

The proofs establish periodic invariance for Matformer’s multi-edge and fully connected crystal graphs by considering periodic copies and distances across cell boundaries. The argument assumes that the supplied lattice describes the minimum repeating pattern.

  • The crystal is represented by atom features, positions, and lattice vectors, with periodic copies generated by integer combinations of the lattice vectors.
  • The proof assumes the provided lattice L is the minimum repeating pattern, excluding inputs of the form L′ = αL for α ∈ N^3+.
  • The multi-edge graph connects atom pairs using all periodic-image distances within a prefixed radius threshold r.
  • Because periodic-boundary shifts do not change these pairwise Euclidean distances, the multi-edge construction satisfies periodic invariance.
  • A deterministic radius function, such as selecting the 12-th smallest distance, preserves invariance because identical distance inputs yield the same radius.
  • The fully connected construction selects the t smallest periodic-image distances, and boundary shifts likewise leave those distances unchanged.

A.3 Dataset descriptions

The experiments use the Materials Project and JARVIS crystal datasets with established train, validation, and test splits. The datasets cover formation energy, band gaps, mechanical moduli, total energy, and Ehull.

  • The dataset descriptions cover two experimental crystal datasets: Materials Project and JARVIS.
  • Materials Project experiments use ALIGNN splits for formation energy and band gap, and GATGNN splits for Bulk Moduli and Shear Moduli.
  • JARVIS evaluation covers formation energy, bandgap(OPT), bandgap(MBJ), total energy, and Ehull using the stated training, validation, and test sets.

A.4 Matformer configurations

The configurations specify Matformer’s graph construction, embeddings, message passing, readout, training setup, and comparison procedures. An ablation further evaluates the aggregation operation used to capture node-degree information in multi-edge crystal graphs.

  • Crystal graph construction: Matformer uses a radius-based multi-edge graph construction with the 12-th smallest distance defining each atom’s neighborhood radius.Nearby atoms within that radius are included, and the construction is stated to satisfy periodic invariance.
  • Node and edge embeddings: Atomic numbers are embedded into 92 dimensions and projected to 128 dimensions, while edge distances use 128 radial basis function kernels over 0.0 to 8.0.
  • Matformer layer: Matformer uses five message-passing layers with four attention heads, and concatenates head outputs before mapping them to a 128-dimensional message.Different heads use separate linear transformations for queries, keys, values, and edge embeddings.
  • Readout layer: Mean pooling aggregates node features, followed by a nonlinear layer with hidden dimension 128 and a linear layer producing the crystal-graph scalar output.
  • Training: Training uses Adam with 1e-5 weight decay, batch size 64, a one-cycle learning-rate scheduler, mean squared error for optimization, and mean absolute error for evaluation.Learning rates and training epochs are adjusted slightly across tasks in The Materials Project and JARVIS.
  • Operation ablation: Sigmoid and normalization significantly improve test MAE over scalar- and vector-attention softmax alternatives in the operation ablation.The operation is designed to capture node-degree information in multi-edge crystal graphs.
Loading 2209.11807v1…