PaperlayerView exact revision on arXiv

Source-linked AI summary

Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs

Yi-Lun Liao, Tess Smidt

arXiv:2206.11990v2cs.LGcs.AIphysics.comp-ph

TL;DR

Transformers have yet to perform well across 3D atomistic-graph datasets despite translational-invariance and rotational-equivariance biases, a gap relevant to machine-learned prediction of atomistic quantum properties. Equiformer uses dot-product attention and linear message passing with SE(3)/E(3)-equivariant irreps features and tensor products, and proposes a novel equivariant graph attention mechanism. Equiformer achieves overall better results across 12 regression tasks than each individual model, with experiments on QM9, MD17, and OC20 demonstrating its effectiveness.

  • Problem

    Transformers have yet to perform well across 3D atomistic-graph datasets despite translational-invariance and rotational-equivariance biases, a gap relevant to machine-learned prediction of atomistic quantum properties.

    Source: §1 · p. 1

  • Method

    Equiformer uses dot-product attention and linear message passing with SE(3)/E(3)-equivariant irreps features and tensor products, and proposes a novel equivariant graph attention mechanism.

    Source: §4 · p. 4

  • Results

    Equiformer achieves overall better results across 12 regression tasks than each individual model, with experiments on QM9, MD17, and OC20 demonstrating its effectiveness.

    Source: §5.1 · p. 7, §6 · p. 9

  • Takeaways & Limitations

    The experiments support the effectiveness of combining Transformer architectures with equivariant features on QM9, MD17, and OC20.

    Source: §6 · p. 9

    Higher irreps degrees can increase feature size, and tensor products can be computationally expensive.

    Source: G LIMITATIONS · p. 30

Abstract

from arXiv · show

Despite their widespread success in various domains, Transformer networks have yet to perform well across datasets in the domain of 3D atomistic graphs such as molecules even when 3D-related inductive biases like translational invariance and rotational equivariance are considered. In this paper, we demonstrate that Transformers can generalize well to 3D atomistic graphs and present Equiformer, a graph neural network leveraging the strength of Transformer architectures and incorporating SE(3)/E(3)-equivariant features based on irreducible representations (irreps). First, we propose a simple and effective architecture by only replacing original operations in Transformers with their equivariant counterparts and including tensor products. Using equivariant operations enables encoding equivariant information in channels of irreps features without complicating graph structures. With minimal modifications to Transformers, this architecture has already achieved strong empirical results. Second, we propose a novel attention mechanism called equivariant graph attention, which improves upon typical attention in Transformers through replacing dot product attention with multi-layer perceptron attention and including non-linear message passing. With these two innovations, Equiformer achieves competitive results to previous models on QM9, MD17 and OC20 datasets.

1 INTRODUCTION

Equiformer argues that Transformers for 3D atomistic graphs need built-in Euclidean symmetry and presents an irreps-based equivariant architecture with redesigned attention and message passing. It reports competitive results on QM9, MD17, and OC20.

1 INTRODUCTION

  • Atomistic graphs call for E(3)-related inductive biases—translation, rotation, and inversion equivariance—because quantities such as energy remain constant under shifts while forces rotate with the system.Equivariant networks use geometric tensors such as vectors, while invariant approaches augment graphs with information such as distances and angles.Source: §1 · p. 1
  • Equiformer reports better QM9 and MD17 results than previous models and improves on OC20 state-of-the-art models, achieving the best IS2RE result with IS2RE/IS2RS data.The best IS2RE result uses only IS2RE and IS2RS data; compared with previous models, Equiformer improves training time by 2.3× to 15.5×.Source: §1 · p. 2
  • The initial Equiformer replaces Transformer operations with equivariant counterparts and tensor products, encoding equivariant information in irreps channels without complicating graph structure.This version uses dot-product attention and linear message passing, with minimal Transformer modifications.Source: §1 · p. 2
  • Equiformer’s equivariant graph attention replaces dot-product attention with MLP attention and adds nonlinear message passing.The proposed combination is described as more expressive than linear or nonlinear messages alone and MLP or dot-product attention alone; it supports vectors of any type and higher-order tensor-product interactions.Source: §1 · p. 2
  • Although Transformers have succeeded in other domains, they have performed poorly across 3D atomistic graph datasets even with relevant inductive biases, motivating architectures tailored to this domain.The introduction also notes that a pure Vision Transformer generalizes poorly on ImageNet relative to CNNs when trained only on that dataset, illustrating the importance of domain-related inductive biases.Source: §1 · pp. 1–2

2 RELATED WORKS

  • Earlier equivariant models use different combinations of equivariant operations: TFN and NequIP use linear graph-convolution messages, SEGNN uses nonlinear messages, and SE(3)-Transformer uses dot-product attention with linear messages.Later equivariant Transformers retain dot-product attention and linear messages but use specialized architectures for type-0 and type-1 vectors; Equiformer supports vectors of higher types and higher-order tensor products.Source: §1 · p. 2

3 BACKGROUND

Equivariant learning for 3D atomistic graphs respects coordinate-system symmetries by encoding features and operations with irreducible representations (irreps). The background explains how spherical harmonics initialize these features and tensor products combine them.

3.1 E(3) EQUIVARIANCE

  • For atomistic graphs, coordinate changes leave scalar properties such as energy unchanged while rotating quantities such as force; Equiformer uses SE(3)/E(3)-equivariant features and operations, benchmarking SE(3) in the main text.Translation is handled through relative positions; inversion and E(3)-equivariance results are discussed in the appendix.Source: §3.1 · pp. 2–3

3.2 IRREDUCIBLE REPRESENTATIONS

  • SO(3) representations decompose into type-L irreps, concatenated as features with channels; spherical harmonics of relative positions initialize them, and equivariant operations propagate them.Degree L determines angular frequency, and scalars and Euclidean vectors are type-0 and type-1, respectively.Source: §3.2 · p. 3

3.3 TENSOR PRODUCT

  • Tensor products combine type-L vectors via Clebsch-Gordan coefficients; learnable weights attach to equivariant paths, outputs above Lmax are discarded, and weights may depend on relative distances.Allowed output degrees satisfy |L1 − L2| ≤ L3 ≤ |L1 + L2|.Source: §3.3 · pp. 3–4

4 EQUIFORMER

Equiformer adapts Transformer operations to SE(3)/E(3)-equivariant irreps features and adds equivariant graph attention, combining these innovations in an architecture for atomistic graphs.

4 EQUIFORMER

  • Equiformer combines equivariant Transformer operations and tensor products with equivariant graph attention, replacing dot-product attention and linear message passing in its proposed attention mechanism.The paper presents an initial architecture with dot-product attention and linear message passing, then introduces equivariant graph attention as a second innovation.Source: §4 · p. 4

4.1 EQUIVARIANT OPERATIONS FOR IRREPS FEATURES

  • Equivariant linear layers transform each irreps type separately, omit biases for non-scalar features, and generalize layer normalization using channel-wise RMS norms.For non-scalar vectors, the normalization removes means and biases.Source: §4.1 · p. 4
  • Gate activation applies SiLU to scalar channels and uses sigmoid-transformed scalars to gate higher-order vectors, preserving equivariance.After gating, the number of scalar channels is reduced to the original scalar-channel count.Source: §4.1 · p. 5
  • Depth-wise tensor products let each output vector depend on one input vector, reducing weights and memory when weights depend on relative distances.Weights can also be input-independent; dependence on all input channels with distance-parameterized weights can cause out-of-memory errors.Source: §4.1 · p. 5

4.2 EQUIVARIANT GRAPH ATTENTION

  • Equivariant graph attention uses tensor products for content and geometric information, MLP attention for invariant weights, and nonlinear message passing for values.It forms node-pair features from target and source nodes, then combines them with spherical harmonics of relative position; attention uses scalar features, while gated transformations and tensor products can produce nonlinear messages.Source: §4.2 · pp. 5–6

4.3 OVERALL ARCHITECTURE

  • The overall architecture embeds atom species and local geometry, uses radial bases and learned radial functions for distance-conditioned weights, and predicts scalar quantities by summing node outputs.Its feed-forward networks use two equivariant linear layers with an intermediate gate; embedding and output sums are scaled by the square root of average degree and average atom count, respectively.Source: §4.3 · p. 6

5 EXPERIMENT

Equiformer is evaluated on QM9, MD17, and OC20, with ablations testing dot-product attention and linear messages against the proposed equivariant graph attention. The reported results show strong benchmark performance and gains from more expressive attention or message passing, with effects varying by dataset.

5.1 QM9

  • On QM9, Equiformer performs better overall across 12 regression tasks than each individual model and improves on most equivariant Transformers.The comparison with SEGNN supports the effectiveness of combining non-linear message passing with MLP attention; Equiformer also uses the same architecture for every task, unlike PaiNN and TorchMD-NET on µ and R2.Source: §5.1 · p. 7

5.2 MD17

  • On MD17, Equiformer achieves better overall results across 8 molecules than each individual model, and Lmax = 3 improves MAE for most molecules except benzene.Equiformer with Lmax = 2 achieves overall lower MAE than NequIP, while Lmax = 3 achieves lower MAE for most molecules; for the last three molecules, reweighting energy and force losses can lower both MAEs.Source: §5.2 · pp. 7–8

5.3 OC20

  • Equiformer achieves the lowest MAE on all four OC20 sub-splits in validation and testing without a node-level auxiliary task.The task predicts relaxed energy from an initial structure; evaluation uses MAE and energy within threshold (EwT), which need not track average-error improvements. Table 3 reports OC20 IS2RE testing results.Source: §5.3 · p. 8, Table 3 · p. 8
  • With IS2RS as an auxiliary task, Equiformer achieves the best IS2RE results as of submission using only IS2RE and IS2RS data, with 2.3× to 15.5× training-efficiency gains over invariant networks and Transformers.The reported testing result uses much less computation; Table 4 covers validation and Table 5 testing.Source: §5.3 · p. 8, Table 4 · p. 8, Table 5 · p. 8

5.4 ABLATION STUDY

  • On QM9, dot-product attention with linear messages is competitive, MLP attention has similar results but is roughly 8% faster, and non-linear messages improve results with MLP attention.Non-linear messages increase tensor-product operations per block from one to two; the authors suggest dot-product attention is expressive enough for QM9's smaller neighborhoods and fewer atom species.Source: §5.4 · p. 9, Table 6 · p. 9
  • On OC20, dot-product attention with linear messages already outperforms previous models, while non-linear messages improve results and MLP attention performs better than dot-product attention and is 8% faster.The authors attribute the advantage of MLP attention on OC20 to its larger graphs and more diverse atom species; Table 7 summarizes the ablation.Source: §5.4 · p. 9, Table 7 · p. 9

6 CONCLUSION

Equiformer combines Transformer architectures with irreducible-representation equivariant features and introduces equivariant graph attention. Experiments on QM9, MD17, and OC20 demonstrate its effectiveness, while ablations show the attention mechanism improves on typical Transformer attention.

6 CONCLUSION

  • Equiformer combines Transformer networks with irreducible-representation features, incorporating equivariant operations such as tensor products.Source: §6 · p. 9
  • Equivariant graph attention uses multi-layer perceptron attention and non-linear messages.Source: §6 · p. 9
  • Experiments on QM9, MD17, and OC20 demonstrate Equiformer’s effectiveness, and ablations show its equivariant graph attention improves on typical Transformer attention.Source: §6 · p. 9

7 ETHICS STATEMENT

The authors argue that Equiformer’s molecular-property prediction capabilities offer more potential for productive applications than adversarial misuse, while noting that deployment remains difficult and the method has limitations.

7 ETHICS STATEMENT

  • Equiformer’s quantum-property approximations could support productive investigations of molecules and materials more than adversarial uses such as creating hazardous chemicals.The authors note that substantial hurdles remain between identifying a useful or harmful molecule and deploying it at scale.Source: §7 · p. 10
  • The authors discuss limitations of Equiformer and equivariant graph attention in Appendix Sec. G.Source: §7 · p. 10

8 REPRODUCIBILITY STATEMENT … B.1 GRAPH NEURAL NETWORKS FOR 3D ATOMISTIC GRAPHS

The merged sections provide reproducibility and appendix information, mathematical background on equivariance and irreps, and context on GNNs for 3D atomistic graphs. They explain how symmetry-aware features and operations represent geometric transformations while preserving the relevant symmetries.

8 REPRODUCIBILITY STATEMENT

  • Equiformer’s QM9, MD17, and OC20 reproduction code is available, with architecture, hyperparameter, and training-time details provided in dataset-specific sections.The referenced details are in Sec. D.1 for QM9, Sec. E.1 for MD17, and Sec. F.2 for OC20.Source: §8 · p. 10

APPENDIX

  • The appendix presents Equiformer architecture and computational-complexity details, training details, SE(3)-versus-E(3) equivariance and training-time/parameter comparisons, and OC20 dataset, IS2RE validation, and error-distribution sections.Source: APPENDIX · p. 17

A ADDITIONAL BACKGROUND

A.1 GROUP THEORY

  • The discussion focuses on E(3), SE(3), O(3), and SO(3); SO(3) representations can be decomposed into irreducible representations, or irreps.E(3) includes rotation, translation, and inversion; SE(3) excludes inversion, while O(3) and SO(3) describe rotation with and without inversion, respectively.Source: §A.1 · p. 18

A.2 EQUIVARIANCE

  • Equivariance requires outputs to transform consistently with inputs, while invariance leaves outputs unchanged; the method uses relative positions and O(3) irreps with equivariant operations.For atomistic graphs, relative positions provide translation invariance, while O(3) irreps and equivariant operations handle rotation and inversion.Source: §A.2 · p. 18

A.3 EQUIVARIANT FEATURES BASED ON VECTOR SPACES OF IRREDUCIBLE REPRESENTATIONS

  • O(3) irreps distinguish even and odd parity: scalars are type-(0, e), pseudo-scalars type-(0, o), Euclidean vectors type-(1, o), and pseudo-vectors type-(1, e).Spherical harmonics project Euclidean vectors into type-L features; for E(3), their parity is even for even L and odd for odd L.Source: §A.3 · p. 19

A.4 TENSOR PRODUCT

  • Tensor products combine type-(L, p) features using Clebsch-Gordan coefficients, preserving symmetry while tracking output parity through even/odd multiplication rules.The appendix describes tensor products as general interactions between O(3) irreps features.Source: §A.4 · pp. 19–20

B.1 GRAPH NEURAL NETWORKS FOR 3D ATOMISTIC GRAPHS

  • 3D spatial graphs encode atom positions and geometry, creating a need for invariant or equivariant predictions; proposed approaches use distances and angles or geometric tensors such as type-L vectors.Chemical bond graphs instead represent covalent bonds without considering 3D geometry.Source: §B.1 · p. 20

B.2 DETAILED COMPARISON BETWEEN EQUIVARIANT TRANSFORMERS · B.3 INVARIANT GNNS · B.4 ATTENTION AND TRANSFORMER

Equiformer combines equivariant irreps features with Transformer operations and equivariant graph attention, achieving strong results across QM9, MD17, and OC20. The comparison emphasizes its simple architecture, flexible representations, and ability to use directional information without complicating graph structures.

B.2 DETAILED COMPARISON BETWEEN EQUIVARIANT TRANSFORMERS

  • Equiformer achieves the best results on MD17, QM9, and OC20, while earlier equivariant Transformers show dataset-dependent performance.The paper notes that SE(3)-Transformer underperforms other equivariant networks on QM9, while TorchMD-NET is not comparable to NequIP on MD17 despite being competitive on QM9.Source: §B.2 · p. 20
  • On OC20, Equiformer is competitive with GNS and Graphormer while saving 2.3× to 15.5× training time.The authors state that, to their knowledge, this is the first application of equivariant Transformers to a large, complicated dataset such as OC20.Source: §B.2 · p. 20
  • Equiformer’s base architecture replaces Transformer operations with equivariant counterparts and tensor products, avoiding extra architectural modifications.Unlike SE(3)-Transformer’s norm nonlinearities, Equiformer uses equivariant layer normalization, which the authors find yields lower errors.Source: §B.2 · pp. 20–21
  • Equiformer and SE(3)-Transformer support vectors of any degree L, whereas other compared equivariant Transformers are limited to L = 0 and 1.Prior work cited in the passage finds that higher degrees can improve QM9 and MD17 performance.Source: §B.2 · p. 21
  • Depth-wise tensor products let Equiformer use more channels without out-of-memory errors than fully connected tensor products.On QM9, SE(3)-Transformer uses 16 channels per degree, while Equiformer uses 128, 64, and 32 channels for degrees 0, 1, and 2, respectively.Source: §B.2 · p. 21
  • Equiformer’s equivariant graph attention improves its base architecture with MLP attention and non-linear message passing.The base architecture uses dot-product attention and linear message passing.Source: §B.2 · pp. 20–21

B.3 INVARIANT GNNS

  • Invariant GNNs use distances, angles, or dihedrals; angle-based methods introduce higher-order atom interactions, while Equiformer encodes direction with irreps without complicating graph structures.Triplet or quadruplet representations can be memory-intensive and require non-trivial modifications to generic GNNs.Source: §B.3 · p. 21

B.4 ATTENTION AND TRANSFORMER

  • Graph-attention methods use MLP or dot-product attention, while graph Transformers incorporate graph biases and Equiformer adapts equivariant Transformers to 3D graphs.Examples of graph biases include message passing, Laplacian-spectrum position embeddings, and degree, distance, or edge-feature encodings.Source: §B.4 · pp. 21–22

C DETAILS OF ARCHITECTURE … D.2 COMPARISON BETWEEN SE(3) AND E(3) EQUIVARIANCE

Equiformer’s appendix explains how its irreps operations and attention variants preserve equivariance, how parity is incorporated for E(3), and why the attention and message choices affect computation. On QM9, the reported comparison finds similar performance with inversion included.

C.1 EQUIVARIANT OPERATION USED IN EQUIFORMER

  • Linear maps, RMS-based layer normalization, scalar-gated nonlinearities, and depth-wise tensor products preserve equivariance by respecting irreps and using invariant scalings or weights.The appendix states the equivariance condition f(DX(g)x) = DY(g)f(x); scalar parts admit nonlinear functions, while non-scalar features are scaled or gated using invariant quantities.Source: §C.1 · pp. 22–23

C.3 DOT PRODUCT ATTENTION

  • In the dot-product attention ablation, edge irreps are split into key and value, the query is linearly projected, and scaled dot product of query and key yields attention weights.Source: Figure 4 · p. 23, §C.3 · p. 24

C.2 EQUIFORMER ARCHITECTURE

  • Architecture dimensions are parameterized by embedding width dembed, spherical-harmonic width dsh, attention heads h and dhead, and FFN and output widths dffn and dfeature.Source: §C.2 · pp. 23–24

C.4 INCORPORATING E(3)-EQUIVARIANCE

  • E(3) Equiformer uses parity-tagged type-(L, p) irreps, treats even and odd types separately in linear maps and normalization, and gates scalars while weighting pseudo-scalars and higher-degree vectors.Source: §C.4 · p. 24

C.5 DISCUSSION ON COMPUTATIONAL COMPLEXITY

  • MLP attention uses scalar features for weights, producing fewer, smaller intermediates and running faster than dot-product attention; nonlinear messages double tensor products from one to two, increasing compute time.Source: §C.5 · pp. 24–25

D.1 TRAINING DETAILS

  • 12.1 minutes per epoch were reported for Equiformer on one A6000 GPU, compared with 7.2 minutes for its linear-message variant, 7.8 for linear-message dot-product attention, and 16.3 for E(3)-Equiformer.Source: §D.1 · p. 26

D.2 COMPARISON BETWEEN SE(3) AND E(3) EQUIVARIANCE

  • Table 9 reports similar QM9 performance for Equiformer with SE(3)-equivariant features and E(3)-Equiformer with inversion included.The paper compares the two variants on the QM9 test set; the supplied passage does not provide table values.Source: Table 9 · p. 26, §D.2 · p. 26

D.3 COMPARISON OF TRAINING TIME AND NUMBERS OF PARAMETERS … E.2 ADDITIONAL COMPARISON TO TORCHMD-NET

The sections compare Equiformer’s training efficiency with SEGNN and TorchMD-NET, then detail its MD17 training setup and additional TorchMD-NET comparison.

D.3 COMPARISON OF TRAINING TIME AND NUMBERS OF PARAMETERS

  • Equiformer trains faster than SEGNN despite having more channels and parameters, with comparable training time attributed to more efficient depth-wise tensor products.Each Equiformer message-passing block uses two DTPs, where an output channel depends on one input channel; SEGNN uses four compute-intensive FCTPs per block, where an output channel depends on all input channels.Source: §D.3 · p. 26
  • Equiformer achieves competitive results after 300 or 600 epochs, compared with TorchMD-NET’s 3,000, while its more expressive messages and higher-degree tensors slow each epoch.Non-linear messages double the number of tensor products relative to linear messages, almost doubling training time; higher-degree tensors such as Lmax = 2 improve performance but slow training.Source: §D.3 · p. 26

E DETAILS OF EXPERIMENTS ON MD17

  • MD17 experiments use TorchMD-NET’s data partition and normalize energy and force targets differently.Energy targets are centered and scaled by their standard deviation; force targets are divided by the standard deviation of ground-truth energy.Source: §E.1 · p. 26

E.1 TRAINING DETAILS

  • MD17 Equiformer configurations use six blocks with Lmax = 2 or 3, and their learning rates, batch sizes, epoch counts, and parameter totals vary by setting.For Lmax = 2, the model has about 3.50M parameters, batch size 8, and 1,500 epochs; for Lmax = 3, it has about 5.50M parameters, batch size 5, and 2,000 epochs. On one A5000 GPU, training takes about 15.4 and 54.4 hours per molecule, respectively. Table 11 summarizes MD17 hyperparameters.Source: §E.1 · pp. 26–27, Table 11 · p. 27

D.3 COMPARISON OF TRAINING TIME AND NUMBERS OF PARAMETERS

  • Training Equiformer for 300 and 600 epochs takes 61 and 122 GPU-hours, respectively.Table 10 compares QM9 training time and parameter counts for SEGNN, TorchMD-NET, and Equiformer.Source: §D.3 · p. 26, Table 10 · p. 27

E.2 ADDITIONAL COMPARISON TO TORCHMD-NET

  • For the additional TorchMD-NET comparison, Equiformer adjusts force-loss weighting per molecule so its Lmax = 2 model can achieve lower MAE for both energy and forces.The comparison is motivated by TorchMD-NET’s use of dot-product attention rather than Equiformer’s equivariant graph attention; Table 12 presents the comparison.Source: §E.2 · p. 27

E.3 COMPARISON OF TRAINING TIME AND NUMBERS OF PARAMETERS · F DETAILS OF EXPERIMENTS ON OC20 · F.1 DETAILED DESCRIPTION OF OC20 DATASET

Equiformer generally outperforms NequIP on most molecules, with training-time tradeoffs tied to model configuration and tensor-product operations; the OC20 experiments focus on predicting relaxed energy from initial structures.

E.3 COMPARISON OF TRAINING TIME AND NUMBERS OF PARAMETERS

  • Equiformer achieves better results than NequIP for most molecules, and its Lmax = 2 model has better overall results than NequIP with linear convolution.The comparison identifies equivariant graph attention as the architectural difference supporting the Lmax = 2 result.Source: §E.3 · pp. 27–28
  • Equiformer with Lmax = 2 is faster per epoch than NequIP with Lmax = 3, which uses a larger Lmax.NequIP is trained for more than 1000 epochs, but its epoch count is not reported, so the comparison is per epoch.Source: §E.3 · p. 27, Table 13 · p. 28
  • Equiformer’s training time is longer than NequIP’s because each equivariant graph attention uses two tensor products, versus one in each linear convolution.The paper compares training time and parameter counts in Table 13, whose caption identifies the dataset as MD17.Source: §E.3 · p. 28, Table 13 · p. 28

F.1 DETAILED DESCRIPTION OF OC20 DATASET

  • OC20 systems pair an adsorbate molecule with a catalyst slab, average more than 70 atoms, and include over 50 atom species.The systems contain more atoms and more diverse atom types than those in QM9 and MD17.Source: §F.1 · p. 28
  • OC20 relaxation updates atom positions using density functional theory forces until the system stabilizes as a relaxed structure.The relaxed energy is correlated with catalyst activity and is used to understand adsorbate–catalyst interactions.Source: §F.1 · p. 28
  • The OC20 task is IS2RE: predict the relaxed energy from an initial structure, using 460k training, 100k validation, and 100k test structures.The initial structure places the adsorbate on top of the catalyst.Source: §F.1 · p. 28

F.2 TRAINING DETAILS … F.6 ERROR DISTRIBUTIONS

The appendix documents OC20 training choices and validation reporting, then examines equivariance, efficiency, and error distributions. It finds Equiformer competitive with comparable training time, while model gains vary by split and metric.

F.2 TRAINING DETAILS

  • With IS2RS auxiliary supervision, Equiformer uses 18 Transformer blocks, a 5×10^-4 maximum learning rate, and a relaxed-structure prediction head.The IS2RS loss weight decays linearly from 15 to 1; with Noisy Nodes augmentation, training runs for 40 epochs.Source: §F.2 · p. 28
  • 24 GPU-days train Equiformer + Noisy Nodes on A6000 GPUs, compared with 56 TPU-days for GNS + Noisy Nodes and 372 GPU-days for Graphormer.Graphormer uses an ensemble of 31 models and the cited training time is on A100 GPUs.Source: §F.2 · p. 29

F.3 RESULTS ON IS2RE VALIDATION SET

  • The appendix reports Equiformer results on the OC20 IS2RE validation set in Table 15.The table caption identifies the results as validation-set results and marks results reported by Liu et al. (2022) with †.Source: §F.3 · p. 29, Table 15 · p. 29

F.4 COMPARISON BETWEEN SE(3) AND E(3) EQUIVARIANCE

  • E(3)-Equiformer has slightly inferior overall performance to SE(3)-Equiformer, despite improving MAE on ID and OOD Cat sub-splits.Inversion degrades performance on other sub-splits; the paper suggests it may not be a bottleneck, splits type-1 vectors into types (1,e) and (1,o) that exchange directional information only through depth-wise tensor products, and may require different hyper-parameters.Source: §F.4 · pp. 29–30, Table 16 · p. 30

F.5 COMPARISON OF TRAINING TIME AND NUMBERS OF PARAMETERS

  • Equiformer achieves better results than SEGNN with comparable training time when IS2RS auxiliary supervision is not used.The comparison covers training time and parameter counts; the paper refers to Table 5 for training-time comparison when IS2RS is used.Source: Table 17 · p. 30, §F.5 · p. 30

F.6 ERROR DISTRIBUTIONS

  • 0.5657 to 0.5545 MAE is the improvement from non-linear messages, while EwT remains almost unchanged because the improved errors stay above 0.02 eV.On OOD Both, the error improvement is concentrated among examples ranked 15,000th through 20,000th; gains also vary by split, with MLP attention and non-linear messages helping ID more than OOD Ads.Source: §F.6 · p. 30

G LIMITATIONS

Equiformer’s limitations include the costs of irreps and tensor products, task-dependent gains from stronger attention, and attention complexity tied to local edges that may require modification for other domains.

G LIMITATIONS

  • Higher irrep degrees enlarge features, and tensor products can be compute-intensive; pruning paths may improve efficiency but is left to future work.The authors attribute some tensor-product cost to kernels that are less optimized than common library operations, a software issue rather than a design limitation.Source: G LIMITATIONS · pp. 30, 32
  • MLP attention’s gains depend on the dataset: they are not significant on QM9 but clear on OC20, while nonlinear messages improve over linear ones on both.The authors surmise that QM9’s smaller molecules and less diverse atom types make linear attention sufficient.Source: G LIMITATIONS · p. 32
  • Equivariant graph attention costs more computation than typical graph convolution, adding a softmax and sum aggregation, while nonlinear message passing doubles tensor products from one to two.When training budgets are constrained, stronger attention may slow training without significant improvement on some tasks or datasets.Source: G LIMITATIONS · p. 32
  • Attention complexity scales with channels times edges; although this matches message and convolution complexity on 3D atomistic graphs, other domains may require modifications.The authors contrast this with computer-vision convolution memory complexity, which scales with pixels or nodes rather than edges.Source: G LIMITATIONS · p. 32

Related papers

Loading 2206.11990v2…