PaperlayerView exact revision on arXiv

Source-linked AI summary

Cormorant: Covariant Molecular Neural Networks

Brandon Anderson, Truong-Son Hy, Risi Kondor

arXiv:1906.04015v3physics.comp-phcs.LGstat.ML

TL;DR

Cormorant targets the computational difficulty of modeling many-body molecular behavior when quantum calculations do not scale and classical force fields are limited. It uses atom-subset neurons with covariant spherical-tensor activations and Clebsch–Gordan nonlinearities. The architecture outperforms competing methods on MD-17 potential-energy-surface learning and is competitive across several QM9 molecular-property targets.

  • Problem

    Quantum calculations become infeasible for larger systems, while classical empirical potentials have inherent limitations for molecular dynamics.

  • Method

    Cormorant uses neurons tied to physical atom subsets, spherical-tensor activations with built-in rotational covariance, and Clebsch–Gordan products as nonlinearities.

  • Results

    Cormorant outperforms competing methods on MD-17 potential-energy-surface learning and achieves leading or competitive results on six of twelve QM9 regression targets.

  • Takeaways & Limitations

    The architecture supports learning molecular potential-energy surfaces and geometric, energetic, electronic, and thermodynamic properties from molecular structures.

  • Takeaways & Limitations

    Training can become unstable or yield poor higher-layer activations, and consistent low error was obtained with at most four Clebsch–Gordan layers under the described initialization scheme.

Abstract

from arXiv · show

We propose Cormorant, a rotationally covariant neural network architecture for learning the behavior and properties of complex many-body physical systems. We apply these networks to molecular systems with two goals: learning atomic potential energy surfaces for use in Molecular Dynamics simulations, and learning ground state properties of molecules calculated by Density Functional Theory. Some of the key features of our network are that (a) each neuron explicitly corresponds to a subset of atoms; (b) the activation of each neuron is covariant to rotations, ensuring that overall the network is fully rotationally invariant. Furthermore, the non-linearity in our network is based upon tensor products and the Clebsch-Gordan decomposition, allowing the network to operate entirely in Fourier space. Cormorant significantly outperforms competing algorithms in learning molecular Potential Energy Surfaces from conformational geometries in the MD-17 dataset, and is competitive with other methods at learning geometric, energetic, electronic, and thermodynamic properties of molecules on the GDB-9 dataset.

1 Introduction

Quantum-mechanical descriptions become computationally infeasible for larger molecular systems, while classical force fields have inherent limitations. Cormorant addresses this gap by combining machine-learned force fields with symmetry-aware neural-network design.

  • Motivation: Quantum mechanics is impractical to solve explicitly at every timestep beyond a few dozen atoms, and DFT struggles beyond a few hundred atoms.These scaling limits motivate approximate approaches for molecular dynamics.
  • Motivation: Practical molecular dynamics therefore commonly uses classical empirical potentials, although these models have inherent limitations.Machine learning methods increasingly learn more descriptive force fields directly from DFT computations.
  • Cormorant: Cormorant combines machine-learned force-field efforts with Fourier-space covariant and equivariant neural-network theory.Its design connects learned operations with physically motivated interactions.
  • Cormorant: Cormorant’s architecture explicitly assigns neurons to physical atom subsets and makes activations covariant to rotations and translations.The authors report that this produces learned neuron-level laws resembling known physical interactions.

2 The nature of physical interactions in molecules

Molecular interactions can be organized by the order of their multipole moments, from scalar interactions through dipoles and quadrupoles to higher-order terms. This hierarchy captures increasingly complex directional and many-body structure, but its formulas rapidly become involved.

  • Scalar interactions: Scalar interactions describe pointlike particles without directional degrees of freedom, such as electrostatic attraction or repulsion between charges.The Coulomb energy is an example of this simplest interaction class.
  • Scalar interactions: The Coulomb potential is invariant under translations and rotations because it depends only on the separation length, not direction or absolute positions.This symmetry structure provides a basic example of physically allowable interactions.
  • Dipole/dipole interactions: Dipole interactions use first moments of charged-particle positions and are important for molecular energetics because most chemical bonds are polarized.Dipole interactions also describe magnetic-spin interactions between electrons.
  • Quadrupole/quadrupole interactions: Quadrupole interactions use second moments of charge density and can describe interactions such as those between benzene rings.Their general potential is more complicated and often requires simplifying structural assumptions.
  • Higher-order interactions: Higher-order interactions involve moment tensors of orders 3, 4, 5, and beyond, with formulas that quickly become involved across electrostatic, dispersion, and exchange interactions.This complexity motivates a more general representation of interaction structure.

3 Spherical tensors and representation theory

Spherical tensors decompose rotational behavior into independently transforming irreducible components, making symmetry constraints explicit. Tensor products and Clebsch–Gordan decompositions then provide a general route to constructing rotationally invariant interaction terms.

  • Spherical tensors: Spherical tensors provide a formalism in which the transformation behavior of molecular interaction quantities under rotations is explicit.The formalism also underlies the Cormorant network design.
  • Representation theory: A k-th order Cartesian moment tensor decomposes into irreducible SO(3) representations, with multiplicities τℓ and τk = 1 while τℓ = 0 for ℓ > k.This decomposition separates rotational components into irreducible parts.
  • Spherical tensors: The spherical components Qℓ,i transform independently as Qℓ,i 7→ Dℓ(R)Qℓ,i under rotations.Although Cartesian tensors have 3^k components, the corresponding spherical moment is represented by a 2ℓ+1-dimensional vector.
  • Spherical tensors: A zeroth-order spherical tensor is a scalar, while a first-order spherical tensor can represent a spatial vector through spherical harmonics.Spherical moments are indexed by components from −ℓ through ℓ.
  • Clebsch–Gordan products: Tensor products are required when interactions couple multiple quantities, such as two dipoles and their relative position vector.Their representation transforms according to the tensor product of the corresponding irreducible representations.
  • Clebsch–Gordan products: The Clebsch–Gordan decomposition breaks tensor-product representations into irreducible components that remain spherical tensors.Combining such a component with another matching tensor can produce a scalar interaction term.
  • Interaction structure: Iterated Clebsch–Gordan products can express almost any rotation-invariant interaction potential, including multipole, magnetic, and higher-order interactions.The discussion uses electrostatics to illustrate a broader algebraic structure rather than restricting the framework to electrostatics.

4 CORMORANT: COvaRiant MOleculaR Artificial Neural neTworks

Cormorant is a molecular neural-network architecture whose atom-subset neurons produce rotation-covariant activations and scalar invariant outputs. Its covariant layers use Clebsch–Gordan products to model interactions while preserving symmetry.

  • Internal activations are SO(3)-covariant vectors composed of spherical-tensor parts, while output neurons are restricted to scalars.A type τ specifies how many components transform under each angular order.
  • Each neuron corresponds to a single atom or physically meaningful atom subset, encouraging physically interpretable interactions.
  • Global rotational invariance is guaranteed by combining covariant activations into scalar outputs, while translation invariance uses relative distances.
  • Clebsch–Gordan products provide the network nonlinearity by combining spherical-tensor parts directly in Fourier space.This avoids the costly inverse-transform, pointwise-nonlinearity, and retransform approach, which can introduce quadrature errors.
  • Cormorant uses a restricted channel-wise Clebsch–Gordan product, mixing corresponding columns to reduce quadratic channel growth.
  • The covariant architecture propagates one-body information through atom activations and two-body information through pairwise edge activations.Pairwise interactions are modulated by learnable radial and cutoff functions and spherical harmonics.
  • Compared with other covariant networks, Clebsch–Gordan nonlinearities enable complete interaction of activation degrees of freedom but make training more difficult.

5 Experiments

Cormorant is evaluated on MD-17 for molecular force fields and potential energy surfaces and on QM-9 for ground-state molecular properties. It outperforms competing methods on MD-17 and achieves leading or competitive results on half of the twelve QM-9 targets.

  • QM-9: Cormorant achieves leading or competitive results on six of twelve QM-9 regression targets: α, ∆ǫ, ǫHOMO, ǫLUMO, µ, and Cv.The remaining four targets are within 40% of the best result, except R2.
  • Evaluation: Table 1 reports mean absolute error for QM-9 prediction targets and MD-17 conformational energies, highlighting results within three Cormorant training-run standard deviations of the best.The table presents QM-9 on the left and MD-17 on the right.
  • MD-17: Cormorant outperforms all competitors on the MD-17 test-set comparisons.The comparison uses a 50k-molecule training set for neural-network baselines, with test-set mean absolute error reported in kcal/mol.

6 Conclusions

The conclusion emphasizes Cormorant’s physically motivated covariant design and its benchmark performance. The architecture outperforms competing MD-17 methods and remains competitive across QM-9 molecular-property targets.

  • Architecture: Cormorant explicitly builds rotational and translational covariance into spherical-tensor activations and combines Clebsch–Gordan products, concatenation, and learnable mixing.These operations are described as covariant operations motivated by physical interactions.
  • Results: Cormorant outperforms competing methods on MD-17, the standard benchmark for force-field learning.The authors envisage integrating learned potentials into molecular-dynamics simulation frameworks.
  • Future work: The authors identify force learning and generalization to other compact symmetry groups as natural extensions.These are presented as future work rather than demonstrated results.

7 Architecture

Cormorant maps molecular charge-position sets to invariant regression targets through input featurization, covariant Clebsch–Gordan layers, and output regression. The input and output modules vary by dataset, while the central covariant layers share their design.

  • Network definition: Cormorant takes a set of N atomic charges and positions, {Zi, ri}, and outputs a single regression target.The network is expressed as OUTPUT(CGNet(INPUT({Zi, ri}))).
  • Network units: The architecture has three units: INPUT, CGNet, and OUTPUT.INPUT produces scalar feature vectors, CGNet produces SO(3)-vectors using Clebsch–Gordan operations, and OUTPUT constructs a permutation-invariant prediction.
  • Modularity: The design separates input featurization, covariant SO(3)-vector layers, and output regression tasks.INPUT and OUTPUT differ between GDB9 and MD17, whereas CGNet has identical design and hyperparameter choice across both datasets.
  • MD-17 input: For MD-17, the input uses one-hot atomic-species features combined with normalized charge features and a learnable mixing matrix.The resulting representation uses scalar irreducible features with Nc channels.
  • MD-17 input: The MD-17 input parametrization was sufficiently expressive without a complex input-featurization network.The authors report that complex input featurization was not significantly beneficial for MD-17.
  • QM-9 input: For QM-9, input featurization uses message passing with a learned soft distance mask, neighbor aggregation, and a one-hidden-layer ReLU MLP.The MLP has 256 hidden neurons and produces a complex SO(3)-vector with Nc channels.

7.4 Covariant SO(3)-vector layers

The covariant SO(3)-vector layers update atom representations through learned pairwise edge features, spherical-harmonic lifting, Clebsch–Gordan aggregation and nonlinearities, skip connections, and channel mixing.

  • Layer configuration: The shared central network uses four layers with maximum angular weight L = 3, followed by one L = 0 SO(3)-vector layer.Input tensors and representation-channel weights use Nc = 16 channels.
  • Edge networks: Edge features combine prior-layer pair activations, dot products of atom features, and learnable radial basis functions.A learnable soft cutoff smoothly drops radial functions to zero.
  • Vertex networks: VertexNetwork updates atom activations by combining Clebsch–Gordan aggregation, a Clebsch–Gordan nonlinearity, a skip connection, and linear feature mixing.The edge representations generated by Edge2Vertex enter the aggregation step.
  • Edge networks: The edge network extends message-passing edge networks by concatenating and mixing three types of pair features.Its weights are indexed by layer and angular degree.
  • Edge-to-vertex mapping: Spherical harmonics convert scalar edge outputs into SO(3)-vector edge representations for the vertex aggregation step.The construction uses the angular direction of each atom pair.

7.5 Output featurization

Cormorant converts covariant activations into scalar invariants before producing molecular predictions. The MD-17 and QM-9 output networks aggregate these scalars differently.

  • The network extracts three scalar invariants from each activation, including its ℓ=0 component and a self-product.These invariants are constructed from the activation components and are later concatenated.
  • The resulting scalar features are concatenated and fed into the output network.
  • MD-17 output: For MD-17, scalar features are summed over atoms and passed through one linear layer to produce the output.The output is y = A (sum_i x_i) + b.
  • QM-9 output: For QM-9, one MLP processes each site’s scalar representation, the results are summed, and a second MLP predicts the regression target.Both MLPs have one hidden layer of size 256, with a 96-neuron intermediate representation.

7.6 Weight initialization

Weight initialization and gain selection are used to keep activations near order unity and preserve useful Clebsch–Gordan contributions during training.

  • The Clebsch–Gordan weights are initialized uniformly in [−1, 1] and then scaled by a gain factor.
  • A gain that is too low suppresses higher-level Clebsch–Gordan products, leaving information to flow mainly through linear one-body operations.
  • A gain that is too high makes Clebsch–Gordan nonlinearities dominate at initialization and increases the change of the instability described in the passage.
  • The gain was hand-tuned so the mean absolute Clebsch–Gordan activation at each level was approximately unity on a random mini-batch.
  • The experiments used gain g = 5.

8 Experimental details

The experiments specify datasets, optimization settings, model sizes, and training behavior for QM-9 and MD-17. Training was sensitive to initialization and optimization choices, with reliable convergence under a restricted depth.

  • Training setup: The networks used AMSGrad with learning rate 5 × 10^-4, mini-batch size 25, and 512 or 256 epochs for MD-17 and QM9 respectively.
  • Data splits: MD-17 used 50k/10k/10k training, validation, and test points per molecule, while QM-9 used 100k training molecules, 10% test molecules, and the remainder for validation.
  • Architecture: Both datasets used S = 4 CG layers, L = 3, and Nc = 16 output channels per CG layer.
  • Architecture: The resulting networks contained 299808 parameters for QM-9 and 154241 for MD-17.
  • Training instabilities: Poor initialization or optimization frequently caused unrecoverable training loss above 10^6 or inactive higher-layer Clebsch–Gordan nonlinearities with poor training error.
  • Training instabilities: Using the stated initialization scheme, the model consistently reached low training and validation error when limited to at most four CG layers.
Loading 1906.04015v3…