Molformer: Motif-based Transformer on 3D Heterogeneous Molecular Graphs

• Atom-level homogeneous graphs underuse motif information, while adding motifs as atom features blurs their semantics and motif-only inputs omit atoms and infrequent substructures.The introduction motivates representing atoms and motifs separately while retaining both.Source: Introduction · p. 1
• HMGs combine atom-level and motif-level nodes, using functional groups for small molecules and reinforcement learning to mine protein amino-acid subsequences.The paper presents this as a shared input representation for both molecule types.Source: Introduction · p. 1
• Molformer uses heterogeneous self-attention to distinguish interactions across node levels and AFPS to aggregate node features into a molecular representation.The model is presented as an equivariant geometric Transformer for HMGs.Source: Introduction · p. 1
• Molformer outperforms or achieves competitive results against state-of-the-art baselines on several small-molecule and protein benchmarks.Source: Introduction · p. 2

• A molecule is represented by its atom coordinates and invariant features, which are embedded, processed into a representation, and passed to a predictor for property estimation.Source: Preliminaries · p. 2
• Standard dot-product self-attention transforms input embeddings into queries, keys, and values, computes token-to-token attention, and produces output embeddings.The passage notes that this non-local mechanism has poor ability to capture fine-grained local patterns.Source: Preliminaries · p. 2
• Relative-distance positional encoding modifies attention scores using pairwise positions; taking the L2 norm of atom-coordinate differences further provides rotation invariance.Source: Preliminaries · p. 2

Heterogeneous Molecular Graphs

• Motifs are recurring substructure patterns that contribute to the biochemical characteristics of whole molecules.Source: Heterogeneous Molecular Graphs · p. 2

Motifs in Small Molecules

• Small-molecule motifs are extracted from SMILES using RDKit and templates for four functional-group categories: hydrocarbons, haloalkanes, oxygen-containing groups, and nitrogen-containing groups.Source: Motifs in Small Molecules · p. 2

Motifs in Proteins

• Protein motifs are mined as task-specific subsequences; the method focuses on four-amino-acid motifs, which the authors describe as the smallest polypeptide with special protein functionalities.The authors note that four amino acids yield 1.6 × 10^5 potential quaternary motifs.Source: Motifs in Proteins · p. 2
• ProtBERT representations and an MLP score candidate quaternions; the top K form a lexicon used to extract motifs, train Molformer, and update the policy from validation performance.Source: Motifs in Proteins · p. 3
• The one-action motif-selection game rewards Molformer performance and lexicon diversity, with the diversity term weighted by γ.Source: Motifs in Proteins · p. 3

Formulation of Heterogeneous Molecular Graphs

• HMG formulation separates motifs from atoms to ease learning motif semantics, while retaining both levels and avoiding errors from inappropriate semantic segmentation.Motifs are treated as higher-level semantic units and as new node types in the graph.Source: Formulation of Heterogeneous Molecular… · p. 3
• Each motif node receives a learnable category embedding, and its position is computed as a weighted sum of the coordinates of its component atoms.Source: Formulation of Heterogeneous Molecular… · p. 3

Molformer

• Molformer applies stacked feature-learning blocks to atom- and motif-level nodes, then uses attentive farthest point sampling to form a molecular representation for prediction.Each block contains heterogeneous self-attention, a feedforward network, and two-layer normalization.Source: Molformer · p. 3

Heterogeneous Self-attention

• Heterogeneous self-attention distinguishes atom–atom, atom–motif, and motif–motif interactions, with pairwise 3D distances also influencing attention scores.A learnable bias indexed by node-pair type is shared across layers.Source: Heterogeneous Self-attention · p. 3
• Distance-based masks at multiple scales extract local patterns, which are concatenated with a global feature to form a multi-scale representation.The distance thresholds accommodate nonuniform molecular spacing, including differences between small molecules and proteins.Source: Heterogeneous Self-attention · p. 4

Attentive Farthest Point Sampling

• AFPS selects points by balancing distance from already selected points with attention importance, then globally averages their features into the molecular representation.Unlike protein-oriented hierarchical sampling, AFPS adaptively samples atoms without prior assumptions about their arrangement.Source: Attentive Farthest Point Sampling · p. 5

Experiments

• Experiments cover seven datasets across small molecules and proteins; QM7, QM8, and QM9 contain about 7K, 22K, and 133K molecules, respectively.PDBbind uses protein-ligand pocket pairs, with complexes split by protein sequence identity at 30%; other datasets use scaffold splits at 8:1:1, while QM9 follows the Townshend et al. (2020) split.Source: Experiments · p. 5

Baselines

• Small-molecule baselines include fingerprint, graph-convolutional, and Transformer-based equivariant models; PDBbind comparisons include SMILES-based, geometric, and 3D methods.The reported PDBbind baselines include DeepDTA, DeepAffinity, Cormorant, HoloProt, Schnet, 3DCNN, and 3DGCN.Source: Baselines · pp. 5–6

Overall Results on Benchmark Tasks

• Molformer records the lowest MAE of 43.2 on QM7 and 0.009 on QM8, performs competitively across QM9 regression tasks, and surpasses all baselines on most classification tasks.On PDBbind, it achieves the lowest RMSD and best Pearson and Spearman correlations.Source: Overall Results on Benchmark Tasks · p. 6, Table 1 · p. 6, Table 2 · p. 7

Ablation Study and Discussion

• HSA with HMGs lowers QM7 MAE from 132.2 to 46.5 and QM8 MAE from 0.0205 to 0.0097; multi-scale reduces QM7 RMSD from 50.1 to 46.5 but helps QM8 less.The authors associate multi-scale’s stronger benefit on small datasets with alleviating inadequate training; they report less need for it as data size increases. AFPS also outperforms virtual-node representation.Source: Ablation Study and Discussion · p. 6, Table 4 · p. 7

What Is the Contribution of HMGs?

• Motif-based HMGs improve over two naive multi-level feature-fusion variants, and all four small-molecule motif categories help, with hydrocarbons most effective and haloalkanes least.Performance is best with all categories; protein motifs sharing a reported backbone occur in pockets at 1.38%, versus 0.73% elsewhere.Source: What Is the Contribution of HMGs? · p. 6, Figure 3 · p. 6

• Motifs have been used for contrastive learning, pretraining, generation, protein design, and drug–drug interaction prediction, but not to build heterogeneous graphs for molecular property prediction.Source: Related Works · p. 7
• Earlier motif-mining methods rely on exact counting or sampling and statistical estimation, while the cited work reports no task-specific motif extraction for improving model performance.Source: Related Works · p. 7
• Although 3D CNNs and GNNs model molecular geometry and Transformers capture long-term dependencies, the described approaches remain limited in local context, long-range dependencies, or heterogeneous graph support.Source: Related Works · p. 7

• Molformer builds heterogeneous molecular graphs from informative motifs extracted using functional groups for small molecules and reinforcement learning for proteins.Source: Conclusion · p. 7
• Its heterogeneous self-attention distinguishes interactions between node levels, while multi-scale spatial processing captures local features and downsampling aggregates molecular representations.Source: Conclusion · p. 7
• Across varied domains, experiments report that Molformer beats a group of baseline algorithms.Source: Conclusion · p. 7

Molformer Architecture

• The standard model uses six heterogeneous self-attention layers, three scales, eight heads, and AFPS settings K=4 and λ=0.1, with smaller configurations for BBBP and ClinTox.The standard input embedding and FFN hidden sizes are 512 and 2048; BBBP and ClinTox use two layers and four heads.Source: Molformer Architecture · p. 10

Training Details

• Models train for 300 epochs except PDBbind, which trains for 30, with dataset-specific batch sizes and learning rates and validation-based hyperparameter tuning.Training uses Adam and a ReduceLROnPlateau scheduler; regression values are normalized using training-set means and standard deviations.Source: Training Details · p. 10

Motif Extraction

• Small-molecule motifs are extracted from SMILES with RDKit, while protein mining considers 29,871 observed PDBbind quaternions and samples 1,000 candidates per iteration.The protein motif lexicon is mined with reinforcement learning; the motif proportion is the number of motifs divided by the number of amino acids.Source: Motif Extraction · p. 10

Conformation Classification

• The task tests whether models can distinguish molecular types across conformations, using 1,000 molecules with 20 conformations each and molecule-specific half-and-half train/test splits.The evaluation reports micro- and macro-average ROC-AUC and accuracy across 1,000 classes.Source: Conformation Classification · p. 10

AFPS vs. FPS.

• In the illustrated molecule, AFPS selects key atoms and the benzene ring while maintaining remote spacing, whereas FPS misses important atoms and samples trivial hydrogens.Source: AFPS vs. FPS · p. 10, Figure 6 · p. 12

Protein

• In the PDBbind example, discovered motifs occur frequently around the protein pocket and surround the ligand.Source: Protein · p. 10, Figure 7 · p. 12

Conformation Classification

• Perfect micro-average and macro-average ROC-AUC, with high accuracy, are reported for Molformer on conformation classification.Source: Conformation Classification · p. 10, Table 8 · p. 11

Protein

• The illustrated RL motif lexicon consists of motifs composed of ten quaternary amino acids.Source: Figure 5 · p. 12