PaperlayerView exact revision on arXiv

Source-linked AI summary

Higher-Order Molecular Grammars for Generative and Foundation Models in Chemistry

Yiming Huang, Yujie Zeng, Vijay Prakash Dwivedi, Simone Foti, Jianmin Wang, Jure Leskovec, Tolga Birdal

arXiv:2610.02186v1cs.LGcs.AI

TL;DR

Sequence and graph representations do not explicitly encode hierarchical higher-order molecular topology, while explicit incidence-based approaches make generation difficult to scale. HGR parses molecular complexes into production-rule sequences for sequence models, and its evaluation reports valid generation, improved distributional alignment, and transferable representations.

  • Problem

    Sequence and graph representations leave hierarchical higher-order topology implicit, while incidence-based representations make scalable molecular generation difficult.

    Source: Main · p. 1

  • Method

    HGR lifts molecular graphs to combinatorial complexes and parses them into compact production-rule sequences that can be decoded exactly and modelled by sequence architectures.

    Source: Main · p. 3

  • Results

    HGR supports chemically valid generation, improved distributional alignment on ring-diverse chemical space, and transferable molecular representations.

    Source: Discussion · p. 13

  • Takeaways & Limitations

    HGR provides a reusable structural vocabulary for making molecular higher-order topology explicit and learnable, while RingDiv evaluates models on complex, ring-rich chemistry.

    Source: Discussion · p. 13

    For very large corpora, lifting is the main upfront computational cost, though it runs once, parallelises well, and is amortised across later training and generation.

    Source: Discussion · p. 13

Abstract

from arXiv · show

Molecular learning models are strongly shaped by their underlying representations. Yet standard sequential and graph formalisms struggle to explicitly encode higher-order topology, such as ring systems and recurring motifs. Existing higher-order representations can capture these structures directly, but they are often computationally demanding and difficult to decode into valid molecules. Here, we introduce Higher-order Grammar Representation (HGR), a principled, topology-aware framework that lifts molecules to combinatorial complexes and parses each complex into a compact sequence of production rules under a context-free higher-order grammar. By serialising higher-order topology into rule sequences, HGR makes these structures directly compatible with standard sequence models, avoiding the computational overhead of explicit higher-order encodings while preserving topological expressiveness. To reduce benchmark bias towards simple ring systems, we construct RingDiv, a ring-enriched benchmark containing 1.18 million molecules, including the curated RingDiv300k subset, and introduce the ring diversity index (RDI) to quantify ring-system coverage. In molecular generation, HGR-based models uniquely combine 100% validity by construction with leading distributional alignment, ranking first in FCD on all five generation benchmarks. In representation learning, HGR-FM achieves the highest mean AUC across seven MoleculeNet benchmarks under both transfer protocols, improving on the strongest baseline by 8.3 and 3.3 AUC points under probing and full fine-tuning, respectively. Collectively, these results establish HGR as an efficient higher-order representation for molecular generation and transferable representation learning.

Main

Molecular representations often leave higher-order topology implicit, despite the importance of rings and other multi-atom structures. HGR serialises hierarchical molecular topology into production-rule sequences, alongside a ring-enriched benchmark for generation evaluation.

Main

  • Rings and functional groups are consequential higher-order units, connecting atom-level connectivity to meso-scale organization and molecule-level properties.Source: Main · pp. 1–2
  • Sequence and graph models leave higher-order topology implicit, fragment methods omit its hierarchy, and incidence-based generation is difficult to scale.Source: Main · pp. 2–3
  • HGR lifts molecules into combinatorial complexes and parses them bottom-up into compact production-rule sequences that preserve hierarchy and enable exact decoding.Source: Main · p. 3
  • RingDiv contains 1.18 million ring-enriched molecules, with RingDiv300k as a topology-balanced subset for evaluating generation in ring-diverse chemical space.Source: Main · p. 3
  • HGR generators are valid by construction and align chemical and graph-structural distributions; ablations attribute transfer gains to explicit higher-order topology.Source: Main · p. 3

Results

HGR-based generation combines full validity with strong distributional alignment, while RingDiv broadens ring-topology coverage and HGR-FM leads transfer AUC across seven benchmarks.

Higher-order grammar representation

  • HGR serializes lifted hierarchical complexes into reusable rule sequences for exact reconstruction; MIG and RSG yield more compact, balanced rule corpora than MHG.The unique-rule corpus grows sublinearly and approaches saturation as molecules are added.Source: Higher-order grammar representation · pp. 4, 6

RingDiv: a ring-diversity benchmark

  • RingDiv and RingDiv300k have higher RDI and more unique ring systems than most benchmarks, alongside high QRCI, uncommon ring types, and the two highest unique-scaffold ratios.Their QRCI distributions concentrate toward the upper end of the approved-drug range without shifting into the overcomplex regime.Source: RingDiv: a ring-diversity benchmark · p. 7

Molecular generation with HGR

  • 100% RDKit validity on all five generation benchmarks accompanies HGR-LDF’s best FCD on GuacaMol and lowest FCD and best KL score on RingDiv300k.GuacaMol FCD: 84.4 versus 78.3 for the strongest diffusion baseline; RingDiv300k FCD: 0.4652 and KL score: 99.2.Source: Molecular generation with HGR · p. 7, Table 1 · p. 9
  • HGR generation preserves the broad RingDiv300k functional-group profile, with residual motif deviations, and its chemical-space density broadly overlaps the test set unlike restricted or shifted diffusion baselines.Nitrogen-containing groups and aromatic motifs show the closest agreement; HGR-LDF reduces the ether enrichment observed with HGR-VAE.Source: Molecular generation with HGR · pp. 9, 11
  • Across sizes, HGR ring-size JS stays ≤0.01 and functional-group JS stays 0.01–0.015, while graph-diffusion baselines deviate more and become less synthetically accessible on large molecules.DiGress reaches JS ≈0.09 above 45 heavy atoms and SA ≈6; HGR-generated molecules retain low SA scores across the size range.Source: Molecular generation with HGR · pp. 9, 11

Molecular foundation model with HGR

  • Removing higher-order rule tokens raises validation loss and lowers mean AUC more than disabling the attention bias, supporting the importance of higher-order tokens.The tokens expose meso-scale and global structural cues beyond atom-level co-occurrence.Source: Molecular foundation model with HGR · p. 11
  • HGR-FM achieves the highest average AUC across seven MoleculeNet benchmarks under probing and full fine-tuning, leading all seven tasks and six, respectively.SIDER is the only full fine-tuning exception.Source: Molecular foundation model with HGR · p. 11, Figure 5 · p. 12
  • Pretraining on RingDiv combined with ZINC2m yields the best downstream performance among the compared data sources.RingDiv contributes more complex, topologically rich molecules, while ZINC2m captures simpler fundamental chemical patterns.Source: Molecular foundation model with HGR · p. 13

Discussion

The discussion presents HGR as a decodable string representation of hierarchical molecular topology and pairs it with RingDiv’s ring-rich evaluation regime. It reports support for valid generation and transferable representations while identifying practical boundaries.

Discussion

  • Ablations removing higher-order rule tokens degrade transfer, while HGR encodes hierarchical topology as a decodable string for standard sequence models.The discussion attributes transfer gains to higher-order topology rather than atom-level co-occurrence alone.Source: Discussion · p. 13
  • RingDiv enriches evaluation with 1.18 million ring-diverse molecules, addressing underrepresentation of complex ring systems in conventional benchmarks.The discussion pairs this more demanding evaluation regime with HGR’s reusable higher-order motifs for generation and transfer learning.Source: Discussion · p. 13
  • HGR’s lifting step is the main upfront cost for very large corpora, and generated molecules remain computational proposals until prospective wet-lab assays establish practical value.The lifting cost runs once, parallelises well, and is amortised across later training and generation; HGR-VAE trains and samples faster than diffusion baselines.Source: Discussion · p. 13
  • HGR offers a reusable structural vocabulary for encoding how local chemical units assemble into motifs and whole-molecule architectures, suggesting sequence-model applications to topology-centric sciences.The discussion also identifies motif-level sampling biases, including ether enrichment under direct latent sampling, which latent-space diffusion reduced.Source: Discussion · p. 13

Methods

The Methods describe how HGR lifts molecules into combinatorial complexes and serializes their topology as invertible grammar-rule sequences, alongside its lifting and decoding procedures, generative and foundation models, and benchmark and evaluation design. They also specify implementation details and reporting conventions for the experiments.

Preliminaries and problem formulation

  • HGR represents molecular topology as a compact, invertible sequence of production rules parsed from a lifted complex paired with its molecular graph.The complex captures hierarchical higher-order structure, while the graph supplies pairwise adjacency during rule construction.Source: Preliminaries and problem formulation · p. 14

Molecular lifting algorithms

  • Molecular lifting uses frequency-guided motif merging, with MIG resolving interfaces exceeding two anchors and RSG staging merges around ring systems.MIG absorbs neighboring subgraphs to simplify ambiguous interfaces; RSG prioritizes within-system merging and treats each ring system as a unit during anchor resolution.Source: Molecular lifting algorithms · pp. 14–15

Higher-order grammar induction and derivation

  • Grammar induction contracts cells in bottom-up post-order into rule tokens whose ordered anchor interfaces preserve attachment correspondence for reverse derivation.Reverse derivation uses a LIFO stack and restores the lifted complex and its molecular graph losslessly.Source: Higher-order grammar induction and… · pp. 15–16

HGR-based generative models

  • The grammar VAE encodes HGR rule sequences into latent vectors and decodes them under interface-matching constraints, while optional latent diffusion further samples the latent space.Completed derivations are grammar-valid and satisfy the local valency and bonding constraints encoded by the rules, without post-hoc filtering for these constraints.Source: HGR-based generative models · pp. 16–17

HGR-based foundation model (HGR-FM)

  • HGR-FM combines higher-order hierarchy and pairwise connectivity in attention, with four pretext tasks spanning atom, fragment, molecular, and 3D-conformation supervision.Its structural signature uses cell rank and grammar-tree distance to distinguish hierarchical relations, while pairwise biases encode molecular graph paths.Source: HGR-based foundation model (HGR-FM) · pp. 17–18

Datasets and benchmarks

  • RingDiv contains 1.18 million ring-enriched molecules, and RingDiv300k is a topology-balanced subset selected for development and ablation studies.Sampling prioritizes rare ring classes and then uses Morgan fingerprints and MaxMin diversity selection; macrocycles are excluded from the drug-like benchmark.Source: Datasets and benchmarks · p. 18
  • Generation is evaluated on five benchmarks, while foundation-model transfer uses seven scaffold-split MoleculeNet classification tasks and pretraining on ZINC2m, RingDiv, or their combination.The generation benchmarks are QM9, ZINC250k, MOSES, GuacaMol, and RingDiv300k; transfer uses 80/10/10 train, validation, and test partitions.Source: Datasets and benchmarks · p. 18

Experimental setup

  • Generation baselines cover sequence, motif, graph, and topology-aware models and are assessed using validity, distributional, chemistry, and topology metrics; transfer uses probing and full fine-tuning measured by AUC.Probing freezes the encoder and trains lightweight task heads, whereas full fine-tuning jointly optimizes the encoder and head.Source: Experimental setup · pp. 18–19
  • HGR-VAE, HGR-LDF, and HGR-FM were implemented in PyTorch; unless otherwise stated, results use three seeds, with Fig. 4 showing standard-deviation error bars and Fig. 5 reporting seed means.Model configurations, optimisers, and training hyperparameters are listed in Supplementary Sec. 3.4; per-task standard deviations for Fig. 5 are in Supplementary Table 12.Source: Experimental setup · p. 19

Author contributions statement

Y.H. conceived the study; Y.H., Y.Z., V.P.D. and T.B. designed it. The authors report distinct research, analysis, visualization, and writing contributions.

Author contributions statement

Competing interests

The authors declare no competing interests.

Competing interests

Supplementary Contents

The supplementary contents are organized around model and algorithm details, RingDiv dataset construction, and experimental and supplementary analyses.

Model and algorithm details

  • Model and algorithm details cover notation, lifting, grammar induction, rule initialization, generation, and HGR-FM.These topics are listed under Section 1.Source: Supplementary Contents · p. 20

RingDiv dataset construction

  • RingDiv materials include dataset construction, source curation, quota-balanced diversity sampling, and dataset variants.These topics are listed under Section 2.Source: Supplementary Contents · p. 20

Experimental setup and supplementary results

  • Experimental setup and supplementary results cover generation metrics, functional-group profiling, transfer benchmarks, and computational efficiency.These topics appear in Sections 3 and 4.Source: Supplementary Contents · p. 20

1 Model and algorithm details

The supplement defines grammar-operation terminology, describes data-driven lifting, and details how lifted complexes are parsed into a grammar that supports exact reconstruction.

1.1 Notation and terminology

  • Parsing produces an HGR string from a lifted complex, while derivation applies its rules in the generative direction.The HGR string is stored in post-order, so reconstruction consumes its reverse.Source: §1.1 · p. 21

1.2 Lifting algorithms

  • MIG merges adjacent partition blocks by frequency, with anchor resolution expanding candidates whose interfaces exceed two anchors; RSG stages merges around ring systems.The two lifting variants share frequency-guided merging primitives.Source: §1.2 · p. 21

1.3 Grammar induction

  • Admissible post-order serializations of the contraction tree yield HGR strings whose ordered rules and anchor interfaces support exact reconstruction of the lifted complex and molecular graph.Sibling-subtree order may vary, producing alternative lossless strings; reverse derivation starts at S and applies the rules in reverse sequence.Source: §1.3 · pp. 22–24

1.4 Structure-aware rule initialisation

Rule tokens receive structure-aware features, combining Weisfeiler–Lehman summaries with random-walk topology features in the foundation model.

Model-specific initialization

  • Generation-model rule tokens combine WL features with learned per-rule embeddings, whereas HGR-FM uses WL features with RWPE for fully structure-derived initialization.The final rule embedding concatenates the WL and RWPE components.Source: §1.4 · pp. 24–25

Structure-aware rule features

  • WL features aggregate right-hand-side graph labels across refinement rounds, while RWPE encodes node return probabilities over multiple walk steps.RWPE is made permutation-invariant by sorting node-feature rows, then truncating or padding and flattening the matrix.Source: §1.4 · p. 25

Weisfeiler–Lehman histogram embedding

  • More than 98% of production rules receive distinct initial features from WL embeddings in the reported grammars.The WL embedding is described as invariant to node and neighbourhood ordering and extendable to unseen rules.Source: §1.4 · p. 25

1.5 HGR-based de novo generation

The HGR generative model encodes HGR strings into a latent representation and decodes production rules under grammar constraints; an optional diffusion model samples in that latent space.

1.5.1 Grammar variational autoencoder

  • The GVAE encodes WL-structured HGR rule tokens into a latent code and autoregressively decodes rules subject to stack and anchor-interface constraints.Sequences unfinished at the dataset-specific length limit are discarded; completed strings decode deterministically to molecular complexes.Source: §1.5.1 · p. 26

1.5.2 Latent diffusion sampling

  • HGR-LDF adds score-based diffusion over standardised GVAE posterior means, sampling from a tractable prior and reversing the noise process to recover latent codes.The grammar decoder remains unchanged.Source: §1.5.2 · pp. 26–27

1.6 HGR-FM architecture and pretraining

HGR-FM combines a Grammar Transformer over HGR strings with structure-aware rule embeddings and higher-order positional biases. Its multi-level pretraining objectives supervise atoms, fragments, molecular properties and 3D geometry.

1.6 HGR-FM architecture and pretraining

  • HGR-FM uses a Grammar Transformer to encode structure-aware rule-token embeddings into contextualized molecular representations.The molecule-level token supplies a fixed-length fingerprint for fragment-, molecule- and 3D-level tasks and downstream transfer.Source: §1.6 · p. 27
  • Higher-order attention bias combines rank-aware rotary encoding with grammar-tree, shortest-path and edge-path structural biases.Attention heads are divided into four equal groups; learnable per-layer scales weight the three additive biases.Source: §1.6 · p. 27
  • Multi-level pretraining covers atom masking, substructure-fingerprint prediction, descriptor regression and conformation-distance prediction.These objectives provide atom-, fragment-, molecule- and 3D-conformation-level supervision, adapted to HGR rule-token inputs where applicable.Source: §1.6 · p. 27
  • The atom-level task masks 25% of atoms, with at least one per molecule, and reconstructs atom types from contextualized atom-token embeddings.A one-layer GIN decoder uses scaled cosine-error loss; this task supervises local atomic environments.Source: §1.6 · p. 27
  • Fragment supervision predicts 1,024-bit Morgan and 166-bit MACCS fingerprints, while molecular supervision regresses standardized RDKit descriptors.The descriptors include molecular weight and logP.Source: §1.6 · pp. 27–28
  • Conformation supervision uses pairwise interatomic distances from optimized conformers, and weighted task losses combine the complementary scales.Per-task coefficients λ are listed in Supplementary Table 6.Source: §1.6 · p. 28

2 RingDiv dataset construction

RingDiv was assembled from a large compound pool, filtered and annotated for drug-like ring-containing molecules, then sampled to balance ring-topology coverage and structural diversity. RingDiv300k is a smaller topology-balanced development subset of the full benchmark.

2.1 Source databases and curation

  • The source pool combined approximately 143 million compounds from PubChem, SureChEMBL and five commercial libraries, with parsing failures and InChI duplicates removed.Structures were standardized and converted to RDKit molecular objects before deduplication.Source: §2.1 · p. 28
  • RingDiv curation retained drug-like compounds with 1–10 rings and molecular weights of 100–900 Da, while excluding macrocycles.Ring size and fusion topology were annotated; macrocycles were excluded because they were sparse in the source pool and differ in flexibility and synthetic accessibility.Source: §2.1 · p. 28

2.2 Quota-balanced diversity sampling

  • Sampling first met quotas for rare, informative ring classes, including medium, spirocyclic and bridged systems, before filling the candidate set by random expansion.This preserved intended marginal balance across topology classes before diversity selection.Source: §2.2 · p. 28
  • A MaxMin criterion on 2,048-bit Morgan fingerprints selected a globally dispersed subset using an approximate Tanimoto threshold of 0.65.The threshold reduced redundancy without eroding rare-topology coverage.Source: §2.2 · p. 28

2.3 RingDiv variants

  • RingDiv is the full benchmark, while RingDiv300k is a smaller topology-balanced subset selected with tighter quotas for rare ring categories.Both share provenance, preprocessing and filtering criteria but differ in scale and rare-ring balance.Source: §2.3 · p. 28

3 Experimental setup

The evaluation combines molecular-generation metrics, functional-group profiling and scaffold-split transfer tests for pretrained encoders. The setup also specifies benchmark-specific sample sizes, preprocessing and shared model-training protocols.

3.1 Molecular generation metrics

  • Generation evaluation groups metrics into basic, chemistry-based and topology-based measures, with topology metrics covering graph similarity, nearest neighbours, scaffold distributions and curvature-filtration descriptors.QM9, ZINC250k, RingDiv300k and GuacaMol use 10,000 generated molecules; MOSES uses 25,000, and non-validity scores use the valid generated subset.Source: §3.1 · pp. 28–29
  • FCD measures distance between ChemNet feature distributions, while NSPDK assesses graph-topological agreement and curvature descriptors provide auxiliary structural checks.FCD is interpreted with validity, novelty and structure-level metrics because it is a Gaussian summary of learned features; curvature descriptors show high sampling variance at evaluated sample sizes.Source: §3.1 · pp. 29–30

3.2 Functional-group profiling

  • Functional-group profiling compares SMARTS-assigned motif occurrence in generated molecules with the RingDiv300k real test set.A group counts as present once per molecule regardless of match count; categories are non-exclusive, so one molecule can contribute to several groups.Source: §3.2 · p. 30

3.3 HGR-FM training and evaluation

  • HGR-FM transfer is evaluated on seven MoleculeNet benchmarks using scaffold splits and both frozen-encoder probing and full fine-tuning.The baseline-aligned comparison uses ZINC2m pretraining; experiments use an 80/10/10 split, a shared three-layer MLP head, three random seeds and no task-specific hyperparameter tuning.Source: §3.3 · pp. 30, 32

3.4 Model configurations and training hyperparameters

  • The supplementary tables document model configurations, including the shared HGR-VAE architecture and optimization recipe and the latent diffusion sampling setup.The MIG-grammar ablation differs from the HGR-VAE recipe in independently searched architecture and optimization values; diffusion uses a variance-preserving SDE and Euler–Maruyama sampling without a corrector.Source: §3.4 · p. 32

4 Supplementary results and analyses

The supplementary analyses extend grammar and generation evaluations, report task-level transfer results, and assess evaluation and model runtimes. They include qualitative grammar trends, benchmark coverage, and targeted evaluation-metric optimizations.

4.1 Extended grammar analysis

  • QM9 grammar statistics show the same qualitative trends as the primary ZINC250k analysis, but the cross-method comparison is restricted to smaller datasets because MIG and MHG induction are substantially slower.MHG also yields a rapidly inflating rule corpus; HGR-VAE and HGR-LDF use the RSG-induced grammar unless otherwise stated.Source: §4.1 · p. 32, Figure 4 · p. 34

4.2 Extended generation benchmarks

  • Extended generation tables cover basic, chemistry- and topology-based metrics, with curvature-filtration descriptors treated as auxiliary rather than rank-highlighted measures.MHG-VAE is included only for QM9 and ZINC250k because induction is slower and its rule corpus rapidly expands; configuration tables cover HGR-VAE and HGR-LDF.Source: §4.2 · p. 32, Table 4 · p. 33, Table 5 · p. 33

4.3 HGR-FM transfer benchmarks

  • Supplementary Table 12 reports original per-task AUC values and across-run standard deviations for pretrained encoders under probing and full fine-tuning.It complements the normalized summary in Fig. 5b.Source: §4.3 · p. 34, Table 12 · p. 37

4.4 Computational efficiency analysis

  • The computational analysis considers both evaluation overhead and model-side training and inference runtime.All runtime experiments used PyTorch on a single NVIDIA L40S GPU and an AMD EPYC 9374F 32-core processor.Source: §4.4 · pp. 34–35

4.4.1 Evaluation optimisation

  • NSPDK computation is accelerated by approximately 5-fold, while curvature-filtration metrics κOR and κBF receive nearly 2-fold speedups.The optimized implementations also show consistent gains across QM9, ZINC250k, MOSES and GuacaMol; Supplementary Fig. 5 reports mean runtimes over ten replicates.Source: §4.4.1 · pp. 35–36, Figure 5 · p. 38

4.4.2 Training and inference time

  • Across the reported settings, training and generation runtimes compare favourably with the baselines.Supplementary Table 13 summarizes end-to-end training time and inference time across molecular datasets; training is reported in hours and inference in minutes.Source: §4.4.2 · pp. 36, 38, Table 13 · p. 38

Related papers

Loading 2610.02186v1…