PaperlayerView exact revision on arXiv

Source-linked AI summary

Procedural Pretraining for Molecular Property Prediction

Moritz Friedemann, Zachary Shinnick, Philip Torr, Bruno Andreis

arXiv:2609.17831v1cs.LGcs.AI

TL;DR

Molecular property prediction often has little labeled downstream data, and it remains unclear whether useful inductive biases must be learned from molecules. This paper tests procedural pretraining before molecular pretraining and fine-tuning, finding that it can improve prediction and that transfer depends on task structure and training budget.

  • Problem

    Labeled molecular property datasets often contain only hundreds or thousands of examples, and it remains unclear whether useful inductive biases must be learned from molecular data.

    Source: §1 · pp. 1–2

  • Method

    The authors pretrain a transformer on procedural masked-token tasks, then molecular SMILES, and finally fine-tune it for molecular property prediction.

    Source: §1 · p. 2, §3 · p. 3

  • Results

    Procedural pretraining improves molecular prediction after molecular pretraining; on Lipophilicity, REVERSE reduces test error by 4.8% relative to the molecular-pretraining-only baseline.

    Source: §1 · p. 2, §4.1 · p. 5, §5 · p. 9

  • Takeaways & Limitations

    Transfer is strongest under downstream data scarcity, depends on procedural structure rather than surface statistics, and typically peaks at an intermediate procedural-training budget.

    Source: §1 · p. 2, §5 · p. 9

    The study finds no clear procedural-pretraining benefit on FreeSolv, whose 642 molecules make differences of this magnitude difficult to resolve.

    Source: §4.1 · p. 5

Abstract

from arXiv · show

Molecular property prediction is often limited by the small size of labeled downstream datasets, motivating pretraining on large corpora of unlabeled molecules. In this work, we ask whether useful inductive biases can instead be learned from abstract, procedurally generated data before a model sees any molecular data. We introduce a three-stage training pipeline consisting of procedural pretraining, molecular pretraining on SMILES, and downstream fine-tuning, and evaluate several procedural tasks spanning sequence structure, cellular automata, and graph reasoning. We find that procedural pretraining can improve molecular property prediction even after subsequent molecular pretraining: on Lipophilicity, \textsc{Reverse} reduces test error by 4.8\%. For context, the magnitude of this improvement is roughly 90\% of the performance difference between our 250K-molecule baseline and the publicly released MoLFormer checkpoint pretrained on approximately 100M molecules. Our analysis shows that the benefit is strongest under downstream data scarcity, depends on the structure of the procedural data rather than only surface-level statistics, and does not increase monotonically with additional procedural training. Instead, transfer typically peaks at an intermediate procedural budget and deteriorates as the model approaches convergence on the procedural task. We further find that, for several tasks, much of the transferable information is localized in the attention layers, while feed-forward layers can contribute to over-specialization. These results show that procedural data can provide transferable structure for molecular learning and offer a complementary route to improving performance when labeled molecular data are limited.

1 Introduction

Labeled molecular datasets are often small, and it remains unclear whether useful inductive biases must be learned from molecular data. The paper inserts procedural pretraining before molecular pretraining and fine-tuning, finding that it can improve downstream prediction.

1 Introduction

  • The three-stage pipeline applies procedural pretraining, molecular pretraining on SMILES, and downstream fine-tuning.The experiments also use a simplified setting without molecular pretraining to isolate transfer mechanisms.Source: Figure 1 · p. 1, §1 · p. 2
  • On Lipophilicity, REVERSE reduces test error by 4.8% relative to a molecular-pretraining-only baseline.This recovers roughly 90% of the performance difference between the 250K-molecule baseline and the publicly released MoLFormer checkpoint pretrained on approximately 100M molecules.Source: §1 · p. 2
  • Procedural pretraining can improve molecular prediction after subsequent molecular pretraining, with the largest benefit under downstream data scarcity.The paper frames procedural pretraining as an inductive bias most valuable when labeled molecular data are limited.Source: §1 · p. 2
  • Across four of five tasks, downstream performance peaks at an intermediate procedural budget and deteriorates as procedural-task performance approaches its optimum.More procedural training does not necessarily produce more downstream benefit.Source: §1 · p. 2
  • Transfer depends on procedural-data structure rather than surface statistics: shuffling token order while preserving vocabulary, length, and frequencies removes the benefit.For several tasks, transferable information is concentrated in attention layers, while feed-forward layers can contribute to over-specialization.Source: §1 · p. 2

2 Related Work

Prior work finds that procedurally generated data can yield transferable representations across language and vision, including across domains. This paper extends the question to molecular modeling, where labeled data are comparatively scarce and procedural pretraining has received little attention.

2 Related Work

  • Procedurally generated languages, algorithmic tasks, and cellular automata have improved downstream natural-language performance, while synthetic images have supported transfer in vision [Papadimitriou and Jurafsky, 2020, Hu et al., 2025, Lee et al., 2026, Jiang et al., 2026] [Kataoka et al., 2021, Baradad Jurjo et al., 2021].Prior examples also include sequence-based formal languages benefiting vision transformers and music pretraining benefiting language models [Nomura, 2026].Source: §2 · pp. 2–3
  • Structural diversity, long-range dependencies, and hierarchy can support transfer, but strong domain alignment is not always necessary [Baradad Jurjo et al., 2021, Chiang and Lee, 2021, Hu et al., 2025].Transferable information may reside in different transformer components depending on the downstream task.Source: §2 · p. 3
  • This work studies whether abstract procedural tasks can complement molecular pretraining, how alignment with SMILES affects transfer, and which task and model properties account for downstream gains.The setting combines comparatively scarce labeled molecular data with little prior attention to procedural pretraining in molecular modeling.Source: §2 · p. 3

3 Methodology

The method inserts masked-token procedural pretraining before optional molecular pretraining on SMILES and downstream fine-tuning. Five procedural tasks cover sequence, hierarchical, dynamical, and graph structure, with transfer experiments designed to isolate task and model-component effects.

3 Methodology

  • The pipeline first pretrains on algorithmically generated data, optionally continues on unlabeled SMILES, and then fine-tunes on a molecular property task.Some experiments omit molecular pretraining to isolate the effect of procedural data.Source: §3 · p. 3
  • Five masked-token tasks cover reversal, nested or crossing bracket sequences, neural cellular-automaton trajectories, and shortest-path prediction on random graphs.DYCK mirrors branching parentheses in SMILES, while DYCK-SHUFFLE mirrors ring-closure digits; Figure 2 illustrates the tasks and SSP input construction.Source: §3 · p. 3, Figure 2 · p. 4
  • The experiments use MoLFormer, an encoder-only transformer with approximately 44M parameters, and keep the architecture fixed across training stages.Its SMILES tokenizer has a vocabulary of 2,362 tokens.Source: §3 · p. 3
  • To approximate molecular pretraining within the available compute budget, the authors train for 12 epochs on a 250K-molecule PubChem subset using masked-token modeling.Procedural and non-procedural initializations receive identical molecular-pretraining data and compute budgets.Source: §3 · p. 3
  • Evaluation covers Lipophilicity, FreeSolv, and QM9 regression, reporting test MAE on standardized targets from the checkpoint with the best validation performance.QM9 includes both the full dataset and controlled training-set subsets; comparisons use identical splits and optimization settings unless stated otherwise.Source: §3 · p. 4
  • Transfer retains the learned transformer backbone but reinitializes token embeddings and the output head; selective experiments transfer attention or feed-forward weights alone.The procedural embeddings remain fixed at random initialization because token semantics differ across domains.Source: §3 · p. 4

4 Experiments

The experiments establish that procedural pretraining can improve molecular prediction after molecular pretraining, then examine when the benefit transfers and where it resides. The analyses find that transfer depends on procedural structure, is strongest with scarce downstream data, and can weaken with excessive training.

4.1 Procedural Pretraining Improves Molecular Property Prediction

  • REVERSE lowers Lipophilicity MAE by 4.80% and improves 1k-molecule QM9 performance by 1.91%; its Lipophilicity gain approaches the 5.35% improvement from the released approximately 100M-molecule MoLFormer checkpoint.The comparison uses the 250k-molecule molecular-pretraining baseline; Table 1 reports standardized test MAE and relative changes.Source: Table 1 · p. 5, §4.1 · p. 5
  • FreeSolv shows no clear benefit: four procedural tasks perform worse than the molecular-pretraining-only baseline, and its 642 molecules may be too few to resolve differences of this magnitude.The released MoLFormer checkpoint, trained on approximately 400× more molecular data, is also worse on FreeSolv by 12.01%.Source: §4.1 · p. 5

4.2.1 Does Structural Alignment With Molecules Matter?

  • SMILES-aligned DYCK, DYCK-SHUFFLE, and SSP gain an early molecular-pretraining advantage that fades, while REVERSE provides the strongest downstream improvement despite little obvious SMILES alignment.SSP eventually becomes detrimental during molecular pretraining despite operating on real SMILES strings.Source: §4.2.1 · pp. 5–6

4.2.2 The Benefit Is Largest Under Data Scarcity

  • DYCK’s transfer advantage grows monotonically as the QM9 training set shrinks, making its inductive bias most valuable in low-data regimes.The comparison spans 1k, 16k, 64k, and the full 108,446-molecule training set.Source: §4.2.2 · p. 6, Figure 3 · p. 6

4.2.3 More Procedural Pretraining Is Not Always Better

  • For four of five tasks, Lipophilicity benefit rises, peaks at an intermediate procedural budget, then deteriorates and often falls below random initialization; NCA is the exception.NCA is also the only task the model does not learn to solve during procedural pretraining.Source: §4.2.3 · p. 6, Figure 4 · p. 7

4.2.4 Transfer Requires Procedural Structure

  • Shuffled-token controls lose the benefit of their structured counterparts, showing that vocabulary, sequence length, and token frequencies alone are insufficient for transfer.In the full pipeline, DYCK-SHUFFLE control reaches Lipophilicity MAE 0.4181 ± 0.0062, a roughly 2.46% degradation, while structured DYCK-SHUFFLE improves performance by 3.53%.Source: §4.2.4 · p. 7

4.3 Where Does the Transfer Reside?

  • Attention-only transfer matches or exceeds full-backbone transfer for DYCK, DYCK-SHUFFLE, and REVERSE, while feed-forward-only transfer removes most benefit and can hurt performance.SSP differs: neither component alone recovers the full-transfer benefit, and NCA shows no significant benefit under any configuration.Source: §4.3 · p. 8, Figure 5 · p. 8
  • Attention-only transfer is more robust to increasing procedural budgets: DYCK and DYCK-SHUFFLE retain benefit at the largest tested budgets, but REVERSE and SSP still converge toward baseline.The results suggest that some degradation from excessive procedural pretraining is associated with feed-forward layers, though attention-only transfer does not eliminate it for every task.Source: §4.3 · p. 8, Figure 6 · p. 8

5 Conclusion and Open Questions

Procedural pretraining on abstract, non-molecular data can improve molecular property prediction after molecular pretraining, with gains strongest in low-data regimes. Transfer depends on procedural structure and training budget, while several task and scope questions remain open.

Conclusion

  • A simple sequence-reversal task recovers roughly 90% of the performance difference between the 250K-molecule baseline and the approximately 100M-molecule MoLFormer checkpoint.The benefit depends on procedural structure rather than token-level statistics alone: shuffling token order removes the transfer advantage.Source: §5 · p. 9
  • Across most tasks, downstream performance peaks at an intermediate procedural budget and deteriorates as the model approaches procedural convergence.For several tasks, transferable information is retained in attention layers, while feed-forward layers appear to contribute more strongly to over-specialization.Source: §5 · p. 9

Open Questions

  • Transferability, stopping-point prediction, FreeSolv and NCA outcomes, and generalization to larger models, other representations, or scientific domains remain open questions.FreeSolv and NCA showed no clear downstream benefit at any procedural budget.Source: §5 · p. 9

A.1 Training details

All three training stages use the same MoLFormer encoder, with shared optimizer settings and stage-specific configuration choices. Procedural and molecular pretraining each use one run, while fine-tuning uses multiple seeds.

A.1 Training details

  • All three stages use the same MoLFormer encoder, with 12 layers, hidden size 768, 12 heads, and a maximum sequence length of 202 tokens.The encoder has 46.8M parameters with the MTM head and 48.0M with the regression head.Source: §A.1 · p. 11
  • All stages use FusedLAMB with β1 = 0.9, β2 = 0.99, ϵ = 10−6, no weight decay, and global gradient-norm clipping at 1.0.Molecular pretraining reduces batch size from 1,200 to 256 and adds a 500-step warm-up; QM9 fine-tuning runs for 400 rather than 500 epochs.Source: §A.1 · p. 11
  • Procedural and molecular pretraining each use a single run with seed 12345; fine-tuning uses seeds 12345, 42, and 43, with three additional seeds for FreeSolv.Token embeddings are fixed at random initialization during procedural pretraining and trainable in the later stages.Source: §A.1 · p. 11

A.2 Procedural-task generation

The procedural tasks use task-specific sequence representations and masking rules. Each task configuration and training budget determines a generated dataset that is traversed once before transferring the final checkpoint.

A.2 Procedural-task generation

  • REVERSE and SSP use the MoLFormer tokenizer and molecular-pretraining masking, while DYCK, DYCK-SHUFFLE, and NCA use integer token IDs with every selected position masked.For REVERSE and SSP, selected positions are replaced by [MASK] 80% of the time, a random token 10% of the time, and left unchanged 10% of the time.Source: §A.2 · p. 11
  • For each task configuration and budget of B steps, the authors generate B × 256 sequences once and train for a single pass.The transferred checkpoint is taken at step B, the end of the cosine schedule, and no validation set is held out.Source: §A.2 · p. 11
  • The task-generation table reports each task's generated sequence, varied complexity parameter, and tokens eligible for masking.Each eligible token is independently selected with probability pmask.Source: Table 3 · p. 12

B Choosing Task Complexity and Procedural Budget

A validation grid varies procedural-task complexity and training budget to guide configuration selection. Its cells report Lipophilicity benefit relative to a randomly initialized baseline.

B Choosing Task Complexity and Procedural Budget

C Procedural Pretraining Losses and Bayes-Floor Estimates

Table 4 reports procedural losses and downstream benefits across task, complexity, and duration settings, alongside Bayes-optimal loss estimates where available.

D Shuffling Token Order Removes Downstream Benefit Across All Procedural Tasks

Shuffling token order removes the downstream benefit of procedural pretraining across all five tasks in the two-stage diagnostic setting.

  • Every procedural task loses its downstream benefit when token order is shuffled before Lipophilicity fine-tuning.The diagnostic setting directly follows procedural pretraining with Lipophilicity fine-tuning, without molecular pretraining.Source: D Shuffling Token Order Removes… · p. 14, Figure 8 · p. 14
  • Shuffling preserves sequence length and token counts while disrupting order-dependent structure.Source: Figure 8 · p. 14
  • Marginal token statistics alone are insufficient to explain the observed downstream benefit.Source: Figure 8 · p. 14
Loading 2609.17831v1…