MolScribe: Robust Molecular Structure Recognition with Image-To-Graph Generation

Introduction

• MolScribe generates a molecular graph from an image by predicting atoms and bonds together with their geometric layouts.Its output can be aligned with the input image at the atom level, helping people interpret predictions.Source: Introduction · pp. 2–3
• Symbolic chemistry constraints support recognition of chirality and parsing of abbreviated functional groups during graph construction.The model determines asymmetric-atom chirality from the predicted graph and layout.Source: Introduction · pp. 2–3
• Data augmentation synthetically generates diverse drawing styles to improve robustness to variation in molecular images.MolScribe combines neural recognition of atoms and bonds with flexible integration of chemistry rules.Source: Introduction · p. 3
• 76–93% recognition accuracy on five public benchmarks, with MolScribe outperforming an image-to-SMILES baseline on both in-domain and out-of-domain images.The paper also reports a new journal-publication benchmark where MolScribe significantly outperforms the baseline and existing systems.Source: Introduction · p. 3

Background

• Prior rule-based systems perform decently on patent images but have room to improve on more diverse journal images; neural image-to-SMILES models handle style variation but make chemistry constraints difficult to incorporate.Small image perturbations can also cause significant performance degradation.Source: Background · p. 4

Task Formulation

• The task maps a single-molecule image to a 2D graph of atoms and typed bonds, with atom labels and image coordinates; output graphs can be stored as SMILES or MOLfiles.Bond types include single, double, triple, and wedge bonds.Source: Task Formulation · pp. 5–6

Model

• MolScribe encodes an image and decodes atoms, bonds, and their geometric layouts jointly to reconstruct a molecular graph.The graph can be saved as a MOLfile with coordinates or as a SMILES string without them.Source: Model · pp. 6, 9
• Chemistry rules infer chirality and cis/trans isomerism from predicted graph geometry, using atom coordinates to determine the relative order of bonds at chiral centers.This addresses the difficulty of deriving SMILES chirality symbols from local image patterns alone.Source: Stereochemistry · p. 9
• Abbreviated labels are split into characters for prediction, then expanded with a greedy algorithm that connects atoms until their valences are full.Because abbreviations lack a rigorous grammar, the algorithm relies on assumptions about how they are typically written.Source: Abbreviated Structures · pp. 10–11

Data

• Training combines one million randomly sampled PubChem molecules rendered synthetically with 680K patent examples, then augments structures and images for broader chemical patterns and drawing styles.Patent data are noisy and lack exact image atom coordinates; the available MOLfile coordinates are normalized to image size.Source: Data · p. 11, Data Augmentation · pp. 11, 13
• Augmentation handles basic Markush structures, but not positional or frequency variation; the authors leave these cases for future work because they cannot be represented in SMILES or MOLfiles.Basic cases include an R-group attached to a particular atom or within a ring.Source: Data Augmentation · p. 12

Experimental Setup

• Evaluation covers synthetic in- and out-of-domain sets, five public realistic benchmarks, and 331 manually annotated ACS images, using whole-graph exact-match accuracy.The synthetic sets share 5,719 molecules; chirality must match, while cis–trans stereoisomerism is ignored.Source: Experimental Setup · pp. 14–15

Results

• MolScribe outperforms the image-to-SMILES baseline and existing systems on most benchmarks, while DECIMER performs slightly better on UOB and MolScribe degrades on blurry, low-resolution Staker images.The table reports exact-match accuracy in percent; the supplied passages do not provide its cell values.Source: Results · pp. 15–16, Table 2 · p. 16
• Under slight rotation and shearing, MolScribe outperforms other methods by large margins, while rule-based systems lose performance significantly.The baseline performs slightly better on UOBp and Stakerp, which contain no chiral molecules.Source: Results · p. 16, Table 2 · p. 16
• MolScribe predicts chirality more accurately than the image-to-SMILES baseline by predicting atom coordinates and bond types, then explicitly determining chiral types.Figure 5 attributes improved recognition on chiral molecules to this design.Source: Figure 5 · p. 17, Results · p. 17
• Ablations show that augmentation improves out-of-domain generalization, discrete sequential coordinate tokens outperform coordinate regression, and removing non-atom tokens slightly lowers performance.The variants use 200K Indigo-generated training images and are evaluated in- and out-of-domain.Source: Table 3 · p. 17, Results · p. 18
• Training with synthetic and patent data performs better than training with either alone; patent-only training works well on patent-image benchmarks but less well in other domains.The synthetic data and augmentation strategies contribute to performance across domains.Source: Figure 6 · p. 18, Results · p. 18
• Lower atom and bond probabilities are usually assigned to incorrect predictions, while molecule-level confidence rises with accuracy and can threshold predictions for greater confidence.Molecule confidence aggregates atom and bond log probabilities; Figure 8 presents confidence-accuracy and precision-recall curves.Source: Results · p. 19, Figure 8 · p. 20
• Students took 137 seconds on average to reconstruct molecules from images alone, 39 seconds with predicted SMILES, and 20 seconds with predicted graphs.The comparison used 20 similarly sized images per setup; the graph preserves layout for visual inspection.Source: Results · pp. 20–21, Figure 9 · p. 21

• MolScribe uses an encoder-decoder architecture to jointly predict atoms, bonds, and geometric layouts for molecular structure recognition.Source: Conclusion · p. 21
• Evaluations on synthetic and realistic benchmarks show strong performance, surpassing existing rule-based and learned systems.Its outputs also let chemists examine and correct predictions, and the authors release code, data, and model with interfaces for use.Source: Conclusion · p. 22
• The work covers single-molecule images; hand-drawn molecules and more complicated Markush cases remain future directions or uncovered cases.The paper notes that some complex Markush structures cannot be represented as SMILES strings or MOLfiles.Source: Conclusion · p. 22