R-GroundBench: A Diagnostic Benchmark for R-Group Groundingin Markush Molecular Editing

• R-group grounding requires locating variable positions, resolving intended substitutions from instructions, and instantiating corresponding molecular structures.Markush structures describe families of related molecules using placeholders such as R1, R2, and X.Source: §1 · p. 1
• The complementary VQA and Generation tracks test candidate selection versus direct molecular instantiation across four image/SMILES input-output modalities.VQA uses controlled difficulty levels, while Generation removes candidate options.Source: §1 · p. 2, Figure 1 · p. 2
• Models perform strongly on simple VQA but degrade substantially on hard grounding and generation, exposing a gap between candidate recognition and autonomous molecular instantiation.The overview also identifies shortcut reliance in current models.Source: §1 · p. 2, Figure 1 · p. 2
• R-GroundBench contains 56,500 instruction records across 12,962 unique Markush scaffolds, filtered from 46,727 patent records into 14,394 clean single-scaffold samples.The benchmark uses the MolParser-7M sft_real subset (Fang et al. 2025).Source: §1 · pp. 1–2

Molecular Structure and Patent Analysis

• MolParser’s E-SMILES preserves Markush placeholders; PatentAgent and MarkushGrapher address patent analysis, while R-GroundBench tests placeholder grounding and instruction-guided editing.MolParser introduces E-SMILES to represent Markush structures alongside molecular scaffolds; the other approaches focus on extraction or patent-level analysis.Source: §2 · p. 2

Molecule-Language Benchmarks

• MolLangBench (Cai et al. 2026), Mol-Instructions (Fang et al. 2023), MolEditRL (Zhuang, Shen, and Sun 2025), and ChemCoTBench address molecular tasks, but R-GroundBench targets variable Markush scaffolds.Grounding placeholder symbols before instantiation mirrors real patent workflows.Source: §2 · p. 3

Shortcut Reasoning in Multimodal Evaluation

• Easy/Medium/Hard splits progressively remove language and visual shortcuts through scaffold-level and instruction-level constraints.Markush editing requires placeholder localization and instruction-based substituent instantiation; localization errors directly lead to incorrect edits, unlike complete-molecule editing benchmarks (Zhuang, Shen, and Sun 2025).Source: §2 · p. 3

3.1 Data Construction

• A three-stage pipeline filters real Markush structures, creates grounded editing instances, and adds molecular annotations, yielding 55,982 operations across 12,962 scaffolds and 56,500 instruction records.Filtering reduced 91,166 source image–E-SMILES pairs to 14,394 clean single-scaffold structures; annotations include six physicochemical descriptors and ChEMBL bioactivity, indication, and mechanism records.Source: §3.1 · p. 3

3.2 Task Formulation

• E-SMILES represents a Markush structure as a backbone SMILES with * attachment points, followed by tokens mapping each attachment point to its R-group label.Source: Definition and Two Tracks · p. 4

Definition and Two Tracks

• R-GroundBench uses complementary VQA and Generation tracks to distinguish candidate-level grounding from direct construction of an edited molecule as a SMILES string.VQA presents four candidate molecules; Generation removes candidate options and requires end-to-end molecular instantiation from the Markush representation and instruction.Source: Definition and Two Tracks · p. 4

Multiple-Choice (VQA) Track

• The VQA track tests Basic substituent edits and Advanced property-guided selection across three difficulty levels, four modality settings, and selected NOTA questions.Easy uses cross-scaffold distractors and direct substituent names, Medium uses same-scaffold distractors with direct names, and Hard uses same-scaffold distractors with chemically descriptive instructions. The four settings are i2i, i2s, s2i, and s2s; NOTA appears in 20% of Basic Easy/Medium questions and not in Hard.Source: Multiple-Choice (VQA) Track · p. 4, Table 2 · p. 4
• Generation evaluates Basic tasks by requiring direct edited-SMILES output, with Easy name-based and Hard chemically descriptive instructions in s2s and i2s settings.Evaluation checks scaffold preservation and instruction-grounded substitution; Generation is limited to Basic tasks to avoid entangling property-reasoning failures with molecular-realization failures.Source: Multiple-Choice (VQA) Track · p. 4

4.3 Generation Results and Analysis

• 59.7% average VLM Exact Match on Hard-Basic VQA drops to 14.7% (s2s) and 5.2% (i2s) in Hard Generation, exposing a selection-to-instantiation gap.VQA supplies explicit candidates, whereas Generation requires autonomous R-group grounding, molecular instantiation, and SMILES construction.Source: Table 3 · p. 5, §4.2 · p. 6, §4.3 · pp. 6–7

4.1 Experimental Setup

• The rule-based template oracle reaches 88.0% EM and 0.969 Tanimoto on Easy Generation, but 0% parse rate and 0.0% EM on Hard, where explicit substituent names are absent.VQA reports splitwise Basic accuracy and Advanced property-type accuracy, including ΔEH; Generation uses canonicalized EM, MCS scaffold match requiring ≥80% scaffold heavy-atom coverage, and Tanimoto similarity.Source: §4.1 · pp. 5–6, Table 4 · p. 6

4.2 VQA Results and Analysis

• 93.4% average s2i Easy-Basic accuracy falls to 58.6% on Hard-Basic; Hard-Basic modality averages are i2s 62.2, s2s 62.0, i2i 60.3, and s2i 58.6.Most VLMs exceed 90% on Easy-Basic, while Hard-Basic uses same-scaffold distractors and shortcut-resistant instructions; comparable i2s and s2s scores indicate visual input is not the primary bottleneck.Source: Table 3 · p. 5, §4.2 · p. 6
• Chemical-domain VLMs perform substantially below general-purpose VLMs on Basic VQA, indicating that molecular representation pretraining alone does not ensure R-group grounding.The evaluated chemical-domain models include ChemVLM-8B, MolVL-7B, and MarkushGrapher-2.Source: §4.1 · p. 5, Table 3 · p. 5, §4.2 · p. 6
• GPT-4.1 scores 98.8% on NOTA-as-distractor but 76.2% when selecting NOTA; bioactivity accuracy is only 26–28%, near the 25% random baseline.Reliable invalid-candidate rejection requires explicit structural checking; bioactivity inference is difficult without assay-specific or SAR context.Source: §4.2 · p. 6

4.3 Generation Results and Analysis

• Average VLM EM falls from 19.1% to 6.8% on Easy and from 14.7% to 5.2% on Hard when generation moves from s2s to i2s.The results suggest visual input adds alignment challenges, while precise structural modification remains difficult.Source: Table 4 · p. 6, §4.3 · p. 7
• MarkushGrapher-2 (Strohmeyer et al. 2026) achieves limited performance because it does not perform instruction-conditioned R-group substitution into complete molecules.Its Markush reconstruction design addresses only part of the required pipeline; task-specific molecular generation does not necessarily imply instruction-grounded editing.Source: §4.3 · p. 7

4.4 Failure Case Studies

• Failure cases include confusing substitution positions, choosing a candidate despite a plausible but incorrect rationale, generating incorrect local connectivity, and altering the molecular backbone.Together, the examples show that plausible explanations or molecules can fail to realize the requested R-group substitution and preserve the Markush scaffold.Source: §4.4 · p. 7

5 Conclusion

• R-GroundBench reveals a gap between recognizing plausible structures and autonomously instantiating valid molecules, with performance degrading under harder grounding and instruction-conditioned Generation.The benchmark evaluates R-group grounding in Markush molecular editing.Source: §5 · p. 7
• The benchmark separates recognition, grounding, and execution through complementary evaluations, including the ∆EH gap and VQA→Generation drop, which expose reliance on superficial cues and limited structure-aware grounding.Source: §5 · pp. 7–8
• Reliable molecular instantiation remains challenging even with visual information, while Advanced bioactivity reasoning requires integrating structural understanding with empirical chemical knowledge.The paper calls for models that jointly ground visual, textual, and symbolic chemical information and execute chemically precise transformations.Source: §5 · p. 8

A Markush Filtering Criteria

• The filtering pipeline applies four sequential criteria: standard R-group tokens, a clean token set, at most 10 R-group positions, and an RDKit-parseable backbone.It filters 91,166 MolParser-7M sft_real samples.Source: A Markush Filtering Criteria · p. 9
• 14,394 clean samples yield 55,982 unique editing operations across 12,962 Markush scaffolds; multi-scaffold structures are excluded because instructions cannot unambiguously refer to one component.Source: A Markush Filtering Criteria · p. 9

B Substituent Pool Design

• The substituent pool contains 22 structurally complex, ring-containing groups organised into seven similarity clusters, with context-aware pools for halogen, nitrogen, and oxygen attachment points.The halogen-position pool has 7 groups, the N-substituent pool 16, and the O-substituent pool 10; each is a strict subset of the general pool.Source: B Substituent Pool Design · p. 9

C Instruction Styles

• Easy and Medium splits use three direct-name instruction templates, while Hard splits use Relational and Chemically descriptive templates that require inference beyond direct name matching.Relational questions compare candidate fragments; descriptive questions map chemical properties and structural constraints to a molecular structure.Source: C Instruction Styles · p. 9
• For scaffolds with multiple R-positions, instructions concatenate one clause per position, and Hard questions are retained only when exactly one of four candidates satisfies all constraints.Source: C Instruction Styles · p. 9

D Dataset Statistics and Diversity Analysis

• 55,982 edited molecules show drug-like profiles, with mean molecular weight 428 Da and 70% between 300–700 Da.Heavy atom count, LogP, ring count, TPSA, and H-bond donors also follow distributions consistent with oral drug space.Source: D Dataset Statistics and Diversity… · p. 9
• Murcko analysis finds 23,822 unique scaffolds, with 82.1% singletons and the top 8,000 scaffolds covering only 62% of molecules.The top 3,618 scaffolds cover 50% of molecules, supporting the reported absence of chemotype bias.Source: D Dataset Statistics and Diversity… · p. 9
• 308 SAR clusters span 13 target classes, and 2,387 pIC50 values have mean 6.92 and standard deviation 1.19.The potency values cover the micromolar-to-sub-nanomolar range relevant to drug discovery.Source: D Dataset Statistics and Diversity… · p. 9
• Seven similarity clusters organize the 22-substituent pool into fine-grained alternatives for Hard splits.Within-group members share a scaffold but differ in ring size, substitution pattern, or heteroatom position.Source: Table 6 · p. 10
• Five instruction templates include direct-name styles for Easy and Medium splits and two Hard styles requiring additional grounding.The Hard styles identify targets through candidate relations or structural descriptions rather than direct names.Source: Table 7 · p. 10

E From VQA to Generation: A Cross-Track Performance Cliff

• Figure 7 visualizes cross-track degradation across representative sequence-to-sequence models.The figure complements the quantitative analysis of the performance cliff from VQA to generation.Source: E From VQA to Generation: A… · p. 10

H Evaluation Scope of Chemical-domain Models

• Chemical-domain models pursue different molecular understanding objectives, so their applicable evaluation settings differ from those of general-purpose LLMs and VLMs.Source: H Evaluation Scope of Chemical-domain… · p. 10
• ChemVLM-8B and MolVL-7B are evaluated on Basic VQA for molecular recognition and candidate selection, reflecting their molecular representation and structure–text training objectives.Their pretraining does not explicitly target instruction-grounded molecular editing.Source: H Evaluation Scope of Chemical-domain… · p. 10
• Advanced VQA adds physicochemical property estimation, structure–activity relationship understanding, and biological activity interpretation, so its results are discussed separately in the Appendix.These requirements go beyond the primary objectives of existing chemical VLM pretraining.Source: H Evaluation Scope of Chemical-domain… · p. 10
• MarkushGrapher-2 is evaluated on the Generation track using molecular images, matching its original Markush reconstruction objective.Its design targets Markush structure reconstruction and molecular generation, not general-purpose molecular question answering or instruction following.Source: H Evaluation Scope of Chemical-domain… · p. 10
• The Generation track additionally tests instruction-grounded R-group resolution, a capability not explicitly targeted by MarkushGrapher-2.Models must interpret natural-language editing instructions and perform the corresponding substituent substitution.Source: H Evaluation Scope of Chemical-domain… · p. 10
• VQA measures discrimination among predefined candidates, whereas Generation measures autonomous molecular instantiation without candidate constraints.The two tracks are therefore not directly comparable for MarkushGrapher-2.Source: H Evaluation Scope of Chemical-domain… · pp. 10–11

I Quantitative Analysis of Shortcut Control

• Easy-Basic’s mean ∆sim is +0.33, versus approximately 0.00 for Medium-Basic and Hard-Basic; Easy-Basic accuracy falls from 99.3% to 94.3% across similarity bins.∆sim measures the correct option’s scaffold-similarity advantage over distractors using ECFP4 fingerprints; Hard-Basic accuracy remains nearly unchanged across similarity bins.Source: I Quantitative Analysis of Shortcut… · p. 11
• The Easy-to-Hard accuracy gap was 36.3 percentage points (95% CI [33.51, 39.10], p < 0.001) under 10,000-iteration bootstrap testing.Mean accuracies were 98.7% for Easy-Basic, 93.7% for Medium-Basic, and 62.4% for Hard-Basic.Source: I Quantitative Analysis of Shortcut… · p. 11
• Spearman ρ was +0.138 (p < 0.001) on Easy-Basic, +0.025 (p = 0.501) on Medium-Basic, and −0.008 (p = 0.815) on Hard-Basic, indicating prediction only on Easy-Basic.The correlations are between per-question ∆sim and mean accuracy across eight VLMs; the Medium- and Hard-Basic correlations are not significant.Source: I Quantitative Analysis of Shortcut… · pp. 11–12

J Evaluation Prompts

• All models use zero-shot evaluation at temperature 0; VQA spans four input-output modalities, while Generation covers smi-smi and img-smi.Source: J Evaluation Prompts · p. 12

J.1 VQA Track

• VQA prompts require a single answer letter with no explanation, and Basic questions provide the Markush structure plus four candidates in modality-appropriate formats.For smi-smi/img-smi, candidates are SMILES strings; for smi-img/img-img, they are molecule images appended after the text.Source: §J.1 · pp. 12–13
• Basic NOTA questions designate option D as “None of the above,” and image-candidate prompts replace a None-of-the-above molecule image with a text label.Source: §J.1 · p. 13
• Advanced VQA replaces the instruction with a property-based question, omits NOTA, and presents Indication/Mechanism options as R-groups rather than full molecules.Candidates remain SMILES in smi-smi/img-smi and images in img-img/smi-img.Source: §J.1 · pp. 13–15
• Figure 7 reports a cross-track performance cliff: models strong on VQA, especially Basic Easy/Medium, often collapse when required to generate edited molecules directly.The caption interprets this pattern as evidence that multiple-choice success can reflect shortcut exploitation rather than robust atomic-level R-group grounding.Source: Figure 7 · p. 14

J.2 Generation Track

• Generation uses smi-smi or img-smi inputs without candidate options and requests only the resulting molecule’s SMILES enclosed in <smiles> tags.The two modalities differ only in whether the Markush structure is supplied as E-SMILES or an image.Source: §J.2 · p. 15

Patent-level complexity

• The benchmark uses well-defined grounding instances, while real patents may require long-range textual dependencies, conditional clauses, and reasoning across heterogeneous layouts.Extending evaluation to full patent-level retrieval and reasoning is an important future direction.Source: K Limitations · p. 15

Multiple-choice and generation scope

• VQA candidate selection isolates grounding but may allow structure-level shortcuts, while generation tests instruction-guided substitutions rather than unconstrained molecular discovery.Future benchmarks could include broader document-level reasoning and more open-ended molecular design scenarios.Source: K Limitations · p. 15