Speak-to-Structure: Evaluating LLMs in Open-domain Natural Language-Driven Molecule Generation

1 Introduction

• Existing molecule–text datasets such as ChEBI-20, PubChem324K, and M3-20M assume one description maps to one predefined molecule, although multiple distinct molecules can share desired properties or activity.The authors question whether this paradigm assesses creative design or encourages memorization and pattern-matching.Source: §1 · p. 2
• S2-Bench evaluates open-domain molecule generation through editing, property-constrained optimization, and customized generation from structural descriptions.MolEdit tests localized edits, MolOpt tests edits under property constraints, and MolCustom tests generation against quantitative and qualitative structural requirements.Source: §1 · p. 2
• An automated evaluation system checks output validity and constraint satisfaction, then measures task-specific quality, including source-molecule similarity or novelty against a reference database.The authors describe this as enabling scalable, objective assessment of instruction following, rational local edits, and chemically meaningful candidates.Source: §1 · p. 2
• The authors introduce OpenMolIns, a large-scale instruction-tuning dataset of up to 120 million instruction–molecule pairs built programmatically from chemical databases.It mirrors S2-Bench’s three tasks and uses diverse prompt templates to address phrasing variation.Source: §1 · p. 2
• The authors report benchmarking 31 LLMs and that OpenMolIns enables Llama3.1-8B to achieve SOTA performance on S2-Bench.They characterize existing targeted-generation datasets as limited and highlight potential for a shift from pattern recall to realistic molecular design.Source: §1 · p. 2

2 Related Work

• Text-based molecule generation relies on paired molecular structures and descriptions; prior approaches include MolT5, KV-PLM, MoMu, and BioT5.MolT5 used large-scale self-supervised learning to generate SMILES, while KV-PLM, MoMu, and BioT5 combined molecular graphs with biochemical text.Source: §2 · p. 2
• MolReGPT and ICMA use in-context examples, MolReFlect applies teacher–student alignment, and Mol-R1 introduces long-CoT reasoning for molecule generation.The passage describes these approaches as supporting adaptive generation, fine-grained structure–language alignment, and robust, interpretable generation, respectively.Source: §2 · p. 2
• Related work extends molecular optimization to textual requirements, adds explicit structural constraints, and studies instruction-following biomolecule design [56].Nicolaou and Brown propose multi-objective optimization; Wu et al. extend it to textual requirements.Source: §2 · pp. 2–3
• Figure 1 contrasts text-based target generation with instructions that may be fulfilled by multiple valid molecules.The illustration places the multiple valid molecules to the right of the arrow.Source: Figure 1 · p. 3

3 Speak-to-Structure (S2-Bench)

• S2-Bench is organized around editing, optimization, and customized generation as core capabilities for real-world molecular design.Source: §3 · p. 3, §3.1 · p. 3

3.1 Task Composition

• Its one-to-many paradigm allows multiple valid molecules per instruction, testing flexible molecular understanding rather than pattern memorization.Source: §3.1 · p. 3
• MolEdit tests precise structural changes, MolOpt tests property-directed edits for LogP, MR, or QED, and MolCustom tests de novo generation under atom, bond, or functional-group constraints.MolEdit includes adding, deleting, or substituting components; MolOpt mirrors hit-to-lead refinement, while MolCustom requires chemically valid novel structures.Source: §3.1 · p. 3

3.2 Data Construction

• Automated construction avoids human annotations, samples Zinc-250K for tests and PubChem for tuning, generates 5,000 MolCustom prompts per subtask, and varies templates to reduce format overfit.Source: §3.2 · p. 4

3.3 Evaluation

• Open-ended evaluation replaces exact-match checks with metrics for valid, relevant, and novel outputs, using different metric sets across tasks.Table 1 lists the metrics and marks which tasks use each one.Source: §3.3 · p. 4, Table 1 · p. 4, §3.3.1 · p. 4

3.3.1 MolEdit & MolOpt Evaluation.

• MolEdit and MolOpt evaluation combines requirement Success Rate, Tanimoto similarity to the source molecule, and SMILES-based validity.Similarity uses Morgan fingerprints; the paper treats high similarity as evidence of a rational, localized edit.Source: §3.3.1 · p. 4

3.3.2 MolCustom Evaluation.

• MolCustom evaluation combines structural-constraint Success Rate with novelty relative to Zinc-250K and chemical validity.Novelty is higher when a generated molecule is less similar to known molecules.Source: §3.3.2 · pp. 4–5

3.3.3 Average Weighted Success Rate.

• Average weighted success rate combines subtask success with similarity for MolEdit and MolOpt or novelty for MolCustom, then averages across nine subtasks.The resulting measure is intended to balance instruction following with rationality or creativity of generation.Source: §3.3.3 · p. 5

3.4 OpenMolIns: Instruction Tuning Dataset

• OpenMolIns is derived from PubChem and uses flexible one-to-many training examples, with zero overlap with Zinc-250K to prevent evaluation leakage.The dataset is intended to teach chemical reasoning rather than memorization of fixed input-output pairs.Source: §3.4 · p. 5
• RDKit supports programmatic construction of equal-sized instruction sets, with 5,000 samples for each of nine subtasks instead of costly human annotation.The dataset construction uses objective molecular properties and rules.Source: §3.4 · p. 5
• MolEdit and MolOpt instructions pair functional-group edits or property-directed changes with their resulting molecules to teach structure-property reasoning.MolEdit changes include adding, deleting, or substituting a modifiable group; MolOpt samples specify directions such as increasing LogP or decreasing MR.Source: §3.4 · p. 5
• MolCustom prompts use extracted structural statistics as criteria and target a molecule generated from scratch rather than a recalled molecule-caption pair.The extracted features include atom counts, bond numbers, and functional-group types.Source: §3.4 · p. 5

3.5 Statistics

• S2-Bench contains nine subtasks with 5,000 test samples each, totaling 45,000 cases designed to assess one-to-many molecule generation.The dataset description presents this scale as supporting statistically robust assessment.Source: §3.5 · p. 5
• OpenMolIns has five data scales from 4,500 to 1,200,000 examples, enabling training at different sizes and systematic investigation of scaling.The tiered design is intended to help examine how model performance changes with data quantity.Source: §3.5 · pp. 5–6, §3.4 · p. 5, Table 2 · p. 5
• S2-Bench is described as the first open-domain natural-language molecule-generation benchmark, and the datasets are reported as larger than prior text-based alternatives.The paper claims OpenMolIns is the largest training set and S2-Bench the largest evaluation benchmark in this domain.Source: §3.5 · p. 6, Table 3 · p. 6
• Selected MolEdit and MolOpt samples align closely with Zinc-250K across average atom, ring, branch, and path-length statistics.Zinc-250K also exhibits broadly similar trends to PubChem, supporting its use as a proxy for discovery-scale small molecules.Source: §3.5 · p. 6, Table 4 · p. 6

4.1 Models

• The evaluation spans proprietary, general open-source, ChEBI-20-tuned, and OpenMolIns-tuned models, excluding multimodal and retrieval-augmented systems.OpenMolIns experiments include Galactica-125M, Llama3.2-1B-Instruct, and Llama3.1-8B-Instruct.Source: §4.1 · pp. 6–7

4.2 Hyperparameters

• Open-source testing used one NVIDIA A100 80G, instruction tuning used four, and closed-source models were accessed through official APIs.Source: §4.2 · p. 7

4.3 Findings

• Claude-3.5 and Gemini-1.5-Pro scored 35.92% and 34.80% average weighted success rate, yet no LLM exceeded 25% on any MolCustom subtask.The findings attribute difficulty with fine-grained structural constraints to the need to reason over discrete, globally coupled constraints; BioT5-base’s high MolEdit and MolOpt success can also accompany low structural similarity, consistent with pattern recall rather than rational editing.Source: §4.3 · pp. 7–8
• 39.33% average weighted success rate let xlarge OpenMolIns-tuned Llama3.1-8B-Instruct surpass all other LLMs; Galactica-125M reached 25.73%.The 39.33% result is for Llama3.1-8B-Instruct fine-tuned on the largest OpenMolIns scale.Source: §4.3 · p. 7, Figure 3 · p. 8
• Galactica-125M’s MolCustom performance rose 286% from large to xlarge, while MolOpt gains were modest and MolEdit gains negative; Llama3.1-8B generally improved except on LogP and MR.The paper concludes that dataset growth may not yield proportional gains and that model scale and data scale both matter.Source: §4.3 · p. 8, Figure 4 · p. 8

4.4 Validating WSR with Human Evaluation

• For BioT5-base, WSR correlated with expert scores at ρ=0.899 (p=0.015) versus SR at ρ=0.551 (p=0.257); for Galactica-125M, the correlations were 0.886 (p=0.019) versus 0.714 (p=0.111).Experts penalized chemically valid outputs that met constraints but departed substantially from the input structure; WSR down-weights such successes.Source: Table 6 · p. 9, §4.4 · p. 9

4.5 Extending MolOpt to Additional Properties

• TPSA and HOMO-LUMO Gap optimization had lower success rates than LogP, MR, and QED, and no model dominated both added properties.The added-property results are reported for representative models because of resource constraints.Source: Table 7 · p. 9, §4.5 · p. 9

• S2-Bench and OpenMolIns are introduced as the first benchmark and instruction-tuning dataset for open-domain natural language-driven molecule generation.Source: §5 · p. 9
• S2-Bench moves beyond one-to-one mappings to accommodate the open-ended nature of molecular discovery and assess realistic molecular reasoning and design.Source: §5 · p. 9
• Benchmarking 31 LLMs highlights challenges for current models and the limitations of existing targeted generation datasets, alongside substantial potential for natural language-driven molecule discovery.Source: §5 · p. 9

• The testing process is used to calculate the success rate for each subtask.Source: A Testing Process · p. 11

• The section presents the S2-Bench leaderboard in Table 8 and detailed subtask results in Tables 9, 10, and 11.Source: B Detailed Results · p. 11
• Vanilla Galactica-125M results are omitted because the model is not instruction-tuned and cannot respond to human instructions; the authors expect its results to be close to zero.Source: B Detailed Results · p. 11

C Limitations

• Empirical atom, bond, and functional-group allocations may not accurately reconstruct real-world data distributions, while important design constraints remain unmodeled.The omitted constraints include stereochemistry, 3D geometry, ring/bridge topology, and basic synthesizability.Source: C Limitations · p. 11
• MolEdit testing takes generated and original molecules plus a subtask as input and includes a check for a one-unit decrease in |Δ| with a one-unit increase in |α|.The supplied MolOpt procedure takes generated and original molecules, a subtask, and an optimization direction, then returns a pass/fail result.Source: C Limitations · p. 11
• Template-generated prompts may not capture the richness of real-world chemist queries.This limits the realism of the instruction setting.Source: C Limitations · p. 11

D Broader Impacts

• The benchmark frames LLMs as chemist assistants and molecule operators for molecule discovery.Its authors hope the benchmark will inspire research in this area, accelerating drug discovery and materials design.Source: D Broader Impacts · pp. 11–12
• MolCustom testing evaluates a generated molecule against atom, bond, and functional-group lists and requirements, returning a pass/fail result.The section also identifies Table 8 as the S2-Bench leaderboard, without supplying its results here.Source: D Broader Impacts · p. 12, Table 8 · p. 12

E GenAI Disclosure

• The authors used LLMs to improve language and readability, then reviewed and edited the content and retained full responsibility for it.Source: E GenAI Disclosure · p. 12
• The detailed-results tables cover MolEdit, MolOpt, and MolCustom and identify success rate and weighted success rate as reported measures.The captions identify the measures but do not provide result values.Source: Table 9 · p. 12, Table 10 · p. 12, Table 11 · p. 12