Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations

Research gap

• Current chemistry evaluations often omit stepwise operational reasoning and real-world challenges such as iteratively optimizing molecular structures under synthesizability or toxicity constraints.The paper also identifies reaction mechanisms involving intermediate transformations as a gap in existing evaluations.Source: §1 · p. 2
• Hybrid evaluations can conflate reasoning, knowledge recall, and computation, obscuring which capability drives model performance.The authors connect this ambiguity to targeted model improvement and downstream tasks such as drug discovery.Source: §1 · p. 2

ChemCoTBench

• ChemCoTBench represents complex chemical tasks as verifiable sequences of molecular-structure operations, including addition, deletion, and substitution.Its tasks progress from molecular understanding and editing to property-guided optimization and multi-molecule reactions, with validation by LLMs and 13 chemists.Source: §1 · p. 2

Results and dataset

• Experiments find room for improvement in reasoning LLMs, particularly open-source and distilled models, while ChemCoTDataset enhances chemical reasoning performance.The authors attribute the difficulty partly to scarce chemical reasoning data and provide 22,000 chain-of-thought instances.Source: §1 · pp. 2–3

LLM Chain-of-Thoughts

• Prior LLM reasoning research covers mathematics, coding, and multimodal tasks, while chemistry studies include synthesis planning and computational chemistry.The review says these efforts lack systematic evaluation spanning spatial reasoning, domain knowledge integration, and multi-step inference.Source: §2 · p. 3

Chemical Benchmarks

• Most chemical benchmarks emphasize factual recall, calculation, or simple prediction rather than multi-step reasoning [39] [40] [53] [66].ChemLLM [73] focuses on knowledge recall, while MolPuzzle evaluates spectral interpretation rather than broader chemical reasoning.Source: §2 · p. 3
• ChemCoTBench uses molecular structures to guide stepwise reasoning, from core chemical arithmetic tasks to cross-context applications.The related-work review presents this as broader coverage than benchmarks focused on isolated tasks or spectral interpretation.Source: §2 · p. 3

3 ChemCoTBench Construction

• The benchmark contains 1,495 samples across 22 tasks, with 22,000 additional chain-of-thought samples forming ChemCoTDataset.Construction involved over 1,800 combined hours of expert and LLM-assisted annotation and followed diversity and quality principles, including scaffold and functional-group coverage and hybrid review.Source: §3 · pp. 3–4

3.1 Task Construction

• Foundational understanding tasks test recognition and counting of functional groups and rings, hierarchical scaffold comprehension, and recognition of equivalent SMILES variants.Molecule editing tests whether models execute instructed additions, deletions, and substitutions while maintaining chemical validity.Source: §3.1 · p. 4
• Application tasks assess property-guided molecule optimization and reaction prediction, including products, retrosynthesis, reaction conditions, and mechanisms.Optimization targets include LogP, solubility, QED, and binding affinity for DRD2, GSK3-β, and JNK3.Source: §3.1 · p. 4

3.2 Benchmark Construction

• Data collection draws molecular structures from published datasets and reactions from patent databases, then filters selected compounds and curates coverage across chemical features.The benchmark spans 38 functional groups, 9 ring types, 57 editing transformations, 100 reaction classes, 175 reaction conditions, and 123 annotated mechanisms.Source: §3.2 · pp. 4–5
• CoT annotations distill stepwise strategies from advanced reasoning models into modular chemical operations across understanding, editing, optimization, and reaction prediction.The annotations are intended for systematic evaluation and supervised fine-tuning.Source: §3.2 · p. 5

3.3 Quality Review & Refinement

• Quality refinement combines scalable LLM-based evaluation with manual chemist review, using task-specific prompts and IUPAC names to improve annotation assessment.GPT-4o verifies consistency between CoT annotations, prompt templates, and IUPAC names.Source: §3.3 · pp. 5–6
• Expert review by 13 chemistry PhD candidates found near-perfect accuracy for understanding and editing samples and over 90% accuracy for optimization and reaction prediction samples.Reviewers assessed chemical entities in the annotations, and errors were corrected.Source: §3.3 · p. 6

4.1 Evaluation Metrics

• Evaluation covers molecule understanding and editing, optimization, and reaction prediction, using reasoning, general-purpose non-reasoning, and biomolecular LLMs.Metrics include MAE, Tanimoto similarity, accuracy, Pass@1, Top-1 accuracy, and fingerprint-based similarity, depending on the task.Source: §4.1 · pp. 6–7, §4.3 · p. 7, §4.2 · p. 7

4.3 LLMs’ Performance on Solving ChemCoTBench

• Claude-3.7-sonnet and Gemini-2.5-pro lead in foundational tasks and also lead in molecule optimization and reaction prediction.The authors describe this pattern as hierarchical skill transfer from foundational molecular understanding and editing to application tasks.Source: §4.3 · pp. 7–8
• Deepseek-R1 improves molecule optimization by >30% over Deepseek-V3, and o3-mini gains >20% over GPT-4o; similar trends appear in reaction prediction.Commercial LLMs with sophisticated reasoning mechanisms outperform their non-reasoning counterparts on the applied tasks.Source: §4.3 · pp. 7–8
• Open-source Llama-3.3-Nemotron and Qwen3 show no significant chemical-task improvement from enabling reasoning modes over their non-reasoning counterparts.The authors attribute this to a lack of domain-specific reasoning training data.Source: §4.3 · p. 8

4.4 Evaluating Distillation Strategies in Chemical Reasoning

• Qwen2.5-Instruct models distilled for CoT show little to no chemical-subtask improvement over their non-distilled counterparts, and smaller base models often perform comparably or better.The authors suggest insufficient volume or specificity of chemical CoT samples; distilled models under 7B also frequently produce lengthy, repetitive, irrelevant thought processes.Source: §4.4 · p. 9, Figure 4 · p. 9

4.5 Effective Methods for Enhancing Chemical Reasoning

• CoT prompting yields stable improvements for functional-group detection across 1.5B–32B models, but unstable effects on other chemical reasoning tasks.The authors attribute the instability to a lack of chemical knowledge.Source: §4.5 · p. 9
• SFT with ChemCoTDataset produces stable, substantial chemical-reasoning gains across evaluated tasks for Qwen-2.5 models from 1.5B to 32B.The table comparison contrasts coarse reasoning-template prompting with supervised fine-tuning using ChemCoTDataset.Source: §4.5 · p. 10, Table 4 · p. 10

5 Conclusion and Discussion

• ChemCoTBench evaluates complex chemical problem-solving across foundational and application tasks, including molecule understanding, editing, optimization, and reaction prediction.The paper describes it as a standardized evaluation platform for complex chemical reasoning.Source: §5 · p. 10
• The 22k-example ChemCoTDataset is provided to enhance LLM chemical reasoning, and experiments across 22 tasks validate its boosting effect.Source: §5 · p. 10
• Experiments find significant room for improvement in complex chemical reasoning by current open-source and distillation-based reasoning LLMs.Source: §5 · p. 10
• Future work could explore policy optimization, distillation strategies, chemical-aware reward mechanisms, and extension to larger biochemical domains.Source: §5 · p. 10

NeurIPS Paper Checklist

• The paper reports no theoretical results, so the checklist marks proof-assumption disclosure as not applicable.Source: NeurIPS Paper Checklist · pp. 16–17
• The checklist reports sufficient experimental details and reproducibility information, with code and data planned for public release upon acceptance.It also says the paper specifies key training and testing details, including data selection, hyperparameters, and optimizer settings.Source: NeurIPS Paper Checklist · pp. 17–18
• The authors report error bars, three runs for model tests, and five runs on a randomly selected 60% of human-experiment data.The checklist says the reported results are means.Source: NeurIPS Paper Checklist · p. 18
• The checklist marks societal-impact discussion as absent because the authors identify no unique or specific impact beyond general AI research considerations..Source: NeurIPS Paper Checklist · p. 19

A.1 LLM Chain-of-Thoughts.

• The related-work discussion says chemical LLM research lacks a comprehensive assessment spanning spatial cognition, domain knowledge assimilation, and complex logical inference.Source: §A.1 · p. 23

B Data Construction Details

• ChemCoTBench’s data construction covers source descriptions, dataset composition, filtering strategies, and construction rationale.Source: B Data Construction Details · p. 23
• The dataset-construction rationale includes Table 5, which visualizes the data distribution of subtasks in ChemCoTBench.Source: B Data Construction Details · p. 24

B.1 Data Collection

• Molecular data come from PubChem, ChEMBL, ZINC, and Deep-Mol-Opt; reaction data come from USPTO, Pistachio, and Reaxys, with mechanism annotation following.Source: §B.1 · p. 24

B.2 Dataset Composition and Filtering Strategies

• Molecular samples make up 25% of the benchmark and were selected from PubChem, ChEMBL, and ZINC for molecular-weight and structural-complexity diversity.The selection addresses the relatively concentrated molecular-weight distribution in ZINC.Source: §B.2 · p. 24
• Molecular optimization pairs comprise 38% of the benchmark; pairs with property improvement below 0.3 or complex polycyclic structures were excluded.Source: §B.2 · p. 24
• Chemical reaction samples comprise 19% of the benchmark, balanced across reaction type and catalyst diversity, with 275 manually curated mechanism examples from.Source: §B.2 · p. 24

B.3 Rationale for Task Construction

• Molecule-understanding and editing tasks have deterministic answers, with ground-truth properties and SMILES transformations extracted programmatically using RDKit.Editing tests adding, deleting, and substituting functional groups under natural-language instructions.Source: §B.3 · p. 24
• Optimization uses predefined molecular pairs from Deep-Mol-Opt to preserve scaffold consistency and verify property improvement, despite a limited molecule sample.The alternative model-generated strategy supports large-scale, multistep paths but can fail to preserve scaffold consistency.Source: §B.3 · pp. 24–25
• Reaction tasks cover forward product and byproduct prediction, single-step retrosynthesis, and SMILES generation for reaction-condition components.The forward task omits reaction type; retrosynthesis supplies reagents and limits prediction to one step.Source: §B.3 · p. 25
• Mechanism prediction tests next elementary-step products and mechanism selection to probe sequential understanding and discrimination among pathways.The selection task considers how reagent or condition changes relate to plausible mechanisms.Source: §B.3 · p. 25

C.1 Hardware Requirements

• Experiments used three computing nodes: RTX A6000 and RTX 3090 for API scheduling and smaller-model deployment, plus an A100 for large-scale inference.The reported storage requirement was approximately 1GB.Source: §C.1 · pp. 25–26

C.2 Evaluation Metrics

• Evaluation measures include accuracy, mean absolute error, scaffold similarity, property improvement, success rate, and SMILES validity.Reaction prediction uses Top-1 exact-match accuracy, while scaffold similarity uses the Tanimoto coefficient.Source: §C.2 · p. 26

• Most models except Gemini-2.5-pro had average ring-counting MAE > 1.0, whereas functional-group errors were mostly 0.0–1.0.Source: Figure 5 · p. 26, §C.3 · p. 26
• The analysis uses Mean Absolute Error to quantify overall accuracy and examines error distributions for a more granular view of both counting tasks.Source: §C.3 · p. 26
• Gemini-2.5-pro was the only model with consistently low errors on ring counting.Source: §C.3 · p. 26
• Ring counting requires precise identification of cyclic structures, making it more complex than detecting localized functional groups.The authors say the disparity underscores the need for further refinement in handling intricate molecular topologies.Source: §C.3 · pp. 26–27

D.1 Case Study for Molecule Understanding

• Commercial models closely parsed SMILES in four examples, with Gemini-2.5-Pro making only 1–2 bond-position errors, while LLaMA-3.1 produced structures largely inconsistent with the sources.Source: §D.1 · pp. 27–28, Table 6 · p. 27
• Commercial models generated connected Murcko scaffolds as instructed, while Llama-3.1 often produced fragmented substructures.Murcko scaffolds retain ring systems while removing non-critical functional groups.Source: §D.1 · p. 28

D.2 Case Study for Molecule Editing

• Gemini-2.5-Pro’s precise functional-group recognition supported accurate editing, while Qwen3-235B often failed to make valid modifications and LLaMA-3.1 struggled with recognition.The authors report a strong correlation between recognition accuracy and downstream success.Source: §D.2 · p. 28, Table 7 · p. 29
• LLMs generally handled extended one-dimensional chains better than complex polycyclic systems with intricate 2D topologies.Source: §D.2 · p. 28

D.3 Case Study for Molecule Optimization

• Optimization examples focus on QED, solubility, and LogP because the models performed poorly on protein-related activations DRD2, JNK3, and GSK3-β.Source: §D.3 · p. 28, Table 8 · p. 30
• Models introduced diverse functional groups in optimization, but some modifications worsened outcomes, which the authors link to limited understanding of physicochemical principles.The authors identify targeted training as a possible way to address this gap.Source: §D.3 · p. 28
• Gemini-2.5-pro often made skeleton-level additions or deletions, whereas Qwen3-235B and llama3.3 favored conservative insertions with minimal structural changes.Source: §D.3 · p. 28

• Figures 6–10 show examples spanning molecule understanding, molecule editing, molecule optimization, and reaction prediction, with red text marking chemical-specific prompt design.Source: E Task Example · p. 31
• Figure 6 depicts ring system counting, and Figure 7 depicts functional-group addition.Source: Figure 6 · p. 31, Figure 7 · p. 32
• Figure 8 shows GSK-3β optimization, while Figures 9 and 10 show next elementary-step product prediction and mechanism route selection, respectively.Source: Figure 8 · p. 32, Figure 9 · p. 33, Figure 10 · p. 34