PaperlayerView exact revision on arXiv

Source-linked AI summary

oMeBench: Towards Robust Benchmarking of LLMs in Organic Mechanism Elucidation and Reasoning

Ruiling Xu, Yifan Zhang

arXiv:2510.07731v4cs.AIcs.CL

TL;DR

Organic mechanism reasoning remains an underexplored test of whether LLMs can generate chemically valid intermediates and maintain coherent multi-step pathways. The paper introduces oMeBench and oMeS to evaluate this ability, finding persistent reasoning difficulties but comparable performance from a fine-tuned 4B model and a closed-source model.

  • Problem

    Existing benchmarks largely assess reactant-to-product mapping rather than the elementary steps needed to evaluate mechanism reasoning.

    Source: §1 · p. 2

  • Method

    The paper introduces oMeBench, an annotated organic-mechanism benchmark, and oMeS, a dynamic framework for scoring predicted mechanisms against gold-standard mechanisms.

    Source: §1 · p. 2

  • Results

    LLMs struggle to sustain multi-step causal logic, while a fine-tuned 4B model achieves performance comparable to Claude-Sonnet-4.6.

    Source: §1 · p. 2, §6 · pp. 8–9

  • Takeaways & Limitations

    oMeBench establishes a foundation for advancing chemically grounded reasoning in large language models.

    Source: §6 · p. 9

    The dataset is smaller than large outcome-prediction corpora, has an overrepresented CH3 substituent, and may retain subtle biases from LLM-based expansion.

    Source: Limitations · p. 9

Abstract

from arXiv · show

Organic reaction mechanisms describe the step-wise elementary processes by which reactants transform into intermediates and products, and are fundamental to understanding chemical reactivity and guiding molecular and reaction de-sign. While large language models (LLMs) have shown promise on chemical tasks such as synthesis design, it remains unclear to what extent this reflects genuine chemical reasoning capabilities: the ability to generate chemically valid intermediates, maintain consistency across reaction steps, and follow logically coherent multi-step pathways. To investigate this, we introduce oMeBench, the first large-scale, expert-curated benchmark for organic mechanism reasoning, comprising over 10,000 annotated mechanistic steps with reaction type labels, intermediate structures, and difficulty ratings. To enable fine-grained evaluation, we further propose oMeS, a dynamic scoring framework that jointly assesses step-level logical consistency and chemical structural similarity. Systematic evaluation of state-of-the-art LLMs reveals that while current models exhibit promising chemical intuition, they struggle to produce correct and consistent reasoning across multi-step mechanisms. Notably, combining prompting strategies with fine-tuning enables smaller-scale models to achieve performance comparable to closed-source frontier models. We hope oMeBench will serve as a rigorous foundation for advancing AI systems toward genuine chemical reasoning.

1 Introduction

Mechanistic reasoning remains underexplored despite LLM progress on chemical tasks and the importance of mechanisms to understanding reactivity and applied chemistry. oMeBench and oMeS support step-level evaluation, and the reported results identify both persistent multi-step reasoning difficulties and gains from prompting and fine-tuning.

1 Introduction

  • Organic mechanisms describe stepwise bond breaking and formation, underpin understanding of chemical reactions, and inform applications such as drug-discovery predictions of metabolic pathways and toxicities.The paper characterizes mechanisms as the natural algorithm of a reaction.Source: §1 · p. 1
  • Existing benchmarks largely map reactants to products without annotating elementary steps, while rule-based mechanism datasets have limited transformation rules and reaction diversity.Natural-language reasoning benchmarks also lack explicit molecule-level details.Source: §1 · p. 2
  • oMeBench contributes over 10,000 annotated mechanistic steps with intermediates, step-type labels, rationales, diverse reaction classes, and difficulty levels.The paper describes the dataset as expert-annotated and released in a standardized format readable by people and machines.Source: §1 · p. 2
  • oMeS uses weighted similarity metrics to compare predicted mechanisms with gold-standard mechanisms, assessing mechanistic faithfulness rather than only final-product correctness.This enables mechanism-level accuracy assessment.Source: §1 · p. 2
  • Exemplar-based prompting and supervised fine-tuning improve mechanistic reasoning, while a fine-tuned 4B model achieves performance comparable to proprietary closed-source models.The evaluation also finds that LLMs struggle to sustain multi-step causal logic, especially in complex or lengthy mechanisms.Source: §1 · pp. 2–3

2 Related Work

Prior chemical datasets and language-model research cover reaction outcomes and several chemistry tasks, but explicit organic mechanism reasoning remains comparatively unaddressed.

2 Related Work

  • Large reaction datasets generally omit explicit mechanisms, while mechanism-aware datasets are often narrow in reaction scope or diversity, difficult to interpret, or simplified into question-answer formats.Some focus on radical or polar reactions, and some broader scientific benchmarks include mechanisms only in simple QA formats.Source: §2 · p. 3
  • Chemistry-specific models use chemical-corpus pretraining, fine-tuning, or representation-level approaches, yet most target property prediction, retrosynthesis, or text-based question answering.The related work characterizes explicit mechanistic reasoning as largely unaddressed.Source: §2 · p. 3
  • Chemistry LLM research has expanded from name translation and property prediction to retrosynthesis, synthesis protocols, prompting, and autonomous agents, while mechanism understanding has received little attention.The paper positions its work as a comprehensive study of LLMs on mechanism reasoning with a large-scale dataset for fine-tuning and evaluation.Source: §2 · p. 3

3 Dataset

oMeBench provides three complementary datasets for evaluation and training, addressing earlier datasets’ limitations in readability, metadata, difficulty, and diversity. Its Gold dataset uses expert-verified mechanisms and annotations, while template abstraction and expansion produce chemically plausible Silver reactions.

3 Dataset

  • Earlier datasets often used unreadable atom-mapped formats, omitted metadata, and lacked difficulty and reaction diversity.The paper describes many large datasets as high-school level, domain-specific, or repetitive.Source: §3 · p. 3
  • oMeBench comprises literature-verified oMe-Gold for evaluation, shared-pathway oMe-Template reactions, and chemically plausible oMe-Silver reactions primarily for training.All three datasets include expert annotations covering difficulty.Source: §3 · p. 3

3.1 Gold Dataset

  • oMe-Gold draws on authoritative textbooks and reaction databases, includes classical or challenging named reactions and out-of-domain examples, and is independently annotated by two expert chemists.Disagreements are resolved by a third expert; difficulty tiers comprise about 20% easy, 70% medium, and 10% hard reactions.Source: §3.1 · pp. 3–4
  • Gold annotations use 8 step types and 30 subtypes, with natural-language reaction descriptions, step rationales, and weights reflecting mechanistic importance and reasoning complexity.Rationales explain the underlying principles of each mechanistic step.Source: §3.1 · p. 4

3.2 Template Abstraction and Expansion

  • Experts reconstruct template mechanisms from representative reactions while preserving conditions, then expand templates with chemically plausible R-group substitutions to create Silver reactions.The template abstractions generalize named-reaction families that share mechanistic pathways.Source: §3.2 · p. 4
  • Each template produces an average of 15 valid Silver reactions; construction filters exact matches, and nearest-neighbor analysis finds no close duplicates of Gold reactions.The authors report that LLM-guided expansion tends to capture more chemically diverse and plausible structures than simple rule-based substitution.Source: §3.2 · p. 5

4 Evaluation

oMeS evaluates mechanistic reasoning through weighted alignment of predicted and gold stepwise mechanisms. Its complementary scores distinguish exact matches from partial structural similarity, while alignment accounts for mismatches and gaps.

4 Evaluation

  • oMeS generates stepwise mechanisms, aligns predictions with gold mechanisms using weighted Needleman–Wunsch, then derives four metrics for fine-grained assessment.The framework addresses unreliable LLM-as-judge evaluations, static string matching, and the need to score partial reasoning.Source: §4 · p. 5

4.1 Definitions

  • oMeS-total gives credit only for exact matches, while oMeS-partial grants credit for meaningful chemical similarity between predicted and gold intermediates.Subtype correctness is a prerequisite for demonstrating understanding of the chemical rationale.Source: §4.1 · p. 5
  • oMeS rankings remain nearly identical with graph-based similarity, and Morgan-fingerprint Tanimoto similarity is used because it reflects local bond changes.The authors identify local bond changes as central to mechanistic reasoning.Source: §4.1 · p. 6

4.2 Alignment Strategy

  • The alignment assigns zero credit and a mild penalty to type mismatches, fixed penalties to gaps, and resolves ties using score, mismatch, and gap priorities.The resulting step tags are match, type_mismatch, skip_gold, or extra.Source: §4.2 · p. 6

5 Experiments

Across the evaluated models, mechanistic reasoning remains limited, especially for difficult or extended pathways, while exemplar prompting and fine-tuning improve performance. The experiments analyze these failures and compare prompting and training approaches.

5.1 Experimental Setup.

  • The baseline evaluation compares proprietary, open-source, and domain-specific LLMs under one-shot prompting using validity, logicality, oMeS-total, and oMeS-partial.The evaluated models span several model families, including GPT-5.5, Claude, Gemini, DeepSeek, LLaMA, Qwen, Mistral, Phi-4mini, OLMo-2, OpenBioLLM, and ChemDFM.Source: §5.1 · p. 6

5.2 LLMs show limited understanding of mechanisms

  • Even the strongest frontier model reaches only 51.3 on oMeS-partial, and scores decline sharply with difficulty; domain-specific models lag general-purpose models across metrics.The reported results characterize LLM outputs as often syntactically valid and loosely following reaction-type sequences, but weak in mechanistic understanding.Source: §5.2 · pp. 6–7

5.3 Why do LLMs fail?

  • The failure analysis groups errors into chemical-knowledge, mechanistic-procedural, and structural-formalization categories, which can interact when misread conditions lead to invalid intermediates.For Gemini-2.5-Pro, 35.4% of errors involve invalid or inconsistent SMILES in multi-step mechanisms, including valence and charge violations accounting for 7.3% of total errors.Source: §5.3 · p. 7, Figure 5 · p. 7
  • Performance drops as mechanisms lengthen; pathways longer than five steps often omit intermediates or rearrangements, and skipped critical steps account for 20.8% of errors.The figure caption also describes longer pathways as amplifying illogical jumps.Source: §5.3 · p. 7, Figure 6 · p. 7
  • Models handle frequent substitution and addition steps better than rearrangements, pericyclic reactions, and radical processes, which demand deeper causal reasoning.The authors identify reasoning and knowledge gaps behind these errors, including difficulty sustaining coherent multi-step logic and limited mechanistic understanding of complex reaction types.Source: §5.3 · pp. 7–8, Figure 7 · p. 8

5.4 Improving LLM Performance

  • Three-shot retrieval of structurally similar reactions improves scores across nearly all models, while random demonstrations yield marginal, unstable gains; supervised fine-tuning also improves performance.The ICL exemplars are selected using DRFP Tanimoto similarity, and fine-tuning is performed on oMe-Silver.Source: §5.4 · p. 8
  • Fine-tuning substantially improves performance across dimensions, and Qwen-3-SFT outperforms some proprietary models; CoT reasoning in SFT adds mechanistic-accuracy gains under ICL.The authors report that explicit mechanistic supervision is effective for transferring mechanism knowledge and reasoning capabilities to open-source LLMs.Source: §5.4 · p. 8

6 Conclusion

The paper presents oMeBench for evaluating organic mechanism reasoning and oMeS for dynamic assessment. Its evaluation identifies persistent reasoning and symbolic-consistency gaps, while prompting and fine-tuning improve performance, including comparable performance by a fine-tuned 4B model and Claude-Sonnet-4.6.

6 Conclusion

  • oMeBench identifies persistent gaps in multi-step causal reasoning and symbolic consistency, while prompting and supervised fine-tuning enhance mechanistic reasoning.The conclusion reports that a fine-tuned 4B model achieves performance comparable to Claude-Sonnet-4.6.Source: §6 · pp. 8–9

Limitations

The paper identifies limitations in its models, dataset, and evaluation framework. These include persistent mechanistic-grounding errors, dataset scale and distribution concerns, and the limits of symbolic molecular representations.

Models

  • Models still misinterpret reaction conditions or generate chemically invalid intermediates, reflecting limited integration of symbolic reasoning with molecular-structure constraints.The authors note that token-level generation lacks explicit enforcement of chemical conservation and contextual awareness.Source: Limitations · p. 9

Dataset

  • The dataset is smaller than large outcome-prediction corpora, has an overrepresented CH3 substituent, and may inherit subtle biases from its LLM generator.The authors also identify expansion to photoredox, enzymatic, and transition-metal catalysis as a way to broaden benchmark coverage.Source: Limitations · p. 9

Evaluation

  • oMeS uses symbolic SMILES and expert annotators, so it cannot fully capture three-dimensional molecular environments and judgments may differ from those of non-experts.The authors suggest graph- or geometry-based similarity measures for a more complete assessment.Source: Limitations · p. 9

Ethical Considerations

This section outlines oMeBench’s ethical safeguards and annotation practices, contrasts its mechanistic data representation with prior resources, and presents the oMe-Template format through an annotated sample and a reaction-type taxonomy.

Ethical Considerations

  • The authors used public data and research-friendly model releases, acknowledge inherited and linguistic or regional biases, and limited work to evaluation and lightweight fine-tuning to reduce computational costs.The tools are intended to assist rather than replace chemists, with results interpreted using expert judgment.Source: Ethical Considerations · p. 9
  • The authors used LLMs for grammar refinement, LaTeX formatting, and occasional concision edits, with human authors reviewing and revising all outputs for accuracy and originality.Source: Use Of Generative AI · p. 9

A Annotation Detail

  • Four chemistry-competition experts with over four years of organic chemistry training annotated reactions after criteria development, instruction, refresher training, and iterative discussion with authors.Their tasks included validating intermediate SMILES and labeling difficulty, reaction type, and subtype.Source: A Annotation Detail · p. 13
  • Each reaction was independently labeled by two experts, with disagreements resolved by a third annotator and consensus; Fleiss’ κ was 0.95 for reaction types and weighted κ was 0.86 for difficulty.The authors describe these results as evidence of reliable, reproducible labels.Source: A Annotation Detail · p. 13

A.1 Instructions to Expert Annotators

  • The annotation protocol required literature-aligned mechanisms, chemically valid and RDKit-compilable SMILES, standardized step types and subtypes, and mechanism entries with ordered steps and rationales.Annotators also supplied intermediates and concise expert-level explanations of mechanistic principles.Source: §A.1 · pp. 13–14

B.1 Related Work

  • oMeBench combines stepwise mechanistic detail, validated intermediates, and natural-language rationales in a unified, diverse dataset designed for challenging reasoning benchmarks.The authors frame it as complementing outcome-oriented corpora and narrower mechanism collections.Source: §B.1 · p. 16

B.2 Data format and comparison

  • The oMe-Gold format records reactants, products, conditions, annotated mechanism steps, intermediate SMILES, fine-grained rationales, and reaction descriptions in interpretable JSON/JSONL.The format is intended to support large-scale evaluation and fine-tuning.Source: §B.2.2 · p. 17

B.2.3 oMe-Template Sample

  • The oMe-Template sample uses [*:i] for substitutable R-groups and illustrates a four-step Nazarov mechanism with proton-transfer, electrocyclization, and elimination steps.The sample also includes step-level metadata and rationales.Source: §B.2.3 · pp. 18–19
  • The sample encodes proton elimination and acid–base proton transfer, with rationales linking deprotonation to restored aromatic conjugation and tautomerization to conjugated ketone formation.Source: §B.2.3 · p. 19

B.3 Types and Subtypes

  • Table 6 presents reaction types, subtypes, and their descriptions.Source: Table 6 · p. 20

B.4 Data Contamination Analysis

This material checks Silver–Gold separation and describes the weighting, scaling, and metric details used in the benchmark. It reports no near-duplicate reactions and outlines how reaction templates are expanded and scored.

B.4 Data Contamination Analysis

  • No Silver–Gold reaction pair exceeds 0.80 Tanimoto similarity; the highest is 0.7969, and the authors conclude that contamination risk is negligible.The highest-similarity pair shares its phosphonate reagent, HMDS base, and Horner–Wadsworth–Emmons transformation but differs in aldehyde substrate.Source: §B.4 · p. 21, Figure 8 · p. 21

C.1 Weighting Strategy

  • Step weights combine a step-type base weight, a subtype modifier, positional adjustments, and bonuses for structurally important events.Terminal steps that mainly copy product structure are down-weighted, while ring closures and key bond-forming events receive positive bonuses; three domain experts curated the scheme.Source: §C.1 · p. 22

C.2 Weighting Implementation

  • Table 7 summarizes the weighting configuration, and final step weights combine the factors described for the metrics.The passage refers readers to Section 4.2 for the combination procedure.Source: §C.2 · p. 22, Table 7 · p. 22

D Scaling Algorithm

  • The scaling algorithm extracts reactive or substitutable groups, queries an LLM for candidate R-groups, enumerates substitutions, constructs instances, and validates them.The procedure is presented as the mechanism-template up-scaling algorithm for oMe-Silver.Source: D Scaling Algorithm · p. 23

E Metrics Details

  • A missing alignment at position k counts as incorrect in the definitions of V and L.The passage introduces the formulas but does not provide their expressions here.Source: E Metrics Details · p. 23

E.1 Robustness to Similarity Choice

The robustness analysis finds that model rankings remain consistent when oMeS uses Tanimoto or GESim. The alignment method then selects mechanism-step matches using a lexicographic score that prioritizes strict and partial chemical agreement.

E.1 Robustness to Similarity Choice

  • ρ = 0.84 Spearman correlation between Tanimoto- and GESim-based Spart scores, with the relative ranking of all 13 models preserved.GESim gives higher absolute scores, while Tanimoto is retained as the primary metric because it more directly captures stepwise bond formation and cleavage.Source: §E.1 · p. 24, Table 8 · p. 24

F Alignment Strategy

  • The specialized dynamic-programming alignment improves on minimum edit distance and simple match-prioritized scoring by incorporating chemical similarity and hierarchical alignment preferences.It aligns predicted mechanism steps against gold-standard steps.Source: F Alignment Strategy · p. 25

F.1 Alignment Process and Multi-Criteria Scoring

  • At each DP cell, oMeS maximizes a lexicographically ordered score vector, prioritizing total score before partial score and subsequent criteria.The available description names strict matches as identical step type and canonical SMILES, and partial matches as fingerprint similarity above τ = 0.60.Source: §F.1 · pp. 25–26
  • When higher-priority scores tie, operation rank favors alignment over skipping, while a −1e−6 penalty minimizes non-perfect alignments.Match ranks above type mismatch, which ranks above skip actions.Source: §F.1 · p. 26

F.2 Edge Cases Demonstrating Robustness of the Alignment Strategy

This section examines the alignment strategy’s handling of edge cases, including redundant steps and terminal type errors, and describes reaction retrieval and prompts for mechanism-related tasks. It also presents results on performance across reaction difficulty and case studies of reasoning and multi-step reconstruction.

F.2 Edge Cases Demonstrating Robustness of the Alignment Strategy

  • The alignment strategy skips chemically valid redundant predictions and isolates a terminal type error, avoiding misalignment cascades and preserving credit for correct steps.In the examples, conventional dynamic programming misaligns redundant predictions or propagates a final-step type error backward, whereas the proposed approach preserves structural and sequential matches.Source: §F.2 · pp. 26–27

G Extracting High-Similarity Reactions via DRFP

  • DRFP-based retrieval uses reaction difference fingerprints and Tanimoto similarity to retrieve structurally analogous reactions from an external database, typically selecting the top three.Fingerprints are cached for database reactions, identical reactions are filtered out, and the method is described as efficient without requiring explicit reaction classification or rule-based matching.Source: G Extracting High-Similarity Reactions… · p. 28, §G.2 · p. 28, §G.3 · p. 29

H.1 Prompt for suggested group

  • The R-group suggestion prompt requests diverse, chemically stable and synthetically accessible substituents in valid SMILES, excluding ring systems when the template already contains them.It also asks for both electron-donating and electron-withdrawing groups and requires JSON-only output.Source: §H.1 · p. 30

H.2 Prompt used for rationale annoation

  • CoT-SFT adds a natural-language rationale to each mechanism step, and its annotation prompt guides chemically grounded explanations while preserving the original step sequence.Rationales are generated by Gemini-2.5-Pro; the prompt’s chemical reasoning hints are described as an effective mind map for more meaningful analysis.Source: §H.2 · p. 31, Prompt used for rationale annoation · pp. 31–32

Prompt Used for Mechanism Generation

  • The mechanism-generation prompt requests a stepwise JSON mechanism with predefined reaction types and subtypes and an RDKit-parsable intermediate SMILES for each step.It requires output as valid JSON only.Source: Prompt Used for Mechanism Generation · p. 33

I Error Discussions; J.1 Case Study 1

  • The appendices include a taxonomy table for model-generated mechanism errors and a case study with an acidic-medium reaction and its generated mechanism.The case study specifies reactants and products, while the supplied passage does not provide the mechanism table’s contents.Source: Table 10 · p. 34, §J.1 · p. 35, Table 11 · p. 35

J.1 Case Study 1

  • Models handle simple ionic reactions more accurately than Lewis acid catalysis, complex ring formation, and radical or catalytic cycles, where pathways may be invalid or reduced to substrate–product mappings.Other recurring errors include misplaced protonation timing, lost hydrogen atoms, and poor stereochemical control.Source: §J.1 · p. 35

J.4 Case Study 4

  • The model produces the correct alkene but omits diazonium cleavage, substituting multiple proton-transfer and elimination steps for a chemically plausible pathway.The gold-standard mechanism proceeds through heterolytic cleavage to a carbocation, followed by base-promoted proton elimination.Source: §J.4 · p. 38

J.5 ICL Case Study

  • With ICL, the model recovers all four gold-standard steps, whereas without ICL it halts early and omits the second proton-transfer step.The example attributes the improvement to exemplar-guided prompting that supports causal continuity in multi-step reasoning.Source: §J.5 · p. 39, Table 15 · p. 39

K.1 Main results

  • Frontier models outperform others, but all models degrade on harder reactions and chemistry-specific models lag despite domain pretraining.The reported comparison covers oMeBench across difficulty levels.Source: Figure 10 · p. 40, Table 16 · p. 40

K.2 CoT results

  • The one-shot CoT setting prompts models to reason through each mechanistic step before predicting the next intermediate or transformation.The supplied passage describes the evaluation setup but reports no CoT outcome.Source: §K.2 · p. 40

K.3 In-context learning (ICL) results

  • The ICL setting supplies annotated mechanism exemplars before models predict mechanisms for new queries.The supplied passages identify the setting and score metrics but provide no ICL comparison values.Source: §K.3 · pp. 40–41

K.4 ICL Retrieval Ablation

The retrieval ablation reports stronger, more consistent improvements from DRFP-similar demonstrations than from random examples. The accompanying subsections describe fine-tuning comparisons and their training setup.

K.4 ICL Retrieval Ablation

  • DRFP-similar demonstrations yield substantial, consistent improvements, whereas random demonstrations produce only marginal, unstable gains.Random examples can introduce irrelevant chemical contexts; high-similarity examples reflect mechanistically relevant patterns and improve structural coherence.Source: §K.4 · p. 42

K.5 Fine-tuning results

  • The fine-tuning results compare selected models before and after tuning under simple-prompt and in-context-prompt conditions.Table 20 covers one-shot simple prompting and three-shot in-context prompting; the supplied passages do not report its result values.Source: §K.5 · p. 43, Table 20 · p. 43

L Fine-Tuning Configuration and Hyperparameters

  • The fine-tuning configuration uses supervised fine-tuning and Low-Rank Adaptation as complementary parameter-efficient strategies.The strategies are described as full SFT and LoRA.Source: L Fine-Tuning Configuration and… · p. 44

L.1 Dataset and Preprocessing

  • The training dataset uses Alpaca instruction, input, and output fields, with a maximum sequence length of 6000 tokens.Inputs contain reactants and products in SMILES format, and outputs contain predicted reaction mechanisms.Source: §L.1 · p. 44

L.2 Optimization Hyperparameters

  • Full SFT uses a learning rate of 1 × 10^-5, while LoRA uses 1 × 10^-4; both use a cosine schedule with 0.1 warmup ratio.The text attributes the different learning rates to updating all weights in full SFT versus only low-rank adaptation matrices in LoRA.Source: §L.2 · p. 44

Distributed Training and Mixed Precision

Related papers

Loading 2510.07731v4…