PaperlayerView exact revision on arXiv

Source-linked AI summary

RxnOptBench: Benchmarking LLMs for Reaction-Condition Optimization in Organic Methodology

Lingli Ge, Yubin Wang, Junyuan Gao, Jiahe Song, Jiaxing Sun, Boyu Zhu, Haote Yang, Jingchao Wang, Lixin Ma, Jiang Wu, Yuqiang Li, Conghui He

arXiv:2610.02242v1physics.chem-phcs.AI

TL;DR

Reaction-condition optimization is an important but underexamined capability for LLMs, and existing benchmarks do not ask models to choose among experimentally screened alternatives. RxnOptBench builds real-table selection tasks with a declared yield-and-selectivity score and evaluates frontier and chemistry-specialized models, which still leave substantial performance headroom.

  • Problem

    Whether LLMs can reliably select experimentally meaningful reaction conditions remains largely unexamined, while existing computational approaches and benchmarks only partially address the task.

    Source: §1 · p. 2

  • Method

    RxnOptBench converts condition-screening tables into multiple-choice and fill-in-the-blank questions, pairing precedent and no-precedent settings and scoring observed rows with a yield-and-selectivity utility.

    Source: §1 · p. 3, §3.4 · p. 6

  • Results

    Substantial performance headroom remains: full-condition selection is harder than single-factor selection, chemistry-specialized models do not consistently outperform general-purpose models, and precedents yield measurable improvements.

    Source: §1 · p. 3

  • Takeaways & Limitations

    The results point to difficulty ordering disclosed candidates and composing multiple condition axes, rather than identifying a single universal cause of model error.

    Source: §5 · p. 9

    The one-year sampling frame is an intended evaluation distribution, not an estimate of historical publication prevalence.

    Source: §3.1 · p. 4

Abstract

from arXiv · show

Chemical reaction-condition optimization -- choosing the catalyst, ligand, solvent, reagent, temperature, time, and atmosphere that jointly maximize yield and stereoselectivity -- is a central, judgement-laden subtask of organic methodology research that large language models are increasingly expected to support. Yet existing chemistry benchmarks evaluate reaction-class labelling, retrosynthesis, or SMILES manipulation, and do not ask models to read a real condition-screening table and pick the best set. We introduce RxnOptBench, a benchmark whose every option and precedent is a real wet-lab entry mined from the optimization tables of organic-methodology papers published in 2025, graded by a continuous relative score derived from a declared headline utility that combines reported yield with enantiomeric excess (ee), diastereomeric ratio (dr), and regioisomeric ratio (rr), and equipped with a paired precedents-vs-no-precedents design that isolates in-context use of literature evidence from parametric memorization. Across nine frontier LLMs and three Chemistry LLMs, even the best models leave substantial headroom: chemistry-specialized models fall to the random-baseline floor on multi-axis selection, while open-weight models have closed most of the gap to proprietary frontier models. We release the final human-reviewed benchmark test set and evaluation code.

1 Introduction

Reaction-condition optimization requires selecting interacting experimental variables to maximize yield and stereoselectivity, yet current methods and benchmarks do not fully test whether LLMs can make experimentally meaningful choices among screened conditions. RxnOptBench evaluates this capability using real condition-screening tables and finds substantial performance headroom across tested models.

1 Introduction

  • Figure 1 pairs questions with prior rows shown or withheld and assigns observed rows relative scores combining yield with selectivity measures.The figure distinguishes multiple-choice Accuracy and Score from fill-in-the-blank Hit, Strict Accuracy, and Score conditioned on Hit.Source: Figure 1 · p. 2, §1 · p. 3
  • Existing computational approaches typically search predefined spaces or assume fixed reaction classes, while existing chemistry benchmarks do not evaluate comparative decisions among tested alternatives.The benchmark gap spans both optimization methods and reaction, property, and scientific-reasoning benchmarks.Source: §1 · p. 2
  • The benchmark treats condition selection as choosing the best disclosed table entry; unreported combinations lack defensible outcome labels and are not treated as failures.The best disclosed row is not a global optimum over chemical space.Source: §1 · pp. 2–3
  • RxnOptBench contains 4,773 questions from 552 screening tables across 232 organic-methodology papers published in 2025, with multiple-choice and fill-in-the-blank tasks.It tests cold-start and precedent-assisted reasoning, as well as single-factor and full-condition selection.Source: §1 · p. 3
  • Even the strongest tested models leave substantial headroom; full-condition selection is harder than single-factor selection, and chemistry-specialized models do not consistently outperform general-purpose LLMs.Providing experimentally related precedents yields measurable improvements.Source: §1 · p. 3

2 Related Work

Prior condition-prediction approaches generally output condition vectors or focus on reaction prediction, rather than selecting the empirically best candidate from a screening table. RxnOptBench is distinguished by its real table-based options, separation of single-axis from whole-condition reasoning, and continuous credit for yield and selectivity.

Reaction datasets and condition-prediction baselines

  • Reaction corpora and condition predictors commonly represent individual reactions or output condition vectors, whereas RxnOptBench frames the task as choosing the empirically best entry from a small set of screened candidates.Prior methods include reaction datasets [8], predictors, and Bayesian optimization workflows.Source: §2 · p. 3

Chemistry LLMs and benchmarks

  • Existing chemistry LLM benchmarks do not separate single-axis from whole-condition reasoning, ground options in real screening-table entries, or provide continuous partial credit over reported yield and selectivity.The cited benchmark landscape includes chemistry-focused suites [11] [13] [14] [18] [19] [21] [22] [23] [24] and condition or yield prediction systems [25] [26] [27].Source: §2 · p. 4

Condition recommendation methods and benchmarks

  • Global and local condition-recommendation methods produce condition vectors rather than selecting the best entry among candidates, making their task framing orthogonal to RxnOptBench.The cited regimes include literature-scale models [33] [34] and HTE-focused methods [1, 2, 35–37].Source: §2 · p. 4

3 RxnOptBench

RxnOptBench converts screening tables from 2025 organic-methodology papers into human-reviewed reaction records and condition-selection questions. It specifies outcome-based scoring, paired precedent/no-precedent comparisons, and an inference protocol for 12 models.

3.1 Source

  • Sources are 2025 papers from 16 prespecified venues; the one-year frame targets contemporary decision problems, not historical publication prevalence.The 552 parent tables span 20 primary transformation classes, 26 condition fields, and 341 canonical products containing 232 non-empty Bemis–Murcko scaffolds.Source: §3.1 · p. 4

3.2 Extraction Pipeline

  • The pipeline converts 232 source PDFs into 552 retained tables and 5,451 row-level records, with expert comparison against source PDFs as the final quality gate.The public release is limited to the final human-reviewed benchmark test set and evaluation code, not the molecular-recognition service or reconstruction code.Source: §3.2 · pp. 4–5

3.3 Question Construction

  • Tables enter question construction only when at least two distinct normalized condition rows remain; MC_single fixes all but one varying axis, while MC_all options are observed rows.FIB predicts one value per varying key, but a Hit requires the recombined tuple to match an observed row; candidate order is shuffled consistently across models.Source: §3.3 · pp. 5–6

3.4 Metrics

  • ∼0.54–0.67 is the matched uniform-random Score floor; Table 2 reports no-precedents results, with FIB Strict over all submissions and Score conditional on Hit.Reported ee/er, dr, and rr are represented by major-product fractions, while an unreported selectivity axis contributes neutral factor 1.Source: §3.4 · pp. 6–7, Table 2 · p. 7

3.5 Precedents vs. No-precedents: Paired Design

  • Each precedents question is paired with a twin sharing its options, answer key, and scoring; adding 2–5 lower-to-mid-yield precedents isolates trend extrapolation without those design confounds.The optimum is withheld from precedents, and FIB precedent combinations are excluded from legal answers.Source: §3.5 · p. 7

3.6 Models and Inference Protocol

  • The evaluation covers nine frontier general-purpose LLMs and three chemistry-specialized LLMs, using shared prompt templates and disabled sampling where available.The public evaluator deterministically scores predictions against the fixed, human-reviewed test set.Source: §3.6 · p. 7

4 Empirical Results

Across 12 models, reaction-condition selection remains far from saturated, with complete multi-axis selection harder than single-factor selection. Per-key results show persistent catalyst difficulty and a gap between selecting individual FIB fields and composing a joint optimum.

4.1 Headline Results

  • Best MC_all Accuracy is Claude-Opus-4.7 at 36.8% (Score 0.725), while best FIB Strict Accuracy is GPT-5.5 at 16.1% (Score | Hit 0.699); FIB Hit peaks at 49.6%.For GPT-5.5, FIB Hit comprises 83 best observed rows among 552 outputs; non-Hits are not labeled chemically infeasible.Source: §4.1 · p. 7
  • Every model loses Score from MC_single to MC_all; frontier-model ∆axis ranges from 0.030 to 0.090, marking joint-condition reasoning as the binding constraint.The per-key analysis likewise finds frontier FIB Accuracy of 44.8–50.1% but Strict Accuracy below 17%.Source: §4.1 · p. 8, §4.2 · p. 9
  • On FIB, ChemLLM’s 6.6% Strict Accuracy / 0.554 Score | Hit is only 0.014 above the 0.540 random floor, versus GPT-5.5’s 16.1% / 0.699.On MC_all, ChemLLM scores 20.7% Accuracy / 0.545 Score against Claude’s 36.8% / 0.725, with its Score within 0.005 of the 0.540 random floor.Source: §4.1 · p. 8
  • DeepSeek-V4-Pro trails Claude on MC_single by 4.0 pp Accuracy / 0.019 Score (44.6% / 0.745 vs. 48.6% / 0.764) and ties GPT-5.5 on MC_all Accuracy at 32.4%.Its FIB Score | Hit is 0.672 against the top proprietary score of 0.683 (∆=0.011).Source: §4.1 · p. 8

4.2 Per-Condition-Key Accuracy

  • Proprietary leads exceed 20 pt on irradiation/light-source and atmosphere keys but compress to ≤12 pt on reagents and reaction time; Qwen3.5 outperforms Claude on reagents.The eight most frequent keys cover 933/1,039 MC questions and 1,561/1,912 FIB key occurrences; Table 3 reports per-key accuracy (%) on no-precedents subsets.Source: §4.2 · p. 8, Table 3 · p. 9
  • Catalysts rank near the bottom across MC and FIB and all model tiers, while frontier FIB per-key Accuracy is 44.8–50.1% but Strict Accuracy remains below 17%.The results establish category-specific catalyst difficulty but do not distinguish catalyst-class knowledge limits from candidate ambiguity or other causes.Source: §4.2 · pp. 8–9

5 Conclusion and Future Work

The conclusion frames RxnOptBench as a real-table benchmark that exposes difficulty in ordering disclosed conditions and composing multiple axes. Future growth is planned through versioned, expert-verified releases and targeted coverage and grading improvements.

5 Conclusion and Future Work

  • RxnOptBench pairs real wet-lab screening tables across precedents/no-precedents and scores choices with a declared aggregate utility over reported yield and stereoselectivity.Across 12 frontier and chemistry-specialized models, the task remains difficult, and the results point to challenges in ordering disclosed candidates and composing condition axes rather than one universal error cause.Source: §5 · p. 9
  • Future releases are planned as versioned and expert-verified rather than unbounded automatic ingestion, with manual verification remaining the deliberate quality-control bottleneck.Automatic stages support batching, concurrency, deterministic checks, and resumable intermediates.Source: §5 · p. 9
  • Planned updates include testing transfer to newly disclosed chemistry, broadening undersampled condition families and non-English venues, and tightening FIB grading with an order-invariant chemical comparator.The proposed transfer test uses periodic, versioned refreshes from newly published methodology papers.Source: §5 · p. 9

A Question Sub-Types and Worked Examples

The appendix formalizes the candidate spaces for multiple-choice and fill-in-the-blank question types. It also specifies how example variants share payload structure while differing in candidate scope or precedent blocks.

A Question Sub-Types and Worked Examples

  • MC_all widens the candidate set to all varying axes, while matched with- and without-precedent variants share item structure and differ by a Reference Conditions And Outcomes block.The appendix works through one MC_single and one all_blank example; the MC example is a single_varying dataset type.Source: A Question Sub-Types and Worked Examples · p. 14
  • MC_single varies one target key within rows sharing the most-covered observed context; MC_all uses distinct projections of observed rows across all varying keys.For FIB, candidate lists expose marginal values by key, while observed-row support is the subset of joint tuples actually present in the table.Source: A Question Sub-Types and Worked Examples · p. 14

A.1 Worked example: MC_single

The worked MC_single item asks models to select one ligand for a fixed Pd-catalyzed reaction. Its four options are real table entries with substantially different measured yields, and models split across them.

A.1 Worked example: MC_single

  • The example comes from Wang et al. (2025), Org. Lett., and its full question payload and gold label are included in the public human-reviewed test set.The cited item is identified as 10_1021_acs_orglett_4c04736_1_table_0_q2.Source: §A.1 · p. 14
  • The reaction holds catalyst [Pd(η3-C3H5)Cl]2, solvent PhCF3, and 50 °C fixed while varying the ligand.The transformation is Pd-catalysed 3,4-addition of pyrazole to a phenyl-carbamate-protected 1,2-dihydropyridine.Source: §A.1 · p. 14
  • Dpephos is the gold choice at 96% yield; BINAP, DPPF, and PPh3 yield 35%, 21%, and 0%, respectively.The next-best option, BINAP, has relative score r≈0.36.Source: §A.1 · p. 14
  • Proprietary LLMs, ChemVLM, and DeepSeek-V4-Pro recover Dpephos, while other models choose BINAP, DPPF, or PPh3.ChemLLM, ChemDFM, GLM-5.1, and Kimi-K2.6 choose BINAP; MiniMax-M2.7 chooses DPPF; Intern-S1-Pro and Qwen3.5-397B-A17B choose PPh3.Source: §A.1 · p. 14

A.2 Worked example: FIB all_blank

The FIB all_blank example shows how models must choose a jointly observed condition tuple, not merely valid values for each field. The benchmark’s review and source-selection procedures define which records enter the released test set.

A.2 Worked example: FIB all_blank

  • All three blanks jointly select temperature, time, and solvent from enumerated candidates, but only four of eight possible tuples have measured-row support.The gold tuple is (35 °C, 24 h, DMSO), with 80% yield; the other four Cartesian combinations are non-Hits, not measured failures.Source: §A.2 · p. 14
  • Models split between the observed but suboptimal tuple and unobserved recombinations, distinguishing row selection from unsupported joint predictions.All three Proprietary LLMs plus kimi-k2.6 recover the gold combination; other models select either an observed suboptimal row or an unobserved tuple.Source: §A.2 · p. 15

B.1 Annotation interface

  • Only accepted entries proceed to multiple-choice and fill-in-the-blank construction; reviewers can edit structured fields before saving their verdict.The review interface supports accept, save edits, skip, reject, and reset actions.Source: §B.1 · p. 15

B.2 Anti-leakage safeguards

  • The source set is restricted to papers first published in 2025, reducing but not eliminating exact-source exposure because model training cutoffs are heterogeneous or undisclosed.The authors do not claim a contamination-free evaluation.Source: §B.2 · p. 15

C Benchmark Composition Across Reaction, Condition, and Product Dimensions

The benchmark spans varied reaction classes, condition spaces, and target products, with its composition summarized at the parent-table level. Its 2025 sampling frame is an intended target distribution, not an estimate of historical publication prevalence.

C Benchmark Composition Across Reaction, Condition, and Product Dimensions

  • Composition statistics use one parent screening table as the counting unit, avoiding repeated counts of chemistry across the 4,773 derived questions.Tables were annotated with one mutually exclusive primary transformation and independent multi-label energy-input and catalyst classes; the audit reports no unresolved records.Source: C Benchmark Composition Across… · p. 15
  • All 20 primary transformation classes appear across 552 tables, and the largest class accounts for 16.7% of tables.The 552 tables contain 341 unique canonical target products across 232 non-empty Bemis–Murcko scaffolds; internal diversity is 0.875.Source: C Benchmark Composition Across… · p. 15, Figure 6 · p. 16
  • Tables vary a median of three condition axes and report a median of seven experimentally observed combinations, within a 26-field schema.Solvent, temperature, time, reagent, and catalyst are the most frequently represented condition families.Source: C Benchmark Composition Across… · p. 15, Figure 7 · p. 17
  • The 2025 sampling frame targets recent methodology problems and their measured local decision spaces rather than estimating historical publication prevalence.The recency lowers exact-source exposure risk but cannot guarantee contamination-free evaluation.Source: C Benchmark Composition Across… · p. 16

D Benchmark Validity and Robustness Audits

The audits bound interpretation of the benchmark’s labels and test results, assess score and candidate-position sensitivity, and describe release and prompt constraints. The prompts preserve shared reaction and condition context while varying task format and whether precedents are included.

D Benchmark Validity and Robustness Audits

  • The best disclosed row is not a global chemical optimum, and unreported combinations remain outside measured support rather than being treated as failures.Across tables, the median best-to-worst yield spread is 69 percentage points (IQR 40.75–83; range 1–100); the model-tier ordering persists across yield-spread quartiles.Source: D Benchmark Validity and Robustness… · p. 17
  • Yield-only scoring preserves model rankings closely, with Spearman correlation 0.965 across all 12 models; alternative utilities are reported as diagnostics, not full multi-objective optimization.On 43 tables with complete selectivity vectors, equal-weight additive and maximin utilities correlate 0.965 and 0.923 with the product score, respectively.Source: D Benchmark Validity and Robustness… · pp. 17–18
  • Gold answers at index zero do not differ significantly from the arity-conditional shuffled null for MC_single, MC_all, or FIB.The respective Monte Carlo p-values are 0.786, 0.063, and 0.569.Source: D Benchmark Validity and Robustness… · p. 18
  • Formal runs freeze prompts, candidate permutations, model IDs, and generation settings, but hosted responses are not claimed to reproduce bit-for-bit and archived outputs are not released.The public evaluator deterministically scores supplied predictions against the fixed test set.Source: D Benchmark Validity and Robustness… · p. 18

E License, Hosting, Intended Use, and Broader Impact

  • The release contains the final human-reviewed test-set records and evaluation code, not source files or benchmark-construction materials; users are cautioned against unvalidated wet-lab use.The records retain DOI/table/page provenance and are provided under a research-use agreement; safeguards cannot eliminate misuse.Source: E License, Hosting, Intended Use, and… · pp. 18–19

F Prompt Templates

  • Three prompt templates cover MC_single, MC_all, and FIB all_blank; reference variants append 2–5 prior wet-lab rows, while controls omit that block.Constant conditions are rendered in every task template, and MC_single also includes question-level fixed conditions.Source: F Prompt Templates · p. 19

F.1 mc_single

  • The prompt asks an expert organic chemist to choose Constant_Current from remaining options given a reaction, one real fixed condition group, and reference condition sets with observed outcomes.The references report yields of 59.0% at 10 mA and 92.0% at 20 mA.Source: §F.1 · pp. 19–20

F.2 mc_all / F.3 fib_all

  • The MC_all and FIB all_blank examples require selecting condition sets or one candidate per required category, with FIB disallowing direct copying of a reference combination.The FIB example requests candidate indices for electrolyte, reagents, scale, selected conditions, and solvents.Source: §F.2 · p. 21, §F.3 · pp. 21–23

G Illustrative Failure Cases

Four illustrative cases highlight different item-level behaviors, including default-like selections, index-zero choices, joint-condition errors, and response to a sibling precedent. They are qualitative examples rather than a prevalence estimate or causal validation.

G Illustrative Failure Cases

  • On one photoredox case, 5/12 models choose a widely used catalyst rather than the gold catalyst; the example alone cannot establish why.The gold is 4CzIPN at index 2; the selected alternative is Ir[dF(CF3)ppy]2(dtbpy)PF6 at index 0.Source: G Illustrative Failure Cases · p. 23
  • In the azulene rearrangement case, the Chemistry trio plus MiniMax-M2.7 choose index zero while several frontier models select the gold solvent.The case is consistent with position sensitivity, but does not establish chemistry-domain finetuning as its cause.Source: G Illustrative Failure Cases · p. 23
  • Every scoring model fails the 10-option catalyst×solvent question, although three strong models recover the catalyst in the paired two-option question.The same reaction’s gold joint condition is β-ICD+THF, so success on one axis does not compose into joint-condition accuracy.Source: G Illustrative Failure Cases · pp. 23–24
  • With a sibling precedent, strict accuracy on the Pd-catalysed cross-coupling case rises from 1/12 to 10/12.The gold catalyst is PdCl2(dppf)·DCM.Source: G Illustrative Failure Cases · p. 24

NeurIPS Paper Checklist

The checklist affirms that the paper’s abstract claims are supported and identifies scope, reproducibility, statistical-significance, and release constraints. It also documents the benchmark’s ethical safeguards and use of VLMs in extraction.

Abstract and introduction claims

  • The abstract’s four claims—real-data paired benchmark, continuous yield-and-selectivity scoring, release of a reviewed test set and evaluation code, and evaluation showing substantial headroom—are supported in the paper.The cited checklist links these claims to the benchmark construction, scoring, release, and results sections.Source: NeurIPS Paper Checklist · p. 25

Limitations and future work

  • Future work targets undersampled condition families and non-English journals, order dependence in the multi-component comparator, single-number scoring, and the gap between possessing and deploying a relevant prior.The paper also plans periodic benchmark refreshes to mitigate training-corpus contamination as the field evolves.Source: NeurIPS Paper Checklist · p. 25

Access and reproducibility

  • The public release includes the final human-reviewed test set and evaluation code, but not the extraction and benchmark-construction implementations or archived model outputs.The test-set records use a lightweight research-use agreement; instructions cover prediction formatting and score reproduction.Source: NeurIPS Paper Checklist · pp. 25–26

Statistical significance and experimental details

  • Models are queried once per question with sampling disabled, so the experiments do not provide seed-level variance estimates.All models use the same 4,773-item benchmark.Source: NeurIPS Paper Checklist · p. 26

Ethics and societal impact

  • The paper reports that source papers were obtained through open-access channels or institutional subscriptions, and that no personal or sensitive data or human-subjects research is involved.The release includes safeguards such as expert-review and experimental-validation warnings for structured records that could enable unvalidated wet-lab recommendations.Source: NeurIPS Paper Checklist · pp. 26–27

LLM usage and experimental setting

  • The extraction pipeline uses VLMs for page selection, table understanding, label-to-structure binding, and row verification, with human-review gating.Evaluated models are queried as-is; no model is trained in this work.Source: NeurIPS Paper Checklist · pp. 26, 28
Loading 2610.02242v1…